0.9.4: faster first snapshots, faster gate #68

Merged
ethan-hann merged 37 commits from integration-094 into main 2026-10-24 12:55:04 -05:00
Owner

Faster first snapshots (#45, D50): the managed repository config sets pack.compression=0 (config stamp 6). Measured on an 85 GiB GOG install of Cyberpunk 2077, a first snapshot took 35 minutes with zlib and 9 without, for a store 2.4% larger, since game archives are already compressed. Routine and reclaim repacks still compress the objects they pack anew at zlib's default level, so small save files stay compressed.

Faster gate (#25): the App module's audits open each app state once for all their checks, the job-pill audit composes each screen once for its four sizes, the scaled-window rows no longer render a PNG nothing reads, and two exit tests use a short exit wait. On the Core side the removed-snapshots world is built once per run and copied per test (with a check that no copy still names the template), HistoryTests lost its stale waits, the delete-responsiveness test decides with a gate instead of a wall clock, and the killed-program test writes its marker into a temp folder. The App module went from about 11 to under 7 minutes on Windows.

Empty folders (#66, D51): each snapshot records its empty folders in a commit-message trailer, and a switch, go back, discard or clean recreates any that are missing, so a folder like Cyberpunk 2077's archive/pc/mod survives switching through a profile that filled it. Found by a real-backend run on the GOG install of Cyberpunk 2077, where all 5,068 files came back identical but the empty folder did not.

First-snapshot progress (#67): git buffers its file list and says nothing while it closes the last pack, so once the store holds 99% of the bytes and has not grown for 8 seconds, the progress text says SavePoint is making sure everything is stored safely instead of showing a stopped figure.

Each part was reviewed by Opus and the findings are fixed in this branch. Version 0.9.4.

Closes #25, closes #45, closes #66, closes #67

Faster first snapshots (#45, D50): the managed repository config sets `pack.compression=0` (config stamp 6). Measured on an 85 GiB GOG install of Cyberpunk 2077, a first snapshot took 35 minutes with zlib and 9 without, for a store 2.4% larger, since game archives are already compressed. Routine and reclaim repacks still compress the objects they pack anew at zlib's default level, so small save files stay compressed. Faster gate (#25): the App module's audits open each app state once for all their checks, the job-pill audit composes each screen once for its four sizes, the scaled-window rows no longer render a PNG nothing reads, and two exit tests use a short exit wait. On the Core side the removed-snapshots world is built once per run and copied per test (with a check that no copy still names the template), HistoryTests lost its stale waits, the delete-responsiveness test decides with a gate instead of a wall clock, and the killed-program test writes its marker into a temp folder. The App module went from about 11 to under 7 minutes on Windows. Empty folders (#66, D51): each snapshot records its empty folders in a commit-message trailer, and a switch, go back, discard or clean recreates any that are missing, so a folder like Cyberpunk 2077's archive/pc/mod survives switching through a profile that filled it. Found by a real-backend run on the GOG install of Cyberpunk 2077, where all 5,068 files came back identical but the empty folder did not. First-snapshot progress (#67): git buffers its file list and says nothing while it closes the last pack, so once the store holds 99% of the bytes and has not grown for 8 seconds, the progress text says SavePoint is making sure everything is stored safely instead of showing a stopped figure. Each part was reviewed by Opus and the findings are fixed in this branch. Version 0.9.4. Closes #25, closes #45, closes #66, closes #67
Point times are stored to the millisecond, so consecutive captures already
sort apart without sleeping. HistoryTests had four such waits (the review
counted 14 across the Core tests; the rest are in the Launch tests and
others and are not touched here).

Timing: HistoryTests class, 12 tests, 14.8 s before, 8.7 s after
(--max-threads 2).

Mutation check: with HistoryReader ordering changed from
OrderByDescending to OrderBy, the class fails
A_point_reports_the_mod_files_that_changed_and_the_total_size_of_the_install_it_recorded
(Expected "More mods", Actual "Vanilla") and
Points_are_told_apart_by_which_folders_a_capture_changed_and_listed_newest_first
(Expected "Second save", Actual "Vanilla"). Restored afterwards.
The save-only install-state test never depended on order; its wait was
unneeded.
The audit row waited out 12 x 25 ms of fade settling and wrote a PNG that nothing reads (no test or doc refers to render-output/scaled), and LayoutAudit ignores opacity. RenderSettleTests still covers a picture waiting for the fade.

ScaledWindowTests (111 tests, --max-threads 2): 1m13s before, 28s after. No assertion changed, so no mutation check; AuditsCanFailTests still covers the layout audit.
AccessibilityAuditTests (screen reader names), TargetSizeTests and the ScaledWindowTests theory each opened all 55 states again. AppStateAuditTests opens each state once and runs the three audits on it in place, then shrinks the window through 960x460 and 910x432, labelling every problem with its audit. The audits are read-only, and the window only shrinks, so nothing carries over except a scroll offset the state's own opening set, as before. The dialog-description theory and the short-window card test stay where they were.

Before (App module, --max-threads 2): AccessibilityAuditTests 74 tests 22s, TargetSizeTests 55 tests 19s, ScaledWindowTests 110 audit rows 28s (after the previous commit). After: AppStateAuditTests 55 tests 18.7s, ScaledWindowTests 1 test 2s.

Mutation check (restored afterwards): AutomationAudit made to report every control, the target-size limit raised to 1000, the layout audit's minimum viewport raised to 96000. AppStateAuditTests failed all 55 rows with the [screen reader name], [target size] and [200% on 1080p, 960x460] / [150% on 1366x768, 910x432] lines in the message.
RemovedWorld.CreateAsync() with no options, no shared workspace and the
default name now builds the world through the real backend once per test
run and gives each caller its own copy of the folder tree. Files that
store absolute paths (the catalog and any text config) are rewritten to
the copy's root in plain, forward-slash and JSON-escaped forms; a binary
file that holds the old root is an error, except git's index, whose
untracked cache names the folder and is discarded by git when it differs.
Worlds that need options, a second game, or a git observer are still
built from scratch. A failed or canceled template build is not kept. The
template folder is deleted when the test process exits.

A copy takes about 40 ms; the one-time template build took about 0.9 s.
Most of the remaining time in these classes is the tests' own work.

Timing (one class at a time, --max-threads 2, duration before -> after):
BringBackTests 37.9 s -> 31.1 s (14 tests, all copy);
RecentlyRemovedTests 25.8 s -> 16.0 s;
FreeUpSpaceStrayBranchTests 16.4 s -> 14.8 s;
FreeUpSpaceRemovedTests 49.4 s -> 47.4 s (most of its tests add a second
game and keep their own build);
ReplaceKeptAsNewTests 2.2 s (one test, unchanged). Timings are noisy on a
shared machine.
The click audit and the pinned-text audit each composed the app for every size (224 compositions for 28 screens x 4 sizes). One test per screen and layout now opens the app once and steps through the sizes from the largest down, so the window only shrinks, and runs both audits at each size, labelling problems (pinned), (pinned text) or (scrolled to the end). Before each size every scroll area goes back to the top and the test asserts its offset is zero. Games failure card and Activity open already scrolled (found by asserting offsets on a fresh open), so they are opened fresh at each size and keep the scroll their opening gave them; any other screen that opens scrolled fails the test and says to add it to the list.

JobPillPlacementTests (--max-threads 2): 237 tests 1m12.7s before, 41 tests 25.2s after.

Mutation check (restored afterwards): the pinned-text overlap test widened to the whole visible area made the 28 rows fail with lines labelled (pinned text) per size. JobPillAudit.TakesTheClick forced true produced no failure, since no pressable control sits under the pill in these screens; the helper itself is covered by AuditsCanFailTests.
AppComposition.Create gets an optional exitWait that it passes to ExitGuard (null keeps the three-second default). The two MainWindowTests that quit during a storage move left the cancelled job running in the fake and waited the whole default; they now pass 100 ms, as ExitGuardTests does. No assertion changed, so no mutation check.

MainWindowTests (11 tests, --max-threads 2): 9.6s before, 2.6s after.
GamesDeleteResponsivenessTests wrote 30,000 padding files and failed when
the calling thread was held longer than 250 ms, which a fast disk could
miss and a loaded machine could trip. The test now holds the folder delete
shut: a new internal GitEngineOptions.BeforeGitDirDelete hook (called by
GitDirRemoval just before the files go, on the deleting thread) blocks on
a gate. A single-thread context stands in for the window; the test asserts
that the DeleteAsync call returned to it while the gate was still shut,
that the delete ran on another thread, and that the folder still existed
at that moment; then it opens the gate and checks the delete finished.
No padding files are needed any more. The patience timeout only matters
when the test is failing.

Observed red, with GamesApi.DeleteAsync deliberately blocking
(DeleteCoreAsync(...).GetAwaiter().GetResult() inside the guard):
Deleting_a_game_hands_the_calling_thread_back_while_the_folder_is_still_being_deleted
failed with "the call did not return to the calling thread while the
folder delete was held shut" (25.9 s, the patience wait). A first mutation
that only removed a ConfigureAwait(false) did not go red because an
earlier await has already left the window's context; it was restored.
Production restored; the test passes.

Timing: GamesDeleteResponsivenessTests 13.6 s -> 2.2 s (--max-threads 2).
The managed repository config now sets pack.compression = 0 and the stamp goes from 5 to 6, so existing repositories take the key the next time they are touched. Files over core.bigFileThreshold are streamed into a pack, and nearly all of them are archives that are already compressed. Measured on an 85 GiB install: 35 min with zlib, 9 min without, for a store 2.4 percent larger.

Observed reds (RepoConfigTests, before the change):
- Init_applies_the_setting_and_git_reads_it_back(pack.compression, 0): Expected "0", Actual null
- Init_applies_the_setting_and_git_reads_it_back(savepoint.repoconfigversion, 6): Expected "6", Actual "5"
- Init_again_reapplies_the_settings_when_the_stamp_differs: Expected "6", Actual "5"
- A_config_nobody_stamped_is_applied_in_full: Expected "6", Actual "5"
- A_repository_stamped_5_takes_pack_compression_off_the_next_time_it_is_touched: Expected "0", Actual null
The killed-program test wrote its marker file into the current directory,
which is the test bin folder. It now creates a fresh temp folder and passes
the marker path as its own argument (cmd mangles quotes written into a
script, and this machine's temp folder has spaces in it). Timings are
unchanged: the kill still comes at 300 ms, 2.7 s before the script's next
step.

A control run first proves the script can write a marker at that path, so
a marker that could not be written would fail loudly instead of passing
the test for the wrong reason. Observed while building this: a first
version with the path quoted inside the cmd script passed even when the
program was left to finish, because cmd never wrote the marker.

Mutation check: with the kill timeout raised to 30 s (so the program runs
to its next step) and the null-output assertion removed, the test fails
with "the program kept running after the timeout"; restored afterwards.

Timing: SystemProcessRunnerTests 7.2 s before and after (--max-threads 2).
pack.compression = 0 is meant for the big files a capture streams, but it also applied to routine and reclaim repacks, which would have stored every small compressible object (saves, configs) uncompressed. Both repack calls now carry pack.compression=-1 through the git config environment (arguments stay as they were); data already in a pack is still reused, so the streamed archives are not recompressed. D50's cost paragraph is corrected.

Red: A_repack_compresses_a_small_file_it_packs_even_though_captures_write_packs_without_zlib, Routine and Reclaim: 'the 2.9 MB save file should be compressed in the pack, but the packs hold 2900638 bytes'. Green after the change.
After the root is rewritten, every file outside the object stores and git's index is searched case-insensitively for the template's unique folder name, so a form of the path the rewrite does not recognise fails loudly instead of letting copies share one folder.

Mutation check: with RewriteRoot disabled, RecentlyRemovedTests failed with 'gitconfig/gitconfig still names the template folder savepoint-backend-...' (4 tests). Restored, class passes (9).
The skip matched any file called index; it now applies only to an index directly inside a git directory (HEAD and objects beside it), so a game file of that name is searched. The comment now gives the real reason objects are left alone: they are named by their hash. Verified by RecentlyRemovedTests (9 pass): had the git index not been skipped, its binary content would have failed the rewrite's assertion.
The template is built under the token of the first test to arrive; if that test is cancelled, every test awaiting the same task failed with a cancellation of its own. A waiter whose own token is fine now asks for the template again, which rebuilds it. No automated test: the case needs a cancellation timed against a concurrent build; the unchanged path is exercised by the Backend classes run in this change.
The test reuses one composition and only shrinks it, which relied on the declaration order of the size list. The list is now sorted by width and then height, descending, so adding a size anywhere keeps the order. JobPillPlacementTests: 41 pass.
With "activity" removed from _opensScrolled, JobPillPlacementTests failed for the activity screen in both layouts: 'activity, Library layout, 1040x660: a scroll area did not start at the top; add the screen to _opensScrolled.' (2 of 41 failed). Restored; 41 pass. No file changes.
pack.compression sat between two core entries, so the block opened [core] twice. It now follows the core entries (including the Windows-only one). The stamp-5 upgrade test still matches its exact text.

Red: The_managed_block_opens_each_section_once failed with 'Assert.Equal() Failure: Values differ' (duplicate [core] header) before the move; RepoConfigTests passes after (67 + 1 skipped).
dotnet format --verify-no-changes flagged the field added for the repack compression setting (IDE1006, missing underscore prefix). Rename only.
Write the managed config block one section at a time
All checks were successful
Gate / linux (pull_request) Successful in 10m1s
f989de2624
Moving pack.compression was not enough: the block interleaved core, index and checkout entries, so [core] still opened several times. The block now groups settings by section in order of first appearance (the stamp stays last), and the test inspects only the managed block, since git init's own [core] sits outside it.

Red (against the previous Block): The_managed_block_opens_each_section_once, Expected: 7 / Actual: 9. Green now: RepoConfigTests 67 pass, 1 skipped.
git does not track empty folders, so a switch that took the last file out of a folder removed the folder, including one the game shipped empty (Cyberpunk 2077's archive\pc\mod).

Each snapshot now records the empty folders it was taken with as a Savepoint-Empty-Folders trailer (a one-line JSON array, innermost folders only) in its own commit message. Being part of the commit it follows the snapshot through Recently removed, Bring back, keep-as-new-profile, Free up space and a storage move. Folders the exclude list names, folders under a folder link or a nested repository, and unsafe paths are neither recorded nor made. The exclude check runs against an empty work tree so a .gitignore inside the game folder cannot widen or narrow it.

A switch (also go back, forced switch and finishing a stopped switch), discard and clean create the target snapshot's missing folders afterwards; nothing is written into them and no folder is removed. Snapshots without the trailer behave as before.

Observed reds (assertion failures) before the change:
- A_snapshot_records_the_empty_folder_it_was_taken_with: expected ["archive/pc/mod"], actual ""
- Switching_back_to_the_profile_that_had_an_empty_folder_brings_the_folder_back: Assert.True failed
- Going_back_to_an_older_snapshot_brings_back_its_empty_folder: Assert.True failed
- A_forced_switch_brings_the_empty_folder_back_too, A_switch_that_was_stopped_and_is_finished_brings_the_empty_folder_back, Empty_folders_inside_empty_folders_come_back_down_to_the_innermost_one, A_switch_does_not_leave_out_an_empty_folder_next_to_files: Assert.True failed
- Discard_brings_back_an_empty_folder_of_the_snapshot_and_still_removes_new_ones: Assert.True failed
- A_folder_the_patterns_name_is_not_recorded_and_not_created, A_pattern_added_after_the_snapshot_keeps_its_folder_from_being_created: trailer expected, actual ""
- A_message_cannot_pass_itself_off_as_the_empty_folder_list: no ArgumentException

Tests that passed first run got a mutation check (all failed under the mutation, then restored): .gitignore negation (probe against the game folder), nested repository (.git skip off), snapshot without a list (fallback list invented).
A trailer that is not a clean list of names could stop a switch half done, and a few names resolved to a different path on Windows.

- A null, number or object in the list is dropped when it is read and the rest stands; malformed JSON is no list. A switch, discard or clean never throws because of the trailer.
- On Windows a name ending in a dot or a space, or a device name (CON, PRN, AUX, NUL, COM1-9, LPT1-9, with or without an extension), is not made, and the joined path must resolve to itself under the game folder. Backslash and colon are refused only where they are path syntax (Windows); elsewhere they are ordinary name characters.
- The exclude check runs in the same empty work tree the other classifications use, and each name gets a "./" in front so a name starting with a colon is not read as pathspec magic (check-ignore refuses GIT_LITERAL_PATHSPECS).
- Empty folders inside cloud placeholder folders are found: only real links are skipped.
- The message guard also refuses a key with blanks before the colon and any case variant of the three trailer keys.
- The switch docs say a switch creates the target's missing empty folders and nothing else.
- The case-insensitive de-duplication on restore is gone: creating a folder twice is harmless and no test could tell it apart.

Observed reds (before the change; all assertion failures except the null case, which was the NullReferenceException being fixed): [null] and ["ok",null] trailers (switch, discard and clean), a message with a blank before the colon (4 cases), IsSafe for ".git.", ".git ", "foo.", "foo ", "NUL", "nul.txt", "a/LPT9.x", "Con" on Windows rules, and for ":(glob)x", "a:b", "a\b" on other systems, ResolvesInside for "a/../b".

Tests that passed on first run were mutation-checked (each failed, then restored): IsSafe/ResolvesInside filter on restore (path escape test), IsInsideLink on restore (link added after the snapshot), the catch around CreateDirectory (a file where a folder goes), and the list passed by the fork, root and move-forward commit path. The pathspec-magic name test skips on Windows (listed in build/expected-skips/windows.txt) and was not run on Linux.
Merge branch 'issue-66-keep-empty-folders' into integration-094
Some checks failed
Gate / linux (pull_request) Failing after 9m6s
6609e52799
A first snapshot of an 85 GiB install showed no file count for 8.5 minutes, then
a burst of names, then a figure frozen at 99% for about 5 minutes. Measured with
the bundled git: git add --verbose prints through a block-buffered stdout, so
the big files (sorted first) show no lines until enough small-file lines fill
the buffer, and the last partial block only appears when git exits. After the
last file git still closes its last pack (re-reading it) and writes the index,
silently. GIT_FLUSH=1 does not change the buffering.

The reporter now marks a capture as finishing once the store has reached its
expected size and then holds still for two samples. FileProgress and JobProgress
carry Finishing, the combiner sets it only when no folder is still storing, the
final report clears it, and the shared progress wording (Activity, save and
unsaved-changes cards) and the wizard say "Making sure everything is stored
safely" instead of a stopped figure. The byte figure still stays under the total.

Observed reds:
- StagingByteSamplerTests.A_store_that_stops_growing_once_it_is_as_big_as_expected_is_reported_as_finishing: Assert.Contains() Failure, Finishing = False
- ProgressReporterBytesTests.Finishing_stays_on_once_said_and_the_last_report_clears_it: Assert.True() Failure
- CaptureProgressCombinerTests.A_single_folder_that_is_finishing_makes_the_job_finishing and ..._once_the_other_folder_has_stored_everything: Assert.True() Failure
- ProgressWordingTests.A_job_that_is_wrapping_up_says_so_...: Expected the finishing text, Actual "90.5 GB of 91.4 GB"
Mutation checks (tests that passed first run): dropping the size condition fails
..._stops_short_of_the_expected_size_is_not_finishing; dropping the steady
condition fails A_store_still_growing_is_not_finishing; dropping the other-folder
condition fails The_job_is_not_finishing_while_the_other_folder_is_still_storing;
raising the steady count to 200 fails the real-git CaptureBytesTests finishing test.
The shared setup deleted a file and captured the folders it left behind, which a point now records as empty folders. Switching to that profile then recreated a/b, so the test could not put its own file there (UnauthorizedAccessException in the test setup, for the Switch and ForcedSwitch cases with path a/b/c and theirs a/b).

The setup now removes the emptied parent folders before capturing, so the profile holds none of the path as the tests intend. No engine change: recording a folder the player left empty is the wanted behavior.

Observed red before the change: FileFolderCollisionTests.A_file_the_ignore_list_names_where_the_other_point_needs_a_folder_stops_the_operation_before_anything_is_written (Switch and ForcedSwitch, a/b/c, a/b) failed in the test's own write of a/b.
Describe the finishing phase of a big first snapshot in D50
All checks were successful
Gate / linux (pull_request) Successful in 9m32s
b59a04f415
Review of the finishing flag: one second of a still store at the cap could be
git hashing one big loose file (under 128m) before it writes it, which on a slow
disk or under a scanner takes several seconds, so the flag could turn on early
and stay on. The window is now measured on the engine's clock (TimeProvider, so
tests inject it) and is 8 seconds: longer than reading and hashing 128 MiB at
about 20 MB/s, and cheap against a wait of minutes. A failed or stopped capture
now sends a last report with the flag cleared (Abandon), so the Core contract
holds without relying on the caller.

Observed reds:
- StagingByteSamplerTests.A_store_that_holds_still_for_less_than_the_window_is_not_finishing: Assert.DoesNotContain failed (two equal samples had turned it on)
- ProgressReporterBytesTests.Finishing_stays_on_once_said_and_the_last_report_clears_it, The_last_report_clears_finishing_even_when_nothing_else_changed, Giving_up_clears_finishing_in_a_last_report: Assert.True failed
Mutation checks: window 8 s -> 0.5 s fails the sampler window test and the just-under-window and growth-restarts tests; removing the '|| _reportedFinishing' clause fails The_last_report_clears_finishing_even_when_nothing_else_changed; disabling Abandon fails the real-git CaptureBytesTests stopped-while-finishing test. The real-git finishing test now uses First instead of Single(Take(1)).
Review follow-ups for the finishing text. Activity no longer guesses a time left
while a job is finishing (at 99% after 20 minutes it said about 12 seconds beside
text that says a few minutes). The wizard hides the stopped file count and file
name while finishing, and reads the flag only: a figure at the old 99% mark with
the flag off shows the bytes again. The launch saving card, the unsaved-changes
card and the wizard speak the finishing text once when it begins (the quarter
announcements never reach 100% there). The wizard uses the shared catalog string,
so its near-duplicate is removed and it also gets the few-minutes sentence.

Observed reds:
- JobRowViewModelTests.No_time_is_guessed_while_the_job_is_wrapping_up: Expected False, Actual True
- LaunchSavingViewModelTests / UnsavedChangesBlockSavingTests The_start_of_finishing_is_announced_once: Expected the finishing text, Actual "75% copied."
- NewProfileWizardViewModelTests.The_start_of_finishing_is_announced_once: Actual "75% copied"; When_the_job_is_finishing_...: old wizard-only text without the few-minutes sentence
- ProgressMilestonesTests.The_start_of_finishing_is_worth_saying_once_until_the_job_starts_over: assertion failure on the stub
Mutation checks: restoring the 99% inference in the wizard fails A_figure_at_the_old_finishing_mark_still_shows_the_bytes_while_the_job_is_not_finishing; dropping the announce-once guard, the files-text hiding, or the current-file hiding each fails the matching wizard test.
Say how long the store must hold still before the finishing phase shows
All checks were successful
Gate / linux (pull_request) Successful in 9m28s
Gate / linux (push) Successful in 9m12s
65075af490
Sign in to join this conversation.
No reviewers
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
ethan-hann/SavePoint!68
No description provided.