0.9.10: Free up space rewrites only what it frees, no git left after quitting, Linux fixes #136

Merged
ethan-hann merged 28 commits from integration-127 into main 2026-10-25 09:53:38 -05:00
Owner

Version 0.9.10.

  • Free up space rewrites only the packs that hold something it frees, and skips the repack when nothing is freed. On a copy of Cyberpunk 2077 (91 GB) a Free up space with nothing to free took about 5 minutes and rewrote the whole store; it now takes well under a second. The space check asks for the floor plus what will be rewritten, not a copy of the store. A pack without its index is skipped, object lists are streamed, and a folder with an unfinished switch is refused. Closes #133.
  • On Linux a cancelled git is killed outright (no SIGSTOP first), and any git still running when SavePoint exits is killed, so none is left stopped under init holding the AppImage's mount. Closes #127.
  • Linux desktop findings: the missing-saves-folder note wraps in its column; a disabled Play says why it is disabled; a session that changed nothing preselects Not now; large file counts use group separators. Closes #128. Closes #129. Closes #130. Closes #131.
  • Details shows the newest 200 lines and Copy takes up to 20,000; names from a stopped group are dropped; the big-file line says it is an estimate. Closes #118.
  • The release checksums job reads only the asset list and the package jobs' own SHA256 lists, so it downloads nothing from the draft (the workflow token cannot read draft files). Closes #134.

The build with warnings as errors and the format check pass locally.

Version 0.9.10. - Free up space rewrites only the packs that hold something it frees, and skips the repack when nothing is freed. On a copy of Cyberpunk 2077 (91 GB) a Free up space with nothing to free took about 5 minutes and rewrote the whole store; it now takes well under a second. The space check asks for the floor plus what will be rewritten, not a copy of the store. A pack without its index is skipped, object lists are streamed, and a folder with an unfinished switch is refused. Closes #133. - On Linux a cancelled git is killed outright (no SIGSTOP first), and any git still running when SavePoint exits is killed, so none is left stopped under init holding the AppImage's mount. Closes #127. - Linux desktop findings: the missing-saves-folder note wraps in its column; a disabled Play says why it is disabled; a session that changed nothing preselects Not now; large file counts use group separators. Closes #128. Closes #129. Closes #130. Closes #131. - Details shows the newest 200 lines and Copy takes up to 20,000; names from a stopped group are dropped; the big-file line says it is an estimate. Closes #118. - The release checksums job reads only the asset list and the package jobs' own SHA256 lists, so it downloads nothing from the draft (the workflow token cannot read draft files). Closes #134. The build with warnings as errors and the format check pass locally.
The note sat in a horizontal stack, which measures its text at infinite width, so it only wrapped at its 470px cap and ran past the column and under the second column at the default window size. A star column gives the text the column's width.

Red: GameEditorViewTests.The_missing_saves_folder_note_wraps_inside_its_column at widths 1040 and 760 failed with 'The note ends at 537px but its column ends at 420px.' and '... 362px.' before the fix.
Play's disabled tip always read 'This game is already running' even while the Switch dialog was open or a switch ran. The tip now says so only when a profile is running; otherwise it uses a new string about the switch or snapshot not being finished.

Red: DisabledReasonTests.Play_does_not_claim_the_game_is_running_while_a_switch_is_waiting_or_running (launch-dialog and launch-progress) failed with Expected 'A switch or snapshot for this game is not finished yet. Play works again when it is.' Actual 'This game is already running. Play works again after you close it.'
When neither folder changed, the after-play dialog said it found no new or changed saves yet still recommended 'Keep this session as a new snapshot' with Keep it as the main button. The session now preselects Not now in that case, the Keep card drops 'Recommended', and Not now says why it is the suggestion. Sessions started from an older point and sessions whose folders could not be read are unchanged.

Red: LaunchSessionTests.A_session_that_changed_nothing_in_either_folder_preselects_Not_now failed with Expected: Skip, Actual: CaptureNew. PostPlayDialogViewModelTests.A_session_that_changed_nothing_recommends_Not_now_and_does_not_recommend_keeping failed with Expected 'Your earlier snapshots stay as they are.' Actual 'Recommended. Your earlier snapshots stay as they a...'
Plural messages printed counts with a bare number, so the changes card read 200000 files while the progress line read 200,000. UiFormatter now passes each whole-number argument as <name>Text, formatted for the language, and the launch dialog, switch and after-play messages use it. The Mac and Linux test script no longer says a notification appears when a game is ready; launches never notify (D10, D49), and it now names the jobs that do.

Red: PostPlayDialogViewModelTests.Large_file_counts_are_written_with_group_separators failed with Expected '200,000 files in your saves folder are new or chan...' Actual '200000 files in your saves folder are new or chang...'. LaunchDialogViewModelTests.A_large_count_of_changed_files_is_written_with_group_separators failed on the same missing separator.
Quitting while a capture was running left the bundled git behind (seen on
Linux as a stopped ls-files, with the AppImage launcher and its mount kept
alive by it). The runner killed a tree with Process.Kill(entireProcessTree),
which stops each process first and ends it later, and nothing made the app
wait for or finish that kill: a host that exited in between left the git
stopped or running, reparented to init.

On Linux the tree is now read from /proc and every member is sent SIGKILL
directly, deepest first and the root last, with no stop signal. Every
started git is also kept in a registry that is killed when the process
exits, so a call that is still running or still being cancelled at exit
cannot outlive the app.

Observed reds (Windows, with an empty KillAll stub):
- GitProcessTreeTests.Ending_every_running_git_ends_a_call_nobody_cancelled:
  "the call did not end after every git was killed".
Tests that passed on first run were mutation checked:
- Cancelling_leaves_no_process_of_the_tree_running_or_stopped fails with
  "the git that was started is still running" when the tree kill is a no-op.
- The_tree_below_a_process_is_read_from_parent_ids_and_leaves_out_what_has_exited
  fails with [11, 12, 14] instead of [11, 12] when zombies are not skipped.
Probe on the Linux VM (a host that cancels a 400,000 file ls-files and exits
within 25 ms, page cache dropped): old runner left a git behind in 4 of 100
runs, this one in 0 of 60.
A reclaim removed every SavePoint keep and ran repack -a -d, so a store
with nothing to free (measured: Cyberpunk 2077, 91 GB in one streamed pack)
was rewritten whole, 298 to 330 s and room for a second copy.

Now the keeps stay through the repack, which leaves kept packs alone. After
the reflogs expire, the reclaim lists every object, every reachable object
(rev-list --objects --all --indexed-objects, so branches, HEAD, the index
and the removed-snapshot records) and, for each SavePoint-kept pack, its
objects (show-index on the .idx; the pack is never read). Only a kept pack
that holds an unreachable object loses its keep and is rewritten without
it. When nothing is unreachable and nothing is loose, no repack starts.
Prune still runs. A keep for a pack that is gone is removed.

Observed reds before the change (ReclaimStreamedPackTests):
- Free_up_space_with_nothing_to_free_...: Assert.Equal() Failure, packs
  differ at index 0 (kept packs merged into one).
- Deleting_a_profile_that_alone_reached_a_big_file_...: "pack-f45d...pack
  was rewritten".
- A_streamed_pack_with_one_unreachable_file_...: "pack-8923...pack was
  rewritten".
- A_recently_removed_snapshots_big_file_survives_...: "pack-f45d...pack was
  rewritten".
Mutations: skipping the held-object check fails the three rewrite tests;
always repacking when nothing is unreachable fails the nothing-to-free test.

Existing tests that encoded merge-everything are updated: the streamed-pack
reclaim tests now expect reached packs and their keeps to stay, the keep
removal failure test uses a pack with an unreachable file, and the
interrupted-reclaim tests start from a store with loose objects so a
repack runs.
The registry is now an instance a runner can be given, so a test that ends
"every git" cannot reach another test class's process; the app still uses
one shared instance that is emptied at process exit. A registration removes
only its own entry. On Linux the root is ended with Process.Kill() (no tree,
no stop signal, and .NET's reaped-process guard applies), each descendant's
parent id is read again just before it is signalled, and a libc that cannot
be bound falls back to the .NET tree kill instead of failing the cancel.

New tests: an exit-time kill leaves a process that only shares a number
alone; a second copy of the test program that exits while a git runs leaves
no git behind (the process-exit hook).

Mutation checks (Windows): without the process-exit hook the exit test
fails (the child never ends, held by the git's pipe, 2 minute limit); with
the stamp check removed the number-takeover test fails with "a process that
only shares the number was killed".
The space check demanded the whole store's size, so a big game on a tight
disk was refused even when nothing would be rewritten. The reclaim now
plans first (which objects nothing reaches, which packs hold them, how many
loose bytes) and asks for the floor plus the bytes it will rewrite: the
unkept packs and kept packs that lose their keep, when a repack of
everything is needed, or only the loose objects otherwise.

The check that runs before the removed-snapshot records are cleared takes
the refs about to be deleted (CheckReclaimAsync overload) and counts them
as already gone, so a kept pack that only those records reach is counted.
The check stays before the clear because a refusal after it would already
have lost the records; the figure can only be too large, never too small.

When nothing in any pack is unreachable and loose objects exist, the
repack is now "repack -q -d" (no -a), so unkept packs are left alone;
unreachable loose objects are pruned as before. "repack -q -a -d" stays
for the case where something packed is freed.

Observed reds before the change (ReclaimStreamedPackTests):
- A_reclaim_with_nothing_to_free_needs_only_the_free_space_floor_...:
  DiskSpaceException, 104931328 bytes needed, 104857600 free.
- A_reclaim_that_must_rewrite_a_streamed_pack_needs_room_for_that_pack_...:
  Assert.InRange failure, range (40060 - 72703), actual 72704 (whole store).
- The_check_before_records_are_cleared_counts_...: DiskSpaceException from
  the check that passes no refs, 104909824 needed, 104857600 free.
- A_reclaim_with_only_loose_objects_to_pack_leaves_the_unkept_packs_alone:
  collections differ at index 2, expected "-d", actual "-a".
Mutations: counting the named refs as still present fails the check test;
making the loose-only plan a full repack fails the loose-only test.

Tests that asserted repack -a -d for a store with nothing packed to free
now expect -d (ReclaimTests) or match any repack (ReclaimInterruptedTests,
FreeUpSpaceStopTests); the refused-for-space test reads the required
figure from the refusal instead of assuming the store size.
The v0.9.9 checksums job failed with "Downloading assets.linux.json answered 404": the
workflow token can upload to a draft release but cannot read its files back.

Upload now hashes every file, every .partN and each split file whole before it sends
anything, uploads that list as SHA256SUMS-<folder>.txt, and writes it to the step output
"sums". The workflow passes the outputs to the checksums job as SUMS_LINUX / SUMS_WIN.
Checksums reads only the asset list (names and sizes) from the API, checks every asset
against the lists (listed, same size), every listed file against the release, and part
numbering, then writes SHA256SUMS.txt and removes the per-job lists (it warns and keeps
them if the delete is refused). Only a run without the variables downloads the small
lists. The three required downloads check and the sources link are unchanged.

Observed reds (before the change, each an assertion failure on the missing function or text):
- Hashing_before_the_upload_lists_each_file_each_part_and_the_joined_file: failed, Get-SumEntries not recognized
- A_sum_list_is_read_back_into_the_entries_that_were_written: failed, Format-SumList not recognized
- The_lists_of_both_jobs_become_one_checksum_file_when_they_match_the_release: expected "aa  a.exe|bb  b.zip|..." actual "ERROR: The term 'Format-SumList' is not recognized"
- A_release_asset_with_a_different_size_than_its_list_is_refused, A_release_asset_no_list_mentions_is_refused, A_listed_file_the_release_lacks_is_refused, A_gap_in_the_numbering_of_the_parts_on_the_release_is_refused, Parts_whose_list_has_no_line_for_a_missing_middle_part_are_still_refused: same not-recognized error instead of the expected refusal text
- Checksums_reads_the_job_lists_from_the_workflow_and_downloads_no_package_file: DoesNotContain failed, browser_download_url found in the Checksums action

Mutation checks (each caught): size comparison disabled, listed-but-absent check disabled,
Assert-PartsNumbered call removed; 4 tests failed, then restored.
Details showed only the newest 200 lines, so a first snapshot of a 5,200-line game showed its last 200. A job now also holds its newest 20,000 lines (about 3 MB at most) behind IJobsApi.GetLines; the list on screen stays at 200 so the view is unchanged. When lines are missing from the list, Details says "Showing the newest 200 of 5,200 lines" and that Copy, with nothing selected, takes all of them or the newest 20,000. Copy with a selection still copies the selection.

The registry now remembers the last few current paths progress named, apart from the 200 shown, so a folder's current path is not added again as a duplicate line after the other folder's names pushed it out.

A cancel or failure used to deliver the names git had printed for a group it never finished. The reporter now holds a group's names and counts until its add ends and drops them on Abandon; names handled before the group (removals) are delivered when the group begins.

On a retry, a file an earlier attempt staged that the leave-out rules now name is already left out of the count by CountStagedBeforeAsync; a regression test now pins it.

The two big-file detail lines now say "Probably" and "about", since the file is worked out from how much is stored.

Observed reds:
- ProgressReporterNamesTests.Names_of_a_group_git_did_not_finish_are_never_delivered_when_the_capture_stops: Assert.Empty failed, collection was [a.bin, b.bin, c.bin]
- ProgressReporterNamesTests.A_group_that_ended_keeps_its_names_when_the_next_one_is_stopped: expected [a.bin, b.bin], actual [a.bin, b.bin, c.bin]
- ProgressReporterNamesTests.A_removal_made_before_the_first_group_is_kept_when_that_group_is_stopped: expected [gone1.bin, gone2.bin], actual [gone1.bin, gone2.bin, a.bin]
- JobLineFormatterTests.Every_byte_count_of_a_step_is_shown_as_a_size: expected "Probably copying Data/big.pak (4.5 GB).", actual "Copying Data/big.pak (4.5 GB)."
- ActivityScreenViewModelTests: Copy_with_nothing_selected_takes_every_line_the_job_holds_not_only_the_ones_shown (expected 251 lines, actual 200), and the two note tests (note empty) against stubs of GetLines and LinesNote.
- JobRegistryTests dedupe and history tests were written against an empty GetLines stub and then implemented together; they were checked by mutation instead.

Mutation checks: removing the staged-count exclusion of named files failed the retry test (expected 9, actual 10); a never-matching seen-path check failed the duplicate-line test (expected 1, actual 2); letting the history grow one past the limit failed the limit test (20000 vs 20001); dropping the count increment at group end failed two reporter tests; ignoring the selection in Copy failed three Copy tests.
- A pack with no index (git renames the pack before its index, so a killed
  add can leave one, and the capture still keeps it) made the plan throw, so
  every Free up space with something unreachable failed. Such a pack is now
  neither read nor rewritten, and its keep stays: nothing can be told about
  what it holds, and git's repack ignores it either way.
- The plan no longer collects whole listings into strings. Object names are
  handed over line by line (OnStdoutLine) into one set, and the reachable
  names are removed from it as they arrive. Only the indexes of kept packs
  are read, and the matches are found as the lines arrive. An unreachable
  object that is neither loose (named from the loose-object folders) nor in
  a kept pack must sit in an unkept pack, so unkept indexes are not read.
- Our keeps whose pack is gone are removed even when nothing is unreachable.
- A reclaim, and the check before records are cleared, are refused with the
  interrupted-switch error while the switch note exists, because the note
  can name a point nothing else reaches and a prune would take it.
- The RepackAsync doc no longer says a reclaim needs room for a copy of the
  store.

Observed reds before the change (ReclaimStreamedPackTests):
- A_pack_without_an_index_...: GitOperationException "A pack's index could
  not be read before the reclaim: Could not find file ...pack-1111....idx".
- A_keep_of_ours_whose_pack_is_gone_...: Assert.False() Failure, the stale
  keep was still there.
- A_reclaim_is_refused_while_a_switch_is_unfinished_...: Assert.Throws()
  Failure, no InterruptedSwitchException was thrown.
A_big_file_added_to_a_streamed_pack_and_never_committed_... passed on its
first run; mutation: dropping --indexed-objects from the rev-list fails it.
Version 0.9.10
Some checks failed
Gate / linux (pull_request) Failing after 11m19s
Gate / windows (pull_request) Successful in 32m25s
e27b997776
A process that forks after the /proc walk and then dies first hands its child to init, where it cannot be traced: the walk saw the shell, the shell forked the sleeper, the shell was killed, the sleeper lived on. The tree is now frozen with SIGSTOP level by level until a pass finds nothing new, and SIGKILL follows in the same call (the exit-time registry runs it before the host goes). The cancel test also waited only for any process carrying the marker, which is true as soon as git starts (before the shell and sleeper exist); on Linux it now waits for the sleeper itself. Not reproduced locally (needs a slow start); CI failure was a sleeper left alive after the cancel.
Describe the stop-then-kill tree walk on Linux
Some checks failed
Gate / linux (pull_request) Failing after 11m46s
Gate / windows (pull_request) Successful in 34m23s
7ad3e3255f
Expect the checksum-list script tests to skip where pwsh is missing
Some checks failed
Gate / linux (pull_request) Failing after 24m9s
Gate / windows (pull_request) Successful in 31m45s
4df453d049
Run the gate again: the Linux job stopped after 7 minutes with no test failure
All checks were successful
Gate / linux (pull_request) Successful in 12m4s
Gate / windows (pull_request) Successful in 31m58s
Wiki / publish (push) Successful in 52s
Release / linux (push) Successful in 2m27s
Gate / linux (push) Successful in 12m15s
Gate / windows (push) Successful in 31m42s
Release / windows (push) Successful in 2m41s
Release / checksums (push) Successful in 36s
4c57e3ed4c
Sign in to join this conversation.
No reviewers
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
ethan-hann/SavePoint!136
No description provided.