Real progress for big snapshots, and keep working while one runs #47

Merged
ethan-hann merged 3 commits from issue-37-snapshot-progress into main 2026-10-23 23:25:12 -05:00
Owner

Fixes #37 and #46.

Reproduced first with three 1.5 GiB random files plus 20 small ones, the bundled git, and SavePoint's repo config: git add printed nothing for the whole add, and both progress reports arrived together at 125 s after about 4.8 GB was written, so a count of files cannot move during a big file. The combined bar also waited for both folders to report a total, which is why the total was missing ("1 files copied so far"). What does grow is objects/pack/tmp_pack_* (files above core.bigFileThreshold stream into it): 9 MB at 0.5 s, 283 MB at 5.6 s, 4.8 GB at 103 s, ending at about the source size for incompressible data; then about 22 s of finalising with nothing growing. On Windows the size is only live when the file is opened with shared access and its length read.

  • Capture progress reports file and byte totals at once, and samples the object store every 500 ms while git add runs (ObjectStoreMeter, 11 to 16 ms per sample at 40,000 loose objects). The figure never goes backwards and stops a hundredth short of the total until the capture finishes.
  • The wizard leads with "3.1 GB of 70 GB copied" on a determinate bar, the file count second, plural rules, "Working out how much there is to copy..." when no total is known, and "Making sure everything is stored safely..." at the cap. The last file line reads "Latest file finished: X".
  • Keep working closes the wizard and leaves the snapshot running as its job (Activity, the status pill, a message when it finishes).
  • #46: the switch dialog, the save views, and Activity share the same ProgressWording; the "so far" keys are gone.

Re-run with the fix on the same data: the first event at 0.1 s with the byte total, 185 progress events, 314 MB at 5.7 s, 2.2 GB at 42.7 s, 4.7 GB at 92.3 s. Not yet seen on a real 70 GB game in the live app. Compressible data reads low until the end, because the store holds compressed bytes (#45 is the pack.compression call).

Fixes #37 and #46. Reproduced first with three 1.5 GiB random files plus 20 small ones, the bundled git, and SavePoint's repo config: git add printed nothing for the whole add, and both progress reports arrived together at 125 s after about 4.8 GB was written, so a count of files cannot move during a big file. The combined bar also waited for both folders to report a total, which is why the total was missing ("1 files copied so far"). What does grow is objects/pack/tmp_pack_* (files above core.bigFileThreshold stream into it): 9 MB at 0.5 s, 283 MB at 5.6 s, 4.8 GB at 103 s, ending at about the source size for incompressible data; then about 22 s of finalising with nothing growing. On Windows the size is only live when the file is opened with shared access and its length read. - Capture progress reports file and byte totals at once, and samples the object store every 500 ms while git add runs (ObjectStoreMeter, 11 to 16 ms per sample at 40,000 loose objects). The figure never goes backwards and stops a hundredth short of the total until the capture finishes. - The wizard leads with "3.1 GB of 70 GB copied" on a determinate bar, the file count second, plural rules, "Working out how much there is to copy..." when no total is known, and "Making sure everything is stored safely..." at the cap. The last file line reads "Latest file finished: X". - Keep working closes the wizard and leaves the snapshot running as its job (Activity, the status pill, a message when it finishes). - #46: the switch dialog, the save views, and Activity share the same ProgressWording; the "so far" keys are gone. Re-run with the fix on the same data: the first event at 0.1 s with the byte total, 185 progress events, 314 MB at 5.7 s, 2.2 GB at 42.7 s, 4.7 GB at 92.3 s. Not yet seen on a real 70 GB game in the live app. Compressible data reads low until the end, because the store holds compressed bytes (#45 is the pack.compression call).
A first snapshot of a game with multi-gigabyte files showed an indeterminate bar,
"1 files copied so far" and a stale file name for many minutes, and the dialog
could not be left.

What the reproduction showed (three 1.5 GiB random files plus 20 small ones,
real bundled git, SavePoint's repo config, Windows 11):
- git add --verbose printed nothing for the whole add. Both progress reports
  (first file, then 23 of 23) arrived together at 125.3 s, after about 4.8 GB was
  written. File counts cannot move while one large file is stored.
- The file total reached the screen only after the first report, and the
  combined bar needs a total from both folders. The folder that never reported
  (here the install folder) kept the total unknown for the whole add; the saves
  folder alone showed "1 files copied so far" and its one file name.
- Files above core.bigFileThreshold (128m) stream into ONE objects/pack/tmp_pack_*
  file that grows steadily: 9 MB at 0.5 s, 283 MB at 5.6 s, 4,834 MB at 103 s,
  about 50 MB/s, ending at almost exactly the source size (4,833 MB for
  4,832 MB of files, because the streamed pack data is barely compressed for
  incompressible input). After 103 s nothing under objects/ grows for about
  22 s while git checksums, indexes and renames the pack (125 s total).
- On Windows a directory entry's size lags or reads as 0 while git holds the file
  open (an earlier run with FileInfo.Length saw 510 bytes at 95 s). Opening the
  file with FileShare.ReadWrite|Delete and asking for its length gave the live
  figure, so the meter does that for tmp_* files.

Changes:
- The capture sizes the files it will stage, reports the file and byte totals
  at once, and while git add runs samples the object store every 500 ms (growth
  since the start, never backwards, held a hundredth under the total until the
  capture completes). One last sample is taken when staging ends. Measuring the
  store cost 11 to 16 ms for 40,000 loose objects.
- The combined bar adds bytes as well as files, and a folder that ends (or had
  nothing to store) no longer holds the total back.
- The wizard leads with "3.1 GB of 70 GB copied" and a determinate bar, keeps
  the file count as a second line, says "Working out how much there is to copy"
  when no total is known, uses plural rules for file counts, labels the file name
  as the latest one finished (git names a file only after it is stored), and says
  "Making sure everything is stored safely" while the last stretch runs.
- A "Keep working" button next to Cancel closes the wizard and leaves the same
  job running; the pill, Activity, the finished toast and the table reload
  already follow the job. Closing never cancels.

Re-run with the fix (same data): first progress event at 0.1 s with
BytesTotal 4,831,838,358; 185 progress events; BytesDone 28 MB at 0.6 s,
314 MB at 5.7 s, 2.2 GB at 42.7 s, 4.7 GB at 92.3 s; the file-count reports still
arrive only at 113.0 s, so the final report reaches 23 of 23 files and 100% of
the bytes at the end. Total 113 s.

Observed reds before the fix (all assertion failures; stubs for the new types
were in place so nothing failed to compile):
- ProgressReporterBytesTests: 6 of 7 failed (no reports at all from Start and
  ObserveWritten; BytesDone null).
- StagingByteSamplerTests: 4 of 4 failed (no reports).
- ObjectStoreMeterTests: 2 of 4 failed (35 expected, 0 actual; 1,000,000
  expected, 0 actual).
- CaptureProgressCombinerTests: 4 of 4 failed (bytes null, total null).
- CaptureBytesTests: 4 of 4 failed (first report was 1 of 2 files with no bytes).
- CaptureProgressTests.A_later_capture_counts_only_the_files_that_changed
  updated for the new first and last reports and failed on them.
- NewProfileWizardViewModelTests: 13 failed (for example expected "3.1 GB of
  70 GB copied", got "0 of 23 files copied"; expected "25% copied", got "25% of
  the files are copied"; ShowKeepWorking false).
Tests that passed on their first run were mutation checked: the meter's two
error paths, the reporter's no-bytes case, the engine's byte total (reported 0),
and the Keep working button's visibility.
Not covered by a test: the multi-gigabyte run itself (manual, above).
The switch progress, the two "saving your session" views and Activity still said
"12 files copied so far" or "812 files put in place so far" when no total was
known, said "1 of 1 files", and the save views ignored the byte totals a
capture now reports.

One helper, ProgressWording, now owns the words: bytes of a known total, files
of a known total with plural rules and grouped numbers, "Working out how much
there is to copy" while no total is known, and the fraction (bytes first, then
files). The switch dialog, LaunchSavingViewModel, UnsavedChangesBlockViewModel,
ActivityText and the wizard use it. The per-screen keys it replaces are removed
from the catalog, and launch.progress.filesWithTime takes the finished "progress"
text instead of its own counts.

Observed reds before the change (all assertion failures; the helper existed as
a stub returning empty/null so nothing failed to compile):
- ProgressWordingTests: 6 of 6 (for example expected "1,200 of 3,430 files",
  got "").
- LaunchDialogViewModelTests: expected "Working out how much there is to copy",
  got "12 files copied so far"; a bytes report left the bar at 0 instead of 50.
- LaunchSavingViewModelTests: expected "35 GB of 70 GB", got "0 of 3 files";
  expected the plain-words text, got "12 files put in place so far".
- SwitchProgressDialogViewModelTests: "812 files put in place so far" and
  "1 of 1 files".
- JobRowViewModelTests: "7 files so far".
The Activity "1 of 1 file" test passed on its first run (Activity already had the
plural); it guards the shared helper that now produces that text.
Merge branch 'worktree-agent-a318d233c8cc2fa62' into integration-091
Some checks failed
Gate / linux (pull_request) Successful in 13m53s
Gate / linux (push) Failing after 13m32s
97e2d5125e
ethan-hann deleted branch issue-37-snapshot-progress 2026-10-23 23:25:13 -05:00
Sign in to join this conversation.
No reviewers
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
ethan-hann/SavePoint!47
No description provided.