Faster first Free up space on Unity games, each file once in job details, Windows gate in CI #93

Merged
ethan-hann merged 8 commits from integration-097 into main 2026-10-24 19:14:58 -05:00
Owner
  • Faster routine Free up space (#92): Unity's block-compressed containers (bundle, unity3d, assetbundle, resource) skip delta search, and the routine and reclaim repacks compress at zlib level 1. Measured on a real 7 Days to Die store (21.5 GB, 18,090 files, 4 pack threads, NVMe): the first routine repack took 316 s instead of 570 s, for a store 1% larger. Unity's assets and resS keep delta search because they compress well; .raw and .wav stay off the list too. D16 and D50 record the numbers.
  • Each file once in a job's details (#91): progress reports repeat the current file until git names the next one, and each report added a details line, so a file showed up to sixteen times in a row. A path already among the recent lines is no longer added again.
  • Windows gate in CI: a new windows job runs build/gate.ps1 on a self-hosted Windows 11 VM (forgejo-runner in host mode under a standard account), with the same documentation-only shortcut as the Linux job. AGENTS.md and CONTRIBUTING.md will say so once the job has passed.

Closes #91
Closes #92

- **Faster routine Free up space (#92):** Unity's block-compressed containers (`bundle`, `unity3d`, `assetbundle`, `resource`) skip delta search, and the routine and reclaim repacks compress at zlib level 1. Measured on a real 7 Days to Die store (21.5 GB, 18,090 files, 4 pack threads, NVMe): the first routine repack took 316 s instead of 570 s, for a store 1% larger. Unity's `assets` and `resS` keep delta search because they compress well; `.raw` and `.wav` stay off the list too. D16 and D50 record the numbers. - **Each file once in a job's details (#91):** progress reports repeat the current file until git names the next one, and each report added a details line, so a file showed up to sixteen times in a row. A path already among the recent lines is no longer added again. - **Windows gate in CI:** a new `windows` job runs `build/gate.ps1` on a self-hosted Windows 11 VM (forgejo-runner in host mode under a standard account), with the same documentation-only shortcut as the Linux job. AGENTS.md and CONTRIBUTING.md will say so once the job has passed. Closes #91 Closes #92
While git stores a large file, the capture's byte count is reported many times a
second and every report carries the same current file. The job registry turned
each report's path into a detail line, so one file showed up as a run of
identical lines (7 Days to Die: the same png about sixteen times). A progress
path that is already among the recent path lines is now skipped. Switches use
the same reporting path and are covered by it.

Observed reds before the fix:
- A_file_that_stays_current_across_many_reports_is_one_recent_line: expected
  [a.png, b.png], actual [a.png, a.png, a.png, a.png, a.png, ...]
- Two_folders_taking_turns_on_their_own_files_still_show_each_file_once:
  expected [a.png, b.sav], actual [a.png, b.sav, a.png, b.sav, a.png, ...]
A 21.5 GB Unity install (7 Days to Die, 285 bundles holding 14 GB) took 570 s to
repack on 4 threads after a 200 s first snapshot. Delta search over the
incompressible bundles was most of it (500 s -> 237 s on the same store at
zlib level 1, and 570 -> 333 s at the default level), and the default zlib
level was most of the rest (333 s -> 237 s with level 1; level 0 gave 68 s but
stored the 2.5 GB of compressible small files raw).

- info/attributes marks bundle, unity3d, assetbundle and resource -delta.
  assets and resS compress to 13-68% in real installs and stay delta-searched.
- Routine and reclaim repacks run with pack.compression = 1 instead of the
  default level. Small saves and configs still compress; data already in a
  pack is copied as it is, so update rollback and old packs are untouched.
- The attributes file is rewritten on every write, so existing stores pick the
  list up without a config stamp bump; the compression level is code.

Observed reds:
- IgnoreFilesTests.Already_compressed_formats_skip_delta_compression for
  bundle, unity3d, assetbundle, resource: Expected "data/big.bundle: delta:
  unset", Actual "...: delta: unspecified".
- MaintenanceTests.A_repack_asks_zlib_for_its_fastest_level_through_the_environment
  (Routine and Reclaim): Expected "1", Actual "-1".
- Unity_data_that_compresses_keeps_delta_compression passed on its first run;
  mutation check: adding assets, resS, sharedAssets to the list failed all
  three with "delta: unset", then restored.
Run the Windows gate in CI on a self-hosted Windows runner
Some checks failed
Gate / windows (pull_request) Failing after 2h0m36s
Gate / linux (pull_request) Has been cancelled
da9045c339
A Windows 11 VM runs forgejo-runner in host mode under a standard
account, labeled windows. The job mirrors the Linux one: it passes at
once when only documentation changed, installs the newest 10.0 SDK into
the runner's tool cache, and runs build/gate.ps1 with its skip list.
Check the base commit outside the condition in the Windows scope step
Some checks failed
Gate / linux (pull_request) Successful in 11m56s
Gate / windows (pull_request) Failing after 37m45s
Gate / linux (push) Successful in 12m42s
Gate / windows (push) Successful in 26m49s
be3821058d
PowerShell does not take two statements inside a parenthesised
expression, so the step failed to parse.
Sign in to join this conversation.
No reviewers
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
ethan-hann/SavePoint!93
No description provided.