Experiment: a content-defined chunk store in place of git for big game installs #102
Labels
No labels
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No assignees
1 participant
Notifications
Due date
No due date set.
Depends on
#71 Use less disk and time for large game installs without losing update rollback
ethan-hann/SavePoint
Reference
ethan-hann/SavePoint#102
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Follow-up to #71, to run after 1.0. AAA installs run to hundreds of gigabytes (Cyberpunk 2077 85 GB, GTA V over 100 GB), and every SavePoint operation on them feels slow. Small installs are fine.
Where git costs time and space today
git addhashes every new or changed file on one core, so on an NVMe drive one core, not the disk, sets the pace of a first snapshot (Cyberpunk: about 150 MB/s).Where git is not the problem
The idea
Content-defined chunking, as restic, kopia and borg do it: files are cut into variable-size pieces by their content, so an update stores roughly what changed, hashing runs on every core, there is no repack step, and a restore can patch a big file instead of rewriting it. This is the most likely way to keep update rollback without a second full copy (#71).
The experiment
No change to the shipped engine before 1.0. Judge the result on a mid-range PC, and never trade away update rollback.
Measurement: content-defined chunking on a real update (No Man's Sky 7.06 to 7.10)
Measured read-only on the existing SavePoint store, where both versions sit in
install.gitas two profiles. 19.6 GB of the 33.7 GB install is byte-identical between the versions and 14.2 GB sits in 29 changed files. Because the repo setscore.bigfilethreshold=128m, git stores every changed file over 128 MB whole, so this update costs the full 14.2 GB today.Each changed file was cut with content-defined chunking (gear hash, about 8 KiB average chunks), counting the bytes of the 7.10 file whose chunks never appear in the 7.06 file:
What this shows:
So chunking alone does not reach "store roughly what changed" on archives that compress across entries. That needs a step that decompresses the archive before chunking and rebuilds it byte-exact on restore, which this run did not measure.
Inventory: what the bytes are (7.7 TB, 297 installs and mod folders)
Read-only scan of every Steam library, GOG, MO2 and Vortex mod folders: header and footer sniffing plus compressibility samples on files over 1 MB. Classification is heuristic.
Mods follow the same pattern: 12 of the 15 GB in MO2 and Vortex folders are LZ4HC Unity bundles.
What this means for the experiment:
Other findings:
.pdbfiles, 22 GB across the library.Suggested next measurements: