Experiment: a content-defined chunk store in place of git for big game installs #102

Open
opened 2026-10-24 22:36:33 -05:00 by ethan-hann · 1 comment
Owner

Follow-up to #71, to run after 1.0. AAA installs run to hundreds of gigabytes (Cyberpunk 2077 85 GB, GTA V over 100 GB), and every SavePoint operation on them feels slow. Small installs are fine.

Where git costs time and space today

  • git add hashes every new or changed file on one core, so on an NVMe drive one core, not the disk, sets the pace of a first snapshot (Cyberpunk: about 150 MB/s).
  • Git stores and switches whole files. A game update that changes 50 MB inside a 4 GB archive stores a new 4 GB object and rewrites 4 GB on every switch. This is the cost behind #71: keeping update rollback means a second copy of the big archives.
  • Loose objects have to be packed afterwards (Free up space), even after #92.

Where git is not the problem

  • A first snapshot is a full copy with any tool. On a mid-range PC with a SATA SSD or a hard drive the disk sets the pace.
  • Later snapshots are already cheap: git skips files whose size and time did not change.

The idea
Content-defined chunking, as restic, kopia and borg do it: files are cut into variable-size pieces by their content, so an update stores roughly what changed, hashing runs on every core, there is no repack step, and a restore can patch a big file instead of rewriting it. This is the most likely way to keep update rollback without a second full copy (#71).

The experiment

  1. Profile a first snapshot, a later snapshot after a game update, and a switch back on the Windows CI VM (4 cores, the mid-tier stand-in) and on a fast PC: CPU-bound or disk-bound, per step.
  2. Prototype the same three operations behind the engine boundary with a chunk store: kopia (Apache-2.0) or restic (BSD-2-Clause) as a bundled tool, or a small chunk store in .NET. Measure time, peak memory and disk on Cyberpunk 2077 (GOG) and 7 Days to Die, including a real game update.
  3. Compare with git, list what the rest of the design would have to change (profiles, history, Go back, Recently removed, empty folders, the safety checks, D-decisions, the test suite), and decide whether to switch, use chunking only for large files beside git, or stay.

No change to the shipped engine before 1.0. Judge the result on a mid-range PC, and never trade away update rollback.

Follow-up to #71, to run after 1.0. AAA installs run to hundreds of gigabytes (Cyberpunk 2077 85 GB, GTA V over 100 GB), and every SavePoint operation on them feels slow. Small installs are fine. **Where git costs time and space today** - `git add` hashes every new or changed file on one core, so on an NVMe drive one core, not the disk, sets the pace of a first snapshot (Cyberpunk: about 150 MB/s). - Git stores and switches whole files. A game update that changes 50 MB inside a 4 GB archive stores a new 4 GB object and rewrites 4 GB on every switch. This is the cost behind #71: keeping update rollback means a second copy of the big archives. - Loose objects have to be packed afterwards (Free up space), even after #92. **Where git is not the problem** - A first snapshot is a full copy with any tool. On a mid-range PC with a SATA SSD or a hard drive the disk sets the pace. - Later snapshots are already cheap: git skips files whose size and time did not change. **The idea** Content-defined chunking, as restic, kopia and borg do it: files are cut into variable-size pieces by their content, so an update stores roughly what changed, hashing runs on every core, there is no repack step, and a restore can patch a big file instead of rewriting it. This is the most likely way to keep update rollback without a second full copy (#71). **The experiment** 1. Profile a first snapshot, a later snapshot after a game update, and a switch back on the Windows CI VM (4 cores, the mid-tier stand-in) and on a fast PC: CPU-bound or disk-bound, per step. 2. Prototype the same three operations behind the engine boundary with a chunk store: kopia (Apache-2.0) or restic (BSD-2-Clause) as a bundled tool, or a small chunk store in .NET. Measure time, peak memory and disk on Cyberpunk 2077 (GOG) and 7 Days to Die, including a real game update. 3. Compare with git, list what the rest of the design would have to change (profiles, history, Go back, Recently removed, empty folders, the safety checks, D-decisions, the test suite), and decide whether to switch, use chunking only for large files beside git, or stay. No change to the shipped engine before 1.0. Judge the result on a mid-range PC, and never trade away update rollback.
ethan-hann added this to the Later milestone 2026-10-24 22:36:33 -05:00
Author
Owner

Measurement: content-defined chunking on a real update (No Man's Sky 7.06 to 7.10)

Measured read-only on the existing SavePoint store, where both versions sit in install.git as two profiles. 19.6 GB of the 33.7 GB install is byte-identical between the versions and 14.2 GB sits in 29 changed files. Because the repo sets core.bigfilethreshold=128m, git stores every changed file over 128 MB whole, so this update costs the full 14.2 GB today.

Each changed file was cut with content-defined chunking (gear hash, about 8 KiB average chunks), counting the bytes of the 7.10 file whose chunks never appear in the 7.06 file:

File Size New bytes New %
NMSARC.audio.pak 5,519 MB 29 MB 0.5
NMSARC.audioBNK.pak 68 MB 2 MB 2.5
NMSARC.TexMisc.pak 1,568 MB 1,166 MB 74
NMSARC.TexSpace.pak 2,888 MB 2,229 MB 77
NMSARC.TexSpacecraft.pak 760 MB 643 MB 85
NMSARC.TexUI.pak 631 MB 565 MB 90
NMS.exe 89 MB 73 MB 81
NMSARC.MeshPlanetBIOMES.pak 977 MB 975 MB 100
NMSARC.MeshCommon.pak 744 MB 743 MB 100
NMSARC.MeshMisc.pak 527 MB 524 MB 100
7 smaller mesh, MBIN and shader paks 380 MB 380 MB 100
Total 14,182 MB 7,359 MB 52

What this shows:

  • Chunking halves this update, but almost all of the saving comes from one file. The 5.5 GB audio pak is 0.5% new, likely because its entries (already compressed Wwise audio) are stored uncompressed inside the archive.
  • Archives whose entries are compressed gain almost nothing. The mesh, MBIN and shader paks are 99 to 100% new, although a point release probably changed little inside them. HGPAK v2 appears to compress one continuous stream in 64 KiB blocks (MeshCommon: 3,935 entries in 36,911 blocks), so one inserted or resized entry shifts every block after it and all later compressed bytes differ. MeshCommon grew from 696 to 744 MB.
  • Texture paks fall in between at 74 to 90% new.

So chunking alone does not reach "store roughly what changed" on archives that compress across entries. That needs a step that decompresses the archive before chunking and rebuilds it byte-exact on restore, which this run did not measure.

Inventory: what the bytes are (7.7 TB, 297 installs and mod folders)

Read-only scan of every Steam library, GOG, MO2 and Vortex mod folders: header and footer sniffing plus compressibility samples on files over 1 MB. Classification is heuristic.

Bucket Size Share
Custom engine formats, not identified (Destiny 2, Call of Duty, Decima, Glacier, idTech, Northlight) 2.8 TB 36%
Known containers: Unreal pak/IoStore 1.2 TB (much of it Oodle), zip 342 GB, BA2/BSA 176 GB, Unity bundles 181 GB, VPK 147 GB, REDengine 121 GB, Wwise 118 GB 2.3 TB 30%
Dense data with no known header (GTA V, Battlefield 6, Avengers, Metro Exodus, Elden Ring) 0.9 TB 12%
Unreal paks with an encrypted index (Black Myth Wukong, Borderlands 3, Lies of P, PUBG and others) 0.5 TB 6%
Video and audio 0.4 TB 5%
Zlib streams inside otherwise unknown files (Shadow of Mordor, Witcher 3, Arkham Knight) 0.3 TB 4%

Mods follow the same pattern: 12 of the 15 GB in MO2 and Vortex folders are LZ4HC Unity bundles.

What this means for the experiment:

  • No small set of format handlers covers most of the bytes. The Unreal family is the largest identifiable group at about 22%, and much of it uses Oodle or an encrypted index, which only the game's own codec or key can unpack and which a third party cannot rebuild byte-exact.
  • Custom engine formats are the largest bucket and spread thin: the top 10 games hold 39% of it.
  • Plan for chunking as the general mechanism, with format-aware unpacking at most a narrow add-on for a few measured formats (Unity LZ4HC bundles, zip, embedded zlib), each verified byte-exact with a raw fallback.

Other findings:

  • Some installs ship debug symbols: Satisfactory carries 19 GB of .pdb files, 22 GB across the library.
  • Git for Windows 2.45 reports "bad object header" on objects over 4 GB (the NMS audio pak is 5.5 GB). The bundled git 2.56 reads them, so any tool that touches the store has to use the bundled git.

Suggested next measurements:

  1. A second version pair from another engine (7 Days to Die or Cyberpunk 2077) with the same chunk measurement.
  2. Unpack HGPAK entries for both NMS versions and measure entry-level overlap, to size what a decompress-then-chunk step would add over plain chunking.
  3. Repeat at 4 and 16 KiB average chunks, since chunk size moves both the saving and the index size.
**Measurement: content-defined chunking on a real update (No Man's Sky 7.06 to 7.10)** Measured read-only on the existing SavePoint store, where both versions sit in `install.git` as two profiles. 19.6 GB of the 33.7 GB install is byte-identical between the versions and 14.2 GB sits in 29 changed files. Because the repo sets `core.bigfilethreshold=128m`, git stores every changed file over 128 MB whole, so this update costs the full 14.2 GB today. Each changed file was cut with content-defined chunking (gear hash, about 8 KiB average chunks), counting the bytes of the 7.10 file whose chunks never appear in the 7.06 file: | File | Size | New bytes | New % | |---|---:|---:|---:| | NMSARC.audio.pak | 5,519 MB | 29 MB | 0.5 | | NMSARC.audioBNK.pak | 68 MB | 2 MB | 2.5 | | NMSARC.TexMisc.pak | 1,568 MB | 1,166 MB | 74 | | NMSARC.TexSpace.pak | 2,888 MB | 2,229 MB | 77 | | NMSARC.TexSpacecraft.pak | 760 MB | 643 MB | 85 | | NMSARC.TexUI.pak | 631 MB | 565 MB | 90 | | NMS.exe | 89 MB | 73 MB | 81 | | NMSARC.MeshPlanetBIOMES.pak | 977 MB | 975 MB | 100 | | NMSARC.MeshCommon.pak | 744 MB | 743 MB | 100 | | NMSARC.MeshMisc.pak | 527 MB | 524 MB | 100 | | 7 smaller mesh, MBIN and shader paks | 380 MB | 380 MB | 100 | | **Total** | **14,182 MB** | **7,359 MB** | **52** | What this shows: - Chunking halves this update, but almost all of the saving comes from one file. The 5.5 GB audio pak is 0.5% new, likely because its entries (already compressed Wwise audio) are stored uncompressed inside the archive. - Archives whose entries are compressed gain almost nothing. The mesh, MBIN and shader paks are 99 to 100% new, although a point release probably changed little inside them. HGPAK v2 appears to compress one continuous stream in 64 KiB blocks (MeshCommon: 3,935 entries in 36,911 blocks), so one inserted or resized entry shifts every block after it and all later compressed bytes differ. MeshCommon grew from 696 to 744 MB. - Texture paks fall in between at 74 to 90% new. So chunking alone does not reach "store roughly what changed" on archives that compress across entries. That needs a step that decompresses the archive before chunking and rebuilds it byte-exact on restore, which this run did not measure. **Inventory: what the bytes are (7.7 TB, 297 installs and mod folders)** Read-only scan of every Steam library, GOG, MO2 and Vortex mod folders: header and footer sniffing plus compressibility samples on files over 1 MB. Classification is heuristic. | Bucket | Size | Share | |---|---:|---:| | Custom engine formats, not identified (Destiny 2, Call of Duty, Decima, Glacier, idTech, Northlight) | 2.8 TB | 36% | | Known containers: Unreal pak/IoStore 1.2 TB (much of it Oodle), zip 342 GB, BA2/BSA 176 GB, Unity bundles 181 GB, VPK 147 GB, REDengine 121 GB, Wwise 118 GB | 2.3 TB | 30% | | Dense data with no known header (GTA V, Battlefield 6, Avengers, Metro Exodus, Elden Ring) | 0.9 TB | 12% | | Unreal paks with an encrypted index (Black Myth Wukong, Borderlands 3, Lies of P, PUBG and others) | 0.5 TB | 6% | | Video and audio | 0.4 TB | 5% | | Zlib streams inside otherwise unknown files (Shadow of Mordor, Witcher 3, Arkham Knight) | 0.3 TB | 4% | Mods follow the same pattern: 12 of the 15 GB in MO2 and Vortex folders are LZ4HC Unity bundles. What this means for the experiment: - No small set of format handlers covers most of the bytes. The Unreal family is the largest identifiable group at about 22%, and much of it uses Oodle or an encrypted index, which only the game's own codec or key can unpack and which a third party cannot rebuild byte-exact. - Custom engine formats are the largest bucket and spread thin: the top 10 games hold 39% of it. - Plan for chunking as the general mechanism, with format-aware unpacking at most a narrow add-on for a few measured formats (Unity LZ4HC bundles, zip, embedded zlib), each verified byte-exact with a raw fallback. Other findings: - Some installs ship debug symbols: Satisfactory carries 19 GB of `.pdb` files, 22 GB across the library. - Git for Windows 2.45 reports "bad object header" on objects over 4 GB (the NMS audio pak is 5.5 GB). The bundled git 2.56 reads them, so any tool that touches the store has to use the bundled git. Suggested next measurements: 1. A second version pair from another engine (7 Days to Die or Cyberpunk 2077) with the same chunk measurement. 2. Unpack HGPAK entries for both NMS versions and measure entry-level overlap, to size what a decompress-then-chunk step would add over plain chunking. 3. Repeat at 4 and 16 KiB average chunks, since chunk size moves both the saving and the index size.
Sign in to join this conversation.
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
ethan-hann/SavePoint#102
No description provided.