Use less disk and time for large game installs without losing update rollback #71
Labels
No labels
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No assignees
1 participant
Notifications
Due date
No due date set.
Blocks
#102 Experiment: a content-defined chunk store in place of git for big game installs
ethan-hann/SavePoint
Reference
ethan-hann/SavePoint#71
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Every game's first snapshot copies its whole game folder into SavePoint's storage, so each game costs its full size twice on disk, and a big game's first snapshot takes minutes (Cyberpunk 2077: 85 GiB, about 9 minutes with D50, 35 before). For a library of hundreds of games that is most of a drive. Extra profiles are already cheap (identical files are stored once per game); the cost is the one base copy per game.
That base copy cannot be shared with the game folder on the usual file systems: git stores files in its own wrapped format, so hard links cannot point at the same bytes, and only copy-on-write file systems (ReFS / Dev Drive, Btrfs, XFS, APFS) could clone them, which most Windows players do not use.
The lever is tracking less. On a modded game nearly all the bytes sit in files no profile ever changes (Cyberpunk 2077: 98.8% in
archive/pc/content); profiles differ in the mod folders (archive/pc/mod,r6/,red4ext/,bin/x64/plugins/,mods/) and the saves.Proposal: when a game is added, the player chooses what SavePoint keeps: Whole game folder (version, mods and saves; what it does today) or Mods and saves only, with the mod folders taken from the game's template (#69 gathers them) and editable. Mods and saves only makes the first snapshot take seconds and use almost no space. The trade-off, said plainly: SavePoint can no longer take the game back to an older version after an update; the store can download the game again. A third option, Saves only, needs no game folder at all.
Open points: compile an include list into layered exclude patterns in
info/exclude(git cannot re-include under an excluded folder), check that previews and the clean check stay fast with most of the folder ignored, decide what a switch does with files outside the tracked folders, and how a game can change its choice later. Record the call in docs/decisions.md.Research findings, 2026-10-24. Not decided; recorded for a later look. Full notes with sources were gathered for both parts below.
Requirements from the maintainer: rolling back a bad game update is a core feature and must stay whole; a per-game "mods and saves only" mode (this issue's original proposal) is rejected for that reason. Any design must hold up on a typical mid-range pre-built PC (one NTFS drive, SATA SSD or hard disk, less RAM and CPU), not only on a high-end machine.
Where the space and time go. Profiles are already cheap: a second profile stores only what differs. The cost is the one base copy of the game folder per game, which git stores in its own format. Measured on Cyberpunk 2077 (85 GiB): 98.8% of the bytes are 17 large archives that never differ between profiles; the first snapshot takes about 9 minutes with D50 on a fast NVMe machine and will take far longer on a mid-range disk.
File systems. Zero-copy clones exist only on ReFS / Dev Drive (Windows), Btrfs, XFS and ZFS 2.2+ (Linux) and APFS (macOS). NTFS has no block cloning, and Microsoft advises against installing apps on a Dev Drive, so most Windows players cannot use it. Hard links are not safe: a program that writes a file in place changes the backup too, and how Steam and GOG apply updates (in place or replace) is undocumented. Volume shadow copies need administrator rights and Windows prunes them.
Stores as the source of old builds. Steam keeps old builds, but downloading one is a console command the player types (
download_depot), works only when the developer kept the build, and cannot be automated without third-party tools (DepotDownloader, GPL-2.0, needs the player's credentials). GOG Galaxy offers rollback only for games whose developers opted in. So the store cannot be relied on for rollback. What a program can read: Steam'sappmanifest_*.acfshows the installed build, the target build and a pending update before it is applied (buildidvsTargetBuildID,StateFlags,BytesToDownload), andAutoUpdateBehavior(1 = update only on launch);depotcache/*.manifestholds per-file hashes, possibly with encrypted names. GOG leavesgoggame-<id>.infowith the build id and a hash file.Ideas considered. (1) Copy a large file only right before it changes (SavePoint's own switches know which files they replace; a pending Steam update is visible at launch when the game updates only on launch); the first snapshot would copy only small files. It splits the rollback guarantee: an update or a tool SavePoint cannot see in advance leaves no old copy of the files it changed. (2) Instant full clones where the drive supports them, with today's full copy elsewhere. (3) Content-defined chunking (as restic, Borg, Kopia do) to store only the changed parts of a large archive when an update touches it; this shrinks the growth after updates, not the first copy. Worth a measurement on two real builds of one game. (4) git-annex-style tracking (large files as hashed entries, bytes kept outside git), the storage shape (1) and (2) would need.
Next step when this is picked up: measure on a mid-range machine (first snapshot time and disk on SATA SSD and hard disk), and measure how much of a large archive one real update changes.
Let a game track only its mods and saves, not the whole game folderto Use less disk and time for large game installs without losing update rollbackRecorded the chunk-store experiment as #102, to run after 1.0. It is the most likely way to keep update rollback without a second full copy of the big archives.