Use less disk and time for large game installs without losing update rollback #71

Open
opened 2026-10-24 12:21:52 -05:00 by ethan-hann · 2 comments
Owner

Every game's first snapshot copies its whole game folder into SavePoint's storage, so each game costs its full size twice on disk, and a big game's first snapshot takes minutes (Cyberpunk 2077: 85 GiB, about 9 minutes with D50, 35 before). For a library of hundreds of games that is most of a drive. Extra profiles are already cheap (identical files are stored once per game); the cost is the one base copy per game.

That base copy cannot be shared with the game folder on the usual file systems: git stores files in its own wrapped format, so hard links cannot point at the same bytes, and only copy-on-write file systems (ReFS / Dev Drive, Btrfs, XFS, APFS) could clone them, which most Windows players do not use.

The lever is tracking less. On a modded game nearly all the bytes sit in files no profile ever changes (Cyberpunk 2077: 98.8% in archive/pc/content); profiles differ in the mod folders (archive/pc/mod, r6/, red4ext/, bin/x64/plugins/, mods/) and the saves.

Proposal: when a game is added, the player chooses what SavePoint keeps: Whole game folder (version, mods and saves; what it does today) or Mods and saves only, with the mod folders taken from the game's template (#69 gathers them) and editable. Mods and saves only makes the first snapshot take seconds and use almost no space. The trade-off, said plainly: SavePoint can no longer take the game back to an older version after an update; the store can download the game again. A third option, Saves only, needs no game folder at all.

Open points: compile an include list into layered exclude patterns in info/exclude (git cannot re-include under an excluded folder), check that previews and the clean check stay fast with most of the folder ignored, decide what a switch does with files outside the tracked folders, and how a game can change its choice later. Record the call in docs/decisions.md.

Every game's first snapshot copies its whole game folder into SavePoint's storage, so each game costs its full size twice on disk, and a big game's first snapshot takes minutes (Cyberpunk 2077: 85 GiB, about 9 minutes with D50, 35 before). For a library of hundreds of games that is most of a drive. Extra profiles are already cheap (identical files are stored once per game); the cost is the one base copy per game. That base copy cannot be shared with the game folder on the usual file systems: git stores files in its own wrapped format, so hard links cannot point at the same bytes, and only copy-on-write file systems (ReFS / Dev Drive, Btrfs, XFS, APFS) could clone them, which most Windows players do not use. The lever is tracking less. On a modded game nearly all the bytes sit in files no profile ever changes (Cyberpunk 2077: 98.8% in `archive/pc/content`); profiles differ in the mod folders (`archive/pc/mod`, `r6/`, `red4ext/`, `bin/x64/plugins/`, `mods/`) and the saves. Proposal: when a game is added, the player chooses what SavePoint keeps: **Whole game folder** (version, mods and saves; what it does today) or **Mods and saves only**, with the mod folders taken from the game's template (#69 gathers them) and editable. Mods and saves only makes the first snapshot take seconds and use almost no space. The trade-off, said plainly: SavePoint can no longer take the game back to an older version after an update; the store can download the game again. A third option, **Saves only**, needs no game folder at all. Open points: compile an include list into layered exclude patterns in `info/exclude` (git cannot re-include under an excluded folder), check that previews and the clean check stay fast with most of the folder ignored, decide what a switch does with files outside the tracked folders, and how a game can change its choice later. Record the call in docs/decisions.md.
ethan-hann added this to the 1.0 milestone 2026-10-24 12:21:52 -05:00
Author
Owner

Research findings, 2026-10-24. Not decided; recorded for a later look. Full notes with sources were gathered for both parts below.

Requirements from the maintainer: rolling back a bad game update is a core feature and must stay whole; a per-game "mods and saves only" mode (this issue's original proposal) is rejected for that reason. Any design must hold up on a typical mid-range pre-built PC (one NTFS drive, SATA SSD or hard disk, less RAM and CPU), not only on a high-end machine.

Where the space and time go. Profiles are already cheap: a second profile stores only what differs. The cost is the one base copy of the game folder per game, which git stores in its own format. Measured on Cyberpunk 2077 (85 GiB): 98.8% of the bytes are 17 large archives that never differ between profiles; the first snapshot takes about 9 minutes with D50 on a fast NVMe machine and will take far longer on a mid-range disk.

File systems. Zero-copy clones exist only on ReFS / Dev Drive (Windows), Btrfs, XFS and ZFS 2.2+ (Linux) and APFS (macOS). NTFS has no block cloning, and Microsoft advises against installing apps on a Dev Drive, so most Windows players cannot use it. Hard links are not safe: a program that writes a file in place changes the backup too, and how Steam and GOG apply updates (in place or replace) is undocumented. Volume shadow copies need administrator rights and Windows prunes them.

Stores as the source of old builds. Steam keeps old builds, but downloading one is a console command the player types (download_depot), works only when the developer kept the build, and cannot be automated without third-party tools (DepotDownloader, GPL-2.0, needs the player's credentials). GOG Galaxy offers rollback only for games whose developers opted in. So the store cannot be relied on for rollback. What a program can read: Steam's appmanifest_*.acf shows the installed build, the target build and a pending update before it is applied (buildid vs TargetBuildID, StateFlags, BytesToDownload), and AutoUpdateBehavior (1 = update only on launch); depotcache/*.manifest holds per-file hashes, possibly with encrypted names. GOG leaves goggame-<id>.info with the build id and a hash file.

Ideas considered. (1) Copy a large file only right before it changes (SavePoint's own switches know which files they replace; a pending Steam update is visible at launch when the game updates only on launch); the first snapshot would copy only small files. It splits the rollback guarantee: an update or a tool SavePoint cannot see in advance leaves no old copy of the files it changed. (2) Instant full clones where the drive supports them, with today's full copy elsewhere. (3) Content-defined chunking (as restic, Borg, Kopia do) to store only the changed parts of a large archive when an update touches it; this shrinks the growth after updates, not the first copy. Worth a measurement on two real builds of one game. (4) git-annex-style tracking (large files as hashed entries, bytes kept outside git), the storage shape (1) and (2) would need.

Next step when this is picked up: measure on a mid-range machine (first snapshot time and disk on SATA SSD and hard disk), and measure how much of a large archive one real update changes.

Research findings, 2026-10-24. Not decided; recorded for a later look. Full notes with sources were gathered for both parts below. Requirements from the maintainer: rolling back a bad game update is a core feature and must stay whole; a per-game "mods and saves only" mode (this issue's original proposal) is rejected for that reason. Any design must hold up on a typical mid-range pre-built PC (one NTFS drive, SATA SSD or hard disk, less RAM and CPU), not only on a high-end machine. **Where the space and time go.** Profiles are already cheap: a second profile stores only what differs. The cost is the one base copy of the game folder per game, which git stores in its own format. Measured on Cyberpunk 2077 (85 GiB): 98.8% of the bytes are 17 large archives that never differ between profiles; the first snapshot takes about 9 minutes with D50 on a fast NVMe machine and will take far longer on a mid-range disk. **File systems.** Zero-copy clones exist only on ReFS / Dev Drive (Windows), Btrfs, XFS and ZFS 2.2+ (Linux) and APFS (macOS). NTFS has no block cloning, and Microsoft advises against installing apps on a Dev Drive, so most Windows players cannot use it. Hard links are not safe: a program that writes a file in place changes the backup too, and how Steam and GOG apply updates (in place or replace) is undocumented. Volume shadow copies need administrator rights and Windows prunes them. **Stores as the source of old builds.** Steam keeps old builds, but downloading one is a console command the player types (`download_depot`), works only when the developer kept the build, and cannot be automated without third-party tools (DepotDownloader, GPL-2.0, needs the player's credentials). GOG Galaxy offers rollback only for games whose developers opted in. So the store cannot be relied on for rollback. What a program can read: Steam's `appmanifest_*.acf` shows the installed build, the target build and a pending update before it is applied (`buildid` vs `TargetBuildID`, `StateFlags`, `BytesToDownload`), and `AutoUpdateBehavior` (1 = update only on launch); `depotcache/*.manifest` holds per-file hashes, possibly with encrypted names. GOG leaves `goggame-<id>.info` with the build id and a hash file. **Ideas considered.** (1) Copy a large file only right before it changes (SavePoint's own switches know which files they replace; a pending Steam update is visible at launch when the game updates only on launch); the first snapshot would copy only small files. It splits the rollback guarantee: an update or a tool SavePoint cannot see in advance leaves no old copy of the files it changed. (2) Instant full clones where the drive supports them, with today's full copy elsewhere. (3) Content-defined chunking (as restic, Borg, Kopia do) to store only the changed parts of a large archive when an update touches it; this shrinks the growth after updates, not the first copy. Worth a measurement on two real builds of one game. (4) git-annex-style tracking (large files as hashed entries, bytes kept outside git), the storage shape (1) and (2) would need. Next step when this is picked up: measure on a mid-range machine (first snapshot time and disk on SATA SSD and hard disk), and measure how much of a large archive one real update changes.
ethan-hann changed title from Let a game track only its mods and saves, not the whole game folder to Use less disk and time for large game installs without losing update rollback 2026-10-24 12:53:32 -05:00
Author
Owner

Recorded the chunk-store experiment as #102, to run after 1.0. It is the most likely way to keep update rollback without a second full copy of the big archives.

Recorded the chunk-store experiment as #102, to run after 1.0. It is the most likely way to keep update rollback without a second full copy of the big archives.
Sign in to join this conversation.
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
ethan-hann/SavePoint#71
No description provided.