Decide pack.compression for large files #45

Closed
opened 2026-10-23 22:59:59 -05:00 by ethan-hann · 0 comments
Owner

A reproduction of a first snapshot with three 1.5 GiB files (random data) measured about 50 MB/s, and most of the time is zlib on the streamed pack: files above core.bigFileThreshold (128m) stream into objects/pack/tmp_pack_*, and core.looseCompression=0 does not cover them; pack.compression does. Setting pack.compression=0 would likely speed large first snapshots a lot (an earlier measurement found looseCompression=0 gave 2.2x on capture) and make the byte progress near exact, at the cost of disk space for large files that do compress (game archives are usually already compressed, uncompressed textures and audio are not). Measure capture time and store size on a real large game (for example a 70 GB install) with and without it, then decide, and record the call in docs/decisions.md.

A reproduction of a first snapshot with three 1.5 GiB files (random data) measured about 50 MB/s, and most of the time is zlib on the streamed pack: files above core.bigFileThreshold (128m) stream into objects/pack/tmp_pack_*, and core.looseCompression=0 does not cover them; pack.compression does. Setting pack.compression=0 would likely speed large first snapshots a lot (an earlier measurement found looseCompression=0 gave 2.2x on capture) and make the byte progress near exact, at the cost of disk space for large files that do compress (game archives are usually already compressed, uncompressed textures and audio are not). Measure capture time and store size on a real large game (for example a 70 GB install) with and without it, then decide, and record the call in docs/decisions.md.
ethan-hann added this to the 1.0 milestone 2026-10-23 22:59:59 -05:00
Sign in to join this conversation.
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
ethan-hann/SavePoint#45
No description provided.