Closed evidence disk recovery, 2026-11-12
Recovery run: scratch/runs/20261112-closed-evidence-disk-recovery-1415/.
The task began with 1,394,286,592 bytes available on /home, below the
1,500,000,000-byte floor. At 14:26:13 UTC, available space was 1,701,826,560
bytes: 201,826,560 bytes above the floor and a measured increase of 307,539,968
bytes despite concurrent shared-tree writes. The 192 wrappers totaled
464,046,793 bytes; 25 packs plus indexes total 147,365,913 bytes, saving
316,680,880 bytes. All 8,280,643,379 raw observation bytes remain local.
The final audit is final-audit.json in that run. The coordinator received a native-resume
message on the append-only messageboard at 14:22:43 UTC, after available space
reached 1,655,918,592 bytes; cleanup continued to improve the margin.
The supplied 187 inactive generated state gzip wrappers (213,977,970 bytes) in seven November 7 Lords2 runs and the November 9 SuperTux prompt run were grouped as raw JSON in local XZ packs. Five explicitly selected, closed November 7–8 observer/graphics/Darkstone diagnostic journals use the same representation. The original observations remain local. C3, Diablo II, Half-Life, Heroes II, native saves, source/media archives, conversation stores, and active logs were excluded. No sandbox or worker was created.
Each *.raw.json index maps original repository-relative paths to a sibling
*.raw.xz pack, raw offsets, lengths, and SHA-256 hashes. It retains the exact
gzip header (including optional filename and timestamp), compression level,
original gzip size/hash, filesystem metadata, and Python/zlib versions. Journal
rows also retain gzip trailer bytes. Raw JSON is compressed directly, allowing
shared observations to compress across file boundaries. Pack creation stages
only compressed output in tmpfs; journal raw bytes stream in 1 MiB chunks.
Before deleting any wrapper, the worker persisted and fsynced its replacement and index, fsynced their directory, reread the index and pack hash, decoded every row, reconstructed the original gzip, and matched the original bytes/hash. The source was checked again immediately before unlink. Source directories were fsynced after unlink. Initial unsupported-header and codec failures left every affected original in place; resumed attempts proved the correct filename header and level-3 journal codec before removal. These failures and resumptions remain in the run logs. A subsequently added no-net-saving guard retains originals when pack plus index would be larger.
The standalone restore proof writes original files into an owned tmpfs
directory, then independently rereads every gzip and checks size, SHA-256, and
mtime. Receipts are in standalone-restore-proof.json. The mixed level-9/level-1
historical container test in mixed-level-full-container-test.json reconstructed
all 14 gzip rows and the entire prior XZ container, matching its original
SHA-256. Raw-JSON restoration of the three-row level-1 pack and refusal to
overwrite existing destinations also passed; see
raw-json-and-overwrite-proof.json.
To restore an index from the repository root:
python3 docs/re-notes/closed-evidence-state-recover-20261112.py \
scratch/runs/20261112-closed-evidence-disk-recovery-1415/INDEX.raw.json \
/dev/shm/owned-restore
Use a separate, owned destination. The output retains repository-relative paths.
Use --raw-json for observation bytes without gzip wrappers. For older Lords2
closed-state-packs/*.raw.json, --container reconstructs the entire original
XZ container; it verifies each gzip and the container's original SHA-256.
The older helper path scratch/usage-watch/recover-raw-state-pack-20261112.py
is synchronized with the version tracked beside this note. The pack creation
helper is closed-evidence-state-pack-20261112.py; its hard allowlist prevents
application to active families or arbitrary filesystem data. Recovery pointers
were added to all thirteen affected closed source directories.
Exact gzip/container reconstruction depends on the Python/zlib versions pinned
in each index. Raw observation recovery only requires an XZ decoder and the
indexed offsets/hashes. Mode, mtime and available atime are restored; original
uid/gid, inode and ctime are recorded, but the helper does not recreate inode,
ctime or ownership. Do not run either helper with Python assertion checks
disabled. Do not expand the 2.58 GB observer journal onto /home; stream the
pack, or use an adequately sized separate destination.
Evidence, manifests, hashes, original plans, failed-attempt logs, helper patches and source Git references remain in the run directory. The two earlier root recode scripts are retained as source patches, not source-tree snapshots. The only Git changes for this task are this note and its two recovery helpers; no runtime code was changed. The 1.5 GB floor remains a live constraint: this recovery provides finite headroom, and concurrent native evidence writes can consume it. Larger capture/source archives and all unproven candidates remain untouched.