Skip to content

turbo-persistence: validate sst_inspect block reads with BlockOffsets - #100313

Merged
lukesandberg merged 1 commit into
canaryfrom
fleet/turbo-persistence-sst-open-bounds
Nov 13, 2026
Merged

lukesandberg merged 1 commit into
canaryfrom
fleet/turbo-persistence-sst-open-bounds

Conversation

@sokra

@sokra sokra commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Summary

#100288 moved SST block-directory validation into BlockOffsets: the whole directory is read with pread and validated once when a file is opened, and every block read goes through BlockOffsets::block_range. That covers the runtime readers this PR originally hardened. What's left here is the sst_inspect tool, which still read SST files with unchecked arithmetic, plus end-to-end tests.

  • sst_inspect reuses the runtime validation. BlockOffsets is now public, with read and block_range (both documented). sst_inspect builds its block directory with BlockOffsets::read and locates every block through block_range. Before this change it panicked on several kinds of corrupt file:

    Corrupt file Before
    File shorter than its block directory subtract with overflow
    Zero blocks block_count - 1 overflow
    Out-of-range block offset slice panic
    Out-of-range block index slice panic
    Index block ending inside the directory reported as a checksum mismatch

    All of these now return a descriptive error.

  • Zero-block files. BlockOffsets::read accepts an empty directory, and the runtime rejects it separately when reading the index block. sst_inspect now rejects zero blocks explicitly before it looks for the index block.

  • Decompressed-length mismatch. In sst_inspect this returns an error instead of failing an assert_eq!.

  • End-to-end open tests (coverage only). These open corrupt files through StaticSortedFile::open in every lookup access mode and through the file-backed StaticSortedFileIter::open (iteration uses pread since turbo-persistence: read compaction inputs with pread instead of mmapΒ #100336):

    • empty and truncated files;
    • zero blocks;
    • an index block that is past the end, overlaps the directory, is too short, ends before it starts, or is header-only;
    • a real SST with each non-index directory entry corrupted.

    Current canary already rejects all of these, so they only guard against regressions.

Runtime reader behavior is unchanged.

Verification

  • cargo test -p turbo-persistence --all-targets, plus runs with --features strict_checks and with --no-default-features.
  • cargo clippy -p turbo-persistence --all-targets in all three configurations, and cargo doc with -D warnings.
  • On canary without this change:
    • the new sst_inspect tests fail as listed above;
    • the runtime open tests pass when only the test code is applied.

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Tests Passed

Commit: 77da597

@github-actions github-actions Bot added created-by: Turbopack team PRs by the Turbopack team. Turbopack Related to Turbopack with Next.js. labels Oct 16, 2026
@sokra
sokra added this pull request to stack #100317 October 16, 2026 14:51
@sokra
sokra force-pushed the fleet/turbo-persistence-sst-open-bounds branch 2 times, most recently from 4872a87 to 65d8c15 Compare October 19, 2026 13:14
@sokra
sokra removed this pull request from stack #100317 October 19, 2026 13:14
@sokra sokra changed the title turbo-persistence: validate SST layout before opening turbo-persistence: validate sst_inspect block reads with BlockOffsets Oct 19, 2026
@sokra
sokra added this pull request to stack #100394 October 19, 2026 13:16
Make `BlockOffsets` public so `sst_inspect` reads SST files with the same
block-directory validation as the database, instead of unchecked offset
arithmetic and slicing. Reject zero-block files and decompressed-length
mismatches with errors instead of panicking. Add end-to-end open tests
for corrupt files through both readers in every access mode.

Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
@sokra
sokra force-pushed the fleet/turbo-persistence-sst-open-bounds branch from 65d8c15 to 77da597 Compare October 23, 2026 15:45

sokra commented Nov 10, 2026

Copy link
Copy Markdown
Member Author

Reviewed 77da597

@sokra
sokra marked this pull request as ready for review November 10, 2026 11:05
@sokra
sokra requested a review from lukesandberg November 10, 2026 11:05
@lukesandberg
lukesandberg merged commit 72a6249 into canary Nov 13, 2026
103 checks passed
@lukesandberg
lukesandberg deleted the fleet/turbo-persistence-sst-open-bounds branch November 13, 2026 01:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

created-by: Turbopack team PRs by the Turbopack team. Turbopack Related to Turbopack with Next.js.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants