Repository navigation
turbo-persistence: validate sst_inspect block reads with BlockOffsets - #100313
Merged
Merged
Conversation
Contributor
Tests PassedCommit: 77da597 |
sokra
force-pushed
the
fleet/turbo-persistence-sst-open-bounds
branch
2 times, most recently
from
October 19, 2026 13:14
4872a87 to
65d8c15
Compare
Make `BlockOffsets` public so `sst_inspect` reads SST files with the same block-directory validation as the database, instead of unchecked offset arithmetic and slicing. Reject zero-block files and decompressed-length mismatches with errors instead of panicking. Add end-to-end open tests for corrupt files through both readers in every access mode. Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
sokra
force-pushed
the
fleet/turbo-persistence-sst-open-bounds
branch
from
October 23, 2026 15:45
65d8c15 to
77da597
Compare
Member
Author
|
Reviewed 77da597 |
sokra
marked this pull request as ready for review
November 10, 2026 11:05
lukesandberg
approved these changes
Nov 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#100288 moved SST block-directory validation into
BlockOffsets: the whole directory is read withpreadand validated once when a file is opened, and every block read goes throughBlockOffsets::block_range. That covers the runtime readers this PR originally hardened. What's left here is thesst_inspecttool, which still read SST files with unchecked arithmetic, plus end-to-end tests.sst_inspectreuses the runtime validation.BlockOffsetsis now public, withreadandblock_range(both documented).sst_inspectbuilds its block directory withBlockOffsets::readand locates every block throughblock_range. Before this change it panicked on several kinds of corrupt file:block_count - 1overflowAll of these now return a descriptive error.
Zero-block files.
BlockOffsets::readaccepts an empty directory, and the runtime rejects it separately when reading the index block.sst_inspectnow rejects zero blocks explicitly before it looks for the index block.Decompressed-length mismatch. In
sst_inspectthis returns an error instead of failing anassert_eq!.End-to-end open tests (coverage only). These open corrupt files through
StaticSortedFile::openin every lookup access mode and through the file-backedStaticSortedFileIter::open(iteration usespreadsince turbo-persistence: read compaction inputs with pread instead of mmapΒ #100336):Current canary already rejects all of these, so they only guard against regressions.
Runtime reader behavior is unchanged.
Verification
cargo test -p turbo-persistence --all-targets, plus runs with--features strict_checksand with--no-default-features.cargo clippy -p turbo-persistence --all-targetsin all three configurations, andcargo docwith-D warnings.sst_inspecttests fail as listed above;