Repository navigation
perf(core): scan for bytes with memchr - #860
Merged
Merged
Conversation
Single-byte searches and two- or three-byte sets in the resolve and load hooks now go through memchr, memchr2, memchr3 and memrchr instead of str::contains / str::find with a char or a char set. A char set decodes every character; memchr compares a vector register's worth of bytes at a time. The %2F / %5C check and the per-line frame-size check walk memchr hits instead of every byte. Substring searches (`/node_modules/`) and byte-class scans are unchanged: str::contains(&str) is already as fast as a memmem::Finder, and memchr cannot express a class. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
βΉοΈ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with π while any review is running, comments if it has suggestions, and reacts with π once all reviews finish with no findings. |
Brooooooklyn
added a commit
that referenced
this pull request
Oct 20, 2026
Follow-up to #860. An audit of every byte and string scan left in `src/`, keeping only the changes a benchmark shows are faster. ## Changed | Where | Before | After | Speed (aarch64, M-series) | |---|---|---|---| | `TryAsStr::try_as_str`: every CommonJS source the load hook checks, up to twice per file | `std::str::from_utf8` | `simdutf8` (`compat` where the error text is shown) | 2.0xβ3.3x on ASCII 2 KBβ200 KB, 6.7x on a 300 KB non-ASCII file | | `encode_file_url_path`: "does this path need `%XX`?", on every resolved file URL | `bytes().any(encoded)` | `any_byte(file_url_encoded)` | 5xβ5.8x on 34β117 byte paths | | `is_bare_specifier`: "any byte β€ space?" | `bytes().any(..)` | `any_byte(..)` | 2.5xβ12x on 18β96 byte specifiers | `any_byte` checks 16 bytes at a time with no early exit inside a chunk, so LLVM emits vector compares (NEON, SSE2). The encode set is rewritten as ranges and compares joined by `|`: the old `matches!` compiles to a bit test per byte, which does not vectorize (that form measured 0.8x, slower than `any`). ## Looked at, left as is | Where | Why | |---|---| | `contains("/node_modules/")` (5 places) | `memmem` with a static `Finder` measured 0.5xβ1.3x against `str::contains` on 34β117 byte paths: no win | | `windows_file_url.rs` host and segment checks | Windows-only, short UNC hosts and segments, rare | | line and column counting for error frames | only on a syntax error | | the remaining `replace` / `split` with one char | already guarded by a `memchr` check, or run once at startup | ## Tests - `file_url_encoded_is_the_path_to_file_url_set`: the compare form matches the old `matches!` set for all 256 bytes. - `any_byte_is_iterator_any`: same answer as `Iterator::any` for lengths 0β40, with the hit at every position. - `cargo test --all-features` (49), clippy with warnings denied, and `vp run test` (272 passed, 7 skipped) all pass. The numbers come from a standalone micro-benchmark (release, LTO, `codegen-units = 1`) on Apple Silicon. They are not measured on x86_64. π€ Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Byte scans in the resolve and load hooks now use the
memchrcrate wherever it is faster.memchrwas already a dependency, anddecode_url_path,has_malformed_escapeand the code-frame helpers already used it.What changed
contains('#'),contains('?'),contains('\\'),find(':')memchrTsconfigSource::discover,is_bare_specifier,file_url_to_path,resolver_request,decode_path_string(Windows)find(['?', '#']),find(['/', '\\'])memchr2url_path,url_to_path/unc_path_to_url(Windows)contains(['\t', '\n', '\r'])memchr3create_resolve,url_to_path(Windows)contains(['%', '\\', '?', '#'])memchr3+memchrcreate_resolvewindows(3).any(β¦)for%2F/%5Cmemchr_iter(b'%')has_encoded_separatorsplit(b'\n').any(β¦)memchr_iter(b'\n')frame_too_largerfind('\n')memrchrline_columniter().position(|b| b == b'%')memchrdecode_path_string,has_forbidden_host_escape(Windows)A
charset decodes one character at a time.memchrcompares a vector register's worth of bytes at a time.Measurements
Release build on Apple Silicon, with typical specifiers, paths and URLs as input. Each figure is the time per call.
contains('#')contains('\\')on a pathfind(':')find(['?', '#'])on a URL%2F/%5Cwindow scanLeft unchanged
contains("/node_modules/").str::contains(&str)is about as fast as a staticmemmem::Finder(4.12 ns against 4.43 ns), and a one-shotmemmem::findis slower (22.79 ns).bytes().any(|b| b <= b' ')and the percent-encode check.memchrcan only search for fixed byte values, not a class.Tests
frame_too_large_measures_each_line. It covers the first, middle and last line at exactly the length limit and one byte over, with and without a trailing newline.has_encoded_separator's existing test already covers a%2at the end of the input.cargo test --all-features,cargo clippy --all-targets --all-features -- -D warnings,vp run test(also withOXC_LOG=debug) andvp run lintall pass.windows_file_url.rschanges compile only on Windows, so they rely on CI.π€ Generated with Claude Code