Repository navigation
fix(core): decode source that is not valid UTF-8 as Node.js does - #854
Merged
Merged
Conversation
Source handed over as bytes was decoded strictly, so any invalid UTF-8 byte failed the module with "Failed to convert Uint8Array to Vec<u8>", naming neither the file nor the problem. Decode it the way Node.js decodes a module instead: UTF-8 with every invalid sequence replaced by U+FFFD. Valid source stays a zero-copy borrow. The strict check stays where it tells source from bytes that are not source at all (a `.node` addon reported as `commonjs`). Closes #843 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
βΉοΈ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with π while any review is running, comments if it has suggestions, and reacts with π once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #843
Problem
TryAsStrdecoded source handed over as bytes withstd::str::from_utf8and mapped every failure to a fixed message. A single invalid UTF-8 byte therefore failed the module with a message that names neither the file nor the problem:Node.js and tsx decode source as UTF-8 and replace invalid bytes with U+FFFD.
Fix
TryAsStr::to_source(): decodes as Node.js does. It returns a borrow when the bytes are valid UTF-8, checked with simdutf8, and falls back toString::from_utf8_lossyotherwise, which replaces each invalid sequence with U+FFFD asTextDecoderdoes. Valid source is still never copied.oxc_transform, behind the load hook and the publictransform/transformAsyncAPIs) and the JSON module path.try_as_str()remains in the two load-hook branches that use invalid UTF-8 to tell source from bytes that aren't source at all, such as a.nodeaddon reported ascommonjs. Those bytes are still left to the CommonJS loader.Tests
decode_source: borrows valid input, replaces lone and truncated sequences.packages/integrate-vitest/__tests__/invalid-utf8.spec.ts: a lone0xE9byte in a.ts, an.mts, a.ctsloaded throughrequireand a JSON module all give the same string as plain Node.js: length 4, with U+FFFD at index 3.Verification (macOS arm64)
nodefor each file type above.transform()/transformAsync()APIs decode aBufferand aUint8Arraythe same way.cargo test --all-features(43 passed),cargo clippy --all-targets --all-features -- -D warnings,vp run test(220 passed) andvp run lintall pass.π€ Generated with Claude Code