Skip to content

fix(core): decode source that is not valid UTF-8 as Node.js does - #854

Merged
Brooooooklyn merged 1 commit into
mainfrom
fix/issue-843
Oct 16, 2026
Merged

Brooooooklyn merged 1 commit into
mainfrom
fix/issue-843

Conversation

@Brooooooklyn

Copy link
Copy Markdown
Member

Closes #843

Problem

TryAsStr decoded source handed over as bytes with std::str::from_utf8 and mapped every failure to a fixed message. A single invalid UTF-8 byte therefore failed the module with a message that names neither the file nor the problem:

Error: Failed to convert Uint8Array to Vec<u8>
  code: 'GenericFailure'

Node.js and tsx decode source as UTF-8 and replace invalid bytes with U+FFFD.

Fix

  • New TryAsStr::to_source(): decodes as Node.js does. It returns a borrow when the bytes are valid UTF-8, checked with simdutf8, and falls back to String::from_utf8_lossy otherwise, which replaces each invalid sequence with U+FFFD as TextDecoder does. Valid source is still never copied.
  • Where it's used: the transform (oxc_transform, behind the load hook and the public transform/transformAsync APIs) and the JSON module path.
  • Where the strict check stays: try_as_str() remains in the two load-hook branches that use invalid UTF-8 to tell source from bytes that aren't source at all, such as a .node addon reported as commonjs. Those bytes are still left to the CommonJS loader.

Tests

  • Rust unit test for decode_source: borrows valid input, replaces lone and truncated sequences.
  • packages/integrate-vitest/__tests__/invalid-utf8.spec.ts: a lone 0xE9 byte in a .ts, an .mts, a .cts loaded through require and a JSON module all give the same string as plain Node.js: length 4, with U+FFFD at index 3.

Verification (macOS arm64)

  • Node.js 24.22 and 26.12: oxc-node prints the same as plain node for each file type above.
  • The public transform()/transformAsync() APIs decode a Buffer and a Uint8Array the same way.
  • cargo test --all-features (43 passed), cargo clippy --all-targets --all-features -- -D warnings, vp run test (220 passed) and vp run lint all pass.

πŸ€– Generated with Claude Code

Source handed over as bytes was decoded strictly, so any invalid UTF-8
byte failed the module with "Failed to convert Uint8Array to Vec<u8>",
naming neither the file nor the problem. Decode it the way Node.js
decodes a module instead: UTF-8 with every invalid sequence replaced by
U+FFFD. Valid source stays a zero-copy borrow. The strict check stays
where it tells source from bytes that are not source at all (a `.node`
addon reported as `commonjs`).

Closes #843

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
πŸ“ Code Review βœ… Completed 2026-10-16T15:32:07.715307Z 9f25308 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with πŸ‘€ while any review is running, comments if it has suggestions, and reacts with πŸ‘ once all reviews finish with no findings.

@Brooooooklyn
Brooooooklyn merged commit ed51b8b into main Oct 16, 2026
67 checks passed
@Brooooooklyn
Brooooooklyn deleted the fix/issue-843 branch October 16, 2026 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Source with invalid UTF-8 fails with "Failed to convert Uint8Array to Vec<u8>"

1 participant