Repository navigation
Report invalid UTF-8 at the first bad byte - #1345
Conversation
String parse used to attach InvalidUnicodeCodePoint to the position after the closing quote, so the column always pointed at the end of the string. Use Utf8Error::valid_up_to so the error matches the first invalid byte.
| self.raw_buffering_start_index | ||
| .saturating_add(err.valid_up_to()) | ||
| .saturating_add(1), |
There was a problem hiding this comment.
What is the situation where raw_buffering_start_index + err.valid_up_to() + 1 would overflow usize::MAX? I am having trouble following what these saturating_add are guarding.
There was a problem hiding this comment.
You are right: valid_up_to() is strictly less than raw.len() for a UTF-8 error, so start + valid_up_to() + 1 is at most self.index, which is already bounded by self.slice.len(). Removed the saturating_add calls in 452afa2; no overflow guard is needed here.
| self.slice.len(), | ||
| ); |
There was a problem hiding this comment.
Similar question, what is this cmp::min accomplishing?
There was a problem hiding this comment.
The clamp was unnecessary for the same bound: the first invalid byte belongs to the raw slice, and its one-past position cannot exceed self.index or self.slice.len(). Removed cmp::min in 452afa2.
| let back = bytes.len() - utf8.valid_up_to(); | ||
| return Err(Error::syntax( | ||
| ErrorCode::InvalidUnicodeCodePoint, | ||
| position.line, |
There was a problem hiding this comment.
Is the invalid unicode code point necessarily in the same line as the end of the raw value? Raw values can span multiple lines i.e. can contain newlines; newlines are just whitespace.
There was a problem hiding this comment.
Correct, it can be on an earlier line. The new multiline RawValue regression failed on this PR head with (line 4, column 0) instead of (line 2, column 3). In 452afa2 IoRead now records the raw buffering start position and derives the error position from the raw prefix through the bad byte, including leading whitespace and later newlines. Both slice and reader cases pass. Validation: 153 tests with raw_value (UI test excluded as in CI), 104 doctests, alloc/raw_value without std, and rustfmt.
Assisted-by: OpenAI Codex
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [serde_json](https://github.com/serde-rs/json) | dependencies | patch | `1.0.152` → `1.0.154` | --- ### Release Notes <details> <summary>serde-rs/json (serde_json)</summary> ### [`v1.0.154`](https://github.com/serde-rs/json/releases/tag/v1.0.154) [Compare Source](serde-rs/json@v1.0.153...v1.0.154) - Support preserve_order in no_std builds ([#​1333](serde-rs/json#1333), thanks [@​razmser](https://github.com/razmser)) ### [`v1.0.153`](https://github.com/serde-rs/json/releases/tag/v1.0.153) [Compare Source](serde-rs/json@v1.0.152...v1.0.153) - Report invalid UTF-8 at the first bad byte ([#​1345](serde-rs/json#1345), thanks [@​00200200](https://github.com/00200200)) </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xNDkuMyIsInVwZGF0ZWRJblZlciI6IjQ0LjE0OS4zIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [serde_json](https://github.com/serde-rs/json) | dependencies | patch | `1.0.151` → `1.0.154` | --- >⚠️ **Warning** > > Some dependencies could not be looked up. Check the [Dependency Dashboard](issues/99) for more information. --- ### Release Notes <details> <summary>serde-rs/json (serde_json)</summary> ### [`v1.0.154`](https://github.com/serde-rs/json/releases/tag/v1.0.154) [Compare Source](serde-rs/json@v1.0.153...v1.0.154) - Support preserve_order in no_std builds ([#​1333](serde-rs/json#1333), thanks [@​razmser](https://github.com/razmser)) ### [`v1.0.153`](https://github.com/serde-rs/json/releases/tag/v1.0.153) [Compare Source](serde-rs/json@v1.0.152...v1.0.153) - Report invalid UTF-8 at the first bad byte ([#​1345](serde-rs/json#1345), thanks [@​00200200](https://github.com/00200200)) ### [`v1.0.152`](https://github.com/serde-rs/json/releases/tag/v1.0.152) [Compare Source](serde-rs/json@v1.0.151...v1.0.152) - Preserve sign of `-0` in arbitrary_precision mode ([#​1346](serde-rs/json#1346), thanks [@​yuxi-liu-wired](https://github.com/yuxi-liu-wired)) - Fix comparison of large non-integer values with f32::INFINITY ([#​1347](serde-rs/json#1347), thanks [@​yuxi-liu-wired](https://github.com/yuxi-liu-wired)) </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - "before 9am on tuesday" - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xNjcuMCIsInVwZGF0ZWRJblZlciI6IjQ0LjE2Ny4wIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJkZXBlbmRlbmNpZXMiLCJyZW5vdmF0ZSJdfQ==--> Reviewed-on: https://code.futo.org/harbor/harbor/pulls/1934 Reviewed-by: Thomas de Zeeuw <thomas@futo.org>
When
from_utf8failed while parsing a string, the error used the reader's current position, which is already past the closing quote. The column therefore always pointed at the end of the string, not at the invalid byte.This uses
Utf8Error::valid_up_tosoInvalidUnicodeCodePointreports the first bad byte, the same way control-character errors already do.Fixes #1083
Fixes #1110
Tests
cargo +nightly testcargo +nightly test --features raw_value --tests -- --skip ui --exactcargo +nightly test --features raw_value --test regression