Skip to content

Report invalid UTF-8 at the first bad byte - #1345

Merged
dtolnay merged 2 commits into
serde-rs:masterfrom
00200200:fix-invalid-utf8-error-column
Oct 11, 2026
Merged

dtolnay merged 2 commits into
serde-rs:masterfrom
00200200:fix-invalid-utf8-error-column

Conversation

@00200200

Copy link
Copy Markdown
Contributor

When from_utf8 failed while parsing a string, the error used the reader's current position, which is already past the closing quote. The column therefore always pointed at the end of the string, not at the invalid byte.

This uses Utf8Error::valid_up_to so InvalidUnicodeCodePoint reports the first bad byte, the same way control-character errors already do.

Fixes #1083
Fixes #1110

Tests

  • cargo +nightly test
  • cargo +nightly test --features raw_value --tests -- --skip ui --exact
  • cargo +nightly test --features raw_value --test regression

String parse used to attach InvalidUnicodeCodePoint to the position after
the closing quote, so the column always pointed at the end of the string.
Use Utf8Error::valid_up_to so the error matches the first invalid byte.

@dtolnay dtolnay left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Comment thread src/read.rs Outdated
Comment on lines +662 to +664
self.raw_buffering_start_index
.saturating_add(err.valid_up_to())
.saturating_add(1),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the situation where raw_buffering_start_index + err.valid_up_to() + 1 would overflow usize::MAX? I am having trouble following what these saturating_add are guarding.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right: valid_up_to() is strictly less than raw.len() for a UTF-8 error, so start + valid_up_to() + 1 is at most self.index, which is already bounded by self.slice.len(). Removed the saturating_add calls in 452afa2; no overflow guard is needed here.

Comment thread src/read.rs Outdated
Comment on lines +665 to +666
self.slice.len(),
);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar question, what is this cmp::min accomplishing?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The clamp was unnecessary for the same bound: the first invalid byte belongs to the raw slice, and its one-past position cannot exceed self.index or self.slice.len(). Removed cmp::min in 452afa2.

Comment thread src/read.rs Outdated
let back = bytes.len() - utf8.valid_up_to();
return Err(Error::syntax(
ErrorCode::InvalidUnicodeCodePoint,
position.line,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the invalid unicode code point necessarily in the same line as the end of the raw value? Raw values can span multiple lines i.e. can contain newlines; newlines are just whitespace.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, it can be on an earlier line. The new multiline RawValue regression failed on this PR head with (line 4, column 0) instead of (line 2, column 3). In 452afa2 IoRead now records the raw buffering start position and derives the error position from the raw prefix through the bad byte, including leading whitespace and later newlines. Both slice and reader cases pass. Validation: 153 tests with raw_value (UI test excluded as in CI), 104 doctests, alloc/raw_value without std, and rustfmt.

@dtolnay dtolnay left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@dtolnay
dtolnay merged commit 8af71cc into serde-rs:master Oct 11, 2026
16 checks passed
dadezzz pushed a commit to dadezzz/kube-dns-rs that referenced this pull request Oct 12, 2026
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [serde_json](https://github.com/serde-rs/json) | dependencies | patch | `1.0.152` → `1.0.154` |

---

### Release Notes

<details>
<summary>serde-rs/json (serde_json)</summary>

### [`v1.0.154`](https://github.com/serde-rs/json/releases/tag/v1.0.154)

[Compare Source](serde-rs/json@v1.0.153...v1.0.154)

- Support preserve_order in no_std builds ([#&#8203;1333](serde-rs/json#1333), thanks [@&#8203;razmser](https://github.com/razmser))

### [`v1.0.153`](https://github.com/serde-rs/json/releases/tag/v1.0.153)

[Compare Source](serde-rs/json@v1.0.152...v1.0.153)

- Report invalid UTF-8 at the first bad byte ([#&#8203;1345](serde-rs/json#1345), thanks [@&#8203;00200200](https://github.com/00200200))

</details>

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - At any time (no schedule defined)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xNDkuMyIsInVwZGF0ZWRJblZlciI6IjQ0LjE0OS4zIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->
github-actions Bot pushed a commit to modify-dev/harbor that referenced this pull request Oct 20, 2026
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [serde_json](https://github.com/serde-rs/json) | dependencies | patch | `1.0.151` → `1.0.154` |

---

> ⚠️ **Warning**
>
> Some dependencies could not be looked up. Check the [Dependency Dashboard](issues/99) for more information.

---

### Release Notes

<details>
<summary>serde-rs/json (serde_json)</summary>

### [`v1.0.154`](https://github.com/serde-rs/json/releases/tag/v1.0.154)

[Compare Source](serde-rs/json@v1.0.153...v1.0.154)

- Support preserve_order in no_std builds ([#&#8203;1333](serde-rs/json#1333), thanks [@&#8203;razmser](https://github.com/razmser))

### [`v1.0.153`](https://github.com/serde-rs/json/releases/tag/v1.0.153)

[Compare Source](serde-rs/json@v1.0.152...v1.0.153)

- Report invalid UTF-8 at the first bad byte ([#&#8203;1345](serde-rs/json#1345), thanks [@&#8203;00200200](https://github.com/00200200))

### [`v1.0.152`](https://github.com/serde-rs/json/releases/tag/v1.0.152)

[Compare Source](serde-rs/json@v1.0.151...v1.0.152)

- Preserve sign of `-0` in arbitrary_precision mode ([#&#8203;1346](serde-rs/json#1346), thanks [@&#8203;yuxi-liu-wired](https://github.com/yuxi-liu-wired))
- Fix comparison of large non-integer values with f32::INFINITY ([#&#8203;1347](serde-rs/json#1347), thanks [@&#8203;yuxi-liu-wired](https://github.com/yuxi-liu-wired))

</details>

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - "before 9am on tuesday"
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about these updates again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xNjcuMCIsInVwZGF0ZWRJblZlciI6IjQ0LjE2Ny4wIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJkZXBlbmRlbmNpZXMiLCJyZW5vdmF0ZSJdfQ==-->

Reviewed-on: https://code.futo.org/harbor/harbor/pulls/1934
Reviewed-by: Thomas de Zeeuw <thomas@futo.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Error position wrong for "unicode code point" error Wrong positions for InvalidUnicodeCodePoint caused by from_utf8

2 participants