Repository navigation
Properly propagate incomplete UTF-8 bytes separately to avoid corruption during string construction - #6895
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6891, Vibe Coded
βΊ Problem
In daemon mode, all output from the Mill daemon is forwarded to the launcher client via RPC. The RPC protocol serializes
output as JSON strings, so bytes must be converted to String at the boundary. This happened in RpcConsole.asStream
(libs/rpc/src/mill/rpc/RpcConsole.scala), which wrapped an OutputStream that called new String(bytes, UTF_8) on every
write call.
Upstream byte sources split writes at arbitrary boundaries β ProxyStream.Output splits into 126-byte packets,
InputPumper reads in 1024-byte chunks. When a multi-byte UTF-8 character (e.g., the 3-byte box-drawing character β)
straddled one of these boundaries, the bytes were split across two independent write calls. Each call converted its
partial byte sequence to a String, producing U+FFFD replacement characters instead of the intended character.
mill -i avoided the issue because -i is an alias for --no-daemon, which bypasses the RPC path entirely.
Fix
Replaced the naive inline asStream implementation with RpcConsole.Utf8OutputStream, which buffers trailing incomplete
UTF-8 byte sequences across write calls. On each write it finds the last complete UTF-8 character boundary, converts
only the complete bytes to a String, and holds back any trailing partial sequence until the next write supplies the
remaining bytes. On flush/close, any remaining pending bytes are emitted as-is.