Skip to content

Properly propagate incomplete UTF-8 bytes separately to avoid corruption during string construction - #6895

Merged
lihaoyi merged 1 commit into
com-lihaoyi:mainfrom
lihaoyi:fix-stream-utf8
Mar 3, 2026
Merged

lihaoyi merged 1 commit into
com-lihaoyi:mainfrom
lihaoyi:fix-stream-utf8

Conversation

@lihaoyi

@lihaoyi lihaoyi commented Mar 3, 2026

Copy link
Copy Markdown
Member

Fixes #6891, Vibe Coded

⏺ Problem

In daemon mode, all output from the Mill daemon is forwarded to the launcher client via RPC. The RPC protocol serializes
output as JSON strings, so bytes must be converted to String at the boundary. This happened in RpcConsole.asStream
(libs/rpc/src/mill/rpc/RpcConsole.scala), which wrapped an OutputStream that called new String(bytes, UTF_8) on every
write call.

Upstream byte sources split writes at arbitrary boundaries β€” ProxyStream.Output splits into 126-byte packets,
InputPumper reads in 1024-byte chunks. When a multi-byte UTF-8 character (e.g., the 3-byte box-drawing character ─)
straddled one of these boundaries, the bytes were split across two independent write calls. Each call converted its
partial byte sequence to a String, producing U+FFFD replacement characters instead of the intended character.

mill -i avoided the issue because -i is an alias for --no-daemon, which bypasses the RPC path entirely.

Fix

Replaced the naive inline asStream implementation with RpcConsole.Utf8OutputStream, which buffers trailing incomplete
UTF-8 byte sequences across write calls. On each write it finds the last complete UTF-8 character boundary, converts
only the complete bytes to a String, and holds back any trailing partial sequence until the next write supplies the
remaining bytes. On flush/close, any remaining pending bytes are emitted as-is.

@lihaoyi
lihaoyi merged commit 1c0cf1a into com-lihaoyi:main Mar 3, 2026
67 of 69 checks passed
@lefou lefou added this to the 1.1.3 milestone Mar 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mill run non-interactive output mangles UTF-8 chars crossing 1k boundaires

2 participants