Repository navigation
Add RawMessage marshal support to unstable - #1084
Merged
Merged
Conversation
pelletier
force-pushed
the
raw-message-marshal
branch
from
June 24, 2026 23:54
f9858c2 to
cfaeacc
Compare
pelletier
force-pushed
the
raw-message-marshal
branch
2 times, most recently
from
June 25, 2026 00:52
93c30ab to
4c63792
Compare
unstable.RawMessage could already capture raw TOML on decode via Decoder.EnableUnmarshalerInterface, but marshaling it emitted the underlying []byte as an integer array, so it could not round-trip. This adds the encode half so RawMessage behaves like json.RawMessage. - unstable.Marshaler interface (MarshalTOML), the encode counterpart of Unmarshaler, implemented by RawMessage. - Encoder.EnableMarshalerInterface(): an opt-in toggle mirroring Decoder.EnableUnmarshalerInterface (both carry the *Unstable* disclaimer). Default Marshal/Encode behavior is unchanged. - The encoder classifies a Marshaler's bytes by parsing them: a single value is spliced inline (key = <raw>), key-value lines become a [key] table body, and a []RawMessage of table bodies becomes an array of tables. Empty output is omitted; table content forced inline is an error. All new public API stays in unstable. The default encode path keeps its exact layout: the entry struct is unchanged at 64 bytes (the classification is packed into valueOptions padding) and all marshaler logic lives in separate methods gated behind the opt-in flag, so existing marshal/unmarshal benchmarks are unaffected. Tests cover every value shape, single/array tables, custom and pointer-receiver marshalers, error propagation, empty omission, inline-context errors, the documented relative-sub-table-header limitation, and full decode->encode->decode round-trips, plus a benchmark and a godoc example. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-ups to the unstable.Marshaler encoder support: - Reject MarshalTOML output that is not valid TOML for its position instead of silently emitting an invalid document, mirroring json.RawMessage's validation of raw bytes. - Comment-prefix spliced table bodies under commented tables and array tables. Previously only the header was commented, so the body leaked into the document as live keys of the parent table. - Support a Marshaler at the document root, emitting its bytes as the whole document: the encode counterpart of the decoder delivering the whole document to a root Unmarshaler (#994). At the root the bytes are parsed as a document, so table headers keep their absolute position and whole-document round-trips are exact. - Re-validate the bytes returned by the emit-time MarshalTOML call, so an implementation returning different content than during classification cannot splice table content into a value position. - Document that MarshalTOML can be called more than once per encode and must be deterministic; include the type in its error messages. The default (interface-disabled) encode path is unchanged: all new work sits behind the existing marshalerOn checks, plus one short-circuited flag test in encodeRoot that runs once per document. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pelletier
force-pushed
the
raw-message-marshal
branch
from
July 3, 2026 02:34
a48d0f0 to
0ba5b18
Compare
pelletier
added a commit
that referenced
this pull request
Jul 3, 2026
Benchmarking the Marshaler-interface commit (#1084) against v2 on linux/amd64 (interleaved rounds, n=12, benchstat) showed statistically significant sec/op regressions on the default (interface-disabled) Marshal path — up to +10% on micro-benchmarks — with allocations unchanged. Bisecting the diff piece by piece isolated the dominant cause: widening the per-type typeEncProps cache struct from 2 to 3 bytes. Its cached p.(typeEncProps) assertion and by-value return run two to three times per encoded value, and the extra field changes the copy size and register shape of every lookup; that lone change reproduced the regression with zero encoder-code modifications. - typeEncProps and encPropsForType are restored to their exact baseline form. The unstable.Marshaler classification lives in its own sync.Map (typeMarshalerCache/marshalerPropsForType), consulted only on EnableMarshalerInterface paths. - encodeTable, encodeKeyValue and encodeArrayTable are restored to their exact baseline source. The Marshaler-aware variants of the table passes live in encodeTableWithMarshalers and encodeArrayTableWithMarshalers, selected by one branch per table instead of one per entry. Marshaler values on a `key =` line flow through appendValue's gated guard, which returns without rejoining the fall-through path and re-classifies the fetched bytes (keeping the emit-time guard), so appendMarshalerInlineValue is gone. - All Marshaler-interface machinery moves to a new marshaler_raw.go so the feature's cold code does not sit between the default path's hot functions. After this change the interface-off encoder differs from baseline only by four single-branch dispatches; typeEncProps, encPropsForType and the hot encoder functions are byte-identical to the pre-#1084 source. allocs/op are identical across the whole ./benchmark suite. Remaining µs-scale sec/op deltas are within this host's address-alignment noise floor, demonstrated by dead-code A/A controls and -randlayout runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
|
Post-merge benchmarking of this feature against the pre-merge base (idle linux/amd64 host, pinned cores, interleaved rounds, n=12) found a default-path (interface-disabled) Marshal regression of up to +10% on µs-scale micro-benchmarks, with allocations unchanged — it was invisible to the alloc counts and the noisy-host Root cause and fix are in #1090 (dominant term: widening the per-type Benchstat: v2 pre-#1084 (79f82e3) vs #1084 + #1090 (c0a09d4) — full ./benchmark suite, go1.26.4, linux/amd64🤖 Generated with Claude Code |
pelletier
added a commit
that referenced
this pull request
Jul 11, 2026
Benchmarking the Marshaler-interface commit (#1084) against v2 on linux/amd64 (interleaved rounds, n=12, benchstat) showed statistically significant sec/op regressions on the default (interface-disabled) Marshal path — up to +10% on micro-benchmarks — with allocations unchanged. Bisecting the diff piece by piece isolated the dominant cause: widening the per-type typeEncProps cache struct from 2 to 3 bytes. Its cached p.(typeEncProps) assertion and by-value return run two to three times per encoded value, and the extra field changes the copy size and register shape of every lookup; that lone change reproduced the regression with zero encoder-code modifications. - typeEncProps and encPropsForType are restored to their exact baseline form. The unstable.Marshaler classification lives in its own sync.Map (typeMarshalerCache/marshalerPropsForType), consulted only on EnableMarshalerInterface paths. - encodeTable, encodeKeyValue and encodeArrayTable are restored to their exact baseline source. The Marshaler-aware variants of the table passes live in encodeTableWithMarshalers and encodeArrayTableWithMarshalers, selected by one branch per table instead of one per entry. Marshaler values on a `key =` line flow through appendValue's gated guard, which returns without rejoining the fall-through path and re-classifies the fetched bytes (keeping the emit-time guard), so appendMarshalerInlineValue is gone. - All Marshaler-interface machinery moves to a new marshaler_raw.go so the feature's cold code does not sit between the default path's hot functions. After this change the interface-off encoder differs from baseline only by four single-branch dispatches; typeEncProps, encPropsForType and the hot encoder functions are byte-identical to the pre-#1084 source. allocs/op are identical across the whole ./benchmark suite. Remaining µs-scale sec/op deltas are within this host's address-alignment noise floor, demonstrated by dead-code A/A controls and -randlayout runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes
unstable.RawMessageround-trip likejson.RawMessageby adding the encode half of the feature.The decode side already existed:
unstable.RawMessagecaptures raw TOML whenDecoder.EnableUnmarshalerInterface()is set (theunstable.Unmarshalerwork from #873 → #940 → #1027, plus root documents in #1083). But marshaling aRawMessageemitted the underlying[]byteas an array of integers (plugin = [110, 97, ...]), so it could not round-trip. This PR adds the missing encoder support.Fixes #796.
API (all new public API stays in
unstable)unstable.Marshaler—MarshalTOML() ([]byte, error), the encode counterpart ofunstable.Unmarshaler, implemented byRawMessage(value receiver, mirroringjson.RawMessage). It can be called more than once for the same value during a single encode, so it must be deterministic.Encoder.EnableMarshalerInterface()— opt-in toggle in thetomlpackage, exactly mirroring the existingDecoder.EnableUnmarshalerInterface()(both carry the*Unstable:*disclaimer). It is the only addition to the stable package, is backward compatible, and the defaulttoml.Marshal/Encodebehavior is unchanged.Behavior
With the interface enabled, the encoder splices a
Marshaler's bytes verbatim, parsing them to pick the valid position:42,"x",[1,2],{a=1})key = <raw>name = "x"\n...)[key]table body[]RawMessageof table bodies[[key]]array of tablesjson.RawMessagewith invalid JSON)SetTablesInline)Spliced bytes are validated by the same parser the decoder uses, so a
Marshalercannot silently corrupt the document; validation runs against the bytes actually emitted, so even a non-deterministic implementation cannot splice table content into a value position. A table-shaped body under acommentedfield is comment-prefixed line by line, like structurally encoded tables.Documented limitation
Bodies are spliced verbatim (the
json.RawMessagephilosophy). A captured table body that itself contains sub-table headers keeps those headers relative to the captured root, so it round-trips exactly only when re-emitted at the same depth. Flat tables, scalars, inline tables, arrays-of-tables — and whole documents at the root, where headers are absolute — round-trip exactly. This is pinned by a test.Performance
No regression on the default (interface-disabled) path:
entrystruct is unchanged at 64 bytes (the classification is packed intovalueOptions' existing padding);encodeRoot, once per document).Order-balanced benchstat of
./benchmark/shows the per-op-sensitiveMarshal/SimpleDocumentcases unchanged (~); allocs/op are identical.Tests
marshaler_raw_test.goadds coverage for every value shape, single/array tables, nested fields, map values, custom and pointer-receiverMarshalers, error propagation, empty omission, inline-context errors, invalid-content errors, commented table bodies, non-deterministic marshalers, root documents, the documented limitation, and full decode→encode→decode round-trips (fields, array tables, and whole documents at the root), plusBenchmarkMarshalRawMessageand a godoc example.🤖 Generated with Claude Code