deserializing inconsistently drops / rewrites whitespace characters (especially \r) since v0.38.0

It appears that the whitespace-preserving changes added to the Deserializer in v0.38.0 also unintentionally (and inconsistently) "normalize" some whitespace on deserialization - but not on serialization.

Minimal reproducer with quick-xml = { version = "0.41", features = ["serialize"] }:

use quick_xml::de::from_str;
use quick_xml::se::to_string;

use serde::{Deserialize, Serialize};

#[derive(Debug, Deserialize, PartialEq, Serialize)]
#[serde(rename = "value")]
struct Value {
    #[serde(rename = "$value")]
    value: String,
}

fn main() {
    let string = "<value>hello\rworld</value>";
    let value = Value { value: String::from("hello\rworld") };

    assert_eq!(to_string(&value).unwrap(), string);
    assert_eq!(from_str::<Value>(string).unwrap(), value);    
}

The first assertion passes (whitespace preserved as-is), the second assertion fails (\r rewritten to \n on deserialization), so it looks like serializing preserves existing whitespace, while deserializing rewrites some existing whitespace characters.

Yes, I know that a lone \r (case 1) does not make much sense here, but the same can also be reproduced when using "hello\r\nworld" using Windows-style CRLF line endings (case 2): This value serializes as expected, but deserializes as "hello\nworld".

I think the \r character should be preserved in either case, and not get inconsistently rewritten (case 1) or dropped (case 2).

So, the XML specification (https://www.w3.org/TR/xml/#sec-line-ends) says the following

XML parsed entities are often stored in computer files which, for editing convenience, are organized into lines. These lines are typically separated by some combination of the characters CARRIAGE RETURN (#xD) and LINE FEED (#xA).

To simplify the tasks of applications, the XML processor MUST behave as if it normalized all line breaks in external parsed entities (including the document entity) on input, before parsing, by translating both the two-character sequence #xD #xA and any #xD that is not followed by #xA to a single #xA character.

So I believe it would be expected and normal for the lone /r to be dropped and replaced with /n - any compliant XML parser would do that.

The actual bug, arguably, is that the serializer should either escape \r to a character reference in order to preserve it or do the same normalization on the output. I need to think about it a bit, I'm recovering from a cold so my brain isn't at 100%.

Either way the normalization is fundamentally "lossy" and I don't think you can expect in == out in the literal byte sense 100% of the time. From the perspective of an XML processor, hello\rworld and hello\nworld and hello\r\nworld are equivalent.

Thanks, that context somewhat explains what's happening.

It would still be great if strings like that could - at least in some limited* sense - be round-trip safe.

limited*: applying some normalization myself to ensure round-trip-ability would be fine, but I don't want to have to replicate whatever logic happens in the internals of quick_xml::de::from_str for this :(

Does encoding (escaping) the return as a character reference () in the input work? (that doesn't "round-trip" exactly, under the precise definition, but it should preserve the \r through the deserialization step)

@decathorpe

@decathorpe On an entirely unrelated note - you may be interested in this - I've been getting the Rust bindings for librpm (https://github.com/rpm-software-management/librpm.rs/) up to parity with (and beyond in some cases) the Python bindings.

And a few other projects at that intersection with respect to packaging, dependency solving, and working with repo metadata. In Rust.

@dralley

Yes, deserialization normalizes EOLs as that is required by the XML specification, as @dralley pointed. That was implemented in #884. The serialization part was never be fully checked for correctness.

It seems to me that this is duplicate of #670.

It seems to me that this is duplicate of #670.

I don't think this is the case, at least not entirely ...
Running the reproducer code snippet from above with quick-xml 0.37.5, there is no failure, and the \r is preserved as-is, both during de- and serialization. So the problem specifically started happening between 0.37.5 and 0.38.0. Issue #670 is much older.

@decathorpe On an entirely unrelated note - you may be interested in this - I've been getting the Rust bindings for librpm (https://github.com/rpm-software-management/librpm.rs/) up to parity with (and beyond in some cases) the Python bindings.

And a few other projects at that intersection with respect to packaging, dependency solving, and working with repo metadata. In Rust.

Oh, interesting. I hadn't realized that this project was getting attention again :)

Does encoding (escaping) the return as a character reference () in the input work? (that doesn't "round-trip" exactly, under the precise definition, but it should preserve the \r through the deserialization step)

Yes, this works for deserialization (i.e. let string = "<value>helloworld</value>"; in the reproducer code).

Should \r get escaped in QuoteLevel::Full serialization mode maybe? That would partly of the mismatch betwen de- and serialization behaviour. (I dropped this from my minimal reproducer because it wasn't necessary to trigger the issue I've reported, but in my actual code I use QuoteLevel::Full).

Interestingly, python's xml.etree.ElementTree does not escape \r when serializing, but lxml does.

import xml.etree.ElementTree as ET

# Build an element with a carriage return in its text
root = ET.Element('value')
root.text = 'hello\rworld'

# Serialize it
xml_bytes = ET.tostring(root, encoding='unicode')
print('Serialized:', repr(xml_bytes))

# Parse it back
parsed = ET.fromstring(xml_bytes)
print('Parsed text:', repr(parsed.text))

# Now try without escaping (manual XML with literal \r)
manual_xml = '<value>hello\rworld</value>'
parsed2 = ET.fromstring(manual_xml)
print('Parsed literal CR:', repr(parsed2.text))

# And CRLF
manual_xml3 = '<value>hello\r\nworld</value>'
parsed3 = ET.fromstring(manual_xml3)
print('Parsed literal CRLF:', repr(parsed3.text))

Serialized: '<value>hello\rworld</value>'
Parsed text: 'hello\nworld'
Parsed literal CR: 'hello\nworld'
Parsed literal CRLF: 'hello\nworld'

from lxml import etree

root = etree.Element('value')
root.text = 'hello\rworld'

xml_bytes = etree.tostring(root, encoding='unicode')
print('lxml serialized:', repr(xml_bytes))

parsed = etree.fromstring(xml_bytes)
print('lxml parsed text:', repr(parsed.text))

manual_xml = '<value>hello\rworld</value>'
parsed2 = etree.fromstring(manual_xml)
print('lxml parsed literal CR:', repr(parsed2.text))

manual_xml3 = '<value>hello\r\nworld</value>'
parsed3 = etree.fromstring(manual_xml3)
print('Parsed literal CRLF:', repr(parsed3.text))

lxml serialized: '<value>hello&#13;world</value>'
lxml parsed text: 'hello\rworld'
lxml parsed literal CR: 'hello\nworld'
Parsed literal CRLF: 'hello\nworld'

I agree it's not an exact duplicate of #670, it's a very similar issue but different contexts, #670 is wr/r/t attribute values.

I'm generally comfortable with assuming that libxml2 (lxml) is correct here and that \r needs to be escaped when serializing. I'll put up a PR later that addresses both this issue and #670

Escape \r in text and \r/\n/\t in attributes during serde serialization

A more comprehensive comparison:

from lxml import etree

print('=== TEXT CONTENT ===')
print()

# CR in text
root = etree.Element('value')
root.text = 'hello\rworld'
xml = etree.tostring(root, encoding='unicode')
print(f'CR in text - serialized: {xml!r}')
parsed = etree.fromstring(xml)
print(f'CR in text - parsed:     {parsed.text!r}')
print(f'CR in text - roundtrip:  {parsed.text == root.text}')
print()

# CRLF in text
root = etree.Element('value')
root.text = 'hello\r\nworld'
xml = etree.tostring(root, encoding='unicode')
print(f'CRLF in text - serialized: {xml!r}')
parsed = etree.fromstring(xml)
print(f'CRLF in text - parsed:     {parsed.text!r}')
print(f'CRLF in text - roundtrip:  {parsed.text == root.text}')
print()

# LF in text (should NOT be escaped)
root = etree.Element('value')
root.text = 'hello\nworld'
xml = etree.tostring(root, encoding='unicode')
print(f'LF in text - serialized: {xml!r}')
parsed = etree.fromstring(xml)
print(f'LF in text - parsed:     {parsed.text!r}')
print(f'LF in text - roundtrip:  {parsed.text == root.text}')
print()

# Tab in text (should NOT be escaped)
root = etree.Element('value')
root.text = 'col1\tcol2'
xml = etree.tostring(root, encoding='unicode')
print(f'TAB in text - serialized: {xml!r}')
parsed = etree.fromstring(xml)
print(f'TAB in text - parsed:     {parsed.text!r}')
print(f'TAB in text - roundtrip:  {parsed.text == root.text}')
print()

print('=== ATTRIBUTE VALUES ===')
print()

# CR in attr
root = etree.Element('root')
root.set('value', 'new\rline')
xml = etree.tostring(root, encoding='unicode')
print(f'CR in attr - serialized: {xml!r}')
parsed = etree.fromstring(xml)
print(f'CR in attr - parsed:     {parsed.get(\"value\")!r}')
print(f'CR in attr - roundtrip:  {parsed.get(\"value\") == root.get(\"value\")}')
print()

# CRLF in attr
root = etree.Element('root')
root.set('value', 'new\r\nline')
xml = etree.tostring(root, encoding='unicode')
print(f'CRLF in attr - serialized: {xml!r}')
parsed = etree.fromstring(xml)
print(f'CRLF in attr - parsed:     {parsed.get(\"value\")!r}')
print(f'CRLF in attr - roundtrip:  {parsed.get(\"value\") == root.get(\"value\")}')
print()

# LF in attr
root = etree.Element('root')
root.set('value', 'new\nline')
xml = etree.tostring(root, encoding='unicode')
print(f'LF in attr - serialized: {xml!r}')
parsed = etree.fromstring(xml)
print(f'LF in attr - parsed:     {parsed.get(\"value\")!r}')
print(f'LF in attr - roundtrip:  {parsed.get(\"value\") == root.get(\"value\")}')
print()

# TAB in attr
root = etree.Element('root')
root.set('value', 'col1\tcol2')
xml = etree.tostring(root, encoding='unicode')
print(f'TAB in attr - serialized: {xml!r}')
parsed = etree.fromstring(xml)
print(f'TAB in attr - parsed:     {parsed.get(\"value\")!r}')
print(f'TAB in attr - roundtrip:  {parsed.get(\"value\") == root.get(\"value\")}')
print()

print('=== CDATA ===')
print()

# CR in CDATA
root = etree.Element('value')
root.text = etree.CDATA('hello\rworld')
xml = etree.tostring(root, encoding='unicode')
print(f'CR in CDATA - serialized: {xml!r}')
parsed = etree.fromstring(xml)
print(f'CR in CDATA - parsed:     {parsed.text!r}')
print()

# CRLF in CDATA
root = etree.Element('value')
root.text = etree.CDATA('hello\r\nworld')
xml = etree.tostring(root, encoding='unicode')
print(f'CRLF in CDATA - serialized: {xml!r}')
parsed = etree.fromstring(xml)
print(f'CRLF in CDATA - parsed:     {parsed.text!r}')
print()

# LF in CDATA
root = etree.Element('value')
root.text = etree.CDATA('hello\nworld')
xml = etree.tostring(root, encoding='unicode')
print(f'LF in CDATA - serialized: {xml!r}')
parsed = etree.fromstring(xml)
print(f'LF in CDATA - parsed:     {parsed.text!r}')
print()

# Space in text (should NOT be escaped)
root = etree.Element('value')
root.text = 'with spaces'
xml = etree.tostring(root, encoding='unicode')
print(f'Space in text - serialized: {xml!r}')

# Space in attr (should NOT be escaped)
root = etree.Element('root')
root.set('value', 'with spaces')
xml = etree.tostring(root, encoding='unicode')
print(f'Space in attr - serialized: {xml!r}')
parsed = etree.fromstring(xml)
print(f'Space in attr - parsed:     {parsed.get(\"value\")!r}')
print(f'Space in attr - roundtrip:  {parsed.get(\"value\") == root.get(\"value\")}')

=== TEXT CONTENT ===

CR in text - serialized: '<value>hello&#13;world</value>'
CR in text - parsed:     'hello\rworld'
CR in text - roundtrip:  True

CRLF in text - serialized: '<value>hello&#13;\nworld</value>'
CRLF in text - parsed:     'hello\r\nworld'
CRLF in text - roundtrip:  True

LF in text - serialized: '<value>hello\nworld</value>'
LF in text - parsed:     'hello\nworld'
LF in text - roundtrip:  True

TAB in text - serialized: '<value>col1\tcol2</value>'
TAB in text - parsed:     'col1\tcol2'
TAB in text - roundtrip:  True

=== ATTRIBUTE VALUES ===

CR in attr - serialized: '<root value="new&#13;line"/>'
CR in attr - parsed:     'new\rline'
CR in attr - roundtrip:  True

CRLF in attr - serialized: '<root value="new&#13;&#10;line"/>'
CRLF in attr - parsed:     'new\r\nline'
CRLF in attr - roundtrip:  True

LF in attr - serialized: '<root value="new&#10;line"/>'
LF in attr - parsed:     'new\nline'
LF in attr - roundtrip:  True

TAB in attr - serialized: '<root value="col1&#9;col2"/>'
TAB in attr - parsed:     'col1\tcol2'
TAB in attr - roundtrip:  True

=== CDATA ===

CR in CDATA - serialized: '<value><![CDATA[hello\rworld]]></value>'
CR in CDATA - parsed:     'hello\nworld'

CRLF in CDATA - serialized: '<value><![CDATA[hello\r\nworld]]></value>'
CRLF in CDATA - parsed:     'hello\nworld'

LF in CDATA - serialized: '<value><![CDATA[hello\nworld]]></value>'
LF in CDATA - parsed:     'hello\nworld'

Space in text - serialized: '<value>with spaces</value>'
Space in attr - serialized: '<root value="with spaces"/>'
Space in attr - parsed:     'with spaces'
Space in attr - roundtrip:  True

Escape \r in text and \r/\n/\t in attributes during serde serialization

Navigation Menu

Repository navigation

deserializing inconsistently drops / rewrites whitespace characters (especially `\r`) since v0.38.0 #990

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

deserializing inconsistently drops / rewrites whitespace characters (especially \r) since v0.38.0 #990

Description

Activity

dralley commented on Jul 30, 2026

decathorpe commented on Jul 30, 2026

dralley commented on Jul 30, 2026

dralley commented on Jul 30, 2026

Mingun commented on Jul 30, 2026

decathorpe commented on Jul 30, 2026

decathorpe commented on Jul 30, 2026

dralley commented on Jul 31, 2026

dralley commented on Jul 31, 2026

dralley commented on Jul 31, 2026

dralley commented on Aug 5, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

deserializing inconsistently drops / rewrites whitespace characters (especially `\r`) since v0.38.0 #990