Repository navigation
BUG: Apply character spacing (Tc) per glyph in layout-mode extraction - #3951
Conversation
In layout-mode text extraction, TextStateParams.word_tx added the
character-spacing amount Tc exactly once per shown string, regardless of
how many glyphs the string contained. PDF 32000-1 (9.3.2, 9.4.4) defines
Tc as a per-glyph quantity: showing an N-glyph string advances the text
matrix by N * Tc, and a TJ numeric adjustment (which shows no glyph)
contributes no Tc at all.
The once-per-call term left every multi-glyph TJ string element short by
(N - 1) * Tc, shifting everything after it in the array, and added a
spurious +Tc for the empty TextStateParams used to advance a leading TJ
numeric adjustment. Multiply Tc by len(word), consistent with the
per-glyph width loop directly above and the existing per-space-glyph
Tw term; len("") == 0 drops the spurious term for pure adjustments.
Closes py-pdf#3948
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3951 +/- ##
=======================================
Coverage 97.89% 97.89%
=======================================
Files 57 57
Lines 10869 10869
Branches 2032 2032
=======================================
Hits 10640 10640
Misses 127 127
Partials 102 102 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
stefan6419846
left a comment
There was a problem hiding this comment.
Please explain why the indentation decreases after this change and why this is correct.
For me, it seems like we might have some empty words? Before changing anything, we should check against other tools like pdftotext from poppler to see how they handle it and compare to our results.
|
Thanks — good questions. I dug into both, including the poppler comparison you asked for. Short version: the indentation change is a global side effect of two words becoming correctly wider, the empty-word case is handled correctly, and it lines up with Why the indentation decreases (and why it's correct)The leading indentation is not per-line. I instrumented every fragment on
So Those two words should be wider. Cross-check with
|
stefan6419846
left a comment
There was a problem hiding this comment.
Thanks for the heads-up.
As a side note, as you are already indicating LLM usage correctly: Please avoid generating comments in discussions with LLMs, especially when answering my questions, as per our AI policy. Some of your comments/answers sound like they are at least partially AI-generated.
## What's new ### Security (SEC) - Detect cycles in TreeObject.insert_child (#3964) by @stefan6419846 ### New Features (ENH) - AppearanceStream: Allow arbitrary rotations and apply rotations for annotation appearance streams (#3917) by @PJBrs - AppearanceStream: Consider more encodings for Type1 core fonts (#3905) by @PJBrs ### Performance Improvements (PI) - Build /ToUnicode source-code bytes directly in parse_bfrange (#3952) by @Anai-Guo ### Bug Fixes (BUG) - Preserve internal Link annotations when appending/merging (#3953) (#3954) by @Anai-Guo - Fix false inline image end markers (#3922) by @ciphercom - Reject typeless non-page /Kids entries in strict mode (#3955) by @Anai-Guo - Apply character spacing (Tc) per glyph in layout-mode extraction (#3951) by @Anai-Guo - Keep the Adobe CMYK inversion when an explicit /Decode is present (#3943) by @RavSinghChandan ### Robustness (ROB) - AppearanceStream: Only escape parentheses for 8-bit fonts (#3961) by @PJBrs - Recover the trailer from a cross-reference stream when rebuilding the xref (#3950) by @Anai-Guo ### Maintenance (MAINT) - Deduplicate CatalogAttributes and CatalogDictionary (#3940) by @j-t-1 ### Code Style (STY) - Type pages as a Sequence rather than a list (#3957) by @RavSinghChandan [Full Changelog](6.15.0...6.16.0)
Closes #3948.
Problem
In layout-mode extraction,
TextStateParams.word_tx(pypdf/_text_extraction/_layout_mode/_text_state_params.py) added the character-spacing amountTconce per shown string, regardless of glyph count:widthis accumulated per glyph andTwis multiplied by the number of space glyphs, butTcwas constant. PDF 32000-1 defines character spacing as a per-glyph quantity (§9.3.2, §9.4.4): showing an N-glyph string advances the text matrix byN * Tc, and a TJ numeric adjustment — which shows no glyph — contributes noTcat all.Consequences when
Tcis nonzero:1 * Tcinstead ofN * Tc, so everything after it in the array was placed(N - 1) * Tctoo far left.TextStateParamscreated in_fixed_width_page.py(value == ""), andword_tx("")still added a spurious+Tc.Fix
wordhas already been decoded to one character per glyph at this point (the per-glyph width loop directly above indexesself.font.width_mapper character), solen(word)is the glyph count and stays consistent with the width accumulation.len("") == 0naturally drops the spurious term for pure numeric adjustments, matching the spec (an advance produced purely by a TJ numeric adjustment carries no character spacing). The one-glyphspace_txpath is unchanged.Verification
Reproducer from the issue — two content streams that place identical glyphs at identical positions per the spec (Courier 12pt,
Tc = 36; the 10-glyph run advances10 * (600/1000 * 12) + 10 * 36 = 432pt, soBBstarts atx = 72 + 432in both):Before, only the Td form kept the (spurious) gap; after the fix both extract identically. A new regression test
test_layout_mode_character_spacing_per_glyphbuilds both streams in memory and asserts they converge.resources/toy.layout.txt(the golden fortest_layout_mode_uncommon_operators, which exercisesTcvia the"operator that setsTc = 120) is regenerated: the corrected per-glyph advance changes the layout-grid scale, shifting a few lines.ruff checkpasses on both changed files; the pre-existingruff formatdrift in those files is unrelated to this change.🤖 Generated with Claude Code