Skip to content

BUG: Apply character spacing (Tc) per glyph in layout-mode extraction - #3951

Merged
stefan6419846 merged 1 commit into
py-pdf:mainfrom
Anai-Guo:fix-layout-tc-per-glyph
Aug 12, 2026
Merged

stefan6419846 merged 1 commit into
py-pdf:mainfrom
Anai-Guo:fix-layout-tc-per-glyph

Conversation

@Anai-Guo

@Anai-Guo Anai-Guo commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Closes #3948.

Problem

In layout-mode extraction, TextStateParams.word_tx (pypdf/_text_extraction/_layout_mode/_text_state_params.py) added the character-spacing amount Tc once per shown string, regardless of glyph count:

return (
    (self.font_size * ((width - td_offset) / 1000.0))
    + self.Tc
    + word.count(self.font.space_char) * self.Tw
) * (self.Tz / 100.0)

width is accumulated per glyph and Tw is multiplied by the number of space glyphs, but Tc was constant. PDF 32000-1 defines character spacing as a per-glyph quantity (§9.3.2, §9.4.4): showing an N-glyph string advances the text matrix by N * Tc, and a TJ numeric adjustment — which shows no glyph — contributes no Tc at all.

Consequences when Tc is nonzero:

  • A TJ string element of N glyphs advanced by 1 * Tc instead of N * Tc, so everything after it in the array was placed (N - 1) * Tc too far left.
  • A leading numeric adjustment in a TJ array is advanced against the empty TextStateParams created in _fixed_width_page.py (value == ""), and word_tx("") still added a spurious +Tc.

Fix

-            + self.Tc
+            + len(word) * self.Tc

word has already been decoded to one character per glyph at this point (the per-glyph width loop directly above indexes self.font.width_map per character), so len(word) is the glyph count and stays consistent with the width accumulation. len("") == 0 naturally drops the spurious term for pure numeric adjustments, matching the spec (an advance produced purely by a TJ numeric adjustment carries no character spacing). The one-glyph space_tx path is unchanged.

Verification

Reproducer from the issue — two content streams that place identical glyphs at identical positions per the spec (Courier 12pt, Tc = 36; the 10-glyph run advances 10 * (600/1000 * 12) + 10 * 36 = 432pt, so BB starts at x = 72 + 432 in both):

PDF A:  BT /F1 12 Tf 36 Tc 72 720 Td [(AAAAAAAAAA) 0 (BB)] TJ ET
PDF B:  BT /F1 12 Tf 36 Tc 72 720 Td (AAAAAAAAAA) Tj 432 0 Td (BB) Tj ET

Before, only the Td form kept the (spurious) gap; after the fix both extract identically. A new regression test test_layout_mode_character_spacing_per_glyph builds both streams in memory and asserts they converge.

resources/toy.layout.txt (the golden for test_layout_mode_uncommon_operators, which exercises Tc via the " operator that sets Tc = 120) is regenerated: the corrected per-glyph advance changes the layout-grid scale, shifting a few lines. ruff check passes on both changed files; the pre-existing ruff format drift in those files is unrelated to this change.

🤖 Generated with Claude Code

In layout-mode text extraction, TextStateParams.word_tx added the
character-spacing amount Tc exactly once per shown string, regardless of
how many glyphs the string contained. PDF 32000-1 (9.3.2, 9.4.4) defines
Tc as a per-glyph quantity: showing an N-glyph string advances the text
matrix by N * Tc, and a TJ numeric adjustment (which shows no glyph)
contributes no Tc at all.

The once-per-call term left every multi-glyph TJ string element short by
(N - 1) * Tc, shifting everything after it in the array, and added a
spurious +Tc for the empty TextStateParams used to advance a leading TJ
numeric adjustment. Multiply Tc by len(word), consistent with the
per-glyph width loop directly above and the existing per-space-glyph
Tw term; len("") == 0 drops the spurious term for pure adjustments.

Closes py-pdf#3948
@codecov

codecov Bot commented Aug 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.89%. Comparing base (f6287fa) to head (b73185c).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3951   +/-   ##
=======================================
  Coverage   97.89%   97.89%           
=======================================
  Files          57       57           
  Lines       10869    10869           
  Branches     2032     2032           
=======================================
  Hits        10640    10640           
  Misses        127      127           
  Partials      102      102           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@stefan6419846 stefan6419846 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please explain why the indentation decreases after this change and why this is correct.

For me, it seems like we might have some empty words? Before changing anything, we should check against other tools like pdftotext from poppler to see how they handle it and compare to our results.

@Anai-Guo

Copy link
Copy Markdown
Contributor Author

Thanks — good questions. I dug into both, including the poppler comparison you asked for. Short version: the indentation change is a global side effect of two words becoming correctly wider, the empty-word case is handled correctly, and it lines up with pdftotext.

Why the indentation decreases (and why it's correct)

The leading indentation is not per-line. fixed_char_width() (_fixed_width_page.py:363) computes a single page-global average character width — the length-weighted mean of (displaced_tx - tx) / len over every fragment — and each group's column is then offset = int(tx // char_width) (:422).

I instrumented every fragment on toy.pdf, old vs. new:

fragment Tc tx (old→new) space_tx (old→new) displaced_tx (old→new)
AWAY again1 … Something/[cited] 0 unchanged unchanged unchanged
Single quote operator 0 74.92 124.48 unchanged
Double quote operator 120 74.92 300.48 175.22 → 271.12
Last Txt 120 74.92 300.48 116.14 → 149.70

So tx and space_tx don't move at all. Only the two Tc=120 fragments get wider — which raises the page-global char_width, and a larger char_width shrinks every line's tx // char_width by roughly the same amount. That's why lines with Tc=0 also shift: they're divided by the same global average.

Those two words should be wider. toy.pdf sets Tc=120 via the " operator on Double quote operator (21 glyphs) and it carries into Last Txt via T*. PDF 32000-1 §9.4.4 defines Tc as advancing the text matrix once per glyph, so a 21-glyph run advances 20·Tc more than the old + self.Tc (constant) counted.

Cross-check with pdftotext -layout (poppler)

AWAY again1
   AW AY again2
Something[cited]
Single quote operator  operator
Double quote
Last Txt

poppler uses a different column model, so exact counts differ, but the physically meaningful signal agrees: poppler renders Double quote and operator as if the word is very wide (it breaks the run), because with Tc=120 per glyph it genuinely is. The old pypdf math treated that word as Tc-narrow, which was the bug.

The empty-word case

You're right that empty words reach word_tx — I logged 3 on toy.pdf. They are: the space_tx fallback word_tx("", -space_width) (_text_state_params.py:98) and a leading TJ numeric adjustment ([300 (A) …] → displacement_matrix(td_offset=300) on the empty initial fragment). In every case the "word" shows zero glyphs, so len(word)·Tc = 0 is the physically correct contribution — a TJ numeric adjustment displays no glyph and per spec gets no Tc, and the space_tx fallback only fires when space_tx≈0 (i.e. Tc≈0 already). The old + self.Tc added one glyph's spacing for a zero-glyph event, which was the spurious term.

I also built a TJ-with-numeric-adjustment page with Tc≠0 ([(AB) -6000 (CD)] TJ) to make sure the per-glyph change doesn't corrupt the adjustment path: it renders AB CD with the gap intact, matching the explicit-positioning baseline.

Happy to add the pdftotext case as a documented comparison in the test if you'd like, or to split the resources/toy.layout.txt update into its own commit for clarity.

@stefan6419846 stefan6419846 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the heads-up.

As a side note, as you are already indicating LLM usage correctly: Please avoid generating comments in discussions with LLMs, especially when answering my questions, as per our AI policy. Some of your comments/answers sound like they are at least partially AI-generated.

@stefan6419846
stefan6419846 merged commit 11d92f2 into py-pdf:main Aug 12, 2026
20 checks passed
stefan6419846 added a commit that referenced this pull request Aug 13, 2026
## What's new

### Security (SEC)
- Detect cycles in TreeObject.insert_child (#3964) by @stefan6419846

### New Features (ENH)
- AppearanceStream: Allow arbitrary rotations and apply rotations for annotation appearance streams (#3917) by @PJBrs
- AppearanceStream: Consider more encodings for Type1 core fonts (#3905) by @PJBrs

### Performance Improvements (PI)
- Build /ToUnicode source-code bytes directly in parse_bfrange (#3952) by @Anai-Guo

### Bug Fixes (BUG)
- Preserve internal Link annotations when appending/merging (#3953) (#3954) by @Anai-Guo
- Fix false inline image end markers (#3922) by @ciphercom
- Reject typeless non-page /Kids entries in strict mode (#3955) by @Anai-Guo
- Apply character spacing (Tc) per glyph in layout-mode extraction (#3951) by @Anai-Guo
- Keep the Adobe CMYK inversion when an explicit /Decode is present (#3943) by @RavSinghChandan

### Robustness (ROB)
- AppearanceStream: Only escape parentheses for 8-bit fonts (#3961) by @PJBrs
- Recover the trailer from a cross-reference stream when rebuilding the xref (#3950) by @Anai-Guo

### Maintenance (MAINT)
- Deduplicate CatalogAttributes and CatalogDictionary (#3940) by @j-t-1

### Code Style (STY)
- Type pages as a Sequence rather than a list (#3957) by @RavSinghChandan

[Full Changelog](6.15.0...6.16.0)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Layout mode: character spacing (Tc) is applied once per TJ string element instead of once per glyph

2 participants