Skip to content

Snippets: split parse_snippets into overridable steps - #3040

Merged
facelessuser merged 4 commits into
facelessuser:mainfrom
joapuiib:snippets-overridable-steps-pr
Nov 11, 2026
Merged

facelessuser merged 4 commits into
facelessuser:mainfrom
joapuiib:snippets-overridable-steps-pr

Conversation

@joapuiib

@joapuiib joapuiib commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

This PR refactors SnippetPreprocessor.parse_snippets into a few smaller methods, so subclasses can extend the snippet syntax and post-process the content without copying the whole parsing loop.

This is a pure refactor: the snippet syntax, the output and the error messages stay exactly as before.

Changes

  • parse_snippet_target(path): parses file.md:1:5, file.md:1:2,8:9 or file.md:section into a dict with path, start, end and section. Returns None if no path is found.
  • extract_lines(...) / select_snippet_lines(lines, target): pick the requested lines or section from the loaded snippet.
  • process_snippet_lines(lines, target): a hook that does nothing by default. It runs after line selection and before nested snippets are parsed.

Existing tests pass unchanged, and new tests cover the extracted methods and a subclass overriding the hooks.

Motivation

I'd like to build a third-party extension on top of snippets that transforms included content with chained filters. For example, it could turn an abbreviations file into a glossary list:

--8<-- "includes/abbreviations.md:abbr | sed /CSS/d | sed s/^\*\[(\w+)\]: /- **\1**: /"

With this PR the whole extension comes down to two overrides:

def parse_snippet_target(self, path):
    path, sep, spec = path.partition('|')
    target = super().parse_snippet_target(path.strip())
    if target is not None and sep:
        target['filters'] = self.parse_filters(spec, path)
    return target

def process_snippet_lines(self, lines, target):
    for name, args in target.get('filters', ()):
        lines = self.filter_funcs[name](lines, args) # Here, implement custom filter logic
    return lines

How this relates to existing issues and PRs

Note

This PR (code, tests and description) was prepared with the help of an LLM. I have reviewed and checked all of it before submitting.

@gir-bot gir-bot added S: needs-review Needs to be reviewed and/or approved. C: snippets Related to the snippets extension. C: source Related to source code. C: tests Related to testing. labels Nov 10, 2026
@joapuiib

joapuiib commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

While splitting out parse_snippet_target, I noticed a small issue with the empty-path error that was already there before this PR (pymdownx/snippets.py on main, lines 331-335):

path = '' if m is None else m.group(1).strip()
# Looks like we have an empty file and only lines specified
if not path:
    if self.check_paths:
        raise SnippetMissingError(f"Snippet at path '{path}' could not be found")

m.group(1) is the file part of the include, without the :lines or :section suffix. For file.md:1:5 it gives file.md. This branch only runs when that part is empty, for example --8<-- ":5:10" or --8<-- ":abbr". So inside it, path is always '', and the error always reads:

Snippet at path '' could not be found

It doesn't say which include caused it.

To keep this PR a pure refactor, I left the message exactly as it was. After the refactor the raw include string (:5:10) is still available at that point. If you'd like, I can send a separate small PR that includes it in the error message.

@joapuiib
joapuiib marked this pull request as ready for review November 10, 2026 11:53
@facelessuser

Copy link
Copy Markdown
Owner

This is likely not exactly what people envisioned when they desired a way to override things, as they weren't coming from the angle of subclassing the extension, but I'm not against refactoring Snippets to allow exactly that, especially if it introduces no adverse effects or performance issues. To be honest, if we are going to do a refactor, we should audit performance issues now, before other people with AI find them and force us to refactor yet again, breaking some of your expectations. It may already be fine, though, but I haven't checked.

To keep this PR a pure refactor, I left the message exactly as it was. After the refactor, the raw include string (:5:10) is still available at that point. If you'd like, I can send a separate small PR that includes it in the error message.

Sure, better errors make sense.

@facelessuser

Copy link
Copy Markdown
Owner

I'm actually going to perform an audit on Snippets. You might have to rebase when I'm done identifying issues.

@joapuiib

Copy link
Copy Markdown
Contributor Author

@facelessuser I'm glad you are considering merging this :)

I will keep an eye out for any updates!

@facelessuser

Copy link
Copy Markdown
Owner

Small refactor to subclassing doesn't cost me much, but my biggest focus is on shutting down the endless AI issues/PR about: quadratic behavior, recursion error, etc. This is a drain on me right now across multiple repositories I maintain, and I am aggressively going after those types of things, before anything else. It's getting less on a number of projects, but the focus usually shifts to another project.

joapuiib and others added 2 commits November 10, 2026 09:36
Extract parse_snippet_target, extract_lines, select_snippet_lines and a
no-op process_snippet_lines hook so subclasses can extend the include
syntax and post-process snippet content. No behavior change.

Rebased onto the Snippets performance work: parse_snippet_target uses the
linear split_path (not the RE_SNIPPET_FILE regular expression) so long
paths with many line ranges stay linear.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Build on the overridable steps in the previous commit.

`process_snippet_lines` runs before nested snippets are expanded, so the
lines it receives still contain snippet notation. A text rewrite, such as
a find and replace, can change the notation of a nested snippet and make
it quietly disappear (or fail with a confusing path when `check_paths` is
enabled). Keep that ordering, as it matches how line and section
selection already work, and make the contract explicit:

- Spell out the ordering and its consequences in the `process_snippet_lines`
  docstring, including that nested snippets are not seen by the hook.
- Add `get_syntax_mask`, which returns a boolean per line that is `True`
  for snippet notation (inline snippets, block markers and the paths
  between them, and section markers). Escaped notation is content. It
  mirrors the loop in `parse_snippets` so hooks can leave notation alone.
- Add an "Extending Snippets" section to the docs describing each step,
  the target keys, the ordering warning, and a complete example that
  renames a term without touching notation.
- Add a changelog entry.

Tests cover the mask directly, check it against what `parse_snippets`
actually keeps as content on randomly generated lines, and use the
documented example to show both that leaving notation alone keeps nested
snippets working and that an unprotected rewrite loses them.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@facelessuser
facelessuser force-pushed the snippets-overridable-steps-pr branch from a5aa730 to 5a4a671 Compare November 10, 2026 17:01
@gir-bot gir-bot added the C: docs Related to documentation. label Nov 10, 2026
@facelessuser

Copy link
Copy Markdown
Owner

@joapuiib I've made a number of performance-related changes and rebased your stuff on top. A couple of additional changes were made on top of that. Please review the changes. If you have concerns or want to make more changes to fit your needs, please commit those, and we can discuss them.

@joapuiib

joapuiib commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

@facelessuser LGTM

I've tested it with my custom extension providing a sed filter and it works :)

@facelessuser

Copy link
Copy Markdown
Owner

@joapuiib If you can add overridable to en-custom.txt in sorted order so the spelling check doesn't fail, I'll try to get this merged.

@facelessuser

Copy link
Copy Markdown
Owner

@gir-bot lgtm

@gir-bot gir-bot added S: approved The pull request is ready to be merged. and removed S: needs-review Needs to be reviewed and/or approved. labels Nov 11, 2026
@facelessuser
facelessuser merged commit 9f07df0 into facelessuser:main Nov 11, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

C: docs Related to documentation. C: snippets Related to the snippets extension. C: source Related to source code. C: tests Related to testing. S: approved The pull request is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants