Skip to content

🚨 Move processing defaults to class - #49089

Merged
ydshieh merged 136 commits into
huggingface:mainfrom
guarin:move-processing-defaults-to-class
Nov 9, 2026
Merged

ydshieh merged 136 commits into
huggingface:mainfrom
guarin:move-processing-defaults-to-class

Conversation

@guarin

@guarin guarin commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

🤗 Transformers CI

What does this PR do?

  1. Moves all multi-model processor defaults from ProcessingKwargs._defaults to Processor class attributes following image, video, and audio processors. Defaults are not saved in the config. This will be handled by 🚨🚧 FeatureExtractor → AudioProcessor #44394
  2. Deprecates all TextKwargs, ImagesKwargs, VideoKwargs classes defined in processing_*.py files. Empty ProcessingKwargs subclasses are also deprecated.

The result are simpler processing_*.py files with a clearer structure on how to customise behavior.

🚨 Breaking

  • External code relying on XYZModelProcessorKwargs._defaults or XYZModelProcessorKwargs.__annotations__
    • It only breaks if a transformers class is imported and one of the above attributes is accessed
    • No breaking change if the class is imported and only used for type checking
    • No breaking change if the class is subclassed
  • Fix: Chameleon return_for_text_completion=True is now honoured
  • Fix: ShieldGemma2Processor now uses the correct processor defaults, previously it used gemma3 defaults by mistake

♻️ Deprecated

  • All TextKwargs, ImagesKwargs, VideoKwargs subclasses defined in processing_*.py files
    • Note: Classes have to be renamed to DeprecatedXYZProcessorKwargs as this routes any imports through __getattr__ which can emit the deprecation warning. This is required because those classes are typed dicts and are never instantiated, therefore normal deprecation warnings in __init__ are not applicable
  • XYZModelProcessorKwargs.text_kwargs/images_kwargs/video_kwargs/audio_kwargs -> Don't set these anymore as their types are impossible to enforce
  • XYZModelProcessorKwargs._defaults -> Defaults are now specified as class attributes on XYZModelProcessor

Other minor changes

  • No more common_kwargs defaults -> Instead set default for each modality explicitly
  • Redundant kwargs in _defaults that matched their modality-specific processor defaults were removed
  • _merge_kwargs automatically merges with self.valid_processor_kwargs which avoids having to pass them manually (lots of hardcoding)

Where to add Use for Default value Configurable at runtime Example
__init__ Settings saved in the config (model properties) Typically has a default No image_seq_length in [Gemma3Processor]
__call__ Inputs only known at runtime No default or None Yes images, labels, suffix, etc.
Custom MyModelProcessingKwargs class Settings that can be configured at runtime Set as processor class attribute Yes use_image_id in [MiniCPMV4_7Processor]

Runtime arguments which can have a default must be added to a custom MyModelProcessingKwargs and will be passed as part of **kwargs to __call__. Defaults for these values can be set as processor class attributes. Defaults for modality-specific processors, for example the tokenizer or image processor, can be defined in the same way. This is usually not needed as modality-specific processors already have defaults defined in their respective classes. Only use this option if the processor must override a modality-specific processor setting. Of course, arguments passed at runtime to __call__ have always higher priority than any defaults.

from ...processing_utils import ProcessingKwargs, ProcessorMixin, Unpack

class MyModelProcessorKwargs(ProcessingKwargs, total=False):
    """
    use_image_id (`bool`, defaults to `False`):
        Whether to prepend an image-id tag (``<image_id>N</image_id>``) before
        each image placeholder.
    """
    use_image_id: bool


class MyModelProcessor(ProcessorMixin):
    valid_processor_kwargs = MyModelProcessorKwargs
    use_image_id = False

    text_kwargs = {"padding": True}   # default passed to tokenizer

    def __init__(self, image_processor=None, tokenizer=None, chat_template=None, image_seq_length: int = 256, **kwargs):
        super().__init__(image_processor, tokenizer, chat_template=chat_template, **kwargs)
        self.image_seq_length = image_seq_length

    @auto_docstring
    def __call__(
        self,
        images: ImageInput | None = None,
        text: TextInput | PreTokenizedInput | list[TextInput] | list[PreTokenizedInput] | None = None,
        suffix: str | None = None,
        **kwargs: Unpack[MyModelProcessorKwargs],
    ):
        images, text, videos, audio = self.prepare_inputs_layout(images=images, text=text, **kwargs)
        self.validate_inputs(images=images, text=text, **kwargs)

        merged_kwargs = self._merge_kwargs(tokenizer_init_kwargs=self.tokenizer.init_kwargs, **kwargs)

        merged_kwargs["use_image_id"] # is `False` as default from class attribute is added by _merge_kwargs
        merged_kwargs["text_kwargs"]["padding"] # is `True`

How is it enforced?

New tests that check:

  • valid_processor_kwargs is set to the correct class (same for image and video processors)
  • defaults for valid_processor_kwargs are not the same as modality-specific processor default
  • explicit __init__ arguments have no default class attribute (avoid multiple defaults)
  • explicit __init__ arguments are not in valid_processor_kwargs (avoid multiple defaults)
  • explicit __call__ arguments are not in valid_processor_kwargs (different responsibility)
  • explicit __call__ arguments are not in class attributes (avoid multiple defaults)
  • explicit __call__ arguments have no or None as default (avoid multiple defaults)

New linter rules that check:

Comment thread src/transformers/processing_utils.py Resolved
Comment thread src/transformers/processing_utils.py Resolved
Comment thread src/transformers/processing_utils.py Resolved
@HuggingFaceDocBuilderDev

HuggingFaceDocBuilderDev commented Sep 24, 2026 •

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Pages changed in this PR:

@zucchini-nlp zucchini-nlp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like seeing empty _defaults, let's make it a standard for new releases. We defi need to keep them somewhere, cls attr makes sense to me but prob a private one - my goal is to stop users from exploiting that for unintended things . Quite sure some args in _defaults rn don't make much sense, like the padding=False

As for saving, I don't think saving all call-time kwargs is necessary, imo it goes arg-by-arg and I left a detailed comment in gemma4 👇🏻

LMK what you think

Comment thread src/transformers/processing_utils.py Resolved
Comment thread src/transformers/processing_utils.py Resolved
Comment thread src/transformers/processing_utils.py Resolved
@guarin

guarin commented Nov 9, 2026

Copy link
Copy Markdown
Member Author

run-slow: aria, audioflamingo3, auto, aya_vision, cohere_asr, colpali

@vasqu vasqu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Last nits but this is good to go imo!

Comment thread docs/source/en/auto_docstring.md Resolved
Comment thread docs/source/ro/multimodal_processing.md Resolved
Comment thread src/transformers/processing_utils.py Resolved
self.assertEqual(video_processor_1.size, modified_copied_size_1)
self.assertEqual(video_processor_1.image_mean, modified_copied_image_mean_1)

def test_valid_kwargs_set(self):

@vasqu vasqu Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good to keep in mind for potentially audio / feat extractor as well?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that requires the audio refactor first though.

@guarin
guarin enabled auto-merge November 9, 2026 15:06
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

[For maintainers] Suggested jobs to run (before merge)

run-slow: align, aria, audioflamingo3, auto, aya_vision, blip, blip_2, bridgetower, bros, canary, chameleon, clipseg, clvp, cohere2_vision, cohere_asr, cohere_compass

@ydshieh
ydshieh disabled auto-merge November 9, 2026 16:17
@ydshieh
ydshieh merged commit 3c3ff8c into huggingface:main Nov 9, 2026
555 of 565 checks passed
@vasqu vasqu mentioned this pull request Nov 9, 2026
4 of 6 tasks
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

CI (Nvidia + AMD)

Nvidia · ✅ No new failures 🎉👏 · ⚙️ Workflow · 📊 Dashboard

AMD · ✅ No new failures 🎉👏 · ⚙️ Workflow · 📊 Dashboard

Failures (9 pre-existing)

    Pre-existing failures (still failing) — 9 total:

        Same error (9):

        - aria:
            tests/models/aria/test_modeling_aria.py::AriaForConditionalGenerationIntegrationTest::test_batched_generation
            tests/models/aria/test_modeling_aria.py::AriaForConditionalGenerationIntegrationTest::test_small_model_integration_test
            tests/models/aria/test_modeling_aria.py::AriaForConditionalGenerationIntegrationTest::test_small_model_integration_test_batch
            tests/models/aria/test_modeling_aria.py::AriaForConditionalGenerationIntegrationTest::test_small_model_integration_test_llama_batched
            tests/models/aria/test_modeling_aria.py::AriaForConditionalGenerationIntegrationTest::test_small_model_integration_test_llama_batched_regression
            tests/models/aria/test_modeling_aria.py::AriaForConditionalGenerationIntegrationTest::test_small_model_integration_test_llama_single
            tests/models/aria/test_modeling_aria.py::AriaForConditionalGenerationIntegrationTest::test_tokenizer_integration

        - aya_vision:
            tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_generate_chat_template
            tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_generate_text_only

tarekziade added a commit that referenced this pull request Nov 10, 2026
#49089 and the deepseek_ocr2, mistral_large4 and nemotron3_5_transcribe fixes are on main now; this branch keeps only the mlinter pin and the esm, pp_doclayout_v2 and pp_formulanet TRF055 fixes.
opensource-fork-sync Bot pushed a commit to cla7aye15I4nd/opensource-huggingface__transformers that referenced this pull request Nov 16, 2026
…0740)

huggingface#49089 removed it as redundant with
`Qwen3ASRFeatureExtractor`, but original checkpoints use
`WhisperFeatureExtractor`, which truncates audio longer than 30 s by default.

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants