Skip to content

[ESMFold2] Don't build the MSA encoder for checkpoints without one - #49890

Merged
Rocketknight1 merged 1 commit into
huggingface:mainfrom
jimburtoft:esmfold2-msa-encoder-enabled
Oct 27, 2026
Merged

Rocketknight1 merged 1 commit into
huggingface:mainfrom
jimburtoft:esmfold2-msa-encoder-enabled

Conversation

@jimburtoft

@jimburtoft jimburtoft commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

🤗 Transformers CI

What does this PR do?

EsmFold2Model.__init__ always builds self.msa_encoder, but checkpoints trained without MSA conditioning don't ship MSA-encoder weights. biohub/ESMFold2-Fast's config has msa_encoder.enabled: false, and its safetensors contain none of the 130 msa_encoder.* tensors. The flag is parsed but never read, so loading the checkpoint leaves the MSA encoder randomly initialized. Because prepare_protein_features always provides msa (a single-row query MSA for infer_protein), _build_msa_kwargs returns MSA inputs, and the trunk runs the random encoder on every loop.

The result is non-deterministic and wrong. With biohub/ESMFold2-Fast on CPU, two loads with different seeds (same sequence, same diffusion noise) give pLDDT 0.272 vs 0.287 and a 48.9 Å maximum coordinate difference. Passing no MSA makes the two loads identical. The reference esm package builds the MSA encoder only if msa_cfg.enabled.

The fix:

  • Adds enabled (default True, so biohub/ESMFold2 and existing configs are unchanged) to EsmFold2MsaEncoderConfig.
  • Builds msa_encoder only when it is enabled, otherwise None.
  • Has _build_msa_kwargs return None when there is no MSA encoder, so MSA features are ignored for such checkpoints.

modeling_esmfold2.py was regenerated from the modular file.

Tests

Adds test_msa_encoder_disabled, which checks that with msa_encoder={"enabled": False} there is no module and no msa_encoder.* keys, that inference runs, and that the setting survives save_pretrained/from_pretrained. The test fails on main and passes with this change. All tests/models/esmfold2 non-slow tests pass on CPU (14 passed, 1 skipped). ruff check / ruff format --check and utils/check_modular_conversion.py are clean.

Before submitting

  • Did you read the contributor guideline?
  • Did you write any new necessary tests?

Who can review?

@ArthurZucker @Cyrilvallez

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution 🤗!

CI Security Gate — waiting for maintainer approval

CI is approved automatically only for authors who already have a contribution merged in this repository. A maintainer will review this PR and approve CI to run on it.

@Rocketknight1 Rocketknight1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this looks good! The port was written working mostly on the non-fast checkpoints, so this slipped through. Thank you!

@Rocketknight1
Rocketknight1 force-pushed the esmfold2-msa-encoder-enabled branch from 8332c75 to 6f4d68a Compare October 27, 2026 12:14
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@github-actions

Copy link
Copy Markdown
Contributor

[For maintainers] Suggested jobs to run (before merge)

run-slow: esmfold2

@Rocketknight1
Rocketknight1 added this pull request to the merge queue Oct 27, 2026
Merged via the queue into huggingface:main with commit b35ae8e Oct 27, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants