Skip to content

Fix sparse embedding normalization for non-default dtypes - #4045

Merged
tomaarsen merged 3 commits into
huggingface:mainfrom
HaokaiDing:fix/sparse-normalization-dtype
Oct 15, 2026
Merged

tomaarsen merged 3 commits into
huggingface:mainfrom
HaokaiDing:fix/sparse-normalization-dtype

Conversation

@HaokaiDing

Copy link
Copy Markdown
Contributor

Sparse normalize_embeddings allocates its row-norm accumulator with the global default dtype, while values**2 retains the input dtype. index_add_ raises when these differ. For example, this fails with the default float32 setting:

import torch
from sentence_transformers.util import normalize_embeddings

embeddings = torch.tensor([[3.0, 4.0], [0.0, 0.0]], dtype=torch.float64).to_sparse()
normalize_embeddings(embeddings)
# RuntimeError: index_add_(): self (Float) and source (Double) must have the same scalar type

Allocate the accumulator from values so its dtype and device match the input. The example then returns sparse rows equivalent to [[0.6, 0.8], [0.0, 0.0]]. This also covers float16/bfloat16 inputs and float32 inputs when the global default is float64.

Tests cover four input dtypes with both float32 and float64 defaults, zero rows, and backward gradients. Before the fix, 7 of the 9 new cases failed; afterward, all 28 tests in tests/util/test_tensor.py pass.

Validation (CPU):

  • python -m pytest tests/util/test_tensor.py -q — 28 passed.
  • Ruff lint and formatting checks for both changed files — passed.

The full model and GPU test suites were not run.

Copilot AI lite review requested due to automatic review settings September 17, 2026 20:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@tomaarsen

Copy link
Copy Markdown
Member

Hello!

This fixes a real dtype mismatch. I've merged main into the branch and updated the fix to normalize half-precision inputs in float32 while preserving the output dtype. Matching the accumulator to the input dtype avoided the crash, but could produce incorrect norms.

I've added regression coverage for those numerical errors and the interaction with single-vector normalization. Should be good to go now!

  • Tom Aarsen

@tomaarsen
tomaarsen merged commit 766b7f4 into huggingface:main Oct 15, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants