Repository navigation
[losses] Consolidate GradCache into one shared engine, fix silently wrong gradients, add mini_batch_num_tokens - #3862
Merged
Merged
Conversation
… onto it The GradCache machinery (RNG-replayed double forward, cached embedding gradients, backward hook) existed in several diverging copies. This adds losses/gradcache.py as the single engine: - RandContext (MPS-aware), _get_batch_size / _create_minibatch (padded, flattened and VLM inputs), and one _backward_hook that receives the cache and random states as arguments, so a second forward pass before the first backward pass can no longer clobber them. - CachedLossMixin: embed_minibatch(_iter), calculate_loss_and_cache_gradients and forward_cached(sentence_features, labels). Consumers implement only calculate_loss(reps, labels=None, *, with_backward=False). RNG snapshots are skipped when gradients are disabled (evaluation). - uses_gradient_cache(loss): capability predicate used by MatryoshkaLoss (to route through CachedLossDecorator) and AdaptiveLayerLoss (to warn), replacing hardcoded isinstance tuples that losses had to be added to. CachedMultipleNegativesRankingLoss moves onto the mixin unchanged in behavior. MegaBatchMarginLoss's mini-batched path (the default) previously crashed on the modality marker and, before that, silently dropped every mini-batch's gradient except the last; it is rebuilt on the engine: - mini_batch_size now changes neither the loss nor the gradient, for any batch/mini-batch size combination, and bounds the loss-stage memory (the similarity matrix is chunked; the dense torch.eye is gone). - Back-propagation happens from the hook, so gradient accumulation and fp16 loss scaling reach every mini-batch. - Exactly 2 input columns are enforced again (a 3rd column used to be silently ignored), the StaticEmbedding rejection sees through a Router, evaluation works under torch.no_grad, and the model's training mode is left untouched.
…loss losses.GradCacheLoss(model, loss, mini_batch_size=...) adds GradCache to any loss that can compute itself from precomputed embeddings, i.e. that exposes compute_loss_from_embeddings(embeddings, labels) -- the seam most losses in the library already have. The loss and gradient are exactly those of training the wrapped loss with the full batch; mini_batch_size only bounds the activation memory. ContrastiveLoss, OnlineContrastiveLoss and the four Batch*TripletLoss classes inline their loss computation in forward; the post-embedding half is extracted into compute_loss_from_embeddings so they can be wrapped too. Dict-valued losses keep their per-component logging: the components are rebuilt around the single gradient-carrying total. Losses that gradient caching cannot support are rejected at construction with a pointed error instead of training incorrectly: - losses that already use gradient caching (double caching), - GISTEmbedLoss (embeds each batch with a second, guide model), - SpladeLoss (scheduled by the sparse trainer via isinstance, and served by CachedSpladeLoss), - losses with their own trainable parameters, e.g. SoftmaxLoss: the backward hook scales the model's gradient by whatever the outer backward provides (gradient accumulation, fp16 loss scale), but the loss's own parameters receive their gradient before that scale is known, - losses whose compute_loss_from_embeddings cannot receive the trainer's labels (e.g. MSELoss's takes teacher embeddings; per-sample losses gain nothing over plain gradient accumulation anyway), - StaticEmbedding-based models (no batch dimension to slice), including behind a Router. The model card config passes the wrapped loss module through, so the inner loss's hyperparameters stay on the card.
CachedMultipleNegativesRankingLoss duplicated MultipleNegativesRankingLoss's entire configuration surface: all 7 hyperparameters, ~50 lines of validation, and the scoring maths. The copies had already drifted -- the cached variant lost the scale > 0 check and crashed get_config_dict on a similarity_fct without __name__. It is now GradCacheLoss around an inner MultipleNegativesRankingLoss, which owns and validates every hyperparameter (read-only properties keep loss.scale, loss.directions, etc. working, and get_config_dict emits the same keys as before). The one thing the class keeps is its chunked calculate_loss, which computes the score matrix mini_batch_size rows at a time -- the wrapper's generic loss stage cannot chunk, and for this loss the full matrix is (batch_size, batch_size * columns). The deprecated CachedMultipleNegativesSymmetricRankingLoss shim is unaffected: it subclasses this class and only fixes two arguments.
…Cache engine CachedGISTEmbedLoss kept its own copies of RandContext (39 lines, identical) and the backward hook. Its hook still read the cache off the loss module at backward time, so a second forward pass before the first backward pass made both hooks back-propagate the second batch's gradients -- silently. It now imports the shared RandContext, uses the shared hook (which receives the cache as an argument), skips the RNG snapshots during evaluation, and skips the guide model in the backward re-embedding, where only the model's own embeddings are needed. Its bespoke pieces stay: the guide model makes the loss genuinely non-wrappable. CachedSpladeLoss duplicated embed_minibatch(_iter) from the engine verbatim; it now inherits them from CachedLossMixin. _compute_total_loss becomes calculate_loss(reps, labels, with_backward), which makes the class actually satisfy the protocol its uses_gradient_cache flag advertises (MatryoshkaLoss would previously have hit an AttributeError trying to decorate it), and its forward collapses onto forward_cached plus the shared per-component dict reconstruction. CachedSpladeLoss had no test coverage at all; tests now pin equivalence with SpladeLoss (loss, components and gradients, for dividing and non-dividing mini-batch sizes), the exactly-one-gradient-carrying-component contract, the two-forwards-before-one-backward case, and the no-grad eval path. The GIST two-forwards regression is pinned too.
…ivesRankingLoss on GPU RandContext can only capture the RNG state of devices it sees tensors for, but the cross-encoder loss handed it the raw string pairs -- before tokenization -- so it captured no device state at all. On CUDA (or MPS), the backward hook's second forward pass therefore drew different dropout masks than the logits the cached gradients belong to: measured logit differences of up to 1.4 between the two passes, i.e. silently biased gradients for every reranker trained with dropout on an accelerator. On CPU the loss was unaffected, which is why the test suite never saw it. predict_minibatch now tokenizes and moves the features to the device *before* snapshotting the RNG state, and snapshots from the feature tensors. A CUDA-gated regression test asserts the forward-pass and backward-pass logits are bitwise identical with dropout active, and fails against the previous code. The loss also moves onto the shared GradCache machinery: the MPS-aware RandContext and the shared backward hook (its private copies were stale on both counts), the cache handed to the hook per forward pass instead of stored on self (two forward passes before one backward pass no longer corrupt each other), and RNG snapshots skipped during evaluation. The prompt/task tokenization context rides along in the hook's per-forward bundle rather than on the module. Loss modules now import SentenceTransformer from its defining submodule: importing it from the top-level package made the losses package un-importable from the cross_encoder import chain (circular import).
…acheLoss prepare_loss only read requires_media_counts off the outermost loss, so a wrapped gradient-cached loss -- e.g. the MatryoshkaLoss(CachedMNRL) in the visual document retrieval example -- never enabled per-sample media counting, leaving VLM mini-batch slicing on its one-item-per-sample fallback. The lookup now scans the loss's submodules. Adds the GradCacheLoss API reference entry and a loss-overview section on scaling the batch size with gradient caching.
An audit of the cached losses against what this branch changed found five uncovered paths; each new test was verified to go red when its fix or mechanism is reverted: - CachedMultipleNegativesRankingLoss attribute back-compat: the delegating properties (scale, directions, ...), the exact get_config_dict keys, the restored scale validation, and get_config_dict with a __name__-less similarity function. - MatryoshkaLoss(CachedMultipleNegativesRankingLoss) equivalence -- the combination the visual document retrieval example ships with, and the one place the *chunked* calculate_loss runs under the CachedLossDecorator. - CachedGISTEmbedLoss end-to-end equivalence with GISTEmbedLoss (2 and 3 columns, dividing and non-dividing mini-batch sizes) and a dropout-replay test for its bespoke embed_minibatch, which now skips the guide model in the backward re-embedding. - The backward hook's frozen-route path, via a real inference-free SPLADE model with a frozen SparseStaticEmbedding query route: the hook must skip that route's mini-batches while the document route still trains. - Cross-encoder: explicit hard negatives (third input column), and the prompt reaching the backward hook's re-prediction -- the prompt changes the tokenization, so dropping it from the hook's bundle would apply the cached gradients to different inputs. Plus a bf16 autocast smoke test for GradCacheLoss.
An independent review of the branch confirmed the core (bit-exact CUDA
dropout replay, correct grad_output scaling on every path, the CE fix,
tolerance calibration) and surfaced the following, all fixed with tests:
- CachedMultipleNegativesRankingLoss hyperparameters had become read-only
properties: `loss.scale = 5.0` raised, and assigning an nn.Module
similarity was silently captured by nn.Module.__setattr__ into _modules
without reaching the wrapped loss. Assignments now delegate to the
wrapped loss.
- A wrapped loss that ignores an input column (e.g. a single-column loss
handed extra columns) crashed on a None gradient deep inside
loss.backward(); the mixin now raises at forward time, naming the unused
column.
- Under autocast, the cached gradients are reduced-precision while the
hook's re-embedding runs in fp32, crashing torch.dot on models whose
embeddings leave autocast in half precision (e.g. Dense-final) --
pre-existing across all cached losses, fixed by computing the surrogate
in fp32. The autocast test now uses a Dense-final model; the previous
LayerNorm-final model passed for a model-specific reason.
- RandContext / _create_minibatch / _get_batch_size stay importable from
cached_multiple_negatives_ranking, their historical home.
- The backward hook's frozen-route guard now skips the mini-batch
(continue) instead of abandoning the column (break): with mixed inputs,
a later mini-batch of the same column may still need backprop.
- GradCacheLoss(modelA, loss(modelB)) is now rejected as a model mismatch
instead of a misleading "has trainable parameters of its own".
- The pre-existing inverted docstring claim ("the larger the mini-batch
size, the more memory efficient") is corrected in all cached losses.
Useful if the model uses FA2 with padding
A public GradCacheLoss(model, loss) wrapper added gradient caching to any loss exposing compute_loss_from_embeddings. Dropped on reflection: most losses it could wrap are per-sample, where plain gradient accumulation is already exactly equivalent without GradCache's extra forward pass, and the batch-coupled losses are served by the dedicated Cached* classes, whose chunked loss stage a generic wrapper cannot provide (at batch size 32768 the un-chunked loss stage of MNRL needs ~17 GB where the chunked one needs ~5 GB). Rather than ship a class whose flagship spelling is a memory trap next to CachedMultipleNegativesRankingLoss, the engine stays internal: CachedLossMixin + the shared primitives in base/losses/gradcache.py are the way to build cached losses. CachedMultipleNegativesRankingLoss returns to CachedLossMixin directly, still composing an inner MultipleNegativesRankingLoss that owns and validates every hyperparameter (the deduplication survives the wrapper). The engine tests move onto CachedMultipleNegativesRankingLoss as the thinnest consumer; the mixin's unused-column guard, which no first-party loss can trigger, is pinned with a minimal consumer defined in the test. ContrastiveLoss, OnlineContrastiveLoss and the Batch*TripletLoss classes keep their extracted compute_loss_from_embeddings: it is the seam SpladeLoss requires of its inner losses.
The Protocol existed only to type the backward hook's loss_obj across its two implementation styles (CachedLossMixin, and the cross-encoder loss's adapter). That is one formalism more than the contract warrants: the hook now documents the required embed_minibatch_iter shape in its docstring, which is also where implementers actually look.
calculate_loss_and_cache_gradients had exactly one caller (forward_cached) and one job between two comments; it is now inlined into step (2), taking a method name off the mixin. The unused-column guard moves with it, unchanged. CachedGISTEmbedLoss and the cross-encoder loss keep their own private versions, which differ in signature and never shared this one. reconstruct_loss_components was extracted for a second consumer that no longer exists; it moves into cached_splade.py, its only caller, as a module-private helper, and leaves the base.losses exports.
Per review style feedback: no em or double dashes, no semicolon-chained prose, and no comment blocks of three or more lines. Longer explanations moved into the nearest docstring. Comment blocks carried over verbatim from main are left untouched.
Pass mini_batch_num_tokens in signature order in the symmetric loss, make the cross-encoder super().__init__ call keyword-explicit, and add the missing mini_batch_num_tokens docstring entry on CachedMultipleNegativesSymmetricRankingLoss.
Detach the values that track_loss_components accumulates: the autograd graph kept each step's state alive until log() reset the accumulators, which for CachedSpladeLoss meant retaining every step's full gradient cache. Run the GISTEmbedLoss guide on shallow copies of the features, as it overwrote the embeddings that MatryoshkaLoss had cached for the smaller dims and silently trained only the largest dim. Use the Router-aware StaticEmbedding guard in CachedGISTEmbedLoss like the other cached losses.
Keep the per-component keys of dict-valued base losses in CachedSpladeLoss, matching SpladeLoss: the separate keys exist for per-component logging. Build per-chunk one-hot mask rows in the chunked CMNRL loss stage instead of indexing a full identity matrix, whose O(batch^2) memory defeated the chunking at the batch sizes it targets, and only when a consuming config needs them. Share the canary-checked disable_dropout helper across the bi-encoder, sparse, and cross-encoder suites. New coverage: symmetric-shim equivalence, a simulated rank-1 gather test that drives the offset-aware masks, SPLADE dict components and dropout replay, and doc_to_doc, query_to_query, and all_negatives equivalence configs. Document that the CUDA replay regression test has no GPU job in CI to run it.
…pe in CachedSplade
Contributor
There was a problem hiding this comment.
Pull request overview
This PR consolidates GradCache/gradient-caching infrastructure into a single shared engine and updates all cached losses to use it, addressing several correctness issues (RNG replay, cache re-entrancy, prompt-mask mutation) and adding token-budget mini-batching via mini_batch_num_tokens.
Changes:
- Add a shared GradCache implementation (
sentence_transformers/base/losses/gradcache.py) and refactor cached losses (bi-encoder, sparse, cross-encoder, andMegaBatchMarginLoss) to use it. - Fix multiple silently-wrong-gradient scenarios (accelerator dropout replay, per-forward cache isolation, and prompt-exclusion mask mutation), plus compatibility logic for
MatryoshkaLoss/AdaptiveLayerLoss. - Add broad regression coverage (equivalence, dropout replay, no-grad, grad scaling, token-budget boundaries) and update docs to describe
mini_batch_num_tokens.
Reviewed changes
Copilot reviewed 30 out of 31 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/sparse_encoder/losses/test_cached_splade.py | Adds end-to-end equivalence and dropout-replay tests for CachedSpladeLoss, including token-budget coverage. |
| tests/sparse_encoder/losses/init.py | Test package init touched (no diff shown). |
| tests/sentence_transformer/modules/test_pooling.py | Updates pooling tests to assert prompt-exclusion does not mutate the caller’s mask tensor. |
| tests/sentence_transformer/losses/utils.py | Introduces shared test helpers for disabling dropout and checking gradients. |
| tests/sentence_transformer/losses/test_mega_batch_margin.py | Adds extensive regression tests for mini-batched MegaBatchMarginLoss correctness and new token-budget option. |
| tests/sentence_transformer/losses/test_gradcache.py | Adds engine-level invariants tests for shared GradCache (equivalence, RNG replay, cache isolation, scaling, token budget, non-mutation). |
| tests/sentence_transformer/losses/test_cmnrl.py | Repoints legacy imports to shared GradCache, and adds back-compat / config / matryoshka / DDP-gather coverage. |
| tests/sentence_transformer/losses/test_cached_mnsrl.py | Adds equivalence tests for deprecated symmetric cached loss, including token-budget mode. |
| tests/sentence_transformer/losses/test_cached_gist_embed.py | Adds equivalence and regression tests for cached GIST loss, including re-entrancy and matryoshka interaction. |
| tests/cross_encoder/losses/test_cached_multiple_negatives_ranking.py | Adds cross-encoder cached-loss correctness tests (equivalence, RNG replay on CUDA, cache isolation, scaling, prompts). |
| tests/base/test_trainer.py | Adds regression test asserting track_loss_components detaches accumulated component values. |
| sentence_transformers/sparse_encoder/losses/cached_splade.py | Refactors CachedSpladeLoss to shared engine and preserves dict-valued base-loss components for logging. |
| sentence_transformers/sentence_transformer/modules/pooling.py | Fixes prompt-exclusion to operate on a cloned attention mask and replaces the key instead of mutating in-place. |
| sentence_transformers/sentence_transformer/losses/online_contrastive.py | Adds compute_loss_from_embeddings seam and routes forward() through it. |
| sentence_transformers/sentence_transformer/losses/mega_batch_margin.py | Rebuilds mini-batched path on shared GradCache engine; adds mini_batch_num_tokens and input validation. |
| sentence_transformers/sentence_transformer/losses/matryoshka.py | Switches cached-loss detection to capability check (uses_gradient_cache) instead of hardcoded class tuples. |
| sentence_transformers/sentence_transformer/losses/gist_embed.py | Avoids guide-model mutation effects by running it on shallow copies of feature dicts. |
| sentence_transformers/sentence_transformer/losses/contrastive.py | Adds compute_loss_from_embeddings seam and routes forward() through it. |
| sentence_transformers/sentence_transformer/losses/cached_multiple_negatives_symmetric_ranking.py | Threads mini_batch_num_tokens through the deprecated symmetric cached shim and tweaks docs. |
| sentence_transformers/sentence_transformer/losses/cached_multiple_negatives_ranking.py | Refactors cached MNRL to shared engine + wrapped inner loss; adds token-budget and avoids O(batch²) identity allocation. |
| sentence_transformers/sentence_transformer/losses/cached_gist_embed.py | Integrates shared hook/RandContext and adds token-budget minibatching + per-forward cache isolation. |
| sentence_transformers/sentence_transformer/losses/batch_semi_hard_triplet.py | Adds compute_loss_from_embeddings seam. |
| sentence_transformers/sentence_transformer/losses/batch_hard_triplet.py | Adds compute_loss_from_embeddings seam. |
| sentence_transformers/sentence_transformer/losses/batch_hard_soft_margin_triplet.py | Adds compute_loss_from_embeddings seam. |
| sentence_transformers/sentence_transformer/losses/batch_all_triplet.py | Adds compute_loss_from_embeddings seam. |
| sentence_transformers/sentence_transformer/losses/adaptive_layer.py | Updates incompatibility detection to uses_gradient_cache capability check and clarifies requirements text. |
| sentence_transformers/cross_encoder/losses/cached_multiple_negatives_ranking.py | Fixes device RNG replay by snapshotting after tokenization; refactors hook usage for per-forward cache safety. |
| sentence_transformers/base/trainer.py | Ensures media-count enabling checks wrapped losses via .modules(); detaches tracked loss components to avoid graph retention. |
| sentence_transformers/base/losses/gradcache.py | New shared GradCache engine (RandContext, minibatch slicing, token-budget packing, backward hook, mixin, capability helpers). |
| sentence_transformers/base/losses/init.py | Exposes shared GradCache utilities at the sentence_transformers.base.losses package level. |
| docs/sentence_transformer/usage/efficiency.rst | Documents mini_batch_num_tokens for token-budget minibatching with GradCache losses. |
| docs/sentence_transformer/loss_overview.md | Adds a section describing cached losses / GradCache for scaling effective batch size. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Widen the cross-encoder CachedMultipleNegativesRankingLoss forward labels annotation to Tensor | None, matching how callers use it. Keep the reconstructed SPLADE component scalars fp32 and document why: the sum(values) == total adjustment is exact in fp32, while building them in the total's autocast dtype would inject rounding into exactly that invariant.
Benchmarks on an RTX 3090 (bert-base, NQ, batch 256, FA2) show throughput plateaus once mini_batch_num_tokens saturates the GPU and regresses when peak memory approaches the card's limit, where the driver spills to system RAM instead of erroring. Recommend the smallest saturating budget in the docstring and the efficiency docs.
One compact four-line description everywhere, in place of the long CMNRL paragraph and the cross-links to it from the other losses. The details (independent loss-stage chunking, budget tuning guidance) move to the Training Efficiency docs, which every description links to. The efficiency page now also states explicitly that the loss-stage chunking is an independent partition of the full batch, not a subdivision of the embedding mini-batches.
The efficiency page is titled Speeding up Inference, so a link named Training Efficiency pointed at nothing. The unified description now links to a token-budget mini-batching anchor placed directly on the paragraph that documents it.
The description now says what the budget speeds up (training on variable-length data) and where it is most effective (models that avoid padded compute, like flash attention with input flattening), in place of tuning advice that lives in the efficiency docs. The hyperlink is gone: its anchor sat inside a docs tab, which fragment navigation cannot reach, so the pointer is now plain text.
Split the run-on paragraph into concept, example, and tuning advice, drop the loss-stage chunking aside, and mention that flash attention can be installed via the kernels package as well.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #3854
Hello!
Heads up, this PR was AI-generated and human-reviewed.
Pull Request overview
RandContext, minibatch slicing, backward hook,CachedLossMixin) into one shared engine (base/losses/gradcache.py) used by all cached lossesCachedMultipleNegativesRankingLosson GPU, and inPooling(include_prompt=False)prompt-training with any cached lossmini_batch_num_tokens(token-budget mini-batching, supersedes [feat] Addmini_batch_num_tokens: token-budget mini-batching for cached losses #3859) to all gradient-cached lossesMegaBatchMarginLosscode path (the default), which crashed on themodalitymarker and previously dropped every mini-batch's gradient except the lastDetails
The GradCache (https://huggingface.co/papers/2101.06983) machinery existed in several diverging copies across the codebase: the ST
CachedMultipleNegativesRankingLoss,CachedGISTEmbedLoss,CachedSpladeLoss, and the cross-encoderCachedMultipleNegativesRankingLosseach carried their ownRandContext, backward hook, and mini-batch embedding loops. The copies had already drifted apart, and the drift caused real bugs:CachedMultipleNegativesRankingLosshandedRandContextthe raw string pairs, before tokenization, so it captured no device RNG state at all. On CUDA/MPS, the backward hook's second forward pass drew different dropout masks than the logits the cached gradients belong to — I measured logit differences of up to 1.4 between the two passes on my GPU. In short: silently biased gradients for every reranker trained with dropout on an accelerator. On CPU the loss was unaffected, which is why the test suite never noticed. A CUDA-gated regression test now asserts the two passes are bitwise identical with dropout active, and fails against the old code.CachedGISTEmbedLossand cross-encoder hooks read the cached gradients off the loss module at backward time, so two forward passes before one backward pass silently back-propagated the wrong batch's gradients. The shared hook now receives the cache as an argument, per forward pass.RandContextnever received the MPS fix, and the STCachedMultipleNegativesRankingLosshad lost thescale > 0validation that its non-cached counterpart has.All of this now lives in
sentence_transformers/base/losses/gradcache.py: the MPS-awareRandContext, the padded/flattened/VLM-aware mini-batch slicing (now includingmini_batch_num_tokenstoken-budget boundaries, adopted from #3859 — thanks @mermerico for the design and the benchmarks), one re-entrancy-safe backward hook that receives each forward pass's cache as an argument, and aCachedLossMixinthat implements the three-step GradCache forward, so a cached loss only implementscalculate_loss(reps, labels, with_backward=False). I experimented with a publicGradCacheLoss(model, loss)wrapper on the same mixin, but dropped it: most losses it could wrap are per-sample, where plain gradient accumulation is already exactly equivalent without GradCache's extra forward pass, and for the batch-coupled ones the dedicated Cached* classes additionally chunk the loss stage, which a generic wrapper cannot. (ContrastiveLoss,OnlineContrastiveLossand the fourBatch*TripletLossclasses did gaincompute_loss_from_embeddings, the seamSpladeLossrequires of its inner losses.)CachedMultipleNegativesRankingLossnow composes an innerMultipleNegativesRankingLossthat owns and validates every hyperparameter — deleting the fully duplicated configuration/validation surface (~180 lines) — and keeps only its chunkedcalculate_loss, which bounds the score-matrix memory. Its public attributes,get_config_dictkeys, and the deprecated symmetric shim are unchanged.MatryoshkaLossandAdaptiveLayerLossnow detect gradient-cached losses via auses_gradient_cachecapability check instead of hardcoded class tuples; as a resultMatryoshkaLoss(MegaBatchMarginLoss(...))works for the first time, andCachedSpladeLossgenuinely satisfies thecalculate_losscontract its flag advertises. NoCached*class is deprecated in this PR — they remain the tuned variants on top of the shared engine.This PR grew out of reviewing #3854, which correctly identified that the mini-batched
MegaBatchMarginLoss(the default!) was broken: it crashed on the non-tensormodalitymarker, and underneath that, its in-loop backward guard meant every mini-batch but the last contributed nothing to the gradient. Many thanks to @vineethsaivs for surfacing that — the fix here rebuilds the loss on the shared engine, somini_batch_sizechanges neither the loss nor the gradient (now tested for dividing, non-dividing and oversized mini-batch sizes), gradient accumulation and AMP loss scaling reach every mini-batch, evaluation works undertorch.no_grad, and a 3-column dataset is rejected instead of silently training without its negatives column.Warning
This is a behavior change for anyone training with the mini-batched
MegaBatchMarginLosstoday: it now trains on the full batch, where it previously applied only the final mini-batch's gradient. Training results will differ (for the better), and learning rates may need retuning.Equivalence is pinned by tests throughout: cached == non-cached (loss and gradients) for
CachedMultipleNegativesRankingLossincludingdirections/partition_mode/hardness_modeconfigs, forCachedGISTEmbedLoss(2- and 3-column), for the mini-batchedMegaBatchMarginLoss, forCachedSpladeLoss==SpladeLoss(its first tests ever), and for the cross-encoder loss with and without hard negatives, plus theMatryoshkaLosscombinations of each. Dropout-replay tests assert the two forward passes match bitwise with dropout active, which no previous test could see because they all disabled dropout first.An adversarial review pass over the branch surfaced a few more items, now included: assignments to the rebased
CachedMultipleNegativesRankingLosshyperparameters delegate to the inner loss (they had silently become read-only), acalculate_lossthat ignores an input column raises a pointed error instead of crashing on aNonegradient insidebackward(), the backward hook computes its surrogate in fp32 so autocast runs with half-precision embeddings (e.g. Dense-final models) no longer crashtorch.dot(a pre-existing issue across all cached losses), andRandContextstays importable from its historical module.A second review pass over the finished branch added three more fixes. The trainer's
track_loss_componentsnow accumulates detached values: it used to keep each step's autograd graph alive until the next logging flush, which forCachedSpladeLossmeant retaining every step's full gradient cache (multi-GB at the defaultlogging_steps=500, on exactly the loss whose purpose is bounding memory).MatryoshkaLoss(GISTEmbedLoss(...))computed silently wrong gradients where only the largest dim trained (pre-existing onmain): the guide model overwrote the embeddings thatMatryoshkaLosshad cached for the smaller dims, so the guide now runs on shallow copies of the features. AndCachedGISTEmbedLossuses the same Router-aware StaticEmbedding guard as the other cached losses.A third, independent review pass contributed a few more improvements:
CachedSpladeLossnow keeps the per-component keys of a dict-valued base loss in its returned dict instead of collapsing them intobase_loss(matchingSpladeLoss, so per-component logging survives caching), the chunked CMNRL loss stage no longer materializes a full(batch, batch)identity matrix for its self-similarity and hardness masks (torch.eye(65536)would be ~17 GB, defeating the chunking at exactly the batch sizes it targets; it now builds(mini_batch, batch)one-hot rows per chunk), the deprecatedCachedMultipleNegativesSymmetricRankingLossgained a direct equivalence test against its non-cached counterpart, the equivalence matrix now also covers thedoc_to_doc/query_to_querydirections andall_negativeshardness, and the test-sidedisable_dropouthelper (with its canary self-check) is shared across the bi-encoder, sparse, and cross-encoder suites. Note for reviewers: the CUDA dropout-replay regression test only runs on GPU machines, as the CI matrix has no GPU job and the bug it guards cannot manifest on CPU.Benchmarks
Measured with a copy of
examples/sentence_transformer/training/prompts/training_nq_prompts.py: bert-base-uncased in bf16, 150 steps at batch size 256 on natural-questions withCachedMultipleNegativesRankingLoss, padded inputs (no flattening), identical seeds, single RTX 3090. "FA2" is the flash-attn 2 kernel; SDPA computes attention on the padded batch. Single-seed numbers, so treat them as indicative.First, the per-mini-batch padding trim, with a fixed
mini_batch_size=16:main, SDPAmain, FA2The trim gives +26% throughput on the padded (SDPA/eager) attention path, the default configuration. With FA2 the delta is zero, as expected: transformers unpads internally there, so trailing padding was already free. Losses and evaluation scores are equivalent throughout (eval loss curves agree within 0.3% at every checkpoint; the remaining differences are bf16 noise).
mini_batch_num_tokensthen adds a larger speedup on top by packing every mini-batch to near-constant work (all rows PR with FA2):mini_batch_size=16mini_batch_num_tokens=4096mini_batch_num_tokens=16384mini_batch_num_tokens=32768mini_batch_num_tokens=49152Throughput plateaus once the budget saturates the GPU, and regresses when peak memory approaches the card's limit: the driver spills to system RAM instead of erroring, which is why the docs now recommend the smallest saturating budget. End to end,
mainwith fixed mini-batches to the PR with FA2 and a tuned token budget is 715 s to 182 s, a 3.9x speedup, with all quality metrics unchanged.Small intentional changes worth knowing about:
calculate_loss_and_cache_gradientsnow returns(loss, cache)instead of settingself.cache(the.cache/.random_statesattributes are gone, they were the re-entrancy bug), loss-modulestate_dict()s of the rebased CMNRL contain the model under bothmodel.*andloss.model.*(parameters dedupe, optimizers unaffected, and nothing in the library persists loss modules), and model cards for CMNRL trainings now additionally carry theloss:MultipleNegativesRankingLosstag since the card generator walks the new.losschild. Not covered by tests:gather_across_devicesunder real multi-GPU DDP, and the MPSRandContextbranch (no hardware).