Skip to content

[losses] Consolidate GradCache into one shared engine, fix silently wrong gradients, add mini_batch_num_tokens - #3862

Merged
tomaarsen merged 26 commits into
mainfrom
feat/gradcache-loss
Jul 16, 2026
Merged

tomaarsen merged 26 commits into
mainfrom
feat/gradcache-loss

Conversation

@tomaarsen

@tomaarsen tomaarsen commented Jul 15, 2026 •

Copy link
Copy Markdown
Member

Supersedes #3854

Hello!

Heads up, this PR was AI-generated and human-reviewed.

Pull Request overview

  • Extract the GradCache machinery (RandContext, minibatch slicing, backward hook, CachedLossMixin) into one shared engine (base/losses/gradcache.py) used by all cached losses
  • Fix silently wrong gradients in the cross-encoder CachedMultipleNegativesRankingLoss on GPU, and in Pooling(include_prompt=False) prompt-training with any cached loss
  • Add mini_batch_num_tokens (token-budget mini-batching, supersedes [feat] Add mini_batch_num_tokens: token-budget mini-batching for cached losses #3859) to all gradient-cached losses
  • Fix the mini-batched MegaBatchMarginLoss code path (the default), which crashed on the modality marker and previously dropped every mini-batch's gradient except the last

Details

The GradCache (https://huggingface.co/papers/2101.06983) machinery existed in several diverging copies across the codebase: the ST CachedMultipleNegativesRankingLoss, CachedGISTEmbedLoss, CachedSpladeLoss, and the cross-encoder CachedMultipleNegativesRankingLoss each carried their own RandContext, backward hook, and mini-batch embedding loops. The copies had already drifted apart, and the drift caused real bugs:

  1. The cross-encoder CachedMultipleNegativesRankingLoss handed RandContext the raw string pairs, before tokenization, so it captured no device RNG state at all. On CUDA/MPS, the backward hook's second forward pass drew different dropout masks than the logits the cached gradients belong to — I measured logit differences of up to 1.4 between the two passes on my GPU. In short: silently biased gradients for every reranker trained with dropout on an accelerator. On CPU the loss was unaffected, which is why the test suite never noticed. A CUDA-gated regression test now asserts the two passes are bitwise identical with dropout active, and fails against the old code.
  2. The CachedGISTEmbedLoss and cross-encoder hooks read the cached gradients off the loss module at backward time, so two forward passes before one backward pass silently back-propagated the wrong batch's gradients. The shared hook now receives the cache as an argument, per forward pass.
  3. The cross-encoder RandContext never received the MPS fix, and the ST CachedMultipleNegativesRankingLoss had lost the scale > 0 validation that its non-cached counterpart has.

All of this now lives in sentence_transformers/base/losses/gradcache.py: the MPS-aware RandContext, the padded/flattened/VLM-aware mini-batch slicing (now including mini_batch_num_tokens token-budget boundaries, adopted from #3859 — thanks @mermerico for the design and the benchmarks), one re-entrancy-safe backward hook that receives each forward pass's cache as an argument, and a CachedLossMixin that implements the three-step GradCache forward, so a cached loss only implements calculate_loss(reps, labels, with_backward=False). I experimented with a public GradCacheLoss(model, loss) wrapper on the same mixin, but dropped it: most losses it could wrap are per-sample, where plain gradient accumulation is already exactly equivalent without GradCache's extra forward pass, and for the batch-coupled ones the dedicated Cached* classes additionally chunk the loss stage, which a generic wrapper cannot. (ContrastiveLoss, OnlineContrastiveLoss and the four Batch*TripletLoss classes did gain compute_loss_from_embeddings, the seam SpladeLoss requires of its inner losses.)

CachedMultipleNegativesRankingLoss now composes an inner MultipleNegativesRankingLoss that owns and validates every hyperparameter — deleting the fully duplicated configuration/validation surface (~180 lines) — and keeps only its chunked calculate_loss, which bounds the score-matrix memory. Its public attributes, get_config_dict keys, and the deprecated symmetric shim are unchanged. MatryoshkaLoss and AdaptiveLayerLoss now detect gradient-cached losses via a uses_gradient_cache capability check instead of hardcoded class tuples; as a result MatryoshkaLoss(MegaBatchMarginLoss(...)) works for the first time, and CachedSpladeLoss genuinely satisfies the calculate_loss contract its flag advertises. No Cached* class is deprecated in this PR — they remain the tuned variants on top of the shared engine.

This PR grew out of reviewing #3854, which correctly identified that the mini-batched MegaBatchMarginLoss (the default!) was broken: it crashed on the non-tensor modality marker, and underneath that, its in-loop backward guard meant every mini-batch but the last contributed nothing to the gradient. Many thanks to @vineethsaivs for surfacing that — the fix here rebuilds the loss on the shared engine, so mini_batch_size changes neither the loss nor the gradient (now tested for dividing, non-dividing and oversized mini-batch sizes), gradient accumulation and AMP loss scaling reach every mini-batch, evaluation works under torch.no_grad, and a 3-column dataset is rejected instead of silently training without its negatives column.

Warning

This is a behavior change for anyone training with the mini-batched MegaBatchMarginLoss today: it now trains on the full batch, where it previously applied only the final mini-batch's gradient. Training results will differ (for the better), and learning rates may need retuning.

Equivalence is pinned by tests throughout: cached == non-cached (loss and gradients) for CachedMultipleNegativesRankingLoss including directions/partition_mode/hardness_mode configs, for CachedGISTEmbedLoss (2- and 3-column), for the mini-batched MegaBatchMarginLoss, for CachedSpladeLoss == SpladeLoss (its first tests ever), and for the cross-encoder loss with and without hard negatives, plus the MatryoshkaLoss combinations of each. Dropout-replay tests assert the two forward passes match bitwise with dropout active, which no previous test could see because they all disabled dropout first.

An adversarial review pass over the branch surfaced a few more items, now included: assignments to the rebased CachedMultipleNegativesRankingLoss hyperparameters delegate to the inner loss (they had silently become read-only), a calculate_loss that ignores an input column raises a pointed error instead of crashing on a None gradient inside backward(), the backward hook computes its surrogate in fp32 so autocast runs with half-precision embeddings (e.g. Dense-final models) no longer crash torch.dot (a pre-existing issue across all cached losses), and RandContext stays importable from its historical module.

A second review pass over the finished branch added three more fixes. The trainer's track_loss_components now accumulates detached values: it used to keep each step's autograd graph alive until the next logging flush, which for CachedSpladeLoss meant retaining every step's full gradient cache (multi-GB at the default logging_steps=500, on exactly the loss whose purpose is bounding memory). MatryoshkaLoss(GISTEmbedLoss(...)) computed silently wrong gradients where only the largest dim trained (pre-existing on main): the guide model overwrote the embeddings that MatryoshkaLoss had cached for the smaller dims, so the guide now runs on shallow copies of the features. And CachedGISTEmbedLoss uses the same Router-aware StaticEmbedding guard as the other cached losses.

A third, independent review pass contributed a few more improvements: CachedSpladeLoss now keeps the per-component keys of a dict-valued base loss in its returned dict instead of collapsing them into base_loss (matching SpladeLoss, so per-component logging survives caching), the chunked CMNRL loss stage no longer materializes a full (batch, batch) identity matrix for its self-similarity and hardness masks (torch.eye(65536) would be ~17 GB, defeating the chunking at exactly the batch sizes it targets; it now builds (mini_batch, batch) one-hot rows per chunk), the deprecated CachedMultipleNegativesSymmetricRankingLoss gained a direct equivalence test against its non-cached counterpart, the equivalence matrix now also covers the doc_to_doc/query_to_query directions and all_negatives hardness, and the test-side disable_dropout helper (with its canary self-check) is shared across the bi-encoder, sparse, and cross-encoder suites. Note for reviewers: the CUDA dropout-replay regression test only runs on GPU machines, as the CI matrix has no GPU job and the bug it guards cannot manifest on CPU.

Benchmarks

Measured with a copy of examples/sentence_transformer/training/prompts/training_nq_prompts.py: bert-base-uncased in bf16, 150 steps at batch size 256 on natural-questions with CachedMultipleNegativesRankingLoss, padded inputs (no flattening), identical seeds, single RTX 3090. "FA2" is the flash-attn 2 kernel; SDPA computes attention on the padded batch. Single-seed numbers, so treat them as indicative.

First, the per-mini-batch padding trim, with a fixed mini_batch_size=16:

main, SDPA PR, SDPA main, FA2 PR, FA2
train runtime 715.0 s 568.3 s 459.7 s 457.8 s
samples/sec 53.7 67.6 83.5 83.9
final eval loss 0.5964 0.5978 0.5976 0.5974
NanoNQ ndcg@10 0.3865 0.3802 0.3855 0.3855

The trim gives +26% throughput on the padded (SDPA/eager) attention path, the default configuration. With FA2 the delta is zero, as expected: transformers unpads internally there, so trailing padding was already free. Losses and evaluation scores are equivalent throughout (eval loss curves agree within 0.3% at every checkpoint; the remaining differences are bf16 noise).

mini_batch_num_tokens then adds a larger speedup on top by packing every mini-batch to near-constant work (all rows PR with FA2):

runtime samples/sec peak VRAM
mini_batch_size=16 457.8 s 83.9 2.15 GiB
mini_batch_num_tokens=4096 268.6 s 142.9 2.43 GiB
mini_batch_num_tokens=16384 190.8 s 201.3 7.06 GiB
mini_batch_num_tokens=32768 182.3 s 210.7 13.25 GiB
mini_batch_num_tokens=49152 256.3 s 149.9 15.22 GiB

Throughput plateaus once the budget saturates the GPU, and regresses when peak memory approaches the card's limit: the driver spills to system RAM instead of erroring, which is why the docs now recommend the smallest saturating budget. End to end, main with fixed mini-batches to the PR with FA2 and a tuned token budget is 715 s to 182 s, a 3.9x speedup, with all quality metrics unchanged.

Small intentional changes worth knowing about: calculate_loss_and_cache_gradients now returns (loss, cache) instead of setting self.cache (the .cache/.random_states attributes are gone, they were the re-entrancy bug), loss-module state_dict()s of the rebased CMNRL contain the model under both model.* and loss.model.* (parameters dedupe, optimizers unaffected, and nothing in the library persists loss modules), and model cards for CMNRL trainings now additionally carry the loss:MultipleNegativesRankingLoss tag since the card generator walks the new .loss child. Not covered by tests: gather_across_devices under real multi-GPU DDP, and the MPS RandContext branch (no hardware).

  • Tom Aarsen

tomaarsen added 18 commits July 13, 2026 14:29
… onto it

The GradCache machinery (RNG-replayed double forward, cached embedding
gradients, backward hook) existed in several diverging copies. This adds
losses/gradcache.py as the single engine:

- RandContext (MPS-aware), _get_batch_size / _create_minibatch (padded,
  flattened and VLM inputs), and one _backward_hook that receives the
  cache and random states as arguments, so a second forward pass before
  the first backward pass can no longer clobber them.
- CachedLossMixin: embed_minibatch(_iter), calculate_loss_and_cache_gradients
  and forward_cached(sentence_features, labels). Consumers implement only
  calculate_loss(reps, labels=None, *, with_backward=False). RNG snapshots
  are skipped when gradients are disabled (evaluation).
- uses_gradient_cache(loss): capability predicate used by MatryoshkaLoss
  (to route through CachedLossDecorator) and AdaptiveLayerLoss (to warn),
  replacing hardcoded isinstance tuples that losses had to be added to.

CachedMultipleNegativesRankingLoss moves onto the mixin unchanged in
behavior. MegaBatchMarginLoss's mini-batched path (the default) previously
crashed on the modality marker and, before that, silently dropped every
mini-batch's gradient except the last; it is rebuilt on the engine:

- mini_batch_size now changes neither the loss nor the gradient, for any
  batch/mini-batch size combination, and bounds the loss-stage memory
  (the similarity matrix is chunked; the dense torch.eye is gone).
- Back-propagation happens from the hook, so gradient accumulation and
  fp16 loss scaling reach every mini-batch.
- Exactly 2 input columns are enforced again (a 3rd column used to be
  silently ignored), the StaticEmbedding rejection sees through a Router,
  evaluation works under torch.no_grad, and the model's training mode is
  left untouched.
…loss

losses.GradCacheLoss(model, loss, mini_batch_size=...) adds GradCache to
any loss that can compute itself from precomputed embeddings, i.e. that
exposes compute_loss_from_embeddings(embeddings, labels) -- the seam most
losses in the library already have. The loss and gradient are exactly
those of training the wrapped loss with the full batch; mini_batch_size
only bounds the activation memory.

ContrastiveLoss, OnlineContrastiveLoss and the four Batch*TripletLoss
classes inline their loss computation in forward; the post-embedding
half is extracted into compute_loss_from_embeddings so they can be
wrapped too. Dict-valued losses keep their per-component logging: the
components are rebuilt around the single gradient-carrying total.

Losses that gradient caching cannot support are rejected at construction
with a pointed error instead of training incorrectly:

- losses that already use gradient caching (double caching),
- GISTEmbedLoss (embeds each batch with a second, guide model),
- SpladeLoss (scheduled by the sparse trainer via isinstance, and served
  by CachedSpladeLoss),
- losses with their own trainable parameters, e.g. SoftmaxLoss: the
  backward hook scales the model's gradient by whatever the outer
  backward provides (gradient accumulation, fp16 loss scale), but the
  loss's own parameters receive their gradient before that scale is
  known,
- losses whose compute_loss_from_embeddings cannot receive the trainer's
  labels (e.g. MSELoss's takes teacher embeddings; per-sample losses
  gain nothing over plain gradient accumulation anyway),
- StaticEmbedding-based models (no batch dimension to slice), including
  behind a Router.

The model card config passes the wrapped loss module through, so the
inner loss's hyperparameters stay on the card.
CachedMultipleNegativesRankingLoss duplicated MultipleNegativesRankingLoss's
entire configuration surface: all 7 hyperparameters, ~50 lines of validation,
and the scoring maths. The copies had already drifted -- the cached variant
lost the scale > 0 check and crashed get_config_dict on a similarity_fct
without __name__.

It is now GradCacheLoss around an inner MultipleNegativesRankingLoss, which
owns and validates every hyperparameter (read-only properties keep
loss.scale, loss.directions, etc. working, and get_config_dict emits the
same keys as before). The one thing the class keeps is its chunked
calculate_loss, which computes the score matrix mini_batch_size rows at a
time -- the wrapper's generic loss stage cannot chunk, and for this loss
the full matrix is (batch_size, batch_size * columns).

The deprecated CachedMultipleNegativesSymmetricRankingLoss shim is
unaffected: it subclasses this class and only fixes two arguments.
…Cache engine

CachedGISTEmbedLoss kept its own copies of RandContext (39 lines, identical)
and the backward hook. Its hook still read the cache off the loss module at
backward time, so a second forward pass before the first backward pass made
both hooks back-propagate the second batch's gradients -- silently. It now
imports the shared RandContext, uses the shared hook (which receives the
cache as an argument), skips the RNG snapshots during evaluation, and skips
the guide model in the backward re-embedding, where only the model's own
embeddings are needed. Its bespoke pieces stay: the guide model makes the
loss genuinely non-wrappable.

CachedSpladeLoss duplicated embed_minibatch(_iter) from the engine verbatim;
it now inherits them from CachedLossMixin. _compute_total_loss becomes
calculate_loss(reps, labels, with_backward), which makes the class actually
satisfy the protocol its uses_gradient_cache flag advertises (MatryoshkaLoss
would previously have hit an AttributeError trying to decorate it), and its
forward collapses onto forward_cached plus the shared per-component dict
reconstruction.

CachedSpladeLoss had no test coverage at all; tests now pin equivalence with
SpladeLoss (loss, components and gradients, for dividing and non-dividing
mini-batch sizes), the exactly-one-gradient-carrying-component contract, the
two-forwards-before-one-backward case, and the no-grad eval path. The GIST
two-forwards regression is pinned too.
…ivesRankingLoss on GPU

RandContext can only capture the RNG state of devices it sees tensors for,
but the cross-encoder loss handed it the raw string pairs -- before
tokenization -- so it captured no device state at all. On CUDA (or MPS),
the backward hook's second forward pass therefore drew different dropout
masks than the logits the cached gradients belong to: measured logit
differences of up to 1.4 between the two passes, i.e. silently biased
gradients for every reranker trained with dropout on an accelerator. On
CPU the loss was unaffected, which is why the test suite never saw it.

predict_minibatch now tokenizes and moves the features to the device
*before* snapshotting the RNG state, and snapshots from the feature
tensors. A CUDA-gated regression test asserts the forward-pass and
backward-pass logits are bitwise identical with dropout active, and fails
against the previous code.

The loss also moves onto the shared GradCache machinery: the MPS-aware
RandContext and the shared backward hook (its private copies were stale
on both counts), the cache handed to the hook per forward pass instead
of stored on self (two forward passes before one backward pass no longer
corrupt each other), and RNG snapshots skipped during evaluation. The
prompt/task tokenization context rides along in the hook's per-forward
bundle rather than on the module.

Loss modules now import SentenceTransformer from its defining submodule:
importing it from the top-level package made the losses package
un-importable from the cross_encoder import chain (circular import).
…acheLoss

prepare_loss only read requires_media_counts off the outermost loss, so a
wrapped gradient-cached loss -- e.g. the MatryoshkaLoss(CachedMNRL) in the
visual document retrieval example -- never enabled per-sample media
counting, leaving VLM mini-batch slicing on its one-item-per-sample
fallback. The lookup now scans the loss's submodules.

Adds the GradCacheLoss API reference entry and a loss-overview section on
scaling the batch size with gradient caching.
An audit of the cached losses against what this branch changed found five
uncovered paths; each new test was verified to go red when its fix or
mechanism is reverted:

- CachedMultipleNegativesRankingLoss attribute back-compat: the delegating
  properties (scale, directions, ...), the exact get_config_dict keys, the
  restored scale validation, and get_config_dict with a __name__-less
  similarity function.
- MatryoshkaLoss(CachedMultipleNegativesRankingLoss) equivalence -- the
  combination the visual document retrieval example ships with, and the one
  place the *chunked* calculate_loss runs under the CachedLossDecorator.
- CachedGISTEmbedLoss end-to-end equivalence with GISTEmbedLoss (2 and 3
  columns, dividing and non-dividing mini-batch sizes) and a dropout-replay
  test for its bespoke embed_minibatch, which now skips the guide model in
  the backward re-embedding.
- The backward hook's frozen-route path, via a real inference-free SPLADE
  model with a frozen SparseStaticEmbedding query route: the hook must skip
  that route's mini-batches while the document route still trains.
- Cross-encoder: explicit hard negatives (third input column), and the
  prompt reaching the backward hook's re-prediction -- the prompt changes
  the tokenization, so dropping it from the hook's bundle would apply the
  cached gradients to different inputs.

Plus a bf16 autocast smoke test for GradCacheLoss.
An independent review of the branch confirmed the core (bit-exact CUDA
dropout replay, correct grad_output scaling on every path, the CE fix,
tolerance calibration) and surfaced the following, all fixed with tests:

- CachedMultipleNegativesRankingLoss hyperparameters had become read-only
  properties: `loss.scale = 5.0` raised, and assigning an nn.Module
  similarity was silently captured by nn.Module.__setattr__ into _modules
  without reaching the wrapped loss. Assignments now delegate to the
  wrapped loss.
- A wrapped loss that ignores an input column (e.g. a single-column loss
  handed extra columns) crashed on a None gradient deep inside
  loss.backward(); the mixin now raises at forward time, naming the unused
  column.
- Under autocast, the cached gradients are reduced-precision while the
  hook's re-embedding runs in fp32, crashing torch.dot on models whose
  embeddings leave autocast in half precision (e.g. Dense-final) --
  pre-existing across all cached losses, fixed by computing the surrogate
  in fp32. The autocast test now uses a Dense-final model; the previous
  LayerNorm-final model passed for a model-specific reason.
- RandContext / _create_minibatch / _get_batch_size stay importable from
  cached_multiple_negatives_ranking, their historical home.
- The backward hook's frozen-route guard now skips the mini-batch
  (continue) instead of abandoning the column (break): with mixed inputs,
  a later mini-batch of the same column may still need backprop.
- GradCacheLoss(modelA, loss(modelB)) is now rejected as a model mismatch
  instead of a misleading "has trainable parameters of its own".
- The pre-existing inverted docstring claim ("the larger the mini-batch
  size, the more memory efficient") is corrected in all cached losses.
Useful if the model uses FA2 with padding
A public GradCacheLoss(model, loss) wrapper added gradient caching to any
loss exposing compute_loss_from_embeddings. Dropped on reflection: most
losses it could wrap are per-sample, where plain gradient accumulation is
already exactly equivalent without GradCache's extra forward pass, and the
batch-coupled losses are served by the dedicated Cached* classes, whose
chunked loss stage a generic wrapper cannot provide (at batch size 32768
the un-chunked loss stage of MNRL needs ~17 GB where the chunked one needs
~5 GB). Rather than ship a class whose flagship spelling is a memory trap
next to CachedMultipleNegativesRankingLoss, the engine stays internal:
CachedLossMixin + the shared primitives in base/losses/gradcache.py are
the way to build cached losses.

CachedMultipleNegativesRankingLoss returns to CachedLossMixin directly,
still composing an inner MultipleNegativesRankingLoss that owns and
validates every hyperparameter (the deduplication survives the wrapper).
The engine tests move onto CachedMultipleNegativesRankingLoss as the
thinnest consumer; the mixin's unused-column guard, which no first-party
loss can trigger, is pinned with a minimal consumer defined in the test.
ContrastiveLoss, OnlineContrastiveLoss and the Batch*TripletLoss classes
keep their extracted compute_loss_from_embeddings: it is the seam
SpladeLoss requires of its inner losses.
The Protocol existed only to type the backward hook's loss_obj across its
two implementation styles (CachedLossMixin, and the cross-encoder loss's
adapter). That is one formalism more than the contract warrants: the hook
now documents the required embed_minibatch_iter shape in its docstring,
which is also where implementers actually look.
calculate_loss_and_cache_gradients had exactly one caller (forward_cached)
and one job between two comments; it is now inlined into step (2), taking
a method name off the mixin. The unused-column guard moves with it,
unchanged. CachedGISTEmbedLoss and the cross-encoder loss keep their own
private versions, which differ in signature and never shared this one.

reconstruct_loss_components was extracted for a second consumer that no
longer exists; it moves into cached_splade.py, its only caller, as a
module-private helper, and leaves the base.losses exports.
Per review style feedback: no em or double dashes, no semicolon-chained
prose, and no comment blocks of three or more lines. Longer explanations
moved into the nearest docstring. Comment blocks carried over verbatim
from main are left untouched.
Pass mini_batch_num_tokens in signature order in the symmetric loss,
make the cross-encoder super().__init__ call keyword-explicit, and add
the missing mini_batch_num_tokens docstring entry on
CachedMultipleNegativesSymmetricRankingLoss.
Detach the values that track_loss_components accumulates: the autograd
graph kept each step's state alive until log() reset the accumulators,
which for CachedSpladeLoss meant retaining every step's full gradient
cache. Run the GISTEmbedLoss guide on shallow copies of the features,
as it overwrote the embeddings that MatryoshkaLoss had cached for the
smaller dims and silently trained only the largest dim. Use the
Router-aware StaticEmbedding guard in CachedGISTEmbedLoss like the
other cached losses.
Keep the per-component keys of dict-valued base losses in
CachedSpladeLoss, matching SpladeLoss: the separate keys exist for
per-component logging. Build per-chunk one-hot mask rows in the chunked
CMNRL loss stage instead of indexing a full identity matrix, whose
O(batch^2) memory defeated the chunking at the batch sizes it targets,
and only when a consuming config needs them. Share the canary-checked
disable_dropout helper across the bi-encoder, sparse, and cross-encoder
suites. New coverage: symmetric-shim equivalence, a simulated rank-1
gather test that drives the offset-aware masks, SPLADE dict components
and dropout replay, and doc_to_doc, query_to_query, and all_negatives
equivalence configs. Document that the CUDA replay regression test has
no GPU job in CI to run it.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR consolidates GradCache/gradient-caching infrastructure into a single shared engine and updates all cached losses to use it, addressing several correctness issues (RNG replay, cache re-entrancy, prompt-mask mutation) and adding token-budget mini-batching via mini_batch_num_tokens.

Changes:

  • Add a shared GradCache implementation (sentence_transformers/base/losses/gradcache.py) and refactor cached losses (bi-encoder, sparse, cross-encoder, and MegaBatchMarginLoss) to use it.
  • Fix multiple silently-wrong-gradient scenarios (accelerator dropout replay, per-forward cache isolation, and prompt-exclusion mask mutation), plus compatibility logic for MatryoshkaLoss/AdaptiveLayerLoss.
  • Add broad regression coverage (equivalence, dropout replay, no-grad, grad scaling, token-budget boundaries) and update docs to describe mini_batch_num_tokens.

Reviewed changes

Copilot reviewed 30 out of 31 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/sparse_encoder/losses/test_cached_splade.py Adds end-to-end equivalence and dropout-replay tests for CachedSpladeLoss, including token-budget coverage.
tests/sparse_encoder/losses/init.py Test package init touched (no diff shown).
tests/sentence_transformer/modules/test_pooling.py Updates pooling tests to assert prompt-exclusion does not mutate the caller’s mask tensor.
tests/sentence_transformer/losses/utils.py Introduces shared test helpers for disabling dropout and checking gradients.
tests/sentence_transformer/losses/test_mega_batch_margin.py Adds extensive regression tests for mini-batched MegaBatchMarginLoss correctness and new token-budget option.
tests/sentence_transformer/losses/test_gradcache.py Adds engine-level invariants tests for shared GradCache (equivalence, RNG replay, cache isolation, scaling, token budget, non-mutation).
tests/sentence_transformer/losses/test_cmnrl.py Repoints legacy imports to shared GradCache, and adds back-compat / config / matryoshka / DDP-gather coverage.
tests/sentence_transformer/losses/test_cached_mnsrl.py Adds equivalence tests for deprecated symmetric cached loss, including token-budget mode.
tests/sentence_transformer/losses/test_cached_gist_embed.py Adds equivalence and regression tests for cached GIST loss, including re-entrancy and matryoshka interaction.
tests/cross_encoder/losses/test_cached_multiple_negatives_ranking.py Adds cross-encoder cached-loss correctness tests (equivalence, RNG replay on CUDA, cache isolation, scaling, prompts).
tests/base/test_trainer.py Adds regression test asserting track_loss_components detaches accumulated component values.
sentence_transformers/sparse_encoder/losses/cached_splade.py Refactors CachedSpladeLoss to shared engine and preserves dict-valued base-loss components for logging.
sentence_transformers/sentence_transformer/modules/pooling.py Fixes prompt-exclusion to operate on a cloned attention mask and replaces the key instead of mutating in-place.
sentence_transformers/sentence_transformer/losses/online_contrastive.py Adds compute_loss_from_embeddings seam and routes forward() through it.
sentence_transformers/sentence_transformer/losses/mega_batch_margin.py Rebuilds mini-batched path on shared GradCache engine; adds mini_batch_num_tokens and input validation.
sentence_transformers/sentence_transformer/losses/matryoshka.py Switches cached-loss detection to capability check (uses_gradient_cache) instead of hardcoded class tuples.
sentence_transformers/sentence_transformer/losses/gist_embed.py Avoids guide-model mutation effects by running it on shallow copies of feature dicts.
sentence_transformers/sentence_transformer/losses/contrastive.py Adds compute_loss_from_embeddings seam and routes forward() through it.
sentence_transformers/sentence_transformer/losses/cached_multiple_negatives_symmetric_ranking.py Threads mini_batch_num_tokens through the deprecated symmetric cached shim and tweaks docs.
sentence_transformers/sentence_transformer/losses/cached_multiple_negatives_ranking.py Refactors cached MNRL to shared engine + wrapped inner loss; adds token-budget and avoids O(batch²) identity allocation.
sentence_transformers/sentence_transformer/losses/cached_gist_embed.py Integrates shared hook/RandContext and adds token-budget minibatching + per-forward cache isolation.
sentence_transformers/sentence_transformer/losses/batch_semi_hard_triplet.py Adds compute_loss_from_embeddings seam.
sentence_transformers/sentence_transformer/losses/batch_hard_triplet.py Adds compute_loss_from_embeddings seam.
sentence_transformers/sentence_transformer/losses/batch_hard_soft_margin_triplet.py Adds compute_loss_from_embeddings seam.
sentence_transformers/sentence_transformer/losses/batch_all_triplet.py Adds compute_loss_from_embeddings seam.
sentence_transformers/sentence_transformer/losses/adaptive_layer.py Updates incompatibility detection to uses_gradient_cache capability check and clarifies requirements text.
sentence_transformers/cross_encoder/losses/cached_multiple_negatives_ranking.py Fixes device RNG replay by snapshotting after tokenization; refactors hook usage for per-forward cache safety.
sentence_transformers/base/trainer.py Ensures media-count enabling checks wrapped losses via .modules(); detaches tracked loss components to avoid graph retention.
sentence_transformers/base/losses/gradcache.py New shared GradCache engine (RandContext, minibatch slicing, token-budget packing, backward hook, mixin, capability helpers).
sentence_transformers/base/losses/init.py Exposes shared GradCache utilities at the sentence_transformers.base.losses package level.
docs/sentence_transformer/usage/efficiency.rst Documents mini_batch_num_tokens for token-budget minibatching with GradCache losses.
docs/sentence_transformer/loss_overview.md Adds a section describing cached losses / GradCache for scaling effective batch size.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Widen the cross-encoder CachedMultipleNegativesRankingLoss forward
labels annotation to Tensor | None, matching how callers use it. Keep
the reconstructed SPLADE component scalars fp32 and document why: the
sum(values) == total adjustment is exact in fp32, while building them
in the total's autocast dtype would inject rounding into exactly that
invariant.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 30 out of 31 changed files in this pull request and generated 1 comment.

Benchmarks on an RTX 3090 (bert-base, NQ, batch 256, FA2) show
throughput plateaus once mini_batch_num_tokens saturates the GPU and
regresses when peak memory approaches the card's limit, where the
driver spills to system RAM instead of erroring. Recommend the
smallest saturating budget in the docstring and the efficiency docs.
One compact four-line description everywhere, in place of the long
CMNRL paragraph and the cross-links to it from the other losses. The
details (independent loss-stage chunking, budget tuning guidance) move
to the Training Efficiency docs, which every description links to. The
efficiency page now also states explicitly that the loss-stage chunking
is an independent partition of the full batch, not a subdivision of the
embedding mini-batches.
The efficiency page is titled Speeding up Inference, so a link named
Training Efficiency pointed at nothing. The unified description now
links to a token-budget mini-batching anchor placed directly on the
paragraph that documents it.
The description now says what the budget speeds up (training on
variable-length data) and where it is most effective (models that avoid
padded compute, like flash attention with input flattening), in place
of tuning advice that lives in the efficiency docs. The hyperlink is
gone: its anchor sat inside a docs tab, which fragment navigation
cannot reach, so the pointer is now plain text.
Split the run-on paragraph into concept, example, and tuning advice,
drop the loss-stage chunking aside, and mention that flash attention
can be installed via the kernels package as well.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 30 out of 31 changed files in this pull request and generated 2 comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants