🏆 Live Soccer TV: vercel/next.js/pull/

sokra · 2026-10-29T05:19:25Z

What?

Replace aggregation-edge removal retries with negative follower counts ("debt"). A removal that overtakes its addition records debt; the late addition cancels it instead of spinning, requeuing, or eventually panicking. Remove retry_loop, the retry job fields, and MAX_RETRIES.

Also add deterministic regression tests, concurrent graph-churn stress tests, fuzz-driver improvements, and benchmarks.

How?

An edge U → F lives either in F.upper[U] (unsigned) or in U.followers[F] (now i32). Only followers can go negative. Only counts greater than zero are real edges.

Removal, without a pair lock: decrement an existing inner edge under F's lock; otherwise decrement the follower count under U's lock, possibly creating debt. If debt remains, recheck F and schedule BalanceEdge if an inner edge appeared meanwhile.
Addition: pay existing debt on the follower side before creating an inner edge. After creating a new inner edge, recheck U for debt unconditionally, including empty-data/no-activeness cases.
Race argument: removal writes U then reads F; addition writes F then reads U, each under the relevant task lock. For a split that survives intervening operations, both reads cannot precede the other write; at least one side schedules balancing. An intervening update may already resolve the split.
Pair-locked balancing: net follower debt against the inner count first, reverse any disappeared inner contribution, then place the remaining count using root precedence and the existing equality-conflict rules.
Semantic effects: aggregate data, follower propagation, and active counts change only on <= 0 ↔ > 0 crossings. Debt creation/payment alone has no semantic edge effect.
GC: outgoing follower debt does not pin its owner. With persistence enabled, normal queues hold operation guards and snapshot/GC exclusion waits for operations to settle. GC finishes parallel deletion before deferred rebalancing. These phase guarantees—not a debt reference—protect in-flight updates. Removing the last raw debt entry still reports possible collectibility.
Helpers: aggregation_edge.rs contains pure checked arithmetic, placement, and netting. In the queue, the handlers for one or several uppers or followers share step helpers: removal remove_inner_edges → on_inner_edges_disappeared and remove_follower_edges; addition follower_add_side / add_follower_edge and add_inner_edge → on_inner_edges_appeared (on_inner_edges_appeared_on_upper for the multi-follower handler). Their id lists use a small TaskIdList trait (TaskIdVec or Option<TaskId>), so single-edge handlers build no vectors.
Verifier: at stop, scan restored resident metadata for residual debt, split edges, and invalid placement. Retain edgeless endpoints' numbers; do not fabricate numbers for unrestored metadata. Root follower edges are invalid even when the other endpoint is unavailable.

Review follow-up

A separate-model correctness review identified four issues, addressed by the follow-up:

Ignore intentional panic-catching/panic tests on aborting WASM targets, using the repository convention; native panic assertions still run.
Replace the helper-only addition race test with three real-handler tests. A per-queue, test-only one-shot hook forces removal to finish between the addition's precheck and inner write, without holding either task lock. Cover both empty and nonempty data. Independently removing each production postcheck callsite now makes its corresponding test fail.
Fix the verifier's false negative for resident, edgeless endpoints. Reinstating the old omission makes the new regression fail. Include valid follower placement and unknown/unrestored endpoint cases.
Correct the GC explanation and add actual collectibility and persistence-operation exclusion regressions. Keep GC semantics unchanged.

Additional focused tests seed dirty/current-session-clean state and nonzero active counters: pure debt has no propagation effects; overpayment creates one appearance per edge; repeated positive references do not duplicate effects; split cancellation reverses dirty and clean contributions once; leaf promotion preserves data, active counts, and ancestor state while renesting dirty containers.

Review order: pure helpers → queue module docs and handlers → snapshot/verifier → deterministic tests → churn/fuzz driver → benchmarks.

Maintainer review round (`0f405c3`, `7b61001`)

0f405c3e: plain i32 arithmetic in net_debt; no follower_ids clone when there are no uppers; removal steps shared between single and multiple edges via TaskIdList; check_optimization_pending on the single upper; test hook type renamed to TestHook.
7b61001b: the three add handlers share step helpers like the removal path. Lock order, debt pre-check and postcheck, job variants, batching and optimize heuristics are unchanged.
Verified on the final state: full crate suite 56/56 groups in default, verify_aggregation_graph and aggregation_update_no_batch; churn 10/10 normal + 10/10 verify (serial); fuzz 10,449 runs / 181 s without crashes; clippy and rustdoc -D warnings. Removing either shared postcheck, or the debt pre-check in follower_add_side, fails the race/debt tests. CI green on 7b61001b.
Queue microbench vs c9d5d79f (3 interleaved rounds, median): ordered −3%, ordered removals −10%, overtake −6%, ordered additions +4–5% (~6 ns per single-edge addition, attributed to the add-path extraction; not the added check_optimization_pending and not recovered by inline hints).

Verification

Follow-up local results:

Full crate suites in default, verify_aggregation_graph, and aggregation_update_no_batch: 56/56 test groups pass in each mode, including 198 unit tests (one ignored benchmark).
aggregation_churn: 20/20 serial runs in normal mode and 20/20 with graph verification.
The known WASI failure on the original tip was deterministic (catch_unwind on panic-abort), not a flaky job. Target guards are fixed; post-push WASI CI passes on e82c325d (job 123330618672). The full build-and-test suite is green, including native units, miri, Rust checks, benches, and Windows tests.
Four mutation probes: each of the three actual add-handler postcheck callsites, plus the verifier's edgeless-endpoint omission, is independently detected by the new tests; all mutations restored.
Follow-up fuzz rerun: 6,929 runs in 181 seconds, no crashes, with graph verification enabled (no sanitizer).
Follow-up formatting, clippy (backend all targets, default and graph-verification modes; fuzz crate), and private rustdoc with -D warnings: pass.

Historical pre-follow-up evidence (not newly measured coverage/performance):

Full crate suites in default, verify_aggregation_graph, and aggregation_update_no_batch modes passed.
Churn 20/20 in normal and graph-verification modes; nine existing stress targets 90/90.
Fuzz: 58,782 runs in 16 minutes, plus a later 14,736-run session, no crashes.
Coverage: helper file 100% of lines; changed queue lines excluding ignored benchmark 96.7%. These figures include in-file test code and were not rerun for the review follow-up.

Benchmarks

Historical same-machine A/B against canary, three interleaved rounds, criterion medians:

Workload	Change
Queue microbench, removals overtake additions	−38%
Queue microbench, ordered add + remove (32 edges)	+3–4% (removals +14–20%, additions −2–8%)
Concurrent end-to-end churn / `scope_stress`	within noise
Ordered end-to-end churn	noisy, no stable direction
`stress/fibonacci/500`	approximately +3–7%

This is a tradeoff, not a blanket performance win. The single-threaded microbench does not measure the old multi-threaded spin-wait. Part of the Fibonacci cost comes from the extra upper lock required by the race argument; reducing that cost without weakening correctness is a separate follow-up. No new performance claim is made for the review follow-up.

Limitations / status

The verifier can only check placement using restored resident endpoint metadata; absent/evicted endpoint numbers remain unknown, except root placement can be checked without them.
Unmatched debt is silent. Debt is only valid if the matching addition is eventually generated and processed; producers must push additions into a queue their operation runs to completion. This is a requirement on producers, separate from the GC/snapshot exclusion rules (debt does not pin tasks). Production aggregation drains are synchronous (no async cancellation point) and now abort on panic rather than returning a partial graph to a per-task catcher. This does not protect jobs abandoned by a producer before the drain starts. If an addition were lost, a later addition of the same task pair would pay the stale debt instead of creating the edge, which can skew propagation. The old retry loop panicked in that case; now only the verify_aggregation_graph stop-time check reports residual debt (on restored resident tasks).
The stress test checks returned per-round counts. A stricter exact-collectibles check also fails on unmodified canary under CPU load and is tracked separately.
A second separate-model correctness review found no new blocking defect; its documentation note on the unmatched-debt assumption is applied above. Neither review is a formal proof or approval.

Removal benchmark and traversal follow-up

Two separate commits add permanent coverage and then the narrow traversal change:

private::aggregation_queue is an explicitly unstable native fixture shared by benches and normal-library profiling. The opt-in TURBOPACK_BENCH_AGGREGATION_QUEUE=1 Criterion group runs ten routine cases; extended exposes 756 filterable configurations: inner/follower/mixed placement, true single-edge/one-ID batch/full-batch jobs, references 1→0 or 2→1, data/no-data, aligned/reversed/seeded-permuted IDs, and total sizes 1/8/16/32/33/64/66. Setup/restoration/checks are untimed. Raw edge counts and aggregate contributions are checked after every phase. Aggregation-number optimizations are deliberately disabled: this is isolated queue work, not an end-to-end workload.
remove_follower_edges consumes its input with existing TaskIdList::retain swap-removal semantics. For aligned small linear maps, visiting the swapped replacement avoids repeated scans. Both result lists, edge multiplicity, positive-crossing effects, lock order, debt prechecks and postchecks remain intact. Emitted batch order intentionally changes; new real-handler tests cover Option/32/33 inputs, ancestor propagation, retained references, duplicate IDs and mixed positive/debt entries.

Normal-dependency profiling (not a backend test build) uses the same frozen fixture, system allocator, one runtime/backend worker, no backing storage, and no dev-only inline statistics. Release settings are explicit (opt3/thin LTO/cgu1 plus the applicable regex-syntax size override). Callgrind/DHAT cover 18 configurations; Criterion uses 30 samples, 1 s warmup, 3 s measurement and three interleaved parent/patch rounds. Payload/context/queue construction and removal drain are timed; restoration/assertions are not. Guest instructions and simulated events are not hardware cycles.

Synthetic removal case	Guest instructions	Median elapsed time
Aligned follower full batch, 32 edges	−9.79%	1922.013 → 1743.011 ns (−9.31%)
Follower full batch, 64 edges	+5.23%	3440.070 → 3670.363 ns (+6.69%, first three rounds)

The 64-edge case had non-overlapping slower 95% intervals in all three rounds. Its extra confirmation pair was 3454.881 → 3569.930 ns (+3.33%), still with non-overlapping intervals: the ≥5% magnitude did not repeat, but a smaller slowdown persists. This tradeoff is retained and disclosed, not described as regression-free. Full33 is +4.78% elapsed/+5.99% instructions; mixed one-ID32 is +4.25% elapsed. Mixed/permuted/retained cases can do more instruction work. Allocation totals were equal across the three revisions in all 18 sampled cases. Remaining normal-library helper/guard/accessor overhead relative to canary is not fully diagnosed; symbol self costs do not prove extra semantic locks. No blanket speedup claim.

Local verification before the canary rebase: all three full feature modes pass 56/56 groups (204 unit tests plus one ignored benchmark); 10/10 normal + 10/10 graph-verified serial churn; fuzz seed 2019458061 completed 14,271 runs in 181 s without failure; both shared addition-postcheck mutations fail the corresponding real-handler race tests and were restored. Formatting, all-target clippy, private rustdoc (warnings denied), fuzz manifest checks, 756-case extended smoke, ten routine cases, disabled gating and the original ignored benchmark all pass. Separate-model correctness and profiling-report reviews found no remaining blocker; these are not human approval. Post-push CI is green on 50e19c34, including native cargo units/benches, miri, WASI/wasm, Rust checks/rustdoc, and Windows tests; no reruns were needed.

Canary rebase (`cc9ae702`)

Rebased the seven existing commits onto canary cc65c414. The only manual integration resolves TurboTasksBackend::stop: keep canary's ActivityMonitor shutdown handling, then run both graph and aggregation-edge verification before dropping storage. All debt, race, GC, helper and traversal behavior is unchanged; the benchmark and traversal commits remain separate. Range-diff shows only this stop-hook adaptation.

Both rebased layers were checked against exact verified source trees. On this parent, full default, verify_aggregation_graph and aggregation_update_no_batch suites each pass 59 test groups, including 216 unit tests (one ignored benchmark). Formatting, all-target clippy, private rustdoc with warnings denied and fuzz manifest checks pass. The historical stress/fuzz/mutation and benchmark results above were not rerun on the new canary base, and their performance percentages are not new-base measurements. Post-push CI is green on cc9ae702, including cargo units/benches, miri, WASI/wasm, Rust checks/rustdoc and Windows tests; no reruns were needed.

GC/no-resurrection canary rebase (`c2d937c1`)

Rebased the same seven commits onto canary 4a0609e1 (#100370). The only manual resolution removes the obsolete resurrect_deleted import while keeping aggregation-edge helpers. Upstream's live-task assertions, deleted-candidate filtering, GC task-cache removal and optimization changes remain intact. The previous ActivityMonitor stop-verification integration is retained. No debt, race, balancing or traversal logic was weakened; final source exactly matches the verified integration tree.

Latest local verification: full default, verify_aggregation_graph and aggregation_update_no_batch suites pass 59 groups each, including 220 unit tests (one ignored benchmark), the updated GC reconnect/replacement tests and all four task_by_type tests. Formatting, all-target clippy, private rustdoc with warnings denied and fuzz manifest check pass. On the child an existing 16-bit random-tag re-execution oracle collided once despite correct 1→2 propagation and logged re-execution; targeted and final full runs passed, with a deterministic-oracle follow-up recorded. No test assertions were removed. Older performance, fuzz and stress numbers above are historical, not measurements on this base. Post-push CI is green on c2d937c1, including cargo units/benches, miri, WASI/wasm, Rust checks/rustdoc and Windows tests; no CI reruns were needed.

Reviewer follow-up (`614290c4`, `75538d53`)

Added a narrow fatal-panic boundary to production aggregation queue drains, including stats and GC entry points. The manager's outer abort wrapper does not by itself protect backend calls reached inside the per-task panic catcher. Native subprocess tests first reproduced containment of an injected panic after a debt write, then verified process abort and normal controls. User-code panic policy is unchanged; synchronous drains have no async cancellation point, and pre-drain job abandonment remains a producer responsibility.
Shortened GC docs; renamed follower-edge propagation and activeness state; used filter_map; clarified the outgoing-diff merge TODO without implementing merging.
Extracted optimization heuristics without changing raw debt-inclusive counts, single power-of-two versus batch ceiling-power triggers, caps or pending flags. Added boundary/debt-count regressions.
Kept singleton peeling and Option/Vec job shapes. The shared iterator helper has no-storage and ready-task paths that do not allocate restore batches; no blanket allocation or speedup claim is made. Existing real-handler tests cover root/root placement and root debt payment.

Local checks on both layers: three full feature modes pass 59 groups each (parent 223 units, child 229, one ignored benchmark); stats libraries, fmt, all-target clippy, private rustdoc with warnings denied, and fuzz manifest checks pass. Both layers pass opt-in benchmark smoke: 10 routine + 756 extended cases, with default omission verified. These are correctness smoke tests, not fresh performance measurements. The historical timings/stress/fuzz results above were not rerun for this update. Pushed-tip CI is green on 75538d53, including cargo units/benches, miri, WASI/wasm, Rust checks/rustdoc and Windows tests; this parent's CI needed no reruns. Review threads remain unresolved for the reviewer.

github-actions · 2026-10-29T05:19:47Z

Tests Passed

Commit: 75538d5
View workflow run

vercel-fleet-prod · 2026-10-29T13:06:07Z

Reproduction results

We checked this PR against the stress reproduction that intermittently panicked (inner_of_upper(s)_lost_follower(s) is not able to remove follower …) or ran away in memory on canary.

Setup

Bench: turbo_tasks_backend_priority_contention/realistic_warm/128x512 from turbo-tasks: use per-priority lock-free queues in PriorityRunner #100561 (added locally as a bench overlay, not part of this PR), Criterion defaults, 32 CPUs, uninstrumented release builds.
Revisions: this PR at 23eb92d7 vs its canary base 7fd2d0a6, with identical bench files.
15 runs on one boot, interleaved pr, pr, base × 5. Every start counted; none were replaced.
Per-run limits: killed above 20 GB RSS (runaway) or at 1800 s (timeout). No run came near the time limit.

	Runs	ok	panic	runaway	timeout / error	Peak RSS (ok runs)	Wall time
This PR (`23eb92d7`)	10	10	0	0	0	0.26–0.33 GB	113–130 s
Canary base (`7fd2d0a6`)	5	4	0	1 (killed at 20.1 GB after 136 s)	0	0.44–1.80 GB	158–203 s

Caveats

These are finite samples, not a rate estimate. The base control reproduced the failure once (a runaway, not a panic) in 5 runs. This PR had no failure in 10 runs.
Criterion medians drifted upward over the session on both binaries (about 42 → 66 ms per iteration), so they say nothing about speed.

### What? When an aggregation update queue holds at least 4096 `jobs`, it now runs all of them in parallel instead of one by one. - Each chunk runs on a child execution context and pushes the work its jobs generate into a private buffer queue. Buffers are never processed. - After all chunks finish, every buffer is merged back into the queue: - aggregation number updates go through the same done-map-aware merging as regular pushes; - balance and find-and-schedule work stays deduplicated; - scheduled tasks keep their highest priority; - optimizations respect the queue's cap and budget. Refused optimizations set `optimization_pending`, as before. - GC queues, fan-out buffers, and contexts with a GC collectible callback never fan out. Stacked on #100873. ### Why? One queue runs on one thread, but a single update can cascade into a very large number of jobs, e.g. when a task with many followers becomes an aggregating node. This spreads such a cascade over the available threads. This relies on #100873. There, an edge removal that is processed before its addition records follower debt instead of retrying. So no job waits for other work: every chunk, including edge removals, can complete on its own, as `scope_bounded` requires. The generated work, including additions that pay debt, is merged back into the queue and runs to completion with it. This is a parallelization change. It is not a fix for the aggregation edge removal panic or memory runaways; that is #100873. ### How? - `JOB_FAN_OUT_THRESHOLD = 4096`, checked in `AggregationUpdateQueue::process`. - `fan_out` runs the jobs via `turbo_tasks::parallel::map_collect_chunked_owned`. - `ChildExecuteContext` is now `Copy`. Its priority is still resolved on the calling thread. - New: `ExecuteContext::has_gc_collectible`. - Aggregation number merging moved into `push_aggregation_number_update`, so `push` and the buffer merge share it. - Tests cover: - the threshold and the exclusions; - every buffer merge rule; - activation with a test-only threshold: a graph with interleaved edge additions and removals (some removals overtake their additions), then an invalidation. They check that fan-out ran and compare against sequential processing, with 1 and 4 runtime workers. ### Verification On top of #100873 `23eb92d7`: - `cargo test -p turbo-tasks-backend` passes in default, `verify_aggregation_graph` and `aggregation_update_no_batch` modes (382 tests each). - clippy `--all-targets -D warnings` (default and `verify_aggregation_graph`), `cargo fmt --check`, and a `cargo check` with `trace_aggregation_update_stats,trace_aggregation_update_queue` pass. - Locally forced threshold of 4 (not committed): the full suite passes in default and `verify_aggregation_graph` modes, with about 28.7k / 28.9k fan-outs, including about 27k in `aggregation_churn`. - `turbo_tasks_backend_priority_contention/realistic_warm/128x512` (the realistic contention bench from #100561, added locally, not part of this PR; Criterion defaults; 32 CPUs), 6 interleaved runs on one boot, #100873 vs this PR: | | Runs | ok | Median ms per run | |---|---:|---:|---| | #100873 | 3 | 3 | 40.69, 58.34, 58.13 | | This PR | 3 | 3 | 55.25, 58.09, 57.40 | No clear difference; this bench shows no measured speed-up.  Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>

… batches ### What? An aggregation update queue now runs large batches in parallel instead of one by one: - **Jobs:** when it holds at least 4096 `jobs`, all of them run in parallel. - **Edge removals:** when at least 4096 edge removals are due, all of them run in parallel. Removals are due when no job, aggregation number update or balancing is pending; this is the same rule #100892 uses to process removals one by one. How the parallel batch runs: - Each chunk runs on a child execution context and pushes the work its jobs generate into a private buffer queue. Buffers are never processed. - After all chunks finish, every buffer is merged back into the queue: - generated jobs are appended to `jobs`; - generated removals are appended to the deferred removals, FIFO; - aggregation number updates go through the same done-map-aware merging as regular pushes; - balance and find-and-schedule work stays deduplicated; - scheduled tasks keep their highest priority; - optimizations respect the queue's cap and budget. Refused optimizations set `optimization_pending`, as before. - GC queues, fan-out buffers and contexts with a GC collectible callback never fan out. Stacked on #100889 (on #100892 and #100873). ### Why? One queue runs on one thread, but a single update can cascade into a very large number of jobs, e.g. when a task with many followers becomes an aggregating node, or loses many followers. This spreads such a cascade over the available threads. Every job of a batch completes on its own, which is what `scope_bounded` requires. A removal that runs before its addition records follower debt (#100873) instead of waiting. Additions that pay that debt come back through the merge and run to completion with the queue. This changes one ordering rule from #100892, and only for removal batches at or above the threshold. Processed one by one, the work a removal generates runs before the next removal. In a parallel batch, it runs after the whole batch, with the normal priorities. A batch still only starts when no job, aggregation number update or balancing is pending, so removals still run after the additions queued before them. Below the threshold, nothing changes. This is a parallelization change. It is not a fix for the aggregation edge removal panic or memory runaways; that is #100873. ### How? - `JOB_FAN_OUT_THRESHOLD = 4096`. `AggregationUpdateQueue::fan_out_kind` picks `FanOutKind::Jobs`, `FanOutKind::Removals` or none. - `fan_out` takes all queued jobs of that kind and runs them via `turbo_tasks::parallel::map_collect_chunked_owned`. Jobs and removals share one implementation. - `ChildExecuteContext` is now `Copy`. Its priority is still resolved on the calling thread. - New: `ExecuteContext::has_gc_collectible`. - Aggregation number merging moved into `push_aggregation_number_update`, so `push` and the buffer merge share it. - The test hook type is now `Send + Sync`. Buffers never inherit hooks. - Tests cover: - the threshold and the exclusions, for both batch kinds; - that no removal runs, and no removal batch starts, while other work is pending; - every buffer merge rule, including removals; - work generated by a removal batch, with the final edges and collectible counts; - activation with a test-only threshold: additions, deferred removals and then an invalidation. They check that both batch kinds fanned out, and compare the graph and data against sequential processing, with 1 and 4 runtime workers. ### Verification On top of #100889 `e44ad7f0`: - `cargo test -p turbo-tasks-backend` passes in default, `verify_aggregation_graph` and `aggregation_update_no_batch` modes (400 tests each). This includes the deferred removal, follower debt and held dirty state tests from the PRs below, unchanged. - clippy `--all-targets -D warnings` (default and `verify_aggregation_graph`), `cargo fmt --check`, and a `cargo check` with `trace_aggregation_update_stats,trace_aggregation_update_queue` pass. - Locally forced threshold of 4 (not committed): the full suite passes in default and `verify_aggregation_graph` modes with no tests excluded, with about 24k jobs fan-outs and 10k removal fan-outs per mode. - `turbo_tasks_backend_priority_contention/realistic_warm/128x512` (the realistic contention bench from #100561, added locally, not part of this PR; Criterion defaults; 32 CPUs), 6 interleaved runs on one boot. All runs completed, with peak RSS around 0.26 GB: | | Runs | ok | Median ms per run | |---|---:|---:|---| | #100889 | 3 | 3 | 67.05, 67.05, 69.79 | | This PR | 3 | 3 | 67.81, 67.49, 66.97 | No clear difference; this bench shows no measured speed-up.  Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>

vercel

Vercel Agent reviewed cc9ae70 and found no issues.

… batches ### What? An aggregation update queue now runs large batches in parallel instead of one by one: - **Jobs:** when it holds at least 4096 `jobs`, all of them run in parallel. - **Edge removals:** when at least 4096 edge removals are due, all of them run in parallel. Removals are due when no job, aggregation number update or balancing is pending; this is the same rule #100892 uses to process removals one by one. How the parallel batch runs: - Each chunk runs on a child execution context and pushes the work its jobs generate into a private buffer queue. Buffers are never processed. - After all chunks finish, every buffer is merged back into the queue: - generated jobs are appended to `jobs`; - generated removals are appended to the deferred removals, FIFO; - aggregation number updates go through the same done-map-aware merging as regular pushes; - balance and find-and-schedule work stays deduplicated; - scheduled tasks keep their highest priority; - optimizations respect the queue's cap and budget. Refused optimizations set `optimization_pending`, as before. - GC queues, fan-out buffers and contexts with a GC collectible callback never fan out. Stacked on #100889 (on #100892 and #100873). ### Why? One queue runs on one thread, but a single update can cascade into a very large number of jobs, e.g. when a task with many followers becomes an aggregating node, or loses many followers. This spreads such a cascade over the available threads. Every job of a batch completes on its own, which is what `scope_bounded` requires. A removal that runs before its addition records follower debt (#100873) instead of waiting. Additions that pay that debt come back through the merge and run to completion with the queue. This changes one ordering rule from #100892, and only for removal batches at or above the threshold. Processed one by one, the work a removal generates runs before the next removal. In a parallel batch, it runs after the whole batch, with the normal priorities. A batch still only starts when no job, aggregation number update or balancing is pending, so removals still run after the additions queued before them. Below the threshold, nothing changes. This is a parallelization change. It is not a fix for the aggregation edge removal panic or memory runaways; that is #100873. ### How? - `JOB_FAN_OUT_THRESHOLD = 4096`. `AggregationUpdateQueue::fan_out_kind` picks `FanOutKind::Jobs`, `FanOutKind::Removals` or none. - `fan_out` takes all queued jobs of that kind and runs them via `turbo_tasks::parallel::map_collect_chunked_owned`. Jobs and removals share one implementation. - `ChildExecuteContext` is now `Copy`. Its priority is still resolved on the calling thread. - New: `ExecuteContext::has_gc_collectible`. - Aggregation number merging moved into `push_aggregation_number_update`, so `push` and the buffer merge share it. - The test hook type is now `Send + Sync`. Buffers never inherit hooks. - Tests cover: - the threshold and the exclusions, for both batch kinds; - that no removal runs, and no removal batch starts, while other work is pending; - every buffer merge rule, including removals; - work generated by a removal batch, with the final edges and collectible counts; - activation with a test-only threshold: additions, deferred removals and then an invalidation. They check that both batch kinds fanned out, and compare the graph and data against sequential processing, with 1 and 4 runtime workers. ### Verification On top of #100889 `e44ad7f0`: - `cargo test -p turbo-tasks-backend` passes in default, `verify_aggregation_graph` and `aggregation_update_no_batch` modes (400 tests each). This includes the deferred removal, follower debt and held dirty state tests from the PRs below, unchanged. - clippy `--all-targets -D warnings` (default and `verify_aggregation_graph`), `cargo fmt --check`, and a `cargo check` with `trace_aggregation_update_stats,trace_aggregation_update_queue` pass. - Locally forced threshold of 4 (not committed): the full suite passes in default and `verify_aggregation_graph` modes with no tests excluded, with about 24k jobs fan-outs and 10k removal fan-outs per mode. - `turbo_tasks_backend_priority_contention/realistic_warm/128x512` (the realistic contention bench from #100561, added locally, not part of this PR; Criterion defaults; 32 CPUs), 6 interleaved runs on one boot. All runs completed, with peak RSS around 0.26 GB: | | Runs | ok | Median ms per run | |---|---:|---:|---| | #100889 | 3 | 3 | 67.05, 67.05, 69.79 | | This PR | 3 | 3 | 67.81, 67.49, 66.97 | No clear difference; this bench shows no measured speed-up.  Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>

Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>

Use plain i32 arithmetic in net_debt, avoid cloning follower ids when no uppers need them, share the removal steps between single and multiple edges via a TaskIdList trait (TaskIdVec or Option<TaskId>), check pending optimizations on the single upper, and rename the test hook type. Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>

Extract the addition steps of the three add handlers into helpers, like the removal steps: follower_add_side, add_follower_edge, add_inner_edge, on_inner_edges_appeared and on_inner_edges_appeared_on_upper. Handlers keep their lock order, debt checks, job variants and batching. Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>

Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>

vercel

Vercel Agent reviewed c2d937c and found no issues.

… batches ### What? An aggregation update queue now runs large batches in parallel instead of one by one: - **Jobs:** when it holds at least 4096 `jobs`, all of them run in parallel. - **Edge removals:** when at least 4096 edge removals are due, all of them run in parallel. Removals are due when no job, aggregation number update or balancing is pending; this is the same rule #100892 uses to process removals one by one. How the parallel batch runs: - Each chunk runs on a child execution context and pushes the work its jobs generate into a private buffer queue. Buffers are never processed. - After all chunks finish, every buffer is merged back into the queue: - generated jobs are appended to `jobs`; - generated removals are appended to the deferred removals, FIFO; - aggregation number updates go through the same done-map-aware merging as regular pushes; - balance and find-and-schedule work stays deduplicated; - scheduled tasks keep their highest priority; - optimizations respect the queue's cap and budget. Refused optimizations set `optimization_pending`, as before. - GC queues, fan-out buffers and contexts with a GC collectible callback never fan out. Stacked on #100889 (on #100892 and #100873). ### Why? One queue runs on one thread, but a single update can cascade into a very large number of jobs, e.g. when a task with many followers becomes an aggregating node, or loses many followers. This spreads such a cascade over the available threads. Every job of a batch completes on its own, which is what `scope_bounded` requires. A removal that runs before its addition records follower debt (#100873) instead of waiting. Additions that pay that debt come back through the merge and run to completion with the queue. This changes one ordering rule from #100892, and only for removal batches at or above the threshold. Processed one by one, the work a removal generates runs before the next removal. In a parallel batch, it runs after the whole batch, with the normal priorities. A batch still only starts when no job, aggregation number update or balancing is pending, so removals still run after the additions queued before them. Below the threshold, nothing changes. This is a parallelization change. It is not a fix for the aggregation edge removal panic or memory runaways; that is #100873. ### How? - `JOB_FAN_OUT_THRESHOLD = 4096`. `AggregationUpdateQueue::fan_out_kind` picks `FanOutKind::Jobs`, `FanOutKind::Removals` or none. - `fan_out` takes all queued jobs of that kind and runs them via `turbo_tasks::parallel::map_collect_chunked_owned`. Jobs and removals share one implementation. - `ChildExecuteContext` is now `Copy`. Its priority is still resolved on the calling thread. - New: `ExecuteContext::has_gc_collectible`. - Aggregation number merging moved into `push_aggregation_number_update`, so `push` and the buffer merge share it. - The test hook type is now `Send + Sync`. Buffers never inherit hooks. - Tests cover: - the threshold and the exclusions, for both batch kinds; - that no removal runs, and no removal batch starts, while other work is pending; - every buffer merge rule, including removals; - work generated by a removal batch, with the final edges and collectible counts; - activation with a test-only threshold: additions, deferred removals and then an invalidation. They check that both batch kinds fanned out, and compare the graph and data against sequential processing, with 1 and 4 runtime workers. ### Verification On top of #100889 `e44ad7f0`: - `cargo test -p turbo-tasks-backend` passes in default, `verify_aggregation_graph` and `aggregation_update_no_batch` modes (400 tests each). This includes the deferred removal, follower debt and held dirty state tests from the PRs below, unchanged. - clippy `--all-targets -D warnings` (default and `verify_aggregation_graph`), `cargo fmt --check`, and a `cargo check` with `trace_aggregation_update_stats,trace_aggregation_update_queue` pass. - Locally forced threshold of 4 (not committed): the full suite passes in default and `verify_aggregation_graph` modes with no tests excluded, with about 24k jobs fan-outs and 10k removal fan-outs per mode. - `turbo_tasks_backend_priority_contention/realistic_warm/128x512` (the realistic contention bench from #100561, added locally, not part of this PR; Criterion defaults; 32 CPUs), 6 interleaved runs on one boot. All runs completed, with peak RSS around 0.26 GB: | | Runs | ok | Median ms per run | |---|---:|---:|---| | #100889 | 3 | 3 | 67.05, 67.05, 69.79 | | This PR | 3 | 3 | 67.81, 67.49, 66.97 | No clear difference; this bench shows no measured speed-up.  Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>

Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>

…tics Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>

vercel

Vercel Agent reviewed 75538d5 and found no issues.

…s and balancing (#100892) Stacked on #100873 (negative follower debt). Only the commits on top of #100873 belong to this PR. ### What? Within one `AggregationUpdateQueue`, aggregation edge removals (`InnerOf{Upper,Uppers}LostFollower(s)`) now wait in a separate removal queue until all ordinary jobs, aggregation number updates and balancing in that queue are done. ### Why? Previously additions and removals were interleaved in one FIFO. When an edge (or a task reachable through another edge) was removed and re-added in the same queue, its aggregated data went `1 → 0 → 1`: data was removed and re-added, collectibles dependents were invalidated, dirty containers flipped and readers could briefly miss collectibles. With removals last it goes `1 → 2 → 1`, so none of that temporary work happens. ### How? - `push` routes the three lost-follower jobs into `removal_jobs`; everything else (data updates, active/parent counts, invalidations, additions) keeps its queue and order. - `process` priority: `jobs` → aggregation number updates → balancing → **removals** → optimize → find dirty → schedule. A small `pop_job` helper picks a removal only when the higher-priority queues are empty, so work generated by a removal runs before the next removal, and generated removals append FIFO. The existing job dispatch is unchanged. - GC: `execute_until_rebalance` must drain removals while balancing stays deferred. It moves balance jobs out of the queue after every step and returns both previously pending and newly generated ones; the caller's deleted-endpoint filtering is unchanged. - Safety: a removal processed later is the ordinary case. A removal that still overtakes its addition (another queue, or an addition generated later) records follower debt as in #100873. The PR relies on the existing debt contract and doesn't weaken it. No new locking, no schema change. ### Scope / limitations - Queue-local only: helps when the keeping addition/balance is already pending in the same queue. No atomicity across separate queues or concurrent operations. - Task completion still removes old children (`cleanup_old_edges`) and connects new children (`connect_children`) in separate queues, so a task switching between children that share a descendant still churns. Combining those is planned as a separate, later stacked PR after measuring this one. ### Tests New `deferred_removal_tests` (real backend, step-by-step queue processing): - same-edge replacement and a moved shared descendant: the upper's collectible never drops to zero at any step, and a collectibles reader is not invalidated; - genuine loss still removes data and invalidates the reader; - removals only run when no ordinary job, aggregation number update or balance is pending; - work generated by a removal runs before the next removal, generated removals append FIFO; - `execute_until_rebalance` drains removals but executes no balancing and returns pending and newly generated balance edges. An existing debt test that mixed remove/remove/add in one queue now uses separate queues so it still forces a real overtake. Mutations: routing removals back into the ordinary FIFO fails 4 of the new tests; not collecting newly generated balances in the GC path fails the GC test. ### Verification - `cargo test -p turbo-tasks-backend` in default, `verify_aggregation_graph`, and `aggregation_update_no_batch`: 56/56 test groups each (including the existing stress tests). - `aggregation_churn` stress: 20/20 serial runs normal and 20/20 with graph verification. - `fuzz_graph`: 8,543 runs in 181 s, no crashes. - clippy (`--all-targets`, default and `verify_aggregation_graph,trace_aggregation_update_stats`), fuzz crate clippy, fmt, private rustdoc with `-D warnings`. ### Work counts and performance Exact counts (`replace_work`: 32 existing edges removed and re-added in one queue, a reader on the upper; feature `trace_aggregation_update_stats`), vs #100873: | | aggregated data update jobs | collectibles invalidation jobs | |---|---:|---:| | #100873 | 2 | 2 | | this PR | 0 | 0 | Queue microbench (release unit-test binary, 3 interleaved same-machine rounds, Criterion central estimates): | case | #100873 | this PR | |---|---|---| | replace (remove + re-add in one queue) | 12.7–12.9 µs | 8.5–10.2 µs (−20…−35%) | | overtake (removals in a separate earlier queue) | 10.2–10.4 µs | 11.0–12.3 µs (+6…+21%, noisy) | | ordered / ordered additions / ordered removals | — | within noise, removals possibly slightly slower | The microbench binary includes inert test-only hooks on this branch, so it does not isolate production overhead. End-to-end benches (`aggregation_churn`, `stress/fibonacci/500`) drifted too much across rounds to draw a conclusion either way. This is not claimed as a general speedup. ### Rebase onto removal benchmark/traversal follow-up Rebased the existing two logical commits onto #100873 at `50e19c34`, retaining the deferred-removal change and separate balance-queue drain refinement. The parent's permanent removal fixture/benchmarks and swap-order traversal are inherited, not additional child scope. Only the new parent's test observer needed adapting to inspect `removal_jobs` rather than the ordinary queue; production priority, GC balance collection and drain behavior are unchanged by the rebase. Local verification before the canary rebase: full default, `verify_aggregation_graph` and `aggregation_update_no_batch` suites each pass **56/56 groups** (210 unit tests plus one ignored benchmark). This includes the parent's 756-configuration fixture test and traversal/multiplicity/ancestor regressions, the actual-handler race tests, and this child's priority, genuine-loss, no-invalidation and GC-drain tests. Formatting, all-target clippy, private rustdoc with warnings denied and fuzz manifest check pass. The older stress/fuzz/performance numbers above are historical, not newly measured on this rebased child. Parent normal-library profiling explicitly reports a 32-edge benefit and a larger-map tradeoff; it is not evidence of an end-to-end child speedup. Post-push CI is green on `c26cd62d`, including native cargo units/benches, miri, WASI/wasm, Rust checks/rustdoc, and Windows tests; no reruns were needed. ### Canary restack (`e9607690`) Replayed the existing two child commits onto #100873 at `cc9ae702` (canary base `cc65c414`). Both child patches are unchanged by range-diff. Queue priority, GC balance collection and separate drain refinement are preserved. The final source tree exactly matches the locally verified integration tree. On this child, full default, `verify_aggregation_graph` and `aggregation_update_no_batch` suites each pass **59 test groups**, including **222 unit tests** (one ignored benchmark). Formatting, all-target clippy, private rustdoc with warnings denied and fuzz manifest checks pass. Prior stress/fuzz/performance results above remain historical; no new-base performance or end-to-end speedup is claimed. Post-push CI is green on `e9607690`, including cargo units/benches, miri, WASI/wasm, Rust checks/rustdoc and Windows tests; no reruns were needed. Base relationship and ready status are unchanged; higher layers were not modified. ### GC/no-resurrection canary restack (`d91b5264`) Restacked the same two child commits onto #100873 at `c2d937c1` (canary base `4a0609e1`, #100370). Both child patches remain unchanged by range-diff: removal priority, deferred GC balance collection and the separate drain refinement are preserved. Upstream no-resurrection behavior and deleted-endpoint filtering are retained. The final source exactly matches the verified integration tree. Latest local verification: full default, `verify_aggregation_graph` and `aggregation_update_no_batch` suites pass **59 groups each**, including **226 unit tests** (one ignored benchmark), updated GC reconnect/replacement and `task_by_type` tests, plus this child's GC/priority regressions. Formatting, all-target clippy, private rustdoc with warnings denied and fuzz manifest check pass. Initial default `recompute_dependency` hit an existing 16-bit random-tag collision: both functions logged re-execution and value correctly changed 1→2, but tags were equal. A targeted rerun and final full runs pass; a deterministic-oracle follow-up is recorded, without changing test or production behavior here. Older stress/fuzz/performance figures remain historical; no new-base speedup is claimed. Post-push CI is green on `d91b5264`, including cargo units/benches, miri, WASI/wasm, Rust checks/rustdoc and Windows tests; no CI reruns were needed. Base, ready status and higher stack layers are unchanged by this update. ### Restack onto reviewer follow-up (`fd4345e9`) Replayed only the existing two logical child commits onto #100873 at `75538d53`. The GC drain keeps its removal-job predicate, deferred balance collection and per-step `drain(..)` inside the parent's new fatal-panic boundary. Final source matches the verified child integration exactly. Queue priority, debt pre/postchecks, deleted-endpoint filtering, PR base/readiness and higher stack branches are unchanged. Local verification: default, `verify_aggregation_graph` and `aggregation_update_no_batch` full suites each pass **59 groups**, including **229 unit tests** (one ignored benchmark); stats library passes (two ignored benchmarks), including ordinary/stats/GC fatal-panic subprocess tests. Formatting, all-target clippy, private rustdoc with warnings denied and fuzz manifest checks pass. Opt-in queue benchmark smoke passes **10 routine + 756 extended cases**, and default omission is checked. Older timing/stress/fuzz figures above remain historical; no new speedup is claimed. Pushed-tip CI is green on `fd4345e9`, including cargo units/benches, miri, WASI/wasm, Rust checks/rustdoc and Windows tests. The first cargo-unit run had one `turbopack-go::live::versioned_dependencies_use_the_module_cache_without_downloads` failure with `Permission denied (os error 13)`; one rerun passed without source changes. That test's error-context/permission investigation is tracked separately.   --------- Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>

… batches ### What? An aggregation update queue now runs large batches in parallel instead of one by one: - **Jobs:** when it holds at least 4096 `jobs`, all of them run in parallel. - **Edge removals:** when at least 4096 edge removals are due, all of them run in parallel. Removals are due when no job, aggregation number update or balancing is pending; this is the same rule #100892 uses to process removals one by one. How the parallel batch runs: - Each chunk runs on a child execution context and pushes the work its jobs generate into a private buffer queue. Buffers are never processed. - After all chunks finish, every buffer is merged back into the queue: - generated jobs are appended to `jobs`; - generated removals are appended to the deferred removals, FIFO; - aggregation number updates go through the same done-map-aware merging as regular pushes; - balance and find-and-schedule work stays deduplicated; - scheduled tasks keep their highest priority; - optimizations respect the queue's cap and budget. Refused optimizations set `optimization_pending`, as before. - GC queues, fan-out buffers and contexts with a GC collectible callback never fan out. Stacked on #100889 (on #100892 and #100873). ### Why? One queue runs on one thread, but a single update can cascade into a very large number of jobs, e.g. when a task with many followers becomes an aggregating node, or loses many followers. This spreads such a cascade over the available threads. Every job of a batch completes on its own, which is what `scope_bounded` requires. A removal that runs before its addition records follower debt (#100873) instead of waiting. Additions that pay that debt come back through the merge and run to completion with the queue. This changes one ordering rule from #100892, and only for removal batches at or above the threshold. Processed one by one, the work a removal generates runs before the next removal. In a parallel batch, it runs after the whole batch, with the normal priorities. A batch still only starts when no job, aggregation number update or balancing is pending, so removals still run after the additions queued before them. Below the threshold, nothing changes. This is a parallelization change. It is not a fix for the aggregation edge removal panic or memory runaways; that is #100873. ### How? - `JOB_FAN_OUT_THRESHOLD = 4096`. `AggregationUpdateQueue::fan_out_kind` picks `FanOutKind::Jobs`, `FanOutKind::Removals` or none. - `fan_out` takes all queued jobs of that kind and runs them via `turbo_tasks::parallel::map_collect_chunked_owned`. Jobs and removals share one implementation. - `ChildExecuteContext` is now `Copy`. Its priority is still resolved on the calling thread. - New: `ExecuteContext::has_gc_collectible`. - Aggregation number merging moved into `push_aggregation_number_update`, so `push` and the buffer merge share it. - The test hook type is now `Send + Sync`. Buffers never inherit hooks. - Tests cover: - the threshold and the exclusions, for both batch kinds; - that no removal runs, and no removal batch starts, while other work is pending; - every buffer merge rule, including removals; - work generated by a removal batch, with the final edges and collectible counts; - activation with a test-only threshold: additions, deferred removals and then an invalidation. They check that both batch kinds fanned out, and compare the graph and data against sequential processing, with 1 and 4 runtime workers. ### Verification On top of #100889 `e44ad7f0`: - `cargo test -p turbo-tasks-backend` passes in default, `verify_aggregation_graph` and `aggregation_update_no_batch` modes (400 tests each). This includes the deferred removal, follower debt and held dirty state tests from the PRs below, unchanged. - clippy `--all-targets -D warnings` (default and `verify_aggregation_graph`), `cargo fmt --check`, and a `cargo check` with `trace_aggregation_update_stats,trace_aggregation_update_queue` pass. - Locally forced threshold of 4 (not committed): the full suite passes in default and `verify_aggregation_graph` modes with no tests excluded, with about 24k jobs fan-outs and 10k removal fan-outs per mode. - `turbo_tasks_backend_priority_contention/realistic_warm/128x512` (the realistic contention bench from #100561, added locally, not part of this PR; Criterion defaults; 32 CPUs), 6 interleaved runs on one boot. All runs completed, with peak RSS around 0.26 GB: | | Runs | ok | Median ms per run | |---|---:|---:|---| | #100889 | 3 | 3 | 67.05, 67.05, 69.79 | | This PR | 3 | 3 | 67.81, 67.49, 66.97 | No clear difference; this bench shows no measured speed-up.  Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>

github-actions Bot added created-by: Turbopack team PRs by the Turbopack team. Turbopack Related to Turbopack with Next.js. labels Oct 29, 2026

sokra mentioned this pull request Oct 29, 2026

turbo-tasks-backend: keep the dirty state of completing tasks in uppers that drop them #100889

Draft

sokra added this pull request to stack #100890 October 29, 2026 14:36

This was referenced Oct 29, 2026

turbo-tasks-backend: fan out large aggregation update job and removal batches #100803

Draft

turbo-tasks-backend: process aggregation edge removals after additions and balancing #100892

Merged

sokra removed this pull request from stack #100890 October 29, 2026 18:30

sokra added this pull request to stack #100904 October 29, 2026 18:30

sokra force-pushed the fleet/aggregation-negative-edges branch 2 times, most recently from 9dff90e to c9d5d79 Compare October 30, 2026 10:29