Skip to content

sync: fix semaphore use-after-free and permit leak on tracing panic - #8542

Merged
Darksonn merged 3 commits into
tokio-rs:tokio-1.51.xfrom
Darksonn:f03
Oct 3, 2026
Merged

Darksonn merged 3 commits into
tokio-rs:tokio-1.51.xfrom
Darksonn:f03

Conversation

@Darksonn

@Darksonn Darksonn commented Oct 2, 2026

Copy link
Copy Markdown
Member

add_permits_locked emits a tracing event once a waiter has been
assigned all of its permits, but before removing it from the wait
queue. If the tracing subscriber panics during this event, the waiter
stays in the queue with zero permits remaining. Polling the Acquire
future again then completes it and clears queued, so dropping it
frees the waiter without unlinking it, and the next release
dereferences the freed waiter.

Similarly, poll_acquire emits tracing events after taking permits from
the semaphore, but before the Acquire future records that it holds
them, so a subscriber panic there leaks the permits. This affects both
the uncontended path, where all permits are taken at once, and the
contended path, where the available permits are taken before the waiter
is queued.

This PR contains one commit per issue:

  • Remove the waiter from the wait queue before emitting the tracing
    event.
  • On the uncontended path, record the permits in the waiter's state and
    set queued before emitting the tracing events, and only clear
    queued in Acquire::poll after the last tracing event. With this,
    queued consistently means that Acquire::drop must clean up after
    the waiter, and num_permits - state is always the number of permits
    it holds. (This is also the state that sync: preserve FIFO lock order when coop budget is exhaustedΒ #8371 needs when the coop
    budget runs out after the permits were taken.)
  • On the contended path, assign the permits to the waiter before
    emitting the tracing event.

All of these require tokio_unstable and the tracing feature. The
regression tests use a subscriber that panics on the first event with a
given target; without the first fix, Miri reports the use-after-free.

Not covered here: a panic in the tracing calls that run after
Acquire::poll has returned (span exits and the poll_op event in
InstrumentedAsyncOp) can still leak permits, and a panic in
add_permits_locked after a waiter was removed from the queue still
loses its wakeup and the remaining permits. Neither is a memory safety
issue, and handling them would require restructuring how these futures
are instrumented.

Fixes: #8541

@Darksonn Darksonn added C-enhancement Category: A PR with an enhancement or bugfix. A-tokio Area: The main tokio crate M-sync Module: tokio/sync M-tracing Tracing support in Tokio labels Oct 2, 2026
@github-actions github-actions Bot added the R-loom-sync Run loom sync tests on this PR label Oct 2, 2026
@Darksonn

Darksonn commented Oct 2, 2026

Copy link
Copy Markdown
Member Author

I'm really not sure about this one ... feel free to give suggestions or even come up with alternate fixes.

@Darksonn Darksonn added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Oct 2, 2026

@ADD-SP ADD-SP left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix and test looks good to me. Actually, I think this is a minimal workable fix while it is not the clean fix.

BTW, it looks like our Miri tests don't cover the --cfg tokio_unstable, we can add it after merge. I ran the tests locally using Miri with tokio_unstable and it passed.

@Darksonn Darksonn removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Oct 2, 2026
@Darksonn
Darksonn changed the base branch from master to tokio-1.51.x October 3, 2026 09:31
…#8542)

Currently, `add_permits_locked` emits a tracing event once a waiter
has been assigned all of the permits it needs, but before the waiter
is removed from the wait queue. If the tracing subscriber panics
during this event, the waiter stays in the queue with zero permits
remaining. Polling the `Acquire` future then completes it and sets
`queued = false`, so dropping the future frees the waiter without
unlinking it from the queue, and the next release dereferences the
freed waiter.

Fix this by removing the waiter from the wait queue before emitting
the tracing event.
Currently, when `poll_acquire` takes all of the requested permits from
the semaphore without queueing the waiter, it emits tracing events
before returning. At this point, neither the waiter's state nor the
`queued` flag record that the permits were taken, so if the tracing
subscriber panics during one of these events, the `Acquire` future is
dropped without returning the permits to the semaphore.

Fix this by recording the permits in the waiter's state and setting
`queued` before emitting the tracing events. `Acquire::poll` then only
clears `queued` once the future completes, after the last tracing
event has been emitted.

With this, `queued` consistently means that `Acquire::drop` must clean
up after the waiter, and `num_permits - state` is always the number of
permits the waiter holds, regardless of whether they were taken
directly from the semaphore or assigned in the wait queue.
…s#8542)

Currently, when `poll_acquire` takes only some of the requested permits
from the semaphore, it emits a tracing event before assigning them to
the waiter. If the tracing subscriber panics during this event, the
`Acquire` future is dropped while unwinding, and `Acquire::drop` only
returns the permits that were assigned to the waiter. The permits that
were taken from the semaphore are therefore leaked.

Fix this by assigning the permits to the waiter before emitting the
tracing event.
@Darksonn
Darksonn merged commit 625c851 into tokio-rs:tokio-1.51.x Oct 3, 2026
83 of 86 checks passed
Darksonn added a commit that referenced this pull request Oct 3, 2026
Currently, `add_permits_locked` emits a tracing event once a waiter
has been assigned all of the permits it needs, but before the waiter
is removed from the wait queue. If the tracing subscriber panics
during this event, the waiter stays in the queue with zero permits
remaining. Polling the `Acquire` future then completes it and sets
`queued = false`, so dropping the future frees the waiter without
unlinking it from the queue, and the next release dereferences the
freed waiter.

Fix this by removing the waiter from the wait queue before emitting
the tracing event.
Darksonn added a commit that referenced this pull request Oct 3, 2026
Currently, when `poll_acquire` takes all of the requested permits from
the semaphore without queueing the waiter, it emits tracing events
before returning. At this point, neither the waiter's state nor the
`queued` flag record that the permits were taken, so if the tracing
subscriber panics during one of these events, the `Acquire` future is
dropped without returning the permits to the semaphore.

Fix this by recording the permits in the waiter's state and setting
`queued` before emitting the tracing events. `Acquire::poll` then only
clears `queued` once the future completes, after the last tracing
event has been emitted.

With this, `queued` consistently means that `Acquire::drop` must clean
up after the waiter, and `num_permits - state` is always the number of
permits the waiter holds, regardless of whether they were taken
directly from the semaphore or assigned in the wait queue.
@Darksonn
Darksonn deleted the f03 branch October 3, 2026 09:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-tokio Area: The main tokio crate C-enhancement Category: A PR with an enhancement or bugfix. M-sync Module: tokio/sync M-tracing Tracing support in Tokio R-loom-sync Run loom sync tests on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tracing subscriber panic leaves a completed semaphore waiter linked (use-after-free)

2 participants