Repository navigation
sync: fix semaphore use-after-free and permit leak on tracing panic - #8542
Merged
Merged
Conversation
Member
Author
|
I'm really not sure about this one ... feel free to give suggestions or even come up with alternate fixes. |
ADD-SP
approved these changes
Oct 2, 2026
Member
There was a problem hiding this comment.
The fix and test looks good to me. Actually, I think this is a minimal workable fix while it is not the clean fix.
BTW, it looks like our Miri tests don't cover the --cfg tokio_unstable, we can add it after merge. I ran the tests locally using Miri with tokio_unstable and it passed.
β¦#8542) Currently, `add_permits_locked` emits a tracing event once a waiter has been assigned all of the permits it needs, but before the waiter is removed from the wait queue. If the tracing subscriber panics during this event, the waiter stays in the queue with zero permits remaining. Polling the `Acquire` future then completes it and sets `queued = false`, so dropping the future frees the waiter without unlinking it from the queue, and the next release dereferences the freed waiter. Fix this by removing the waiter from the wait queue before emitting the tracing event.
Currently, when `poll_acquire` takes all of the requested permits from the semaphore without queueing the waiter, it emits tracing events before returning. At this point, neither the waiter's state nor the `queued` flag record that the permits were taken, so if the tracing subscriber panics during one of these events, the `Acquire` future is dropped without returning the permits to the semaphore. Fix this by recording the permits in the waiter's state and setting `queued` before emitting the tracing events. `Acquire::poll` then only clears `queued` once the future completes, after the last tracing event has been emitted. With this, `queued` consistently means that `Acquire::drop` must clean up after the waiter, and `num_permits - state` is always the number of permits the waiter holds, regardless of whether they were taken directly from the semaphore or assigned in the wait queue.
β¦s#8542) Currently, when `poll_acquire` takes only some of the requested permits from the semaphore, it emits a tracing event before assigning them to the waiter. If the tracing subscriber panics during this event, the `Acquire` future is dropped while unwinding, and `Acquire::drop` only returns the permits that were assigned to the waiter. The permits that were taken from the semaphore are therefore leaked. Fix this by assigning the permits to the waiter before emitting the tracing event.
Darksonn
added a commit
that referenced
this pull request
Oct 3, 2026
Currently, `add_permits_locked` emits a tracing event once a waiter has been assigned all of the permits it needs, but before the waiter is removed from the wait queue. If the tracing subscriber panics during this event, the waiter stays in the queue with zero permits remaining. Polling the `Acquire` future then completes it and sets `queued = false`, so dropping the future frees the waiter without unlinking it from the queue, and the next release dereferences the freed waiter. Fix this by removing the waiter from the wait queue before emitting the tracing event.
Darksonn
added a commit
that referenced
this pull request
Oct 3, 2026
Currently, when `poll_acquire` takes all of the requested permits from the semaphore without queueing the waiter, it emits tracing events before returning. At this point, neither the waiter's state nor the `queued` flag record that the permits were taken, so if the tracing subscriber panics during one of these events, the `Acquire` future is dropped without returning the permits to the semaphore. Fix this by recording the permits in the waiter's state and setting `queued` before emitting the tracing events. `Acquire::poll` then only clears `queued` once the future completes, after the last tracing event has been emitted. With this, `queued` consistently means that `Acquire::drop` must clean up after the waiter, and `num_permits - state` is always the number of permits the waiter holds, regardless of whether they were taken directly from the semaphore or assigned in the wait queue.
This was referenced Oct 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
add_permits_lockedemits a tracing event once a waiter has beenassigned all of its permits, but before removing it from the wait
queue. If the tracing subscriber panics during this event, the waiter
stays in the queue with zero permits remaining. Polling the
Acquirefuture again then completes it and clears
queued, so dropping itfrees the waiter without unlinking it, and the next release
dereferences the freed waiter.
Similarly,
poll_acquireemits tracing events after taking permits fromthe semaphore, but before the
Acquirefuture records that it holdsthem, so a subscriber panic there leaks the permits. This affects both
the uncontended path, where all permits are taken at once, and the
contended path, where the available permits are taken before the waiter
is queued.
This PR contains one commit per issue:
event.
set
queuedbefore emitting the tracing events, and only clearqueuedinAcquire::pollafter the last tracing event. With this,queuedconsistently means thatAcquire::dropmust clean up afterthe waiter, and
num_permits - stateis always the number of permitsit holds. (This is also the state that sync: preserve FIFO lock order when coop budget is exhaustedΒ #8371 needs when the coop
budget runs out after the permits were taken.)
emitting the tracing event.
All of these require
tokio_unstableand thetracingfeature. Theregression tests use a subscriber that panics on the first event with a
given target; without the first fix, Miri reports the use-after-free.
Not covered here: a panic in the tracing calls that run after
Acquire::pollhas returned (span exits and thepoll_opevent inInstrumentedAsyncOp) can still leak permits, and a panic inadd_permits_lockedafter a waiter was removed from the queue stillloses its wakeup and the remaining permits. Neither is a memory safety
issue, and handling them would require restructuring how these futures
are instrumented.
Fixes: #8541