📺 Info Siaran: cloudamqp/lavinmq/pull/

carlhoerberg · 2026-10-08T23:59:24Z

Why

Re-applying a policy first reset a queue's or exchange's settings to their argument values, then applied the policy on top. Anyone reading in between saw the resource with no policy at all:

Publishes got past a policy's max-length with overflow=reject-publish. In the new spec, about 600k–880k publishes got through in 2s on main.
Messages sent to an exchange whose alternate exchange came from a policy were dropped as unroutable. About 1.8M were dropped in 2s on main.
Readers could see torn Int64? limits, and effective-argument arrays while they were being changed in place.

This already happens on main, whenever policies change while the queue is in use. It is one of the queue-internals steps towards running the default execution context on several threads.

What

clear_policy_arguments and apply_policy_argument now only stage the new values. A new PolicyTarget#commit_policy_arguments hook publishes them, and clear_policy uses the same path.
Queues keep their argument- and policy-derived settings in a single Queue::Settings object: limits, TTLs, overflow mode, consumer timeout, DLX/DLRK, max-age and effective arguments.
- The object is built separately, swapped in by reference, and never changed after it is published.
- Hot-path readers do one reference load and allocate nothing.
- Side effects run after the swap: the expire fibers, limit enforcement, and the stream store's limits under @msg_store_lock.
Exchanges stage their alternate exchange and effective arguments the same way.
PolicyTarget replaces effective_policy_args instead of clearing it in place.

Specs:

New multi-threaded regression specs in spec/queue_policy_churn_spec.cr and spec/alternate_exchange_spec.cr fail on main and pass with this change.
spec/policies_spec.cr now reads values through @settings.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Mqv3W4rh4BxJDC4Y587DV9

Generated by Claude Code

Re-applying a policy first reset a queue's or exchange's settings to the argument values and then applied the policy on top. Anyone reading in between saw the resource without its policy: publishes slipped past a policy's max-length with overflow=reject-publish, and messages to an exchange whose alternate exchange came from a policy were dropped as unroutable. Readers could also observe torn Int64? limits and effective-argument arrays that were being mutated in place. Policy arguments are now staged by clear_policy_arguments and apply_policy_argument and published by a new commit_policy_arguments hook. Queues keep their argument and policy derived settings in one Settings object that is built off to the side and swapped in by reference, and side effects (expire fibers, limit enforcement, stream store limits) run after it is published. Exchanges stage their alternate exchange and effective arguments the same way, and PolicyTarget replaces effective_policy_args instead of clearing it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mqv3W4rh4BxJDC4Y587DV9

claude · 2026-10-09T00:01:10Z

No issues found.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mqv3W4rh4BxJDC4Y587DV9

commit_policy_arguments copied the staged DLX and DLRK into the DeadLetterer as two separate stores, and DeadLetterer#route read them separately, so a route on another thread could pair a new DLX with an old DLRK. The DeadLetterer now keeps both in one immutable Target that is swapped in by reference, and route reads it once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mqv3W4rh4BxJDC4Y587DV9

carlhoerberg · 2026-10-09T00:10:09Z

Fixed the DLX/DLRK pairing in f3b8c69.

DeadLetterer now keeps the DLX and DLRK together in one immutable Target. commit_policy_arguments swaps it in with a single store (set_target), and route reads @target once.
The new spec "never pairs a new dead letter exchange with an old routing key" (spec/queue_dead_lettering_spec.cr) switches between (dlx-a, rk-a) and (dlx-b, rk-b) while 3 threads dead-letter messages. It then checks that nothing reached the queue that only a mixed pair would route to. With the old two separate reads in route it failed 3/3 runs, and with the fix it passes 3/3.

amqp-quickcheck property tests is not caused by this PR. The failing properties are mqtt_wildcard_routing_matches_model and mqtt_new_subscription_gets_retained, both about $-prefixed MQTT topics. The same job fails on main at bce014f ("wildcard filters do not match '$' topics"), because the quickcheck model hasn't been updated for that rule. The fix belongs in lavinmq-quickcheck, and none exists yet to port. Re-running would fail the same way.

Generated by Claude Code

The publishers in the policy re-apply, alternate exchange and DLX/DLRK specs spun without yielding, so on the macOS runner, with fewer cores than the spec's threads, the fibers re-applying the policy were starved and the spec timed out. Publishers now yield every iteration and stop at the deadline themselves. They still fail on the code before the fix with two cores (taskset -c 0,1). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mqv3W4rh4BxJDC4Y587DV9

carlhoerberg · 2026-10-09T00:19:09Z

Spec macOS failed because one of this PR's own new specs, "keeps enforcing policy limits while the policy is re-applied", timed out. Its publisher threads never yielded, so on the macOS runner, which has fewer cores than the spec has threads, the threads re-applying the policy were starved.

Fixed in e9caa4d. The publishers in that spec, in the alternate-exchange spec and in the DLX/DLRK spec now yield every iteration and stop at the deadline themselves.

With the fix, all 3 pass 5/5 runs restricted to 2 cores (taskset -c 0,1).
On the old code they still fail 3/3 runs under the same 2-core limit.

Generated by Claude Code

The queue Settings, the dead letter target and the effective argument arrays were built and then published with a plain store. On a weakly ordered CPU (arm64) a reader on another thread could see the new reference before the writes that filled it in, e.g. a Settings without its max-length or overflow mode, which let a publish past the limit on the macOS (arm64) runner. They are now held in Atomics, set with :release and read with :acquire (an ldar on arm64, a plain load on x86). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mqv3W4rh4BxJDC4Y587DV9

carlhoerberg · 2026-10-09T00:24:54Z

Spec macOS on f3b8c69 found a real problem. One publish got past the limit (accepted was 1) in "keeps enforcing policy limits while the policy is re-applied". That run was on the arm64 runner.

Likely cause: a missing release barrier.

The new Settings object, the dead-letter Target and the effective-argument arrays were filled in and then published with a plain store.
On a weakly ordered CPU like arm64, a reader on another thread can see the new reference before the writes that filled its fields. It then reads, for example, a Settings without its max-length or overflow mode. x86 orders those stores, which is why Linux never showed it.
I can't run arm64 here, so the macOS run on the new commit is the real test.

Fixed in 3013aff.

They're now held in Atomics, set with :release and read with :acquire. On arm64 that read is an ldar; on x86 it's a plain load.
The queue exposes settings for readers, and the specs now use it.
The related specs pass (1095 examples), and the 3 race specs pass 3/3 on 2 cores.

Generated by Claude Code

carlhoerberg · 2026-10-09T00:32:00Z

Spec macOS on 3013aff: the new re-apply race specs passed. I don't think this PR caused the two failures:

server_spec "restarts fast even with large messages": it measured 2.1 MB allocated during the restart, against a 1 MB limit.
- Benchmark.memory counts allocations from every thread while the server restarts, not only the restart itself.
- This PR adds one small Settings object and array per queue, a few hundred bytes for the queues in this spec.
- The same spec passed on this PR's earlier macOS runs.
permission_groups_spec "accepts a group name with hyphens and underscores": pthread_mutex_lock: Resource deadlock avoided, raised from inside Crystal's Fiber::ExecutionContext::Isolated#start_thread while the Persister was starting its thread. That's a runtime or runner thread-startup error in code this PR doesn't touch.

I've re-run the failed jobs once. If either fails again I'll treat it as real.

Generated by Claude Code

…er-f2xdva-p3-policy-values

carlhoerberg · 2026-10-09T16:56:15Z

The main (Docker image build) check is failing because of the depot.dev build service, not this PR.

It failed on the merge-with-main commit and again on its one re-run. Both times LavinMQ compiled, then the build lost its connection to depot.dev: rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout.
The same failure hit all 13 open multi-threading PRs within the same minutes.
There's nothing to port. It needs depot.dev to recover, then a re-run.

Generated by Claude Code

…er-f2xdva-p3-policy-values # Conflicts: # CHANGELOG.md

# Conflicts: # CHANGELOG.md # src/lavinmq/amqp/queue/queue.cr

carlhoerberg · 2026-10-12T11:53:25Z

The Docker main job failed, and the cause isn't this PR. The image builder's connection dropped (rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout) at the same moment on every open PR after main was merged in. I re-ran the failed job once. There's no code fix to port.

Generated by Claude Code

…er-f2xdva-p3-policy-values The delayed retry arguments from main move into Queue::Settings with the other argument and policy values: settings_from_arguments parses them, applies the default delivery limit and creates the retry queue, and the retry code reads them through settings.

carlhoerberg · 2026-10-12T16:59:07Z

Merged main, which includes #2254 (retries with backoff), in 53f2618. The x-delayed-retry-min/max/multiplier values move into Queue::Settings with the other argument and policy values. settings_from_arguments parses them, applies the default delivery limit and creates the retry queue, and the retry code reads them through settings. The retry queue, policy, policy-churn and dead-lettering specs all pass.

Generated by Claude Code

…er-f2xdva-p3-policy-values # Conflicts: # CHANGELOG.md

carlhoerberg · 2026-10-12T22:03:07Z

After merging main into this branch to resolve the CHANGELOG conflict, Spec amd64 and Spec arm64 are expected to fail, and the cause is not this PR. Since today, CI runs on Crystal 1.22.0 (the 84codes/crystal:latest-ubuntu-26.04 image). On that version, spec/etcd_spec.cr segfaults with Process terminated because of an invalid memory access, and main fails the same way: Spec amd64 and Spec arm64 on 85cf7aa.

I reproduced the crash locally on main with Crystal 1.22.0. It happens in the Etcd::Lease isolated execution context while it reads from an etcd socket, inside the polling event loop's waiter list. The same specs pass on Crystal 1.21. There is no fix to port yet. I'll merge it in, or re-run, once main has one.

Generated by Claude Code

Crystal 1.22.0's Fiber::ExecutionContext::Isolated#wait_for sets `done = true` before it runs the event loop, so when the loop returns without a ready fiber, the isolated fiber is resumed anyway. That happens after a write: the socket is registered edge triggered for both read and write, and its write readiness event wakes nobody. A fiber waiting to read then retries with its event still queued on the fd, and the next wait_readable segfaults in Waiters#add. The Etcd::Lease keepalive does exactly that, so the etcd specs crashed the spec run on 1.22.0. Override wait_for on 1.22 with `done = false`, and require it where isolated contexts are started. (cherry picked from commit f2309a7)

Before Crystal 1.22, HTTP::Request#body = nil set Content-Length: 0 on POST and PUT requests, also on requests the server parsed, so parse_body returned an empty object for them. On 1.22 such a request has neither a body nor a content length, and parse_body answered "Request body required", e.g. to PUT /api/queues/vhost/name without a body. (cherry picked from commit 8bf5a08)

(cherry picked from commit 92aa32b)

Crystal's Options::ALL (0x80000BFF) includes bit 3, which is SSL_OP_ENABLE_KTLS in OpenSSL 3. Crystal 1.21 cleared ENABLE_KTLS again right after adding ALL in Context#initialize, 1.22 doesn't, so every TLS context had kTLS enabled, also when tls_ktls was off. On kernels with the tls module, the kTLS specs then found kTLS in effect. Clear ENABLE_KTLS after Context#initialize, as 1.21 did. The launcher still adds it when tls_ktls is on. (cherry picked from commit 36e16dd)

carlhoerberg · 2026-10-13T02:11:01Z

I ported #2440 ("Fix the spec suite on Crystal 1.22.0") into this branch as cherry-picks of its four commits, so CI is green again on the Crystal 1.22.0 image. #2440 has the Isolated#wait_for override that fixes the etcd segfault, plus the fixes for a POST/PUT without a body and for kTLS being on by default. Its own Spec amd64, arm64 and macOS checks pass, and locally the etcd, isolated-wait and queue specs pass on 1.22.0 with it. Once #2440 (or another fix) lands on main, merging main in drops these commits from this diff.

Generated by Claude Code

…er-f2xdva-p3-policy-values

carlhoerberg · 2026-10-13T08:23:11Z

Stress test on 795efec failed, but not because of this PR. The stress client starved itself:

What the log shows: bin/stress printed no stats line for 97 s (08:08:03 → 08:09:40). In that window only the publishers ran: pub=84539644 ack=0, pol+/-=2/0, vh+/-=2/0, q_pause+/-=1/0. Every other worker in the client and its 2 s stats printer never got scheduled.
Why: the :fast producer in extras/stress.cr (producer, mode == :fast) never yields. basic_publish only suspends when the socket write would block, so when the broker keeps up, those fibers monopolise the client's single thread.
What followed: 84M persistent-but-unconsumed messages filled the runner disk, and the broker died with Invalid memory access (signal 7) on an mmap'd segment. That's a SIGBUS, typical for a full disk under an mmap. Sibling runs on the same main show normal activity (hundreds of policy and vhost operations, ~30–36M publishes, Aliveness OK).
Local check: I ran this PR's merge head locally 3 times, all with Aliveness OK. In one of them the client went silent for 60 s the same way.

No fix for this exists yet. I've re-run the job once. Proposed patch for extras/stress.cr, to land separately rather than widen this PR:

        case mode
        when :fast
          Fiber.yield if seq % 100 == 0
        when :slow

Generated by Claude Code

…er-f2xdva-p3-policy-values # Conflicts: # CHANGELOG.md

…er-f2xdva-p3-policy-values # Conflicts: # CHANGELOG.md # src/lavinmq/amqp/argument/dead_lettering.cr

carlhoerberg · 2026-10-15T11:52:15Z

Spec amd64 on 96e6101 failed in spec/queue_spec.cr:83, "should expire itself after last consumer disconnects". The queue expired after 150.18 ms; the spec allows 100 ± 50 ms. This PR only changes where the expiry loop reads the TTL: settings.expires, one atomic load, instead of @expires. Its waits and timeouts are unchanged, so the extra 50 ms is runner timing on a timing assertion. I've re-run the job once.

Generated by Claude Code

…er-f2xdva-p3-policy-values # Conflicts: # CHANGELOG.md

carlhoerberg · 2026-10-15T23:44:46Z

@codex review

chatgpt-codex-connector · 2026-10-15T23:44:58Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-15T23:47:38.607595Z	`8d0a734`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector · 2026-10-15T23:47:37Z

Codex Review: Didn't find any major issues. What shall we delve into next?

Reviewed commit: 8d0a734379

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

get, with_delivery_count_header and delete_message read the settings reference up to four times per delivered message. Load it once in get and pass it down, and check delivery counting with Settings#counts_deliveries?.

# Conflicts: # CHANGELOG.md

github-actions Bot assigned carlhoerberg Oct 8, 2026

claude added 2 commits October 9, 2026 00:04

Add CHANGELOG entry

924b3c8

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mqv3W4rh4BxJDC4Y587DV9

Merge remote-tracking branch 'origin/main' into claude/admiring-meitn…

cf7633f

…er-f2xdva-p3-policy-values

claude added 2 commits October 11, 2026 01:04

Merge remote-tracking branch 'origin/main' into claude/admiring-meitn…

e364278

…er-f2xdva-p3-policy-values # Conflicts: # CHANGELOG.md

Merge remote-tracking branch 'origin/main' into HEAD

5891732

# Conflicts: # CHANGELOG.md # src/lavinmq/amqp/queue/queue.cr

Merge remote-tracking branch 'origin/main' into claude/admiring-meitn…

6b46e05

…er-f2xdva-p3-policy-values # Conflicts: # CHANGELOG.md

dentarg added 4 commits October 13, 2026 02:06

Add CHANGELOG entries for the Crystal 1.22.0 fixes

c8fdda1

(cherry picked from commit 92aa32b)

Merge remote-tracking branch 'origin/main' into claude/admiring-meitn…

795efec

…er-f2xdva-p3-policy-values

Merge remote-tracking branch 'origin/main' into claude/admiring-meitn…

e33625d

…er-f2xdva-p3-policy-values # Conflicts: # CHANGELOG.md

carlhoerberg mentioned this pull request Oct 15, 2026

Keep queue consumer bookkeeping and decisions under @consumers_lock #2404

Merged

Merge remote-tracking branch 'origin/main' into claude/admiring-meitn…

96e6101

…er-f2xdva-p3-policy-values # Conflicts: # CHANGELOG.md # src/lavinmq/amqp/argument/dead_lettering.cr

carlhoerberg mentioned this pull request Oct 15, 2026

Make queue close/delete/pause state transitions atomic #2395

Open

carlhoerberg mentioned this pull request Oct 15, 2026

Don't let an older policy apply overwrite a newer one #2398

Merged

claude added 2 commits October 15, 2026 12:03

Merge remote-tracking branch 'origin/main' into claude/admiring-meitn…

43c4f12

…er-f2xdva-p3-policy-values # Conflicts: # CHANGELOG.md

Merge remote-tracking branch 'origin/main' into claude/admiring-meitn…

8d0a734

…er-f2xdva-p3-policy-values # Conflicts: # CHANGELOG.md

carlhoerberg marked this pull request as ready for review October 15, 2026 23:44

claude added 2 commits October 16, 2026 00:01

Load queue settings once per delivery

3a1de9f

get, with_delivery_count_header and delete_message read the settings reference up to four times per delivered message. Load it once in get and pass it down, and check delivery counting with Settings#counts_deliveries?.

Merge branch 'main' into claude/admiring-meitner-f2xdva-p3-policy-values

58c0d8c

# Conflicts: # CHANGELOG.md

carlhoerberg added this to the 2.11.0 milestone Oct 16, 2026

carlhoerberg merged commit 1391ea8 into main Oct 16, 2026
24 checks passed

carlhoerberg deleted the claude/admiring-meitner-f2xdva-p3-policy-values branch October 16, 2026 11:22

carlhoerberg mentioned this pull request Oct 16, 2026

Route without locks through copy-on-write bindings #2376

Draft

Conversation

carlhoerberg commented Oct 8, 2026

Why

What

Uh oh!

claude Bot commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carlhoerberg commented Oct 9, 2026

Uh oh!

carlhoerberg commented Oct 9, 2026

Uh oh!

carlhoerberg commented Oct 9, 2026

Uh oh!

carlhoerberg commented Oct 9, 2026

Uh oh!

carlhoerberg commented Oct 9, 2026

Uh oh!

carlhoerberg commented Oct 12, 2026

Uh oh!

carlhoerberg commented Oct 12, 2026

Uh oh!

carlhoerberg commented Oct 12, 2026

Uh oh!

carlhoerberg commented Oct 13, 2026

Uh oh!

carlhoerberg commented Oct 13, 2026

Uh oh!

carlhoerberg commented Oct 15, 2026

Uh oh!

carlhoerberg commented Oct 15, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot commented Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

claude Bot commented Oct 9, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 15, 2026 •

edited

Loading