Fixed since 2.11.0-rc.2

• Declaring a durable queue fsynced the vhost directory right away, creating an MQTT session fsynced its directories and its .metadata file, and a priority queue fsynced its directory on every start, which made bulk declares, definition imports, MQTT client connect storms and starts with many priority queues slower than in 2.10. They are now fsynced once with the next publish confirm that depends on them, and not at all for clean MQTT sessions #2544

• Every start rewrote each queue's .queue file and fsynced a rewrite of the MQTT retained-message index, which on ext4 forced the .queue writes out to disk inside the startup window and made starts with many queues slower than in 2.10, most of all on IOPS-limited disks. The .queue file is now written once, when the queue is declared, and the index only when it changed #2550

• A follower fsynced the parent of every directory it created inline on the replication stream, so each new queue or MQTT session on a cluster held up replication for an fsync. The new entries are now fsynced with the next fsync request, before the follower acks it #2555

• A cold start read each queue's last segment twice since 2.11.0-rc.1, once for the schema check and once to count its messages, as segments are mapped without readahead. On Linux the segment is now fetched in one readahead, bounded by what it occupies on disk, before the check, which halves the reads of a start with many queues #2551

• The tcp_nodelay setting only applied to the AMQP and MQTT listeners. The HTTP listeners (management API, WebSocket and metrics) ignored it and kept Nagle's algorithm on, which could delay responses on kept-alive connections, and deliveries to AMQP/MQTT WebSocket clients that ack, by up to ~40 ms #2494

• The HTTP API docs at /docs/ listed no endpoints when LavinMQ is behind a cookie-based auth proxy such as Cloudflare Access, as the spec's referenced files were fetched without cookies #2539

• Importing definitions that contain a federation-upstream-set parameter failed with 400 "Cast from Array(JSON::Any) to Hash(K, V) failed", so the broker's own export couldn't be imported #2536

• A TLS 1.2 connection that sent a servername bypassed the listener's tls_ciphers when any [sni:host] section was configured: OpenSSL picks the TLS 1.2 cipher after the servername callback, and the swapped-in SNI context carried OpenSSL's default cipher list. The listener's tls_ciphers and tls_ciphersuites now apply to every SNI context #2540

• With tcp_nodelay on (the default since 2.11.0-rc.1), a delivery that didn't fit what was left of the socket's write buffer went out as a small TCP segment of the buffered frames and then the rest, about a quarter of the throughput at 16 KiB messages with one publisher and one consumer. A write that doesn't fit now tops the buffer up first, and the socket is corked until the flush that ends the batch #2568

• HTTP API list endpoints built the details of every item before paging, which with tens of thousands of queues could grow the heap by gigabytes or stall the broker #2452

• Since 2.11.0-rc.1, reading an AMQP frame from a connection's pooled socket buffer went through a read that isn't inlined for every field, about 3% lower throughput with 16-byte messages. A read served from the buffer now copies from it directly #2578

• Since 2.11.0-rc.1, the clock was read for every received frame and every delivered message to track heartbeats, about 3% lower throughput with 16-byte messages. It's now read once per socket read and once per socket flush #2579

• Since 2.11.0-rc.1, the segment being written was mapped without readahead, so a queue written fast could use up to twice the CPU per message. It's mapped with readahead again until a publish to the queue is synced (publish confirm, MQTT QoS 1 or transaction), from then on it stays random access so a sync writes a page instead of a large folio #2589

• An HTTP re-PUT of an existing internal exchange, such as any x-mqtt-topic exchange, answered 400 "Not allowed to publish to internal exchange". An identical redeclare now answers 204, like an AMQP redeclare #2541

• A confirmed publish that created a new segment file, like the first publish to a queue or a segment rollover, while the process was out of file descriptors stopped the broker: the persister exited when it couldn't open the directory it had to fsync, and a follower did the same for the leader's fsync requests. Both now sync the whole data dir through the already open data dir descriptor instead, and the broker stays up, as the accept loops already do since #2422 #2542

Added since 2.11.0-rc.2

• [clustering] etcd_lease_ttl config option (default 10 seconds, the previous hardcoded value). A leader that does not renew its etcd lease within the TTL loses leadership, so a higher value lets large nodes survive longer GC pauses, at the cost of slower failover on crashes #2582

Changed since 2.11.0-rc.2

• Large JSON API responses and Prometheus scrapes used 32 to 48% more CPU than needed: every byte went through two virtual IO dispatches and a one-byte slice write. Controllers now write to the response output and the Prometheus writer formats into a buffer #2569

• Lower CPU usage when scanning message segments at startup #2559

• The HTTP API docs at /docs/ now match the management UI: they use the Inter font and LavinMQ colors, share the UI's light/dark/system theme and have a switcher for it, and have a header linking back to the management UI #2526

• Use amq-protocol 1.3.2, which parses incoming AMQP frames with less CPU #2577

Added

• [clustering] etcd_lease_ttl (--clustering-etcd-lease-ttl, LAVINMQ_CLUSTERING_ETCD_LEASE_TTL) sets the etcd lease TTL, default 10 seconds as before. The lease is kept alive every TTL/3, so a pause longer than about 2/3 of the TTL loses leadership; a higher TTL lets large nodes survive longer GC pauses, at the cost of slower failover on crashes #2582

Changed

• Large JSON API responses and Prometheus scrapes used 32 to 48% more CPU than needed: every byte went through two virtual IO dispatches and a one-byte slice write. Controllers now write to the response output and the Prometheus writer formats into a buffer #2569

Fixed

• HTTP API list endpoints built the details of every item before paging, which with tens of thousands of queues could grow the heap by gigabytes or stall the broker #2452

Fixed since 2.11.0-rc.1

• Removing an entry with overridden settings from a federation-upstream-set left its federation queue and exchange on the upstream broker, where the queue kept collecting messages. The resources are now deleted when no entry in the updated set reuses them #2531

• Deleting a federated exchange deleted the link's queue and exchange on the upstream broker while holding the vhost's definitions lock. With an unreachable upstream, every declare, delete and binding in the vhost waited up to 60 seconds per link. Upstream resources are now deleted in the background #2531

• Any policy or parameter change in a vhost restarted every federation link in it, and for exchange federation the restart deleted the link's queue and exchange on the upstream broker, with the messages buffered there. A policy re-apply now keeps an unchanged link running, and replacing an upstream parameter keeps the resources on the upstream broker for the new links to reuse, as long as the new config names the same resources, where credentials and query options in the URI don't count. A re-created upstream also stays in the named upstream sets that reference it. Previously the sets lost the entry, and federation through them silently stopped until the set was re-PUT #2509

• Exchange federation forwarded a binding to the upstream and waited for the reply inside the fiber that processes the downstream bind. In two-way or ring federation with max-hops 3 or more, the brokers deadlocked and the links stayed in starting. Bindings are now forwarded without waiting for a reply #2508

• A consumer that caught up with the publisher of its queue ran madvise(MADV_DONTNEED) over the segment being written each time it emptied the queue, so the publisher's next write faulted the page back in. A segment's pages are now only dropped once the consumer has moved on to the next segment #2486

• Persistent messages in a delayed exchange whose delay had passed while the broker was down were dropped at startup, because they were released before the exchange's bindings were restored. Delayed queues now wait for the vhost to finish loading before releasing messages #2511

• The broker no longer exits when it runs out of file descriptors while accepting a connection. All listeners, including the HTTP API, metrics, clustering and proxy listeners, pause and keep accepting once connections close, and the stats loop, which also stops publishers when the disk is about to fill up, keeps running through errors #2422

• Publishes over the HTTP API and MQTT ignored the disk alarm and kept writing to a full disk. The HTTP API now answers 412, and MQTT connections that publish are closed, as MQTT 3.1.1 can't refuse a single publish. Last will messages aren't published during the alarm either #2510

• A leader stopped gracefully usually exited with code 1 after an Unhandled exception: Channel is closed, leaving its systemd unit failed. It now finishes its shutdown before exiting #2520

• A leader stopped gracefully could remove every follower from the in-sync replica set on its way out, so the survivors won the election, found they weren't in the ISR and exited, leaving the cluster down until the old leader came back. The etcd lease is now revoked before the shutdown goes on, so the ISR writes from the disconnecting followers are refused #2500

• A follower promoted to leader could exit with Could not bind ... Address already in use, leaving the cluster without a leader for several seconds. The leader watcher and the controller both close the replication client on promotion, and the second close returned while the first still held the proxy ports. It now waits for the first close to finish #2528

• Followers in a cluster now accept AMQPS, MQTTS and HTTPS connections and pass them on to the leader, which terminates TLS. Previously the TLS ports were only bound on the leader, so clients that reached a follower were refused #2475

• Retained MQTT messages on non-ASCII topics no longer overwrite each other. The topic tokenizer mixed byte and character offsets, so topics like é/1 and é/2 collapsed into one retain index entry and a subscriber could get a payload from another topic #2512

• The MQTT retain store no longer keeps an open file per retained topic. Many retained topics could exhaust the file descriptor limit, make accept() fail with "Too many open files" and stop the broker #2513

• An MQTT topic longer than 255 bytes closed every queue it was routed to, MQTT sessions and queues bound to an x-mqtt-topic exchange, and the next PUBACKed message to them was lost. Such a publish now closes the connection with a logged warning, a CONNECT with such a will topic is refused, a topic filter that long gets SUBACK Failure, and a retain store poisoned by an older version heals on the first restart #2516

• An MQTT QoS 1 publisher that stopped reading PUBACKs is now disconnected when the PUBACK write times out. Previously the PUBACK writer swallowed the error and the connection stayed open without ever getting another PUBACK #2504

• An MQTT client over WebSocket that disconnected cleanly could have its Will published, as a DISCONNECT that arrived together with the close frame was dropped before it was read #2495

• WebSocket connections are closed with a close frame again, instead of the TCP connection just being dropped, which clients reported as an abnormal closure (1006) #2495

• A queue TTL (x-message-ttl argument or message-ttl policy) so large that publish time plus TTL overflows a 64-bit integer, such as 2^63-1, crashed and restarted the message expire fiber in a loop, flooding the log and making the broker unresponsive. Such messages now never expire #2517

• Closing an AMQP connection through the HTTP API waited up to tcp_send_timeout when a delivery to a client that doesn't read was in progress. The connection is now shut down, with a log line, if sending the Close frame hasn't completed within 5 seconds #2515

• tls_min_version, tls_ciphers, tls_ciphersuites and tls_prefer_server_ciphers in [sni:host] sections now log a warning instead of being silently ignored. OpenSSL selects the TLS version and cipher before the SNI certificate switch, so these keys never took effect; the listener's settings apply to every SNI host. An SNI section supports the certificate and mTLS settings. When a TLS listener is configured, an unreadable certificate in an SNI section now stops startup with a clear error, and a reload with one is refused, instead of failing that host's handshakes. SNI connections keep the listener's TLS options, such as tls_ktls, which the SNI switch used to clear #2506

• Messages redelivered after a retry delay are flagged redelivered, and a message whose TTL runs out while it waits in the retry queue is dead-lettered when the TTL expires instead of when the backoff ends #2503

Changed since 2.11.0-rc.1

• Downgrading to 2.10.x after using new 2.11 features needs care. 2.10.x doesn't know the x-mqtt-topic exchange type: it logs "unknown exchange type", skips those exchanges and their bindings, and starts anyway; the next time 2.10.x compacts its definitions file they're removed for good. 2.10.x also doesn't create the retry queues of queues with x-delayed-retry-min, so messages waiting for a retry are neither shown nor delivered until the node runs 2.11 again #2532

Fixed

• A leader stopped gracefully usually exited with code 1 after an Unhandled exception: Channel is closed, leaving its systemd unit failed. It now finishes its shutdown before exiting #2520
• A leader stopped gracefully could remove every follower from the in-sync replica set on its way out, so the survivors won the election, found they weren't in the ISR and exited, leaving the cluster down until the old leader came back. The etcd lease is now revoked before the shutdown goes on, so the ISR writes from the disconnecting followers are refused #2500
• Followers in a cluster now accept AMQPS, MQTTS and HTTPS connections and pass them on to the leader, which terminates TLS. Previously the TLS ports were only bound on the leader, so clients that reached a follower were refused #2475
• Built with Crystal 1.22.0, LavinMQ could crash with a segfault in clustering, when the etcd lease keepalive waited for a reply #2440
• Built with Crystal 1.22.0, the HTTP API answered 400 Request body required to a POST or PUT without a body, e.g. PUT /api/queues/vhost/name #2440
• Built with Crystal 1.22.0, LavinMQ enabled kTLS on every TLS connection, also when tls_ktls was off #2440

This release adds the x-mqtt-topic exchange type, automatic retries with backoff for rejected messages, and the tcp_send_timeout, tls_ciphersuites and syncfs_threshold config options. The management UI gets new charts and a refresh control. Publish confirms sync only the affected files, message store segments are cached in page-sized folios and connections share pooled socket buffers, lowering write amplification and memory usage. It also fixes many races around policies, consumers, dead-lettering and message store segments that could corrupt state or crash the broker.

Release Highlights

New charts and a refresh control in the management UI

The charts are redrawn on uPlot. The legend is a table under the plot that shows the latest value, or the value under the cursor, and hovering one chart moves the cursor on every chart on the page. Charts rebuild from the server history log on every refresh, so a slower or paused refresh leaves no gaps. The nodes page gets a Disk usage chart with capacity lines. #2448

A refresh control in the header pauses and resumes auto-refresh, sets the refresh rate to 5, 10, 30 or 60 seconds, and shows when the UI cannot reach the server. #2389

Route MQTT publishes into AMQP with the x-mqtt-topic exchange

A new exchange type, x-mqtt-topic, bridges MQTT into AMQP. Bind queues or exchanges to it with MQTT topic filters (+, #), and matching MQTT publishes are delivered with the MQTT topic as the routing key. The exchange is always internal, and malformed filters are refused at bind time. #2346

Automatic retries with backoff

A queue declared with x-delayed-retry-min holds messages that are rejected with requeue=true in an internal retry queue, and redelivers them after a growing delay. The optional x-delayed-retry-multiplier and x-delayed-retry-max arguments control the growth and the cap, and x-delivery-limit (default 20) dead-letters a message after too many attempts. #1815

Lower write amplification and memory usage

Publish confirms sync only the segments the confirmed messages were written to, instead of a syncfs of the whole data dir, so unrelated traffic on other queues is no longer flushed with every confirm #2296. Message store segments are cached in page-sized folios, so each sync writes about 4 KiB instead of up to 128 KiB #2323. Client connections share per-thread pools of socket buffers, lowering memory usage with many connections #2343.

Static Linux binaries on every release

Statically linked binaries for amd64 and arm64 are attached to this release, so you can test the RC without packages or Docker. #2478

Changes Worth Mentioning

• tcp_send_timeout config option (default 15 seconds): a client that does not read what the server sends for that long is disconnected. Previously such a write blocked forever #2363
• tcp_nodelay now defaults to true, removing up to ~40 ms of latency on deliveries to consumers that ack in batches #2336
• Message timestamps and message TTL expiry have millisecond precision; they were previously rounded down to 100 ms #2344
• tls_ciphersuites config option to select the allowed TLS 1.3 ciphersuites #2243
• LavinMQ-Stats-Interval header on every HTTP API response, so clients can place the history logs on a time axis #2447
• Prometheus metrics for dead-lettered messages, per reason and strategy #2413
• Internal queues (e.g. delayed exchange queues) are protected from AMQP clients, and are no longer included in definitions exports #2252
• LavinMQ exits at startup if the data directory lock is held by another process, instead of waiting for the lock #2350

Notable Fixes

• Many races that could corrupt state or crash the broker: overlapping policy changes #2397 #2398, consumer bookkeeping #2404, dead-lettering #2403, unacknowledged basic.get messages #2402, stream requeues #2406, delivery counts #2401 and flow-control flags #2400
• Crashes when a message store segment was unmapped while a message from it was still being delivered. Segments are now kept mapped until in-flight deliveries finish #2324
• Publishing resumes after a disk alarm only once free disk space is well above the threshold, avoiding crashes from the burst of writes right after recovery #2457
• With tls_ktls enabled, idle AMQP connections were closed at half the heartbeat interval instead of being sent a heartbeat #2432
• MQTT subscriptions to a/# now also get messages on the parent topic a, and filters that start with # or + no longer match $SYS/... topics, as the MQTT spec requires #2377
• Three bugs when built with Crystal 1.22.0: a clustering segfault, rejected bodyless POST and PUT requests, and kTLS enabled against the config #2440

Upgrade Notes

• tcp_nodelay now defaults to true. Set tcp_nodelay = false in [main] for the old behaviour.
• A client that does not read what the server sends for tcp_send_timeout (default 15 seconds) is disconnected.
• MQTT QoS 1 PUBACKs are sent once the publish is persisted to disk, so QoS 1 publish throughput is lower than in 2.10.x #2296
• Followers sync files before acking (replication protocol v2). Nodes on v1 remain compatible, so mixed-version clusters work during a rolling upgrade.

For the full list of changes, see the changelog.

Fixed

• Memory growth with clustering enabled when vhosts are repeatedly created and deleted: files of a deleted vhost stayed in the replication index, and followers kept them on disk with an open file descriptor each #2335
• A vhost deleted on a cluster could reappear with its queues, messages, exchanges and policies if the vhost was created again under the same name after a failover: followers kept the deleted vhost's files on disk #2335

Fixed

• Memory growth with clustering enabled when vhosts are repeatedly created and deleted: files of a deleted vhost stayed in the replication index, and followers kept them on disk with an open file descriptor each #2335
• A vhost deleted on a cluster could reappear with its queues, messages, exchanges and policies if the vhost was created again under the same name after a failover: followers kept the deleted vhost's files on disk #2335

This patch release works around a Linux kernel bug that stalled queue churn on affected kernels, and fixes messages above x-max-priority that could not be acked, stream segments with an incomplete trailing record that prevented startup, and streams that max-length retention could empty on rollover. It reduces stream memory usage during replays, resends unacked MQTT QoS 1 publishes under their original packet IDs, keeps a locked-down MQTT definitions import locked down, and fixes several queue policy, HTTP API, management UI and logging issues.

Added

• A startup warning when the data directory's block device has a read ahead above 1 MiB, as a large read ahead stalls publishers at segment rollover #2337

Changed

• The MQTT default permission group is saved to mqtt_permissions.json when a vhost is created instead of when it closes. A definitions import no longer removes the default group from an existing vhost, and a vhost created by an import that has an mqtt_permissions key gets only the groups listed for it, so an exported vhost without groups stays locked down. lavinmqctl definitions export includes the default group for a vhost without mqtt_permissions.json #2330
• max_inflight_messages must be at least 1; 0 is now rejected at startup and on config reload instead of leaving every MQTT session accepting publishes it can never deliver #2233
• Stream max-length and max-length-bytes retention only drops a segment if the stream still meets the limit without it, so a stream now keeps at least the limit (up to one extra segment) instead of possibly being emptied on segment rollover. max-age segments are dropped when they expire, also on streams that receive no new messages #2250
• TLS handshake failures and invalid TLS configuration are logged with an error message instead of a stack trace. A failed TLS configuration reload logs the error and keeps the previous configuration #1762

Fixed

• Severe queue-churn stalls on affected Linux kernels, observed on arm64: segment memory is released in 1 MiB chunks to avoid a kernel TLB flush bug that caused repeated page faults #2288
• Messages published with a priority above the queue's x-max-priority could not be acked, rejected, requeued or purged, leaving the queue inconsistent even after a restart. Store lookups now clamp the priority to the queue's maximum, matching how the messages are stored #2293
• An incomplete trailing record in a stream segment could prevent broker startup when segment metadata was rebuilt. Recovery now drops the incomplete record and replicates the shortened segment to followers #2286
• Unacknowledged MQTT QoS 1 publishes are resent under the packet IDs the client already holds, with dup set, instead of being assigned new ones [MQTT-4.4.0-1]. The IDs are remembered in-process, so a session resumed after a broker restart is still redelivered under fresh IDs #2233
• Stream queue memory usage while consuming: segments are released from memory as soon as no consumer is reading them, instead of by a sweep every 60 seconds, which could grow to hundreds of MB during a fast replay #2250
• Repeated queue policy updates spawned duplicate expiration and limit-enforcement workers. Updates are now coalesced, and overflow and delivery-limit enforcement run independently so a failure in one cannot skip the other #2291
• HTTP routes with percent-encoded unreserved characters, such as /api/nodes/gc%5Fstats, now match the intended endpoint. Encoded slashes remain part of parameter values, including vhost names #2277
• Sorting boolean columns through the HTTP API returned an error, including sorting the connections list by TLS in the management UI #2248
• Management UI tooltips were clipped or misplaced near viewport edges and inside scrollable containers #2235

This patch release works around a Linux kernel bug that stalled queue churn on affected kernels, and fixes messages above x-max-priority that could not be acked, stream segments with an incomplete trailing record that prevented startup, and purged messages that came back after a restart. It also fixes connections dropped by long reply_text values, case-sensitive booleans in [sni:] config sections, stream replays starving other work, and several queue policy, HTTP API and logging issues.

Fixed

• Severe queue-churn stalls on affected Linux kernels, observed on arm64: segment memory is released in 1 MiB chunks to avoid a kernel TLB flush bug that caused repeated page faults #2288
• Messages published with a priority above the queue's x-max-priority could not be acked, rejected, requeued or purged, leaving the queue inconsistent even after a restart. Store lookups now clamp the priority to the queue's maximum, matching how the messages are stored #2293
• An incomplete trailing record in a stream segment could prevent broker startup when segment metadata was rebuilt. Recovery now drops the incomplete record and replicates the shortened segment to followers #2286
• Purged messages that had been requeued came back after a restart, because purge_all dropped them from memory without writing an ack record. The queue size counter could also underflow #2247
• An AMQP reply_text longer than 255 bytes broke the frame after the header was written and dropped the connection. A passive declare of a missing queue with a long name reaches it, as does the X-Reason header on DELETE /api/connections/:name. The text is now truncated on a codepoint boundary #2263
• Boolean values in an [sni:...] config section were parsed case-sensitively, so TRUE read as false for the tls_verify_peer keys and silently disabled mTLS #2264
• A stream consumer replaying from an old offset yielded to other fibers only every 32768 messages, so a fast replay of large messages could starve publishers, other consumers and GC #2237
• Repeated queue policy updates spawned duplicate expiration and limit-enforcement workers. Updates are now coalesced, and overflow and delivery-limit enforcement run independently so a failure in one cannot skip the other #2291
• HTTP routes with percent-encoded unreserved characters, such as /api/nodes/gc%5Fstats, now match the intended endpoint. Encoded slashes remain part of parameter values, including vhost names #2277
• Sorting boolean columns through the HTTP API returned an error, including sorting the connections list by TLS in the management UI #2248

Changed

• TLS handshake failures and invalid TLS configuration are logged with an error message instead of a stack trace. A failed TLS configuration reload logs the error and keeps the previous configuration #1762
• RPM builds install openssl-devel, zlib-devel and git explicitly, since the Fedora crystal package no longer pulls them in #2278

This release adds MQTT topic permissions, negative x-stream-offset values to read the last N stream messages, a state filter on the queue list endpoints and an API endpoint to close a single channel. It adds Prometheus metrics for per-queue deliveries and inter-node replication. Shovels get reworked HTTP destinations and error handling, with classified delivery outcomes, a dest-timeout setting and an aborted state. It also fixes purged messages that came back after a restart, a stream consumer that could starve other fibers during a fast replay, and an AMQP reply text over 255 bytes that broke the frame it travelled in.

Release Highlights

MQTT Topic Permissions

MQTT clients can now be restricted to specific topics. Permission groups are defined per vhost and map members (usernames, or * for every user) to topic filter rules with read and write flags. Patterns can reference the connection, e.g. chat/{client_id}/#. Groups are managed via /api/mqtt/permission-groups and are included in definitions export and import.

Every vhost gets a default group that allows all topics, so an upgraded server keeps every topic open. Delete or narrow the default group to lock a vhost down. There is no administrator bypass. #2126

The new client_id_validation = username option under [mqtt] requires the MQTT client ID to match the authenticated username, which pairs well with {client_id} patterns. #2038

Shovel Delivery Outcomes

Every shovel delivery is now classified into an outcome that decides what happens to the source message:

• 2xx HTTP responses and publisher confirms ack the message
• 408, 429, 5xx, transport failures and on-confirm publisher nacks requeue it, with backoff (0.5s doubling to 30s)
• Statuses that describe the message itself (400, 413, 415, ...) dead-letter it via the source queue's dead-letter exchange
• Repeated unusable-destination outcomes stop the shovel in a new aborted state, which is resumed via the API or the management UI

HTTP destinations get a dest-timeout setting, editable in the management UI, and the shovel API exposes runtime delivery outcome counters. See shovels.md for the full mapping. #2128

Read the Last N Stream Messages

A negative x-stream-offset starts a stream consumer N messages before the end of the stream, e.g. -100 reads the last 100 messages. #1941

Changes Worth Mentioning

• state query parameter on GET /api/queues and GET /api/queues/:vhost to filter queues by state, e.g. ?state=closed or ?state=paused,closed #2234
• API endpoint and management UI action to close a single channel #2212
• Per-queue delivered and acked totals in Prometheus metrics #1837
• Inter-node replication byte metrics in Prometheus #2051
• tls_prefer_server_ciphers config option that makes the server's cipher order decide the negotiated cipher #2204
• Tab navigation on queue detail pages and updated overview cards in the management UI #2006 #2145

Notable Fixes

• Purged messages that had been requeued came back after a restart. See the upgrade note below #2247
• A stream consumer replaying from an old offset could starve publishers, other consumers and GC #2227
• An AMQP reply_text longer than 255 bytes, e.g. from a passive declare of a missing queue with a long name, broke the frame and dropped the connection #2263
• Boolean values in an [sni:...] config section were parsed case-sensitively, so tls_verify_peer = TRUE silently disabled mTLS #2264

Breaking Changes / Upgrade Notes

Shovels retry instead of dead-lettering

In 2.9.x, with ack-mode on-confirm or on-publish, any non-2xx response from an HTTP destination rejected the message without requeue: it was dead-lettered, or silently dropped if the source queue had no dead-letter exchange. An AMQP destination in on-confirm mode acked the source message even when the destination nacked it, so a nacked message was lost.

In 2.10.0:

• 408, 429, 5xx and transport failures requeue the message with backoff, and retry indefinitely
• Statuses that describe the message itself (400, 411, 413, 414, 415, 422, 431) still dead-letter it, or drop it if the source queue has no dead-letter exchange
• Any other non-2xx (401, 403, 404, 3xx, ...) requeues the message. After 10 in a row the shovel stops in the aborted state until it is resumed
• A publisher-confirm nack from an AMQP destination in on-confirm mode requeues the message

A shovel pointed at a failing endpoint now keeps messages in the source queue instead of dead-lettering them, so the source queue can grow. Consumers of the dead-letter queue no longer receive messages that failed with a transient error. Monitor shovels for the aborted state. #2128

Purged messages can come back once

In 2.9.3 and earlier, a purge of a queue that held requeued messages did not write ack records for those messages. The messages stayed in the segment files and returned at the next restart. This is fixed in 2.10.0 #2247.

The upgrade in itself is a restart. If you purged such a queue before upgrading, those messages may return once even after the upgrade. If your consumers must not receive them, do this for each queue that you purged before upgrading:

1. Let the consumers drain the queue.
2. Disconnect the consumers.
3. Upgrade to 2.10.0.
4. Purge the queue again.
5. Reconnect the consumers.

MQTT topic permissions and downgrades

• A downgrade to 2.9.x ignores mqtt_permissions.json, so every MQTT topic is open again. Only the AMQP permission check from permission_check_enabled still applies, if it is set.
• A lockdown made by deleting a vhost's default group is not reliably carried over by a definitions export and import. For an import through the API or lavinmqctl, keep the default group and remove or narrow its rule instead. See mqtt.md. #2272

For the full list of changes, see the changelog.