Skip to content

docs: deduplication strategies documentation - #276

Merged
aludwiko merged 2 commits into
mainfrom
deduplication-docs
Mar 14, 2025
Merged

aludwiko merged 2 commits into
mainfrom
deduplication-docs

Conversation

@aludwiko

@aludwiko aludwiko commented Mar 5, 2025

Copy link
Copy Markdown
Contributor

First version to start the discussion. I would suggest reading everything first, and adding comments in the second round. Unless it's typo or gramma.

@github-actions github-actions Bot added documentation documentation related java-sdk labels Mar 5, 2025

private List<String> addCommandId(String commandId) {
// end::deduplication[]
// To avoid infinite growth of the list with limit the size to 1000.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How to exclude comments from adoc source section?

- *You gain security*. Services serve as a security boundary both in your system overall and between teams.
- *You get granular visibility into costs*. Services are all billed separately, so it's easier to see and understand costs and billing on a per-service basis if you break your services up in some way that matches your organizational needs overall.

== Messages deduplication

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if this is a good place for this section.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems ok, unless the page is becoming too large.
we should link to the section from other component documentation, or places where we already mention at-least once

@patriknw patriknw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking good


Keep in mind that a View row corresponds to a single entity instance, that's why we can use the sequence number as a deduplication token.

The same technique can be used not only for the Akka Views, but for any external storage mechanism. As the matter of fact, Akka Views has a built-in support for sequence number tracking, so you don't have to implement it yourself. The example above just shows how it can be done manually.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find it somewhat strange that we use views in the example, and then say that it's already handled automatically. I understand that it's an easy way to illustrate things, but still confusing with the risk that people with start implementing idempotency manually for views, because people don't read everything. If we continue with views as example we should have a clear note in the beginning similar to this sentence.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we would use a Consumer instead, we would have some kind of key-value interface corresponding to an external storage.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have the same doubts, since you confirmed my thoughts I will change it.

----
<1> Get the CloudEvent ID from the event metadata.

Unfortunately, according to the CloudEvents https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/spec.md#id[specification], the ID is not guaranteed to be the same for redelivery of the same event, hence we can't use it for deduplication.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we even mention this, since it's not a working solution?
If we keep it, I think we can remove the code snippet.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I moved this section to Java docs. Should be enough.

- Useful for cross-service or cross-component communication.

==== Challenges
- Choosing the right hashing algorithm (e.g., SHA-256 vs. MD5) for a balance between collision resistance and performance.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we also mention that another challenge is that the receiver must keep track of used tokens, and it must typically evict tokens after a while to keep memory or storage within some boundaries.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.

----
include::example$transfer-workflow-compensation/src/main/java/com/example/wallet/domain/Wallet.java[tag=deduplication]
----
<1> List of processed command IDs.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A slightly better implementation would be to use LinkedHashSet? if Jackson can serialize it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really, because with Set I don't have a method to remove the oldest entry https://github.com/akka/akka-sdk/pull/276/files#diff-366d9a64a9076550a7ad186849c2003a9410ff145905120d75cf47d7ea816350R62

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's why it's a Linked set. then you have removeFirst or removeLast

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aaaa right, updated to LinkedHashSet

To ensure system integrity, consumers must be capable of handling duplicate messages gracefully. Effective deduplication is not just an optimization - it's a core architectural requirement, transforming the potential complexity of message redeliveries into a predictable and manageable system behavior.

There is no one-size-fits-all solution to this challenge. Usually it's a mix of business requirements and possible technical tricks in a given context.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we repeat that Consumers, Views and Entities are single writers for a given entity id? There are no concurrent updates. Messages (events or commands) are processed sequentially by a single instance for the entity id.

A consumer can read-validate-modify a value corresponding to the entity id and know that nobody else is changing the value during the read-validate-modify cycle. Therefore, there is no need for things like optimistic locking.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a note

@aludwiko

aludwiko commented Mar 7, 2025

Copy link
Copy Markdown
Contributor Author

I think I addressed all the comments.

@aludwiko

aludwiko commented Mar 7, 2025

Copy link
Copy Markdown
Contributor Author

One sample is failing, which is good because we must wait with the merge to the runtime release.

<4> Defines the query.
<5> Uses the new type `CustomerSummary` to return the result of the query.

NOTE: Some `TableUpdater` implementation might update the view model in a non-idempotent way. For example, the view model adds an element to the list. When the source of the changes is an Event Sourced Entity, Key Value Entity or another Akka service, the View component has a build-in deduplication mechanism to ensure that the same event is not processed twice. In other cases, you should add the deduplication mechanism in the `TableUpdater` implementation. See xref:dev-best-practices.adoc#message-deduplication[message deduplication] for some suggested solutions.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can backport this section to kalix docs (once approved).

----
include::example$event-sourced-counter-brokers/src/main/java/counter/application/CounterStore.java[tag=seq-tracking]
----
<1> A view model keeps track of the last processed sequence number.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"view model"? Is that from previous example?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed to read model


NOTE: Akka View component has a built-in support for sequence number tracking. {sample-base-url}/event-sourced-counter-brokers/src/test/java/counter/application/CounterByValueViewTest.java[CounterByValueViewTest] demonstrates how it can be verified.

Let's assume that we want to populate a view storage outside Akka ecosystem. To focus on the deduplication aspect, the following snippet shows the in-memory implementation of the `CounterStore`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good, but we still have the view example in the first two sections. Can we use this consumer example throughout all sections? You could maybe just use an interface CounterStore with get and set in the earlier sections?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could remove the View component, but I would need a CustomerStore, because that section is about idempotent updates and I would prefer not to confuse users with the counter used later for sth else.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, check now.

@octonato octonato left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some thoughts and suggestions.

[#message-deduplication]
== Message deduplication

In the realm of distributed systems, Akka embraces an at-least-once delivery guarantee, for components like Consumer or Views (view updaters). Message redeliveries are not just possible, but inevitable.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we are jumping too fast into saying that duplicates can be delivered and that's inevitable.

Instead, we should explain why message redelivery can happen and explain how at-least-once is guaranteed. Users need to understands the causes first, IMO.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, changed this paragraph a bit, but I don't want to explain how we implemented the message outbox pattern under the hood.

----
include::example$event-sourced-counter-brokers/src/main/java/counter/domain/CounterEvent.java[tag=events-enrichment]
----
<1> `ValueMultiplied` event contains not only delta information under `multiplier` field but also pre-calculated `currentValue` of the counter.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe newValue or updatedValue will better express that's the value after the event was applied.
The text here says 'pre-calculated', but we don't have this information when we look at the code.

On a side note, what is odd in this technique is that it's the same as ValueChanged(someNewValue). When applying this event, we don't need to know that it was a multiplication nor by how much. We just need to update the value.

The use case kind of helps here because the state is just a counter, but if you have a more complex model, you will have to add the state in the event each time.

Also, the entity itself doesn't need this. We are tailoring the event because of the consumers.

@aludwiko aludwiko Mar 14, 2025 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are tailoring the event because of the consumers.

Exactly, that's the real-life it's always a dance between what is required on the entity level (only delta) and how to improve the consumers' processing.

Also, it's not the case that it will be a full state in each event, that's an overuse of this technique. Maybe I will be more clear about it.


NOTE: When consuming from Akka components like Event Sourced Entity, Key Value Entity or another Akka service, the Akka runtime not only guarantees at-least-once delivery, but also the order of messages. Meaning that a series of duplicated messages might be redelivered but always in the same order as they were produced.

To ensure system integrity, consumers must be capable of handling duplicate messages gracefully. Effective deduplication is not just an optimization - it's a core architectural requirement, transforming the potential complexity of message redeliveries into a predictable and manageable system behavior.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To ensure system integrity, consumers must be capable of handling duplicate messages gracefully. Effective deduplication is not just an optimization - it's a core architectural requirement, transforming the potential complexity of message redeliveries into a predictable and manageable system behavior.
To ensure system integrity, consumers must be capable of handling duplicate messages gracefully. Effective deduplication is not just an optimization — it's a core architectural requirement, turning the challenge of message redeliveries into a structured and predictable system behavior.

Disclaimer, I asked AI to re-phrase this. 😄

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's funny, because the origin paragraph was also composed by AI, you have a better AI.

@aludwiko
aludwiko force-pushed the deduplication-docs branch from 5553080 to d0b1897 Compare March 14, 2025 09:00
small improvements

replacing sample for seq number tracking

another batch of improvements

improving deduplication impl

Update docs/src/modules/java/pages/dev-best-practices.adoc

Co-authored-by: Patrik Nordwall <patrik.nordwall@gmail.com>

small improvements

fixing antora warnings

additional batch of improvements
@aludwiko
aludwiko force-pushed the deduplication-docs branch from d0b1897 to 4d5752f Compare March 14, 2025 09:32
@aludwiko
aludwiko marked this pull request as ready for review March 14, 2025 09:32

@patriknw patriknw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@aludwiko
aludwiko merged commit a272331 into main Mar 14, 2025
@aludwiko
aludwiko deleted the deduplication-docs branch March 14, 2025 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation documentation related java-sdk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants