Skip to content

feat: add conversation memory - #457

Merged
patriknw merged 10 commits into
agent-devfrom
add-conversation-memory
May 23, 2025
Merged

patriknw merged 10 commits into
agent-devfrom
add-conversation-memory

Conversation

@efgpinto

Copy link
Copy Markdown
Contributor

Refs https://github.com/lightbend/kalix-runtime/issues/3831

Still very draft and haven't really tested ergonomics. Opening for visibility.

@github-actions github-actions Bot added documentation documentation related java-sdk labels May 19, 2025
@efgpinto
efgpinto changed the base branch from main to agent-dev May 19, 2025 23:15
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "@type")
@JsonSubTypes({
@JsonSubTypes.Type(value = UserMessage.class, name = "UM"),
@JsonSubTypes.Type(value = AiMessage.class, name = "AIM")})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not possible to use sealed interface and @TypeName in same way as we recommend for users?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For some reason it was not working .. need to try it again and debug a bit further.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could it be that we automatically registerTypeHints in the JsonSerializer for events, but this is ordinary nested objects and therefore ordinary Jackson stuff is needed

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TypeName will not work in this case

* @param sessionId The unique identifier for the conversation session
* @param message The content of the user message to add
*/
void addUserMessage(String sessionId, String message);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should probably also track the agent id (ComponentId) that emitted the message

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, will had.

Also, I have been wondering a bit if we won't need an actual agent id.. I mean, an identifier of the agent instance. Even if it's only internal to start. For instance, maybe when having two instances of the same agent working in parallel with the same session id.

* Copyright (C) 2021-2024 Lightbend Inc. <https://www.lightbend.com>
*/

package akka.javasdk.agent.impl;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this be in akka.javasdk.impl.agent package?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

most likely yes, good call 👍

* The maximum number of entries in the history can be set dynamically with command setLimitedWindow.
*/
@ComponentId("akka-conversation-memory")
public class ConversationMemory extends EventSourcedEntity<History, Event> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"If you like it, make it final"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tks, updated

.thenReply(__ -> Done.done());
}

public ReadOnlyEffect<History> getHistory() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it perhaps be good for evolvability to have either an interface the public type here and keep the impl internal/package private?

@JsonSubTypes({
@JsonSubTypes.Type(value = UserMessage.class, name = "UM"),
@JsonSubTypes.Type(value = AiMessage.class, name = "AIM")})
public interface Message {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Message sounds a bit too open to me, even when it is in the agent namespace. Could it be more specific?

Also, sealed in Java can work with multiple classes from separate files, you just have to list the allowed subtypes here.

public sealed class Shape
    permits Circle, Square, Rectangle {
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ExchangeMessage? ConversationMessage? I'm not too confident on the current naming as well to be honest.. I will think a bit more.


import com.fasterxml.jackson.annotation.JsonTypeInfo;

@JsonTypeInfo(use = JsonTypeInfo.Id.CLASS)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we need to use Jackson type info we shouldn't use the concrete class but a short identifier

Comment on lines +12 to +13
public static AiMessage of(String content) {
return new AiMessage(new MessageContent.TextContent(content));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
public static AiMessage of(String content) {
return new AiMessage(new MessageContent.TextContent(content));
public static AiMessage of(String text) {
return new AiMessage(new MessageContent.TextContent(text));

public sealed interface Event {

@TypeName("limited-window-set")
record LimitedWindowSet(int maxSize) implements Event {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this shouldn't be more generic, like HistoryInitiated?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My idea here is that you could change the Window size also later after initiating.. thus the focus on the limit and not exactly it being an initialization.

}

public Effect<Done> delete() {
return effects()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check if not deleted already

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tks, added

aludwiko and others added 5 commits May 21, 2025 22:15
@efgpinto
efgpinto force-pushed the add-conversation-memory branch from 7d6f89d to a264f65 Compare May 21, 2025 22:50

@efgpinto efgpinto left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed some comments and update to actually worl. Still need to check a few things around naming and API.

public sealed interface Event {

@TypeName("limited-window-set")
record LimitedWindowSet(int maxSize) implements Event {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My idea here is that you could change the Window size also later after initiating.. thus the focus on the limit and not exactly it being an initialization.

* The maximum number of entries in the history can be set dynamically with command setLimitedWindow.
*/
@ComponentId("akka-conversation-memory")
public class ConversationMemory extends EventSourcedEntity<History, Event> {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tks, updated

}

public Effect<Done> delete() {
return effects()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tks, added

* @param sessionId The unique identifier for the conversation session
* @param message The content of the user message to add
*/
void addUserMessage(String sessionId, String message);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, will had.

Also, I have been wondering a bit if we won't need an actual agent id.. I mean, an identifier of the agent instance. Even if it's only internal to start. For instance, maybe when having two instances of the same agent working in parallel with the same session id.

@JsonSubTypes({
@JsonSubTypes.Type(value = UserMessage.class, name = "UM"),
@JsonSubTypes.Type(value = AiMessage.class, name = "AIM")})
public interface Message {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ExchangeMessage? ConversationMessage? I'm not too confident on the current naming as well to be honest.. I will think a bit more.

* {@link akka.javasdk.client.ComponentClient} can be used to interact directly with this entity.
*/
@ComponentId("akka-conversation-memory")
public final class ConversationMemory extends EventSourcedEntity<ConversationHistory, Event> {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if there is a better way to still hide the implementation.. Maybe only doing the conversation from ConversationHistoryImpl to ConversationHistory on getHistory instead? But then the Impl specific one will be on emptyState and here..

@patriknw patriknw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking good, some comments can be considered for follow up instead of here if you like

@JsonSubTypes.Type(value = UserMessage.class, name = "UM"),
@JsonSubTypes.Type(value = AiMessage.class, name = "AIM")})
public sealed interface ConversationMessage permits AiMessage, UserMessage {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can define the AiMessage and UserMessage records inside here, to keep them all together

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was a bit afraid they would grow out of hand.. but maybe not. I also prefer to have them here tbh. will update

@Override
public ConversationHistory emptyState() {
// FIXME: load default from config?
return new ConversationHistoryImpl(1000, new LinkedList<>());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should base the removal of old entries on content size instead of number of messages? In the end we have a hard limit of how much we can transfer over Akka cluster for a message (the ConversationHistory response). The number of LLM tokens might also be a driving factor, and that corresponds more closely to content size than to number of messages.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, my idea was to start with the number of messages, just because it's the easiest to reason about, even for the user... But I think it makes sense to support other options. What I had thought about was a token based limit but what you mention about the content size makes sense.. I think I can work on that right after this as follow up if you agree.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good as follow up.
Not sure we should provide too many options (in the beginning). Content size is more useful, in my opinion, so I'd start with that one.

I also thought about tokens, but that depends on the model afaik, so more difficult.

val spiModelProvider = toSpiModelProvider(req.modelProvider)
val metadata = MetadataImpl.toSpi(req.replyMetadata)

// FIXME refactor to persist both user and ai message on transform method

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, that is not only more efficient, but also important for the error case, when we don't want to save the user message

@patriknw patriknw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking good

* @param sessionId The unique identifier for the conversation session
* @param message The content of the user message to add
*/
void addUserMessage(String componentId, String sessionId, String message);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the purpose of this interface is to have custom implementations of the memory used by an Agent I don't think we need to define anything more than what the agent implementation requires, which is addInteraction and getFullHistory.

Follow up would be to remove addUserMessage and addAiMessage, when we can use addInteraction.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I was building the ask akka sample, one thing that became clear to me when trying it out, is that we only want to add the user message if we have an AI message.

That's why I introduced there a single addExchange method.

The call to the AI might fail and if that happens, we don't want to save the user message.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, that is in the follow up plan


}
public Effect<Done> addInteraction(AddInteractionCmd cmd) {
log.debug("Adding interaction: user={}, ai={}", cmd.userMessage, cmd.aiMessage);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We must not log the content of user and ai messages, since those may contain sensitive information that shouldn't end up in logging infra.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch, tks

@github-actions github-actions Bot removed the documentation documentation related label May 23, 2025
@efgpinto
efgpinto marked this pull request as ready for review May 23, 2025 09:50
@efgpinto

Copy link
Copy Markdown
Contributor Author

Ok, I think this would be good to go unless someone objects.

Things to follow up:

  • refactor to store both user and ai message (requires change in runtime)
  • allow for custom implementation
  • different strategy for compaction (token or size based)

@patriknw patriknw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@patriknw
patriknw merged commit 247f126 into agent-dev May 23, 2025
@patriknw
patriknw deleted the add-conversation-memory branch May 23, 2025 11:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants