Repository navigation
feat: add conversation memory - #457
Conversation
| @JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "@type") | ||
| @JsonSubTypes({ | ||
| @JsonSubTypes.Type(value = UserMessage.class, name = "UM"), | ||
| @JsonSubTypes.Type(value = AiMessage.class, name = "AIM")}) |
There was a problem hiding this comment.
not possible to use sealed interface and @TypeName in same way as we recommend for users?
There was a problem hiding this comment.
For some reason it was not working .. need to try it again and debug a bit further.
There was a problem hiding this comment.
could it be that we automatically registerTypeHints in the JsonSerializer for events, but this is ordinary nested objects and therefore ordinary Jackson stuff is needed
There was a problem hiding this comment.
@TypeName will not work in this case
| * @param sessionId The unique identifier for the conversation session | ||
| * @param message The content of the user message to add | ||
| */ | ||
| void addUserMessage(String sessionId, String message); |
There was a problem hiding this comment.
we should probably also track the agent id (ComponentId) that emitted the message
There was a problem hiding this comment.
Agree, will had.
Also, I have been wondering a bit if we won't need an actual agent id.. I mean, an identifier of the agent instance. Even if it's only internal to start. For instance, maybe when having two instances of the same agent working in parallel with the same session id.
| * Copyright (C) 2021-2024 Lightbend Inc. <https://www.lightbend.com> | ||
| */ | ||
|
|
||
| package akka.javasdk.agent.impl; |
There was a problem hiding this comment.
shouldn't this be in akka.javasdk.impl.agent package?
There was a problem hiding this comment.
most likely yes, good call 👍
| * The maximum number of entries in the history can be set dynamically with command setLimitedWindow. | ||
| */ | ||
| @ComponentId("akka-conversation-memory") | ||
| public class ConversationMemory extends EventSourcedEntity<History, Event> { |
There was a problem hiding this comment.
"If you like it, make it final"
| .thenReply(__ -> Done.done()); | ||
| } | ||
|
|
||
| public ReadOnlyEffect<History> getHistory() { |
There was a problem hiding this comment.
Would it perhaps be good for evolvability to have either an interface the public type here and keep the impl internal/package private?
| @JsonSubTypes({ | ||
| @JsonSubTypes.Type(value = UserMessage.class, name = "UM"), | ||
| @JsonSubTypes.Type(value = AiMessage.class, name = "AIM")}) | ||
| public interface Message { |
There was a problem hiding this comment.
Message sounds a bit too open to me, even when it is in the agent namespace. Could it be more specific?
Also, sealed in Java can work with multiple classes from separate files, you just have to list the allowed subtypes here.
public sealed class Shape
permits Circle, Square, Rectangle {
}There was a problem hiding this comment.
ExchangeMessage? ConversationMessage? I'm not too confident on the current naming as well to be honest.. I will think a bit more.
|
|
||
| import com.fasterxml.jackson.annotation.JsonTypeInfo; | ||
|
|
||
| @JsonTypeInfo(use = JsonTypeInfo.Id.CLASS) |
There was a problem hiding this comment.
If we need to use Jackson type info we shouldn't use the concrete class but a short identifier
| public static AiMessage of(String content) { | ||
| return new AiMessage(new MessageContent.TextContent(content)); |
There was a problem hiding this comment.
| public static AiMessage of(String content) { | |
| return new AiMessage(new MessageContent.TextContent(content)); | |
| public static AiMessage of(String text) { | |
| return new AiMessage(new MessageContent.TextContent(text)); |
| public sealed interface Event { | ||
|
|
||
| @TypeName("limited-window-set") | ||
| record LimitedWindowSet(int maxSize) implements Event { |
There was a problem hiding this comment.
I wonder if this shouldn't be more generic, like HistoryInitiated?
There was a problem hiding this comment.
My idea here is that you could change the Window size also later after initiating.. thus the focus on the limit and not exactly it being an initialization.
| } | ||
|
|
||
| public Effect<Done> delete() { | ||
| return effects() |
There was a problem hiding this comment.
Check if not deleted already
* feat: system message from template * fixing impl * small improvemnts * test * reverting pom.xml * reverting
7d6f89d to
a264f65
Compare
efgpinto
left a comment
There was a problem hiding this comment.
Addressed some comments and update to actually worl. Still need to check a few things around naming and API.
| public sealed interface Event { | ||
|
|
||
| @TypeName("limited-window-set") | ||
| record LimitedWindowSet(int maxSize) implements Event { |
There was a problem hiding this comment.
My idea here is that you could change the Window size also later after initiating.. thus the focus on the limit and not exactly it being an initialization.
| * The maximum number of entries in the history can be set dynamically with command setLimitedWindow. | ||
| */ | ||
| @ComponentId("akka-conversation-memory") | ||
| public class ConversationMemory extends EventSourcedEntity<History, Event> { |
| } | ||
|
|
||
| public Effect<Done> delete() { | ||
| return effects() |
| * @param sessionId The unique identifier for the conversation session | ||
| * @param message The content of the user message to add | ||
| */ | ||
| void addUserMessage(String sessionId, String message); |
There was a problem hiding this comment.
Agree, will had.
Also, I have been wondering a bit if we won't need an actual agent id.. I mean, an identifier of the agent instance. Even if it's only internal to start. For instance, maybe when having two instances of the same agent working in parallel with the same session id.
| @JsonSubTypes({ | ||
| @JsonSubTypes.Type(value = UserMessage.class, name = "UM"), | ||
| @JsonSubTypes.Type(value = AiMessage.class, name = "AIM")}) | ||
| public interface Message { |
There was a problem hiding this comment.
ExchangeMessage? ConversationMessage? I'm not too confident on the current naming as well to be honest.. I will think a bit more.
| * {@link akka.javasdk.client.ComponentClient} can be used to interact directly with this entity. | ||
| */ | ||
| @ComponentId("akka-conversation-memory") | ||
| public final class ConversationMemory extends EventSourcedEntity<ConversationHistory, Event> { |
There was a problem hiding this comment.
I wonder if there is a better way to still hide the implementation.. Maybe only doing the conversation from ConversationHistoryImpl to ConversationHistory on getHistory instead? But then the Impl specific one will be on emptyState and here..
patriknw
left a comment
There was a problem hiding this comment.
looking good, some comments can be considered for follow up instead of here if you like
| @JsonSubTypes.Type(value = UserMessage.class, name = "UM"), | ||
| @JsonSubTypes.Type(value = AiMessage.class, name = "AIM")}) | ||
| public sealed interface ConversationMessage permits AiMessage, UserMessage { | ||
|
|
There was a problem hiding this comment.
I think you can define the AiMessage and UserMessage records inside here, to keep them all together
There was a problem hiding this comment.
I was a bit afraid they would grow out of hand.. but maybe not. I also prefer to have them here tbh. will update
| @Override | ||
| public ConversationHistory emptyState() { | ||
| // FIXME: load default from config? | ||
| return new ConversationHistoryImpl(1000, new LinkedList<>()); |
There was a problem hiding this comment.
Maybe we should base the removal of old entries on content size instead of number of messages? In the end we have a hard limit of how much we can transfer over Akka cluster for a message (the ConversationHistory response). The number of LLM tokens might also be a driving factor, and that corresponds more closely to content size than to number of messages.
There was a problem hiding this comment.
Yeah, my idea was to start with the number of messages, just because it's the easiest to reason about, even for the user... But I think it makes sense to support other options. What I had thought about was a token based limit but what you mention about the content size makes sense.. I think I can work on that right after this as follow up if you agree.
There was a problem hiding this comment.
Sounds good as follow up.
Not sure we should provide too many options (in the beginning). Content size is more useful, in my opinion, so I'd start with that one.
I also thought about tokens, but that depends on the model afaik, so more difficult.
| val spiModelProvider = toSpiModelProvider(req.modelProvider) | ||
| val metadata = MetadataImpl.toSpi(req.replyMetadata) | ||
|
|
||
| // FIXME refactor to persist both user and ai message on transform method |
There was a problem hiding this comment.
yes, that is not only more efficient, but also important for the error case, when we don't want to save the user message
| * @param sessionId The unique identifier for the conversation session | ||
| * @param message The content of the user message to add | ||
| */ | ||
| void addUserMessage(String componentId, String sessionId, String message); |
There was a problem hiding this comment.
Since the purpose of this interface is to have custom implementations of the memory used by an Agent I don't think we need to define anything more than what the agent implementation requires, which is addInteraction and getFullHistory.
Follow up would be to remove addUserMessage and addAiMessage, when we can use addInteraction.
There was a problem hiding this comment.
When I was building the ask akka sample, one thing that became clear to me when trying it out, is that we only want to add the user message if we have an AI message.
That's why I introduced there a single addExchange method.
The call to the AI might fail and if that happens, we don't want to save the user message.
There was a problem hiding this comment.
yes, that is in the follow up plan
|
|
||
| } | ||
| public Effect<Done> addInteraction(AddInteractionCmd cmd) { | ||
| log.debug("Adding interaction: user={}, ai={}", cmd.userMessage, cmd.aiMessage); |
There was a problem hiding this comment.
We must not log the content of user and ai messages, since those may contain sensitive information that shouldn't end up in logging infra.
|
Ok, I think this would be good to go unless someone objects. Things to follow up:
|
Refs https://github.com/lightbend/kalix-runtime/issues/3831
Still very draft and haven't really tested ergonomics. Opening for visibility.