Repository navigation
feat: Compaction of session memory - #526
Conversation
* replace the history with one user message and one system message * how to make the actual summary of the history is up to the application, for example using an agent (LLM) to create a summary of the historical interactions
| var userMessages = new ArrayList<String>(); | ||
| var aiMessages = new ArrayList<String>(); | ||
|
|
||
| userMessages.add("What are the core components of Akka?"); |
There was a problem hiding this comment.
I wanted to try it with something realistic
| var history = componentClient | ||
| .forEventSourcedEntity(context().sessionId()) | ||
| .method(SessionMemoryEntity::getHistory) // <2> | ||
| .invoke(new SessionMemoryEntity.GetHistoryCmd(Optional.empty())); |
There was a problem hiding this comment.
Shouldn't we have 2 get methods?
- getHistory - all history
- getLimitedHistory(lastN: int)
| .forEventSourcedEntity(sessionId) | ||
| .method(SessionMemoryEntity::compactHistory) // <3> | ||
| .invoke(new SessionMemoryEntity.CompactionCmd( | ||
| new SessionMessage.UserMessage(now, summary.userMessage(), summary.userMessageTokens()), |
There was a problem hiding this comment.
I wonder if that always makes sense, to have the user message + ai message as the compaction result pair. My initial thought was that this should be just the AI message that collects and summarizes all the previous conversations in one output message.
There was a problem hiding this comment.
I'm not sure if the next interaction with the LLM would see any difference between one user message + one ai message vs only one message in the context. I thought it was best to preserve the user vs ai relation also in the compacted history.
Looking at the CompactionAgent example that I provided I feel like it preserves more original intent to make a summary of all questions and a separate summary of all answers.
There was a problem hiding this comment.
It feels to me that the logic should be only here.
The CompactionAgent should not fetch the history, but be more generic. It should receive a bunch text and summarize it.
As it is now, there is some logic in the agent and some logic in the consumer. The agent needs comply to some format that can then be used to build a command.
I think it will be better if the consumer takes the whole history and send it as a single string (user message) and receives back a summarized text (ai message).
But I see another issue as well.
If, in the meantime, other agents add more messages to the memory, the compaction will drop them.
There was a problem hiding this comment.
Maybe we should have this out of the box instead.
Something like a memory compaction rule.
The user defines the limit and which LLM they want to use for compaction, and we take care of the whole dance.
As such, we can lock the memory and avoid that other agents add extra messages while we are compacting it.
Other agents running concurrently will have to 'wait' before the compaction is done.
There was a problem hiding this comment.
ok, I can move the retrieval of the history to the consumer. Actually, is there an indication of that we are missing the possibility for an agent to perform a side effect after calling the LLM? Then the agent could be fully responsible for retrieving and updating the session memory.
If, in the meantime, other agents add more messages to the memory, the compaction will drop them.
Good point. We would have to include a sequence number, so that the compaction update relates to specific sequence number and any additional events that were added would be added again after the compaction event.
Maybe we should have this out of the box instead.
I feel like there could be many application specific requirements and concerns of creating a summary and therefore better to give the application freedom of when and how to compact.
Even if provided it, we would have challenges. We shouldn't call agents from inside an entity, and then we would have the concurrency problem anyway.
There was a problem hiding this comment.
We shouldn't call agents from inside an entity, and then we would have the concurrency problem anyway.
I don't see the agent being called from inside the entity.
My thought was that the consumer would lock the session, call the agent, change the memory and release the lock. Since it's a consumer, if it fails, it would retry and the session will be unlocked.
But now that I wrote it down, I realise how bad is this. A single Agent calling a LLM will be locking the session memory and we also have the consumer backoff in that case increasing the delay.
octonato
left a comment
There was a problem hiding this comment.
LGTM
There are many ways one can compact the memory. As an starting example this is doing a good job. I guess once users understand how the mechanism works they can explore other compaction techniques.
The most important is that once the SessionMemoryEntity receives a compaction command, doesn't matter how it was fabricated, the entity do now lose concurrent messages.
| .userMessage(concatenatedMessages) | ||
| .responseAs(Result.class) | ||
| .thenReply(); |
There was a problem hiding this comment.
Ok, I didn't remember this when I made my comment during the stand-up today.
There is no synthetic user message, but the user message is the full history concatenated, but what we add to the session as user message is what the LLM will add to the Result.userMessage.
I think this is better than a generic "Summarize previous conversation".
It's kind of synthetic conversation, but it will display nicely.
And, it's on the user to decide how they prefer to do this. We are just giving some guidance.
Also better to not ask for the tokens.
The token usage will be returned in the AI message and that's what counts. And there is no real user tokens, but only input token.
Anyway, that will change once we get #530 merged.
There was a problem hiding this comment.
We also need to concatenate the tool responses, but again, we do it after #530
There was a problem hiding this comment.
thanks, I'll merge this as first step
No description provided.