Skip to content

feat: Compaction of session memory - #526

Merged
patriknw merged 5 commits into
agent-devfrom
wip-compact-memory-patriknw
Jun 10, 2025
Merged

patriknw merged 5 commits into
agent-devfrom
wip-compact-memory-patriknw

Conversation

@patriknw

@patriknw patriknw commented Jun 5, 2025

Copy link
Copy Markdown
Contributor

No description provided.

patriknw added 2 commits June 5, 2025 13:32
* replace the history with one user message and one system message
* how to make the actual summary of the history is up to the application,
  for example using an agent (LLM) to create a summary of the historical
  interactions
@github-actions github-actions Bot added documentation documentation related java-sdk labels Jun 5, 2025
var userMessages = new ArrayList<String>();
var aiMessages = new ArrayList<String>();

userMessages.add("What are the core components of Akka?");

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to try it with something realistic

var history = componentClient
.forEventSourcedEntity(context().sessionId())
.method(SessionMemoryEntity::getHistory) // <2>
.invoke(new SessionMemoryEntity.GetHistoryCmd(Optional.empty()));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we have 2 get methods?

  • getHistory - all history
  • getLimitedHistory(lastN: int)

.forEventSourcedEntity(sessionId)
.method(SessionMemoryEntity::compactHistory) // <3>
.invoke(new SessionMemoryEntity.CompactionCmd(
new SessionMessage.UserMessage(now, summary.userMessage(), summary.userMessageTokens()),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if that always makes sense, to have the user message + ai message as the compaction result pair. My initial thought was that this should be just the AI message that collects and summarizes all the previous conversations in one output message.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if the next interaction with the LLM would see any difference between one user message + one ai message vs only one message in the context. I thought it was best to preserve the user vs ai relation also in the compacted history.

Looking at the CompactionAgent example that I provided I feel like it preserves more original intent to make a summary of all questions and a separate summary of all answers.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It feels to me that the logic should be only here.
The CompactionAgent should not fetch the history, but be more generic. It should receive a bunch text and summarize it.

As it is now, there is some logic in the agent and some logic in the consumer. The agent needs comply to some format that can then be used to build a command.

I think it will be better if the consumer takes the whole history and send it as a single string (user message) and receives back a summarized text (ai message).

But I see another issue as well.
If, in the meantime, other agents add more messages to the memory, the compaction will drop them.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should have this out of the box instead.
Something like a memory compaction rule.
The user defines the limit and which LLM they want to use for compaction, and we take care of the whole dance.

As such, we can lock the memory and avoid that other agents add extra messages while we are compacting it.

Other agents running concurrently will have to 'wait' before the compaction is done.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, I can move the retrieval of the history to the consumer. Actually, is there an indication of that we are missing the possibility for an agent to perform a side effect after calling the LLM? Then the agent could be fully responsible for retrieving and updating the session memory.

If, in the meantime, other agents add more messages to the memory, the compaction will drop them.

Good point. We would have to include a sequence number, so that the compaction update relates to specific sequence number and any additional events that were added would be added again after the compaction event.

Maybe we should have this out of the box instead.

I feel like there could be many application specific requirements and concerns of creating a summary and therefore better to give the application freedom of when and how to compact.

Even if provided it, we would have challenges. We shouldn't call agents from inside an entity, and then we would have the concurrency problem anyway.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated: 413f599

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't call agents from inside an entity, and then we would have the concurrency problem anyway.

I don't see the agent being called from inside the entity.

My thought was that the consumer would lock the session, call the agent, change the memory and release the lock. Since it's a consumer, if it fails, it would retry and the session will be unlocked.

But now that I wrote it down, I realise how bad is this. A single Agent calling a LLM will be locking the session memory and we also have the consumer backoff in that case increasing the delay.

@octonato octonato left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

There are many ways one can compact the memory. As an starting example this is doing a good job. I guess once users understand how the mechanism works they can explore other compaction techniques.

The most important is that once the SessionMemoryEntity receives a compaction command, doesn't matter how it was fabricated, the entity do now lose concurrent messages.

Comment on lines +67 to +69
.userMessage(concatenatedMessages)
.responseAs(Result.class)
.thenReply();

@octonato octonato Jun 10, 2025 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I didn't remember this when I made my comment during the stand-up today.

There is no synthetic user message, but the user message is the full history concatenated, but what we add to the session as user message is what the LLM will add to the Result.userMessage.

I think this is better than a generic "Summarize previous conversation".
It's kind of synthetic conversation, but it will display nicely.

And, it's on the user to decide how they prefer to do this. We are just giving some guidance.

Also better to not ask for the tokens.
The token usage will be returned in the AI message and that's what counts. And there is no real user tokens, but only input token.

Anyway, that will change once we get #530 merged.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also need to concatenate the tool responses, but again, we do it after #530

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, I'll merge this as first step

@patriknw
patriknw merged commit 6bec237 into agent-dev Jun 10, 2025
@patriknw
patriknw deleted the wip-compact-memory-patriknw branch June 10, 2025 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation documentation related java-sdk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants