Skip to content

feat: standalone tool support - #568

Merged
octonato merged 1 commit into
agent-devfrom
octonato/standalone-tool-support
Jun 13, 2025
Merged

octonato merged 1 commit into
agent-devfrom
octonato/standalone-tool-support

Conversation

@octonato

Copy link
Copy Markdown
Contributor

No description provided.

public record ToolCallRequest(String name, String arguments) {
}

public record AiAnswer(String message, List<ToolCallRequest> requests) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to extend this with AiAnswer and ToolCallRequest to be able to support tool call.

It turns out that in order to really simulate how a model will react, users will need to have a better understanding of it. For example, will the model return one single ToolCallRequest or many? In which situation will the model ask for parallel tool call?

This will make the mocking harder.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw, the reason I'm called it AiAnswer is to avoid name clash with LC4J. Users could wrongly import classes from LC4J.

And I realise now that ToolCallRequest is also being used by use in the SessionMemory and can be wrongly imported as well.

if (chatMessage instanceof UserMessage userMessage) {
return userMessage.singleText();
} else {
ToolExecutionResultMessage toolExecutionResultMessage = (ToolExecutionResultMessage) chatMessage;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two kinds of messages that can be sent to LLM, the orginal user message or the response from a tool.

We should treat both as input.

Comment on lines +157 to +183
public void mockResponseForExactMatch(String response) {
Predicate<String> predicate = (value) -> value.equals(response);
mockResponse(predicate, response);
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personally, I find it non-intuitive and I'm not happy to have it in public API.

Looking for ideas to make it easier.

The goal of this method is to add a predicate that returns the response if the input matches that response. Ok, not hard to understand the code.

The question is why we need this?
The reason is, a tool will respond with for example "abc". This will be an input to the LLM. I want the LLM to answer back with "abc".

Maybe instead of adding such a non-intuitive method, it will be better to add a type so we can have more clear predicates. If will be also easier if we could use the input message to fabricate a response.

Something like a Predicate and a Function<Input, Output> where the input is either a UserMessage or a ToolResponse and the output is an AiAnwser.

We will be doubling the underlying model, but might be necessary to cover all aspects of tool calling.

override def types: Array[Type] =
method.getGenericParameterTypes

override def invoke(obj: Any, args: Array[Any]): Any = {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aludwiko, I know I said I would preserve this change, but it's not possible.

The main reason is that for some tool calling, we won't have an instance at all. For example, when calling a Component, we will have to use the class to build a method ref and then call it as we do inside the component client.

We also want to add tools by passing the .class only. When it comes the moment to initialize it we will need to either use a default constructor or use the DependencyProvider to create an instance. It will be better to have this done inside the FunctionToolInvoker to shield away the specifics of object initialization.

@aludwiko aludwiko Jun 13, 2025 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I see that it should work as expected with tracing, because the agent used to create the tools lambda will have the proper context in place.

@octonato
octonato force-pushed the octonato/standalone-tool-support branch from 6303429 to 29cfb7f Compare June 13, 2025 08:25
@octonato
octonato marked this pull request as ready for review June 13, 2025 08:25

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in TestModelProvider are more impactful than I originally intended.

But we need to extend it to be able to mock some more advanced interactions with tooling.

I kept the previous mockResponse methods because of the convenience.

Again, not the main topic of this PR. I think we need to brainstorm and re-work the TestModelProvider in another PR. The changes here are sufficient to test tool for now.

Note that I'm introducing some types and trying to avoid conflicts and wrong imports with LC4J and memory session types.

Alternatively, we could make those new types not part of the testkit, but prod code. And they would become the types for the SessionMemory public API, but internally we save the data using another type in order to have some separation between public types and persistent representation.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea, made some suggestions here: https://github.com/akka/akka-sdk/pull/568/files#r2144509511

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, the types are quite different.
The session memory message have timestamp, original agent id (component id), input and output tokens.

Still, improving the TestModelProvider is needed.
In follow up PRs.

Comment on lines +99 to +100
new TestModelProvider.AiResponse(
new TestModelProvider.ToolInvocationRequest("getDateOfToday", "")));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we could extract that to some static factory method:

toolInvocationRequest("","")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, but I would like to improve the TestModelProvider in follow-up PRs. This is not the main topic of this current PR.

new TestModelProvider.ToolInvocationRequest("getDateOfToday", "")));

// when receiving the date back, model asks for calling getWeather
testModelProvider.mockResponseToToolResult(

@aludwiko aludwiko Jun 13, 2025 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we could make it more user-friendly with a fluent builder pattern:

testModelProvider.when(result -> result.content().equals("2025-01-01"))
                               .thenReplyWithToolRequest(...)

and to cover the one below:

testModelProvider.when(result -> result.content().startsWith("The weather is"))
                               .thenReplyWithResult()

@aludwiko aludwiko Jun 13, 2025 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could then extend it with common patterns:

testModelProvider.whenToolResultEquals("2025-01-01").then..
testModelProvider.whenToolResultContain("2025-01-01").then..

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, I see that we need when for 2 main branches, when we are mocking the tool result, and when we are mocking based on user+system message 🤔

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW we shouldn't solve that in that PR.

@aludwiko aludwiko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea, made some suggestions here: https://github.com/akka/akka-sdk/pull/568/files#r2144509511

@octonato
octonato merged commit cc3c434 into agent-dev Jun 13, 2025
@octonato
octonato deleted the octonato/standalone-tool-support branch June 13, 2025 08:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants