Repository navigation
feat: standalone tool support - #568
Conversation
| public record ToolCallRequest(String name, String arguments) { | ||
| } | ||
|
|
||
| public record AiAnswer(String message, List<ToolCallRequest> requests) { |
There was a problem hiding this comment.
I had to extend this with AiAnswer and ToolCallRequest to be able to support tool call.
It turns out that in order to really simulate how a model will react, users will need to have a better understanding of it. For example, will the model return one single ToolCallRequest or many? In which situation will the model ask for parallel tool call?
This will make the mocking harder.
There was a problem hiding this comment.
Btw, the reason I'm called it AiAnswer is to avoid name clash with LC4J. Users could wrongly import classes from LC4J.
And I realise now that ToolCallRequest is also being used by use in the SessionMemory and can be wrongly imported as well.
| if (chatMessage instanceof UserMessage userMessage) { | ||
| return userMessage.singleText(); | ||
| } else { | ||
| ToolExecutionResultMessage toolExecutionResultMessage = (ToolExecutionResultMessage) chatMessage; |
There was a problem hiding this comment.
There are two kinds of messages that can be sent to LLM, the orginal user message or the response from a tool.
We should treat both as input.
| public void mockResponseForExactMatch(String response) { | ||
| Predicate<String> predicate = (value) -> value.equals(response); | ||
| mockResponse(predicate, response); | ||
| } |
There was a problem hiding this comment.
Personally, I find it non-intuitive and I'm not happy to have it in public API.
Looking for ideas to make it easier.
The goal of this method is to add a predicate that returns the response if the input matches that response. Ok, not hard to understand the code.
The question is why we need this?
The reason is, a tool will respond with for example "abc". This will be an input to the LLM. I want the LLM to answer back with "abc".
Maybe instead of adding such a non-intuitive method, it will be better to add a type so we can have more clear predicates. If will be also easier if we could use the input message to fabricate a response.
Something like a Predicate and a Function<Input, Output> where the input is either a UserMessage or a ToolResponse and the output is an AiAnwser.
We will be doubling the underlying model, but might be necessary to cover all aspects of tool calling.
| override def types: Array[Type] = | ||
| method.getGenericParameterTypes | ||
|
|
||
| override def invoke(obj: Any, args: Array[Any]): Any = { |
There was a problem hiding this comment.
@aludwiko, I know I said I would preserve this change, but it's not possible.
The main reason is that for some tool calling, we won't have an instance at all. For example, when calling a Component, we will have to use the class to build a method ref and then call it as we do inside the component client.
We also want to add tools by passing the .class only. When it comes the moment to initialize it we will need to either use a default constructor or use the DependencyProvider to create an instance. It will be better to have this done inside the FunctionToolInvoker to shield away the specifics of object initialization.
There was a problem hiding this comment.
Yeah, I see that it should work as expected with tracing, because the agent used to create the tools lambda will have the proper context in place.
6303429 to
29cfb7f
Compare
There was a problem hiding this comment.
The changes in TestModelProvider are more impactful than I originally intended.
But we need to extend it to be able to mock some more advanced interactions with tooling.
I kept the previous mockResponse methods because of the convenience.
Again, not the main topic of this PR. I think we need to brainstorm and re-work the TestModelProvider in another PR. The changes here are sufficient to test tool for now.
Note that I'm introducing some types and trying to avoid conflicts and wrong imports with LC4J and memory session types.
Alternatively, we could make those new types not part of the testkit, but prod code. And they would become the types for the SessionMemory public API, but internally we save the data using another type in order to have some separation between public types and persistent representation.
There was a problem hiding this comment.
good idea, made some suggestions here: https://github.com/akka/akka-sdk/pull/568/files#r2144509511
There was a problem hiding this comment.
hmm, the types are quite different.
The session memory message have timestamp, original agent id (component id), input and output tokens.
Still, improving the TestModelProvider is needed.
In follow up PRs.
| new TestModelProvider.AiResponse( | ||
| new TestModelProvider.ToolInvocationRequest("getDateOfToday", ""))); |
There was a problem hiding this comment.
maybe we could extract that to some static factory method:
toolInvocationRequest("","")
There was a problem hiding this comment.
Sure, but I would like to improve the TestModelProvider in follow-up PRs. This is not the main topic of this current PR.
| new TestModelProvider.ToolInvocationRequest("getDateOfToday", ""))); | ||
|
|
||
| // when receiving the date back, model asks for calling getWeather | ||
| testModelProvider.mockResponseToToolResult( |
There was a problem hiding this comment.
maybe we could make it more user-friendly with a fluent builder pattern:
testModelProvider.when(result -> result.content().equals("2025-01-01"))
.thenReplyWithToolRequest(...)
and to cover the one below:
testModelProvider.when(result -> result.content().startsWith("The weather is"))
.thenReplyWithResult()
There was a problem hiding this comment.
we could then extend it with common patterns:
testModelProvider.whenToolResultEquals("2025-01-01").then..
testModelProvider.whenToolResultContain("2025-01-01").then..
There was a problem hiding this comment.
ok, I see that we need when for 2 main branches, when we are mocking the tool result, and when we are mocking based on user+system message 🤔
There was a problem hiding this comment.
BTW we shouldn't solve that in that PR.
There was a problem hiding this comment.
good idea, made some suggestions here: https://github.com/akka/akka-sdk/pull/568/files#r2144509511
No description provided.