Repository navigation
feat: status of a tool call in the ledger record - #2186
Conversation
|
Only nits! 🎉 🤖 says... [nit]
The javadoc says only "a result that is not an error result". The runtime SPI scaladoc for A function tool that catches an exception and returns [nit]
A search tool that returns the text [nit]
The transcript renders |
e147fc8 to
0c8d1cf
Compare
|
@octonato incorporated feedback, pls take another look |
|
@patriknw, approved, but there is a merge conflict |
0c8d1cf to
3a4182d
Compare
SDK side of https://github.com/lightbend/akka-runtime/pull/6062
🤖
ToolCallin the public ledger records gainsstatus, aToolCall.Statuswith the valuesCOMPLETED,FAILED,NOT_RUN,PENDING,NO_RESULTandUNSPECIFIED. An evaluator reads the tool calls of the interaction it judges through the ledger client. Before this change it could not tell a call without a result from a call with an empty result, or an error result from a result. The status is the value that the runtime records for each call. A status that this SDK does not know, and a record that an older runtime wrote, read asUNSPECIFIED. Code written against SDK 3.7 that constructs aToolCallwith four components still compiles, and that call has the statusUNSPECIFIED.InteractionRecord.transcript()shows the status of a call that did not complete:-> failed: TEXT,-> not run,-> pendingor-> no result. A judge model can then tell a call that was not run from a call that returned nothing. A completed call reads as before. The javadoc of each status describes what the runtime records. A failed tool round records no results, so every call of such a round isNO_RESULT, also a call that completed. A tool that fails fails the round on a single reply and a streamed reply. An autonomous agent turns the failure into an error result instead, so that call isFAILEDwith the error text, and the other calls keep their results. The documentation exampleHallucinationJudgeEvaluatorbuilds the reference text for its judge from the completed calls only, so the judge does not read an error text as a fact.