Skip to content

feat: status of a tool call in the ledger record - #2186

Merged
octonato merged 9 commits into
mainfrom
wip-tool-call-status-patriknw
Nov 10, 2026
Merged

octonato merged 9 commits into
mainfrom
wip-tool-call-status-patriknw

Conversation

@patriknw

@patriknw patriknw commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

SDK side of https://github.com/lightbend/akka-runtime/pull/6062


🤖

ToolCall in the public ledger records gains status, a ToolCall.Status with the values COMPLETED, FAILED, NOT_RUN, PENDING, NO_RESULT and UNSPECIFIED. An evaluator reads the tool calls of the interaction it judges through the ledger client. Before this change it could not tell a call without a result from a call with an empty result, or an error result from a result. The status is the value that the runtime records for each call. A status that this SDK does not know, and a record that an older runtime wrote, read as UNSPECIFIED. Code written against SDK 3.7 that constructs a ToolCall with four components still compiles, and that call has the status UNSPECIFIED.

InteractionRecord.transcript() shows the status of a call that did not complete: -> failed: TEXT, -> not run, -> pending or -> no result. A judge model can then tell a call that was not run from a call that returned nothing. A completed call reads as before. The javadoc of each status describes what the runtime records. A failed tool round records no results, so every call of such a round is NO_RESULT, also a call that completed. A tool that fails fails the round on a single reply and a streamed reply. An autonomous agent turns the failure into an error result instead, so that call is FAILED with the error text, and the other calls keep their results. The documentation example HallucinationJudgeEvaluator builds the reference text for its judge from the completed calls only, so the judge does not read an error text as a fact.

@github-actions github-actions Bot added the documentation documentation related label Nov 4, 2026
@octonato

octonato commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Only nits! 🎉


🤖 says...

[nit]
The COMPLETED javadoc leaves out that the runtime does not read the result text

akka-javasdk/src/main/java/akka/javasdk/ledger/ToolCall.java:35

The javadoc says only "a result that is not an error result". The runtime SPI scaladoc for Completed says more: a result that reports an error only in its text also reads as completed.

A function tool that catches an exception and returns "Error: not found" as a normal string records COMPLETED. The sample at samples/doc-snippets/src/main/java/com/example/evaluator/HallucinationJudgeEvaluator.java:81 then puts that text in the reference text as a fact.

[nit]
The status markers in the transcript can read the same as tool output

akka-javasdk/src/main/java/akka/javasdk/ledger/InteractionRecord.java:186

renderResult writes the status as bare words: failed: ..., not run, pending and no result.

A search tool that returns the text no result renders the same line as a call with the status NO_RESULT. A judge model that reads the transcript cannot tell the two apart. The same class renders its other placeholders in brackets: [image ...] at line 198 and [binary content] at line 202.

[nit]
No test checks that the four-component constructor gives UNSPECIFIED

akka-javasdk/src/main/java/akka/javasdk/ledger/ToolCall.java:73

akka-javasdk/src/test/scala/akka/javasdk/ledger/InteractionRecordSpec.scala and akka-javasdk-testkit/src/test/java/akka/javasdk/testkit/eval/Targets.java call the four-component constructor, but no test reads status() of the result.

The transcript renders COMPLETED and UNSPECIFIED the same way, so InteractionRecordTest.java does not catch it either. If ToolCall.java:74 used Status.COMPLETED, no test would fail, and the contract for code written against SDK 3.7 would break silently.

@patriknw
patriknw force-pushed the wip-tool-call-status-patriknw branch from e147fc8 to 0c8d1cf Compare November 9, 2026 15:07
@patriknw

patriknw commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

@octonato incorporated feedback, pls take another look

@octonato
octonato enabled auto-merge (squash) November 10, 2026 15:40
@octonato

Copy link
Copy Markdown
Contributor

@patriknw, approved, but there is a merge conflict

@patriknw
patriknw force-pushed the wip-tool-call-status-patriknw branch from 0c8d1cf to 3a4182d Compare November 10, 2026 17:41
@octonato
octonato merged commit 7810bca into main Nov 10, 2026
13 checks passed
@octonato
octonato deleted the wip-tool-call-status-patriknw branch November 10, 2026 17:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation documentation related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants