Repository navigation
docs: replace LLM evaluation page with Evaluator examples - #2149
Merged
Merged
Conversation
johanandren
reviewed
Oct 28, 2026
johanandren
left a comment
Contributor
There was a problem hiding this comment.
Looks good
🤖 review
Blocking
- Broken xref:
examples.adoc:51links toagents.adoc#_configuring_the_model, but that section has the explicit id[#model]. Usexref:sdk:agents.adoc#model[...]like the other pages do. - Public API left without docs:
ToxicityEvaluator,SummarizationEvaluator,HallucinationEvaluator(with their config keys and prompt ids) andEvaluationResultare still public and not deprecated. After this PR, no page documents them.samples/multi-agentEvaluationConsumerstill uses them. Either deprecate them (here or in a stacked PR) or keep a short reference for them somewhere.
Should-fix
nav.adocremovesllm_evalbut does not addevaluators/examples.adoc. Add it as a child under Evaluators.HallucinationJudgeEvaluator:ToolCall.response()is empty when the ledger has no result for a call. A line likename:is still added, soreferenceText.isEmpty()is false and the judge gets no real reference text. This likely gives a false "hallucinated", and it does not match callout <2>. Filter out blank responses before joining.- Evaluation timeout: a stateless evaluation times out after 2 minutes by default. One judge call can take longer:
response-timeout = 1mwithmax-retries = 2, and the prompts ask for step-by-step explanations. On timeout the runtime records a failed evaluation, but the model call keeps running and is still billed. Mention the timeout, and suggest a shorter judge timeout or aDurableEvaluatorfor slow judges. - Sampling and throughput: all stateless evaluators share one trigger projection with low default parallelism. A slow judge limits throughput and delays the other evaluators. Say that
sampling-ratioalso bounds evaluation lag, not only cost. In the binding example, the summarization and hallucination evaluators have the largest prompts but no sampling. - Unbounded prompt size: the hallucination evaluator joins all tool responses, and the summarization evaluator sends the whole input. The prompt can exceed the judge's context window or cost more than the original interaction, and the judge interaction stores the payload again in the ledger. Add a note to cap the text, since users will copy these samples as they are.
- "Reacting to a verdict in a workflow": a short workflow step snippet that branches on
verdict.label()would help more than the pointer "in this way". The section says a direct call records nothing. IfVerdictimplementedEvaluationResult, direct calls would still show up inakka.agent.evaluation.result. That ties in with 2.
Nits
- Comma splices before "see xref" in
examples.adoc:34,:51andconcepts/evaluation.adoc:19. Use "; see". :page-related:inexamples.adocuses bare targets. The parentevaluators.adocusesxref:...[].- Callout <1> says "empty reply", but the code checks
isBlank(). examples.adoc:109: if the reference text comes frominputText(), it is the same as the query. Rephrase.- The NOTE and the
agentic-ai.adocchecklist say "agent or workflow must react", but the section only covers workflows. HallucinationJudgeEvaluator: move thefailed() || answer.isBlank()check above thetoolCalls()stream, like the other two evaluators.- The session suffix
-judgediffers from-quality-judgein the sibling samples.-label-judgewould match. ToxicityJudgeEvaluatorIntegrationTest: checkoutcome()beforeevaluation().orElseThrow(), as callout <3> and the sibling test do.- Prompt typo copied from the built-in: "must be single word" is missing "a".
AGENTS.mdcould also listevaluators/examples.html.md.
Questions
- Is the intent to replace the built-in evaluator agents with copy-paste samples?
- Should
samples/multi-agentEvaluationConsumerbe updated or removed along with this change? QualityJudgeandJudgeDelegatingEvaluatorare not included by any page. Should they be removed in a follow-up, now thatLabelJudgeexists?- FYI: no TestKit test can start in
doc-snippetsbecause of an existing type-name collision on main (com.example.ttl.ShoppingCartEvent$ItemAddedandShoppingCartEntityWithNotifications$CartEvent$ItemAdded). CI only runstest-compilethere, so the new test never runs. It passes when copied into a minimal project.
Contributor
Author
It did that, but I thought it became too deep nesting so I removed it from nav
yes
That is part of #2127 so I didn't touch it here |
patriknw
force-pushed
the
wip-eval-docs-patriknw
branch
from
October 29, 2026 11:09
d94d683 to
73ca321
Compare
Contributor
Author
|
Incorporated feedback and created issue https://github.com/lightbend/akka-runtime/issues/6026 |
johanandren
approved these changes
Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes https://github.com/lightbend/akka-runtime/issues/5871