Skip to content

docs: replace LLM evaluation page with Evaluator examples - #2149

Merged
patriknw merged 1 commit into
mainfrom
wip-eval-docs-patriknw
Oct 29, 2026
Merged

patriknw merged 1 commit into
mainfrom
wip-eval-docs-patriknw

Conversation

@patriknw

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot added the documentation documentation related label Oct 28, 2026

@johanandren johanandren left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

🤖 review

Blocking

  1. Broken xref: examples.adoc:51 links to agents.adoc#_configuring_the_model, but that section has the explicit id [#model]. Use xref:sdk:agents.adoc#model[...] like the other pages do.
  2. Public API left without docs: ToxicityEvaluator, SummarizationEvaluator, HallucinationEvaluator (with their config keys and prompt ids) and EvaluationResult are still public and not deprecated. After this PR, no page documents them. samples/multi-agent EvaluationConsumer still uses them. Either deprecate them (here or in a stacked PR) or keep a short reference for them somewhere.

Should-fix

  1. nav.adoc removes llm_eval but does not add evaluators/examples.adoc. Add it as a child under Evaluators.
  2. HallucinationJudgeEvaluator: ToolCall.response() is empty when the ledger has no result for a call. A line like name: is still added, so referenceText.isEmpty() is false and the judge gets no real reference text. This likely gives a false "hallucinated", and it does not match callout <2>. Filter out blank responses before joining.
  3. Evaluation timeout: a stateless evaluation times out after 2 minutes by default. One judge call can take longer: response-timeout = 1m with max-retries = 2, and the prompts ask for step-by-step explanations. On timeout the runtime records a failed evaluation, but the model call keeps running and is still billed. Mention the timeout, and suggest a shorter judge timeout or a DurableEvaluator for slow judges.
  4. Sampling and throughput: all stateless evaluators share one trigger projection with low default parallelism. A slow judge limits throughput and delays the other evaluators. Say that sampling-ratio also bounds evaluation lag, not only cost. In the binding example, the summarization and hallucination evaluators have the largest prompts but no sampling.
  5. Unbounded prompt size: the hallucination evaluator joins all tool responses, and the summarization evaluator sends the whole input. The prompt can exceed the judge's context window or cost more than the original interaction, and the judge interaction stores the payload again in the ledger. Add a note to cap the text, since users will copy these samples as they are.
  6. "Reacting to a verdict in a workflow": a short workflow step snippet that branches on verdict.label() would help more than the pointer "in this way". The section says a direct call records nothing. If Verdict implemented EvaluationResult, direct calls would still show up in akka.agent.evaluation.result. That ties in with 2.

Nits

  • Comma splices before "see xref" in examples.adoc:34, :51 and concepts/evaluation.adoc:19. Use "; see".
  • :page-related: in examples.adoc uses bare targets. The parent evaluators.adoc uses xref:...[].
  • Callout <1> says "empty reply", but the code checks isBlank().
  • examples.adoc:109: if the reference text comes from inputText(), it is the same as the query. Rephrase.
  • The NOTE and the agentic-ai.adoc checklist say "agent or workflow must react", but the section only covers workflows.
  • HallucinationJudgeEvaluator: move the failed() || answer.isBlank() check above the toolCalls() stream, like the other two evaluators.
  • The session suffix -judge differs from -quality-judge in the sibling samples. -label-judge would match.
  • ToxicityJudgeEvaluatorIntegrationTest: check outcome() before evaluation().orElseThrow(), as callout <3> and the sibling test do.
  • Prompt typo copied from the built-in: "must be single word" is missing "a".
  • AGENTS.md could also list evaluators/examples.html.md.

Questions

  • Is the intent to replace the built-in evaluator agents with copy-paste samples?
  • Should samples/multi-agent EvaluationConsumer be updated or removed along with this change?
  • QualityJudge and JudgeDelegatingEvaluator are not included by any page. Should they be removed in a follow-up, now that LabelJudge exists?
  • FYI: no TestKit test can start in doc-snippets because of an existing type-name collision on main (com.example.ttl.ShoppingCartEvent$ItemAdded and ShoppingCartEntityWithNotifications$CartEvent$ItemAdded). CI only runs test-compile there, so the new test never runs. It passes when copied into a minimal project.

@patriknw

Copy link
Copy Markdown
Contributor Author
  1. nav.adoc removes llm_eval but does not add evaluators/examples.adoc. Add it as a child under Evaluators.

It did that, but I thought it became too deep nesting so I removed it from nav

Is the intent to replace the built-in evaluator agents with copy-paste samples?

yes

Should samples/multi-agent EvaluationConsumer be updated

That is part of #2127 so I didn't touch it here

@patriknw
patriknw force-pushed the wip-eval-docs-patriknw branch from d94d683 to 73ca321 Compare October 29, 2026 11:09
@patriknw

Copy link
Copy Markdown
Contributor Author

Incorporated feedback and created issue https://github.com/lightbend/akka-runtime/issues/6026

@patriknw
patriknw merged commit 71c6679 into main Oct 29, 2026
38 checks passed
@patriknw
patriknw deleted the wip-eval-docs-patriknw branch October 29, 2026 14:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation documentation related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants