Skip to content

feat: evalkit run cases multiple times - #1979

Merged
patriknw merged 6 commits into
mainfrom
evalkit-repeat-runs
Oct 6, 2026
Merged

patriknw merged 6 commits into
mainfrom
evalkit-repeat-runs

Conversation

@aludwiko

@aludwiko aludwiko commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot added the documentation documentation related label Oct 5, 2026

| `Gate.allCasesShouldPass()`
| No case has a failed result. This is the gate that applies when none is given.
| No turn has a failed result. This is the gate that applies when none is given.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I struggling with the decision if this should be a turn or an attempt ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

attempt seems good, also motivated by 🤖 review since turn is an overloaded term

@johanandren johanandren left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Review

Should fix

  • Experiment.name javadoc regressed. This PR replaces the detailed contract from 6bb7ffa (max 200 characters, no < > : " / \ | ? * or control characters, @throws IllegalArgumentException) with an older text: "Not blank and without a path separator". requireFileName still enforces the stricter rules.
  • Report.cases() relies on a hidden invariant. It rebuilds per-case groups by slicing results in chunks of runs. That only works because run() produces exactly cases × runs results in case-major order. run() already has the grouping in turnsByCase and then flattens it. Keeping List<CaseSummary> in Report and deriving results() from it removes the slicing, the results.size() / runs in render() and the recomputation on every cases() call.
  • CaseSummary.outcome is a record component. The public canonical constructor accepts an outcome that contradicts turns. Derive it from turns, the same way passedRuns() and failedRuns() are derived.

Nits

  • With runs(1) the output still shows the new run wording: 3 cases, 1 run: …, failed cases [x (run 1)], case x run 1 FAILED, slowest x run 1. Leaving out the run parts when runs == 1 would keep the default output close to what it was.
  • Naming of the static helpers ExperimentRunner.runs(List<Integer>) and count(long, String):
    • runs(...) clashes with runs(), runs(int) and the runs component. Inside CaseSummary, runs(e.getValue()) reads like an accessor.
    • Something like describeRuns(...) and plural(...) would be clearer.
  • CaseSummary.failures() returns a rendered string. A name like renderFailures() would make that clear.
  • Gate infers the number of runs from max(run). It works, but it is a second derivation of a value the runner already knows.
  • In run(), for (var ignored : cases) turnsByCase.add(new ArrayList<>()) reads oddly. This goes away with the grouping change above.
  • Javadoc for CaseSummary and ReportDocument.Case outcome: "…failed in every run, or both". It should be "…or passed in some runs and failed in others".
  • gates-and-reports.adoc callout <1>: "consecutive calls of one case do not follow each other" is not true with a single case.
  • Tests:
    • The new tests use fully qualified names (java.util.concurrent.atomic.AtomicInteger, java.util.ArrayList, ExperimentRunner.CaseSummary.Outcome.…) instead of imports.
    • The AtomicInteger call-counting targets are duplicated across GateTest and ExperimentRunnerTest.
    • No test covers render() leaving out the inconsistent across runs section when every case is consistent.

Open questions

  • "Turn" as the name for one case in one run. EvalTarget.Turn already uses the word, but in agent vocabulary "turn" usually means one exchange in a conversation. That will be confusing if multi-turn cases arrive later. Would "trial" or "attempt" be better? Renaming is cheap now, before the JSON fields (turns, passedTurns, turnsWithEvidence) are released.
  • Runtime state between runs. The docs say stubs serve every run, but entity or other runtime state that tools write also carries over. Because execution is run-major, run 2 of a case sees what every case did in run 1. Should the docs mention this?
  • Duplicate case ids. They are not rejected. With runs(n), Gate.failedCases groups by id, and the JSON turns[].id is ambiguous. Should case ids be required to be distinct, like evaluator labels are?
  • JSON formatVersion. The format changes meaning (cases[], casesWithEvidence → turnsWithEvidence) while formatVersion stays 1. That is fine only if this PR is released together with the unreleased JSON report base PR. Is that the plan?
  • Wall clock and failure output grow with n. Wall clock grows n×, and render() prints full evidence for every failed turn, so an unstable case with large traces gives a long assertion message. Should the docs mention test duration? Should render() print evidence only once per case, or cap the number of failed turns it prints?

@patriknw

patriknw commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

🤖 review

Findings

Blockers

  • [blocker] akka-javasdk-tests/src/test/java/akkajavasdk/eval/SupportAgentEvalTest.java:386,476, SupportAgentRedTeamEvalTest.java:301,320: this change causes the CI failure. These four tests assert on the old rendered text ("over 6/6 cases with evidence", "case wrong-customer FAILED", "6/6 cases passed", "case direct-injection FAILED"), and the branch did not update them.

    • On the branch: akka-javasdk-tests/testOnly akkajavasdk.eval.* gives 19 tests, 4 failed.
    • On the base: the same command gives 19 passed. The CI sbt-test job runs sbt test, which includes this module.
  • [blocker] ExperimentRunner.java:466,697,773, Gate.java:156: an experiment that does not call runs(n) now renders different text (requirement 4). Repeated runs do not need any of these changes. Tests in this repo and user tests written against 3.6.6 assert on this output, and the four failures above prove it. The changes, all verified by running the same experiment on base and branch:

    • Header: 1/3 cases passed (33%) becomes 3 cases, 1 run: 1/3 turns passed (33%).
    • Gate details:
      • over 3 cases becomes over 3 turns (3 cases, 1 run).
      • judged cases becomes judged turns.
      • failed cases [bad, boom] becomes failed cases [bad (run 1), boom (run 1)].
      • target failed on [boom] becomes target failed on [boom (run 1)].
    • Spend line: slowest good at becomes slowest good run 1 at, and cases with evidence becomes turns with evidence.
    • describe(): case bad FAILED becomes case bad run 1 FAILED.

    Numbers, verdicts and the order of results() are identical. Fix: print the run parts only when runs > 1. For describe(), the report can render the run itself, and the public CaseResult.describe() keeps its format, or adds the run only when the run is above 1.

Should fix

  • [should-fix] ExperimentRunner.java:539-590 (CaseSummary): requirement 2 asks for the requirements that passed in some runs and failed in others. The branch decides at the case level, so a failing check hides an inconsistent one. Verified:

    • Check A fails only in run 1 and check B fails only in run 2. Each check is inconsistent, but the case is FAILED_EVERY_RUN, the text has no "inconsistent" section, and summary.inconsistentCases is 0.
    • Check A fails every run and check B fails in run 2 of 3. B's inconsistency is hidden in the same way.
    • The per-label lines (check-b 2/3) pool all cases, so they cannot name the case.
    • JSON cases[].evaluators has per-case label counts, but it merges checks that share a label inside one case. Replayed cases often carry several tool-arguments checks. A tool that flags passed>0 && failed>0 therefore reports false inconsistencies even with one run.

    See decision D1.

  • [should-fix] Requirement 3 is not delivered. Nothing in the rendered text, the JSON, the schema, the javadoc or the docs states what n passes out of n runs establish. The only related sentence is gates-and-reports.adoc:136 ("One run ... is one sample"). A reader can still take 3/3 as proof. The PR must not close the issue without it. See D3 for what the statement needs.

  • [should-fix] ExperimentRunner.java:549: inconclusive results are not counted as unjudged.

    • A check that passes in run 1 and is inconclusive in runs 2 and 3 makes the case PASSED_EVERY_RUN. The outcome claims three passes for a requirement that was judged once.
    • A check that fails once and is inconclusive otherwise is listed as j passed 2/3 runs: custom-eval:judgy failed in run 1. The check never passed.

    A per-requirement outcome must count passed, failed and inconclusive separately, and label a requirement only on its conclusive runs.

  • [should-fix] ExperimentRunner.java:449: CaseResult is a public record on main (3.6.6). Inserting int run as the second component removes the released constructor CaseResult(String, Interaction, List<EvalResult>), which breaks source and binary compatibility for any test that builds one (requirement 4). Fix: add that constructor back, delegating with run = 1.

  • [should-fix] Experiment.java:23-26: this hunk does not belong to the feature. It puts back the name() javadoc from 4c08d6e ("Not blank and without a path separator"), which base commit 6bb7ffa replaced. The code still calls requireFileName (ExperimentRunner.java:245,412). That method rejects names over 200 characters, < > : " / \ | ? * and control characters. The javadoc now understates the rule and has lost its @throws. Restore the base text. This looks like a conflict resolution that went the wrong way.

  • [should-fix] samples/doc-snippets/src/test/java/com/example/eval/OrderAgentQualityIntegrationTest.java:218: aFailedCaseShowsWhatTheAgentDid asserts contains("case wrong-order FAILED"). It fails on this branch when the build runs with -Deval=true, so this reader of the output was also missed.

  • [should-fix] OrderAgentQualityIntegrationTest.java:140 and gates-and-reports.adoc:145: the sample gate is passRateShouldBeAtLeast(0.9) over 3 cases × 3 runs = 9 turns. One failed turn gives 8/9 = 0.889 < 0.9, so the gate requires 9/9 and is the same as allCasesShouldPass(). The callout "at least 90% of them must pass" suggests a tolerance that does not exist. Against a real model, that is the flaky test that repeated runs are meant to explain. Use 0.85 to tolerate one failure, or state that it tolerates none.

  • [should-fix] gates-and-reports.adoc:148, eval-cases.adoc:46: the docs say the same stubs serve every run. They do not tell the developer to keep stubs and evaluators free of state that a turn writes. The @BeforeEach reset runs once per experiment, not once per run. The docs' own NoRefundIssued evaluator (adversarial.adoc:119) reads stub state that the agent writes. Verified with an equivalent evaluator: the agent causes a side effect in run 1 only, and the report says FAILED_EVERY_RUN, 0/3, not inconsistent. Leftover state looks like a defect of the agent, and the report cannot tell the two apart. See D6.

Nits

  • [nit] ReportDocument.java:124: outcome is described as "passed in every run, failed in every run, or both". The third value means "passed in some runs and failed in others", not both.

  • [nit] ExperimentRunner.java:601: the EvalReport javadoc still says "per-case results". The results are now per turn.

  • [nit] eval-report.example.json: it shows only runs: 1 and no INCONSISTENT case. The docs link the example in place of a format description, so a repeated-run report has no example.

  • [nit] Size. Every turn keeps its full evidence. 30 runs of one case with 8 KB of model messages per turn gives a 263 KB JSON file (9.7 KB for one run). The text prints every failed turn in full (30 blocks). Consider rendering the first failed turn per case and listing the run numbers of the others.

  • [nit] Cost.

    • Judge model calls run in their own session (ModelBasedJudge.java:124), so the spend never includes them. This was already true on the base, but runs multiply them. "The spend grows with the number of runs" (gates-and-reports.adoc:149) suggests that the spend covers them.
    • A target failure on every turn (for example an unreachable model) still runs all n × cases turns.
    • The per-turn INFO log (ExperimentRunner.java:278) gives progress, and it is also new output for every one-run experiment.
  • [nit] ExperimentRunner.java:382-385 and gates-and-reports.adoc callout <1>: on the first point, Gate.java:151-165 gets the number of runs from max(run), and the N cases count assumes that results.size() divides evenly by it. Report already holds runs, so passing it in is simpler. On the second point, "consecutive calls of one case do not follow each other" is false for a one-case experiment. Neither comment says why run-major order matters.

  • [nit] Untested behaviour:

    • inconclusive results across runs
    • a setup failure in one run only
    • runs(-1)
    • CaseResult with run < 1
    • the schema against a multi-run report (theJsonReportMatchesTheSchemaOnTheClasspath uses one run)
    • unchanged one-run text (instead, the existing assertions were rewritten to the new text)

    All the edited assertions follow from the text change. None was weakened or dropped.

Needs a decision

  • D1. Unit of inconsistency. Options:
    • (a) Keep the case level. This costs nothing, but it misses the hidden cases above and does not meet requirement 2.
    • (b) Key requirements by (case id, label). This is cheap, but it merges same-label checks in one case and shows false inconsistency.
    • (c) Key them by (case id, position of the check in the case) and carry the label. This is exact, because evaluators run in a fixed order. It costs one more public type (for example RequirementSummary(caseId, index, label, passed, failed, inconclusive)), one more JSON array, and a rendered line per inconsistent requirement. With (b) or (c), keep the case outcome as well.
  • D2. Gates over repeated runs. The pass-rate and evaluator-rate gates pool turns. Verified with 3 cases × 3 runs and passRateShouldBeAtLeast(0.6): one case that never passes passes the gate (0.67), exactly like three cases that each fail once. allCasesShouldPass reads as expected: "failed cases [c2 (run 2)]". No gate can act on what requirement 2 reports. Options:
    • (a) Keep pooling and document it.
    • (b) Add a per-case gate, such as everyCaseShouldPassAtLeast(rate) over each case's runs.
    • (c) Add a gate on inconsistency, such as noRequirementShouldBeInconsistent().

Base automatically changed from evalkit-json-report to main October 5, 2026 13:12
@aludwiko
aludwiko force-pushed the evalkit-repeat-runs branch from 8f32522 to dca3426 Compare October 5, 2026 13:27
@aludwiko

aludwiko commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor Author

@patriknw I prefer to keep a unified report for runs with and without repeat config, so I ignored those comments.

Done

  • Feature. Experiment.repeat(n) runs every case n times in a fresh session, run-major. One execution of one case is an attempt. The gate, the rates and the spend count attempts.
  • Report model. CaseResult carries the run and keeps the released three-argument constructor. EvalReport gains runs() and cases(). A CaseSummary holds a case's attempts, its outcome and its RequirementSummary
    list, one per evaluator at one position, with the runs it passed, failed and was inconclusive in. Outcomes are derived, and one Outcome enum serves cases and requirements.
  • Rendered text. One format for one and several runs, per your decision over the reviewers: cases, runs and attempts on the first line, runs named in the gate detail, the spend line and each failed attempt,
    and an inconsistent across runs section listing inconsistent requirements and target or setup failures of inconsistent cases.
  • JSON report. Still formatVersion 1: top-level runs, attempt and requirement counts in summary, cases[] with requirements[], and attempts[] holding the evidence. Schema and example updated.
  • Review fixes. Experiment.name javadoc restored, duplicate case ids rejected, inconclusive runs neither pass nor fail a requirement, sample gate 0.85, docs on what a clean sweep establishes and on state
    surviving between runs, helper names, imports, and the added tests the review listed.
  • Readers of the text. The four integration tests, the doc-snippets sample assertion and the docs listings on four pages use the unified text.

@aludwiko

aludwiko commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

@patriknw, referring to D1 and D2, I ran some analyses again, and honestly, every option has some consequences. I would postpone both of them and wait for some feedback from actual users.

@patriknw

patriknw commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

okey

@patriknw patriknw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@patriknw
patriknw merged commit 086ec79 into main Oct 6, 2026
38 checks passed
@patriknw
patriknw deleted the evalkit-repeat-runs branch October 6, 2026 09:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation documentation related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants