Repository navigation
feat: evalkit run cases multiple times - #1979
Conversation
|
|
||
| | `Gate.allCasesShouldPass()` | ||
| | No case has a failed result. This is the gate that applies when none is given. | ||
| | No turn has a failed result. This is the gate that applies when none is given. |
There was a problem hiding this comment.
I struggling with the decision if this should be a turn or an attempt ?
There was a problem hiding this comment.
attempt seems good, also motivated by 🤖 review since turn is an overloaded term
johanandren
left a comment
There was a problem hiding this comment.
🤖 Review
Should fix
Experiment.namejavadoc regressed. This PR replaces the detailed contract from 6bb7ffa (max 200 characters, no< > : " / \ | ? *or control characters,@throws IllegalArgumentException) with an older text: "Not blank and without a path separator".requireFileNamestill enforces the stricter rules.Report.cases()relies on a hidden invariant. It rebuilds per-case groups by slicingresultsin chunks ofruns. That only works becauserun()produces exactlycases × runsresults in case-major order.run()already has the grouping inturnsByCaseand then flattens it. KeepingList<CaseSummary>inReportand derivingresults()from it removes the slicing, theresults.size() / runsinrender()and the recomputation on everycases()call.CaseSummary.outcomeis a record component. The public canonical constructor accepts an outcome that contradictsturns. Derive it fromturns, the same waypassedRuns()andfailedRuns()are derived.
Nits
- With
runs(1)the output still shows the new run wording:3 cases, 1 run: …,failed cases [x (run 1)],case x run 1 FAILED,slowest x run 1. Leaving out the run parts whenruns == 1would keep the default output close to what it was. - Naming of the static helpers
ExperimentRunner.runs(List<Integer>)andcount(long, String):runs(...)clashes withruns(),runs(int)and therunscomponent. InsideCaseSummary,runs(e.getValue())reads like an accessor.- Something like
describeRuns(...)andplural(...)would be clearer.
CaseSummary.failures()returns a rendered string. A name likerenderFailures()would make that clear.Gateinfers the number of runs frommax(run). It works, but it is a second derivation of a value the runner already knows.- In
run(),for (var ignored : cases) turnsByCase.add(new ArrayList<>())reads oddly. This goes away with the grouping change above. - Javadoc for
CaseSummaryandReportDocument.Caseoutcome: "…failed in every run, or both". It should be "…or passed in some runs and failed in others". gates-and-reports.adoccallout<1>: "consecutive calls of one case do not follow each other" is not true with a single case.- Tests:
- The new tests use fully qualified names (
java.util.concurrent.atomic.AtomicInteger,java.util.ArrayList,ExperimentRunner.CaseSummary.Outcome.…) instead of imports. - The
AtomicIntegercall-counting targets are duplicated acrossGateTestandExperimentRunnerTest. - No test covers
render()leaving out theinconsistent across runssection when every case is consistent.
- The new tests use fully qualified names (
Open questions
- "Turn" as the name for one case in one run.
EvalTarget.Turnalready uses the word, but in agent vocabulary "turn" usually means one exchange in a conversation. That will be confusing if multi-turn cases arrive later. Would "trial" or "attempt" be better? Renaming is cheap now, before the JSON fields (turns,passedTurns,turnsWithEvidence) are released. - Runtime state between runs. The docs say stubs serve every run, but entity or other runtime state that tools write also carries over. Because execution is run-major, run 2 of a case sees what every case did in run 1. Should the docs mention this?
- Duplicate case ids. They are not rejected. With
runs(n),Gate.failedCasesgroups by id, and the JSONturns[].idis ambiguous. Should case ids be required to be distinct, like evaluator labels are? - JSON
formatVersion. The format changes meaning (cases[],casesWithEvidence→turnsWithEvidence) whileformatVersionstays 1. That is fine only if this PR is released together with the unreleased JSON report base PR. Is that the plan? - Wall clock and failure output grow with
n. Wall clock grows n×, andrender()prints full evidence for every failed turn, so an unstable case with large traces gives a long assertion message. Should the docs mention test duration? Shouldrender()print evidence only once per case, or cap the number of failed turns it prints?
|
🤖 review FindingsBlockers
Should fix
Nits
Needs a decision
|
8f32522 to
dca3426
Compare
|
@patriknw I prefer to keep a unified report for runs with and without repeat config, so I ignored those comments. Done
|
|
@patriknw, referring to D1 and D2, I ran some analyses again, and honestly, every option has some consequences. I would postpone both of them and wait for some feedback from actual users. |
|
okey |
refs: https://github.com/lightbend/akka-runtime/issues/5628