Skip to content

feat: fail-modes - FailClosed and FailOpen - #2069

Merged
patriknw merged 5 commits into
mainfrom
octonato/guardrail-fail-mode-sdk
Oct 22, 2026
Merged

patriknw merged 5 commits into
mainfrom
octonato/guardrail-fail-mode-sdk

Conversation

@octonato

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot added the documentation documentation related label Oct 13, 2026
@patriknw

Copy link
Copy Markdown
Contributor

🤖 review

Findings

  1. [should-fix] akka-javasdk/src/main/java/akka/javasdk/agent/Decision.java:36: @param reason human-readable explanation surfaced to the user is untrue for fail-open. The caller never sees the reason. It goes to the service log as AK-01213 and to the guardrail decision in the ledger. docs/src/modules/sdk/pages/agents/guardrails.adoc:229 says "logs a warning and records the error" without naming either place. Say what a developer sees after a fail is let through, and where. Also say that fail-mode does not apply to a guardrail that never completes, since the interaction then waits (guardrails.adoc:61).
  2. [should-fix] Tests: only one fail route runs fail-open against the runtime: a throwing AgentResponseGuardrail on a request-based agent. These parts are in the hands of the SDK, and no test covers them:
    • The fail mode of ModelCallSimilarityGuard (GuardrailProvider.scala:136). A change that hands fail-closed there fails no test.
    • The streaming startup check with a fail-open AgentResponseGuardrail (GuardrailProvider.scala:148). Startup must still fail, because a fail-open guardrail can deny.
    • Fail-open on a ToolCallGuardrail against the runtime, for a function tool and for an MCP tool.
  3. [nit] akka-javasdk/src/main/scala/akka/javasdk/impl/agent/GuardrailProvider.scala:502: "A TextGuardrail is always fail-closed" is untrue for a report-only TextGuardrail. Say that a TextGuardrail has no fail mode.
  4. [nit] Reading and describing the setting:
    • An enabled TextGuardrail with fail-mode fails startup. One that is enabled for no agent is accepted, for example the shipped default jailbreak entry. Its value then reaches the runtime (GuardrailProvider.scala:528), and the listing of configured guardrails shows FAIL_OPEN for an old guardrail.
    • There are two reactions to a setting without effect: a startup failure for a TextGuardrail, and a startup warning for report-only. Both are reasonable, but akka-javasdk/src/main/resources/reference.conf:211 names neither.
    • reference.conf:199 and guardrails.adoc:166 describe report-only for a deny only. It also decides a fail, and it overrides fail-mode.

Existing before this change

  • Decision.java:29 says that the reason of a Deny reaches the user as the failure message. That is untrue for report-only.

@octonato
octonato force-pushed the octonato/guardrail-fail-mode-sdk branch from 2c06654 to 10fe69f Compare October 19, 2026 09:33
octonato added a commit that referenced this pull request Oct 19, 2026
Fixes round 1 F1 from the PR #2069 review: no test ran a fail-open
`ToolCallGuardrail` through the runtime.

- Add `ThrowingToolGuard`, a `ToolCallGuardrail` that throws from `decide`.
- Configure it with `fail-mode = fail-open` for a function-tool agent and
  for an MCP agent.
- Check in `AgentIntegrationTest` that each tool runs and that the reply
  holds the tool result.
Fixes round 1 F1 from the PR #2069 review: no test ran a fail-open
`ToolCallGuardrail` through the runtime.

- Add `ThrowingToolGuard`, a `ToolCallGuardrail` that throws from
`decide`.
- Configure it with `fail-mode = fail-open` for a function-tool agent
and
  for an MCP agent.
- Check in `AgentIntegrationTest` that each tool runs and that the reply
  holds the tool result.
@octonato
octonato force-pushed the octonato/guardrail-fail-mode-sdk branch from 24d8ee0 to 5b3fd97 Compare October 19, 2026 09:45
@octonato

Copy link
Copy Markdown
Contributor Author

@patriknw, review suggestions applied

@octonato
octonato marked this pull request as ready for review October 19, 2026 09:46
@octonato
octonato marked this pull request as draft October 19, 2026 09:47
@octonato

Copy link
Copy Markdown
Contributor Author

Back to draft because it's using a runtime snapshot

@octonato
octonato marked this pull request as ready for review October 22, 2026 07:48

@patriknw patriknw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@patriknw

Copy link
Copy Markdown
Contributor

we'll use akka-runtime snapshot on main for a few days, until 1.7.3 is released

@patriknw
patriknw merged commit af979ff into main Oct 22, 2026
116 of 119 checks passed
@patriknw
patriknw deleted the octonato/guardrail-fail-mode-sdk branch October 22, 2026 11:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation documentation related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants