Skip to content

feat(ci): apply the dependency remediation without a model - #3845

Merged
rossnelson merged 3 commits into
mainfrom
drop-claude-from-remediation
Aug 20, 2026
Merged

rossnelson merged 3 commits into
mainfrom
drop-claude-from-remediation

Conversation

@rossnelson

Copy link
Copy Markdown
Collaborator

Description & motivation 💭

The remediation asked Claude to author package.json, and Claude was permitted to reproduce the output of a pure function. The planner already computed the result with applyRemediation, and the next step rejected any candidate whose bytes did not match that result.

So the model could add nothing the function did not already produce, and it could not fix anything the planner refused. The planner now writes the manifest itself.

flowchart TD
  subgraph N["Now"]
    A2["Plan the remediation"] --> B2["The planner writes package.json"]
    B2 --> C2["Prove that nothing<br/>outside package.json changed"]
  end

  subgraph B["Before"]
    A1["Plan the remediation"] --> B1["Claude edits package.json.<br/>Needs an API key."]
    B1 --> C1["Byte-match gate.<br/>Reject any bytes that differ<br/>from the planned result"]
  end
Loading

What the step cost

Run 32312572416 failed at the Claude step. The cause was not the prompt and not the tools:

"api_error_status": 401,
"result": "Failed to authenticate. API Error: 401 API key is invalid.",
"num_turns": 1, "total_cost_usd": 0, "duration_ms": 169583

Ten retries, 169 seconds, no work done. The same 401 appears in temporalio/cloud-ui, which last summarized a pull request at 2026-08-18T22:07 and has failed since. A company-wide change deactivated the standing API keys on 2026-08-18.

New keys are short-lived by default, at 14 days. A weekly workflow cannot hold a 14-day key, so this step would need a director-approved exception for a credential that adds no capability.

The change

  • The plan step keeps its read-only behavior, so a dry run is unchanged.
  • A new step applies the plan in apply mode, then proves the planner changed nothing outside package.json.
  • The report records deterministic-planner as the resolver. The publish gate requires that value for an npm action, where it required claude-code.
  • The Go path is unchanged. go get and go mod tidy already owned that change.

Removed:

  • validate-claude-dependency-remediation.mjs and its test. The script constrained an untrusted author. With no untrusted author it compares a function against itself.
  • ANTHROPIC_API_KEY from the module and from both callers.

The safety model gets stronger

Only deterministic code writes the manifest. Every later gate is unchanged: the file allowlist, the lockfile resolution, the proof after the install that each remediated alert escapes its advisory range, the hash attestation, and a publishing job that runs no package code.

claude.yml still uses the key to review pull requests. This change does not affect that workflow.

Testing 🧪

How was this tested 👻

  • Manual testing
  • E2E tests added
  • Unit tests added

77 tests pass. Prettier reports no problem. All three workflow files parse.

Two tests cover the new resolver: one for an applied plan and one for a plan that holds no action.

The whole apply path was run by hand against the live alert data, in the order the workflow runs it:

  1. The plan authorized one action, to raise the protobufjs override to ^7.6.5.
  2. The apply step wrote package.json and recorded resolver: deterministic-planner.
  3. The guard confirmed that no file outside package.json changed.
  4. pnpm install --lockfile-only --ignore-scripts resolved protobufjs to 7.6.5.
  5. Verification passed and recorded 7.6.5 for alerts 338, 300, and 299.
  6. The publish gate returned true.

Steps for others to test: 🚶🏽‍♂️🚶🏽‍♀️

Open Actions, select Weekly On-Call Review PR Test, select Run workflow on this branch, and set the mode to apply. The rehearsal sets publish to false, so no pull request is touched. The run needs no Anthropic credential.

Checklists

Merge Checklist

  • Run the apply-mode rehearsal on this branch and confirm it reaches the build step.
  • Decide whether to keep requesting an Anthropic key for claude.yml, which is a separate workflow and still needs one.

Issue(s) closed

None.

Docs

Any docs updates needed?

.github/WORKFLOWS.md is updated in this change. The safety model section no longer describes a model, and ANTHROPIC_API_KEY is removed from the required configuration.

The remediation ran through Claude, and Claude was permitted to reproduce the
output of a pure function. The planner already computed the exact result with
`applyRemediation`, and the validation step then rejected any candidate whose
bytes did not match that result. The model could therefore add nothing that the
function did not already produce, and it could not fix anything the planner
refused.

That step cost the workflow an API key, 175 seconds of retries when the key
failed, and a dependency on a credential that company policy now issues for 14
days at a time. A weekly job cannot hold a 14-day key.

The planner now writes the manifest itself.

- The plan step keeps its read-only behavior, so a dry run is unchanged.
- A new step applies the plan in `apply` mode, then proves that the planner
  created no unexpected file and changed nothing except `package.json`.
- The report records `deterministic-planner` as the resolver. The publish gate
  requires that value for an npm action, where it required `claude-code`.
- The Go path is unchanged. The Go toolchain already owned its own change.

Removed with the step:

- `validate-claude-dependency-remediation.mjs` and its test. The script existed
  to constrain an untrusted author. There is no untrusted author now, so the
  byte-match compares a function against itself.
- `ANTHROPIC_API_KEY` from the module and from both callers. The module no
  longer requests it.

The safety model is stronger, not weaker. Only deterministic code writes the
manifest, and every later gate is unchanged: the file allowlist, the lockfile
resolution, the post-install proof that each remediated alert escapes its
advisory range, the hash attestation, and a publishing job that runs no package
code.

`claude.yml` still uses the key for pull request review. This change does not
affect that workflow.

77 tests pass.
The apply step refused any untracked file other than the two evidence reports.
That guard existed because a model could create a file. Deterministic code
writes one path, so the guard protects nothing and it fails for a person who
runs the step in a working tree that holds other new files.

The check that no tracked file outside package.json changed stays, because it
still proves where the planner wrote.
The fixture named claude-code as the resolver. The verification script does not
read that field, so the test passed either way, but the value no longer matches
what the planner writes.
@rossnelson
rossnelson requested a review from a team as a code owner August 20, 2026 13:29
@vercel

vercel Bot commented Aug 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
holocene Ready Ready Preview Aug 20, 2026 1:29pm

Request Review

@rossnelson
rossnelson merged commit dd80eca into main Aug 20, 2026
32 checks passed
@rossnelson
rossnelson deleted the drop-claude-from-remediation branch August 20, 2026 13:42
laurakwhit added a commit that referenced this pull request Sep 9, 2026
Auto-generated version bump from 2.53.3 to 2.54.0

Bump type: minor

Changes included:
- [`7a5684a5`](7a5684a) Check if api has been upgraded for minimum minor bump (#3823)
- [`af39e483`](af39e48) refactor(events): derive event groups on read in the grouped-event-buffer (#3790)
- [`3ce42ebb`](3ce42eb) Add the developer Workflow Catalog (#3785)
- [`a1234e36`](a1234e3) chore: split Catalog into a focused Temporal UI plugin (#3835)
- [`060eacf7`](060eacf) Get total slots based on available plus used (#3833)
- [`283d3cfc`](283d3cf) fix(timezone): use regular font weight for the timezone switcher (#3834)
- [`88d61845`](88d6184) Manual on-call dependency review with constrained Claude remediation (#3816)
- [`f87d239e`](f87d239) ci: validate the weekly review candidate the way the PR checks do (#3841)
- [`dd80eca0`](dd80eca) feat(ci): apply the dependency remediation without a model (#3845)
- [`403e5b97`](403e5b9) refactor(icons)!: replace holocene's icon registry with Io icon components (#3822)
- [`d6823cc4`](d6823cc) fix(ci): correct the provenance claim in the remediation PR body (#3847)
- [`f1b05e99`](f1b05e9) feat(ci): post the weekly review only when it needs a person (#3848)
- [`b993c04b`](b993c04) ci: run the on-call review test only for an on-call change (#3850)
- [`4307d20f`](4307d20) fix(copyable): pin the copy icon to 16px (#3855)
- [`62e3014f`](62e3014) Update SAA heartbeat info (#3856)
- [`3a797592`](3a79759)  Headers spacing alignment and Small Standalone Activity UI fixes  (#3854)
- [`41687b1f`](41687b1) Add total Worker count (#3641)
- [`68a3bc2f`](68a3bc2) fix(catalog): keep isolated test fixtures out of the repo root (#3859)
- [`9a56d3d9`](9a56d3d) fix: don't render an empty selected-count badge in combobox (#3853)
- [`4b97f35b`](4b97f35) Make only keyboard focus pin Tooltip open (#3861)
- [`2fd63c0e`](2fd63c0) Add Rust SDK to worker heartbeats SDK warning (#3863)
- [`4b43f7f9`](4b43f7f) Move Saved Views above Filtering (#3858)
- [`cdfd92c9`](cdfd92c) Add option to maximize paginated tables (#3864)
- [`ea4d3f14`](ea4d3f1) Fix ConfigurableTableHeadersDrawer for SAA nexus operations (#3867)
- [`10f50c8b`](10f50c8) Fix duplicate schedule action keys (#3868)
- [`9feeb756`](9feeb75) feat(DT-3944): SignalWithStart system Nexus obfuscation — decode infrastructure + UI (#3356)
- [`5af59f03`](5af59f0) feat(workers): let users set the Cloud Run scale-down stabilization window (#3860)
- [`bb7d6fbd`](bb7d6fb) Support secondary text on Combobox options (#3866)
- [`e0925297`](e092529) Fix paginated table height and mobile scrolling (#3871)
- [`6dbd9b96`](6dbd9b9) fix: categorize local activity markers as local-activity (#3872)
- [`353fbbc8`](353fbbc)  Wrap long filter chips instead of overflowing the filter bar (#3877)
- [`f4c71240`](f4c7124) Fix workflow list timestamp formatting (#3880)
- [`fa9c51e9`](fa9c51e) Open saved views in a new tab on modifier-click (#3881)
- [`376af414`](376af41) feat(workers): warn about Cloud Run deployment latency when creating a version (#3879)
- [`1bde3187`](1bde318) Fix extra horizontal scroll from filter bar dropdowns (#3885)
- [`dd1dcd9b`](dd1dcd9) Add Next Retry Delay under Retry State (#3886)
- [`66c57a40`](66c57a4)  Make system saved views additive on custom saved views (#3876)
- [`ffa1f2fb`](ffa1f2f) Audit fields for defaults and order (#3887)
- [`9ab9c40b`](9ab9c40) fix(server): update labstack/echo/v4 to 4.15.4 (FE-493) (#3874)
- [`e4449991`](e444999) Make Build ID filterable and show worker status counts (#3889)
- [`0a5e01f5`](0a5e01f) SAA fields audit part 2 (#3890)
- [`0d604b5a`](0d604b5) Fix payload input focus loss (FE-643) (#3891)
- [`06712381`](0671238) Keep portal menus anchored when the layout shifts (#3893)

Co-authored-by: laurakwhit <15069288+laurakwhit@users.noreply.github.com>

This branch was successfully deployed

1 active deployment
Preview — 0a0bcaf8 Deployed Aug 20, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant