Skip to content

feat(builtins): classify command effects - #1167

Merged
jdx merged 2 commits into
agent/step-effectsfrom
agent/builtin-effects
Aug 10, 2026
Merged

jdx merged 2 commits into
agent/step-effectsfrom
agent/builtin-effects

Conversation

@jdx

@jdx jdx commented Aug 10, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • classify every command exposed by the current builtin catalog: 236 command fields across 144 command-bearing modules
  • wrap builtin commands at the generated Builtins.pkl boundary so overrides replace both the command and its safety claim
  • classify fixers as writes and checks that populate caches, compile, or synchronize data as writes; other checks are reads
  • fail builtin generation if any exposed check, check_list_files, check_diff, or fix command lacks an effect

Stack

Depends on #1166 (feat(step): add command effect safety). This is layer 4 of the agent-native hk stack.

Test evidence

  • python3 -m py_compile scripts/gen_builtins.py
  • mise exec ruff@0.15.20 -- ruff check scripts/gen_builtins.py
  • mise exec ruff@0.15.20 -- ruff format --check scripts/gen_builtins.py
  • mise run pkl:gen (includes 100% effect-coverage validation)
  • mise run build
  • mise run test:bats test/builtin_effects.bats
  • mise run test:bats test/builtins_tests.bats was attempted locally; catalog loading and effect wrapping succeeded, while 27 tool executions failed because this host lacks configured Node, Ruby, or Java runtimes and one network-dependent zizmor check timed out. CI has the project toolchain and remains authoritative for the complete catalog.

Rollout notes

Builtin consumers receive explicit effects automatically. A user override such as replacing Builtins.trailing_whitespace.check_diff with a string becomes unknown rather than inheriting the builtin's read claim. No builtin is marked destructive because current builtin commands only inspect or update recreatable/project state.

AI-assisted β€” Tool: Codex; model: unavailable/unavailable; version: unavailable.


Note

Medium Risk
Large mechanical change across 100+ builtin modules; misclassified read vs write could block or allow the wrong commands under safe mode, though generation enforces complete coverage.

Overview
Builtin commands now declare safety effects so hk can reason about read-only vs mutating steps (e.g. --safe, plan JSON).

Across the builtin catalog, check, check_list_files, check_diff, and fix are no longer bare strings or Command objectsβ€”they use Config.CommandSpec with effect = "read" for typical linters/format checks and effect = "write" for fixers and checks that compile, sync, or write caches (cargo, tsc, vale, rubocop, etc.). No builtins are marked destructive.

scripts/gen_builtins.py runs validate_effect_coverage() after generating Builtins.pkl, evaluating the catalog as JSON and erroring if any exposed command field lacks a valid effect. Deprecated builtin aliases now re-export the canonical builtin directly instead of re-importing the module path.

test/builtin_effects.bats asserts plan output includes per-step effects, safe mode runs a read builtin, and overriding a builtin command without CommandSpec surfaces an unknown effect instead of inheriting the builtin’s read claim.

Reviewed by Cursor Bugbot for commit c77afc6. Bugbot is set up for automated code reviews on this repo. Configure here.

@coderabbitai

coderabbitai Bot commented Aug 10, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

βš™οΈ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 4884ca5e-0d68-46bd-8361-46a126012c19

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • πŸ” Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❀️ Share

Comment @coderabbitai help to get the list of available commands.

Comment thread scripts/gen_builtins.py Resolved
@greptile-apps

greptile-apps Bot commented Aug 10, 2026 •

Copy link
Copy Markdown

Greptile Summary

This PR adds generated effect metadata to builtin commands and validates complete effect coverage during generation.

  • Wraps builtin command fields with read/write effect declarations.
  • Preserves effect handling across deprecated aliases and user command overrides.
  • Adds integration coverage for plans, safe-mode execution, fix effects, and overridden commands.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
scripts/gen_builtins.py Generates effect-wrapped builtin steps, validates command-effect coverage, and redirects deprecated aliases to wrapped canonical definitions; no eligible follow-up defect was established.
test/builtin_effects.bats Verifies read/write plan metadata, safe execution of read commands, fix classification, and denial of an override whose effect becomes unknown.

Reviews (6): Last reviewed commit: "feat(builtins): classify command effects" | Re-trigger Greptile

@jdx
jdx force-pushed the agent/builtin-effects branch from 1b1aea7 to b8dcd03 Compare August 10, 2026 02:51
@jdx
jdx force-pushed the agent/builtin-effects branch 2 times, most recently from d7b71ae to d4e9174 Compare August 10, 2026 03:16
Comment thread scripts/gen_builtins.py Resolved
@jdx
jdx force-pushed the agent/builtin-effects branch from d4e9174 to 25421e2 Compare August 10, 2026 03:19
@jdx
jdx force-pushed the agent/builtin-effects branch from 25421e2 to cfa662a Compare August 10, 2026 03:27
@jdx
jdx force-pushed the agent/builtin-effects branch from cfa662a to 5c91143 Compare August 10, 2026 03:31
@greptile-apps

greptile-apps Bot commented Aug 10, 2026

Copy link
Copy Markdown

Too many files changed for review (146 files, 100 file limit).

Bypass the limit by tagging @greptile-apps to review.

@jdx
jdx force-pushed the agent/builtin-effects branch from 5c91143 to 30a8600 Compare August 10, 2026 03:46
@jdx
jdx force-pushed the agent/builtin-effects branch from 30a8600 to bf82639 Compare August 10, 2026 04:06
@jdx
jdx force-pushed the agent/builtin-effects branch 2 times, most recently from 6e91f9f to f616d0b Compare August 10, 2026 04:27
@jdx
jdx force-pushed the agent/builtin-effects branch from f616d0b to 6594b1e Compare August 10, 2026 04:30
@jdx
jdx force-pushed the agent/builtin-effects branch from 6594b1e to e163ca0 Compare August 10, 2026 04:32
Comment thread scripts/gen_builtins.py Resolved
@jdx
jdx force-pushed the agent/builtin-effects branch from e163ca0 to 438ba8e Compare August 10, 2026 04:39
Comment thread pkl/builtins/trailing_whitespace.pkl Resolved
@jdx
jdx force-pushed the agent/builtin-effects branch 2 times, most recently from 82eadce to 76ce0ea Compare August 10, 2026 04:53
@jdx
jdx force-pushed the agent/builtin-effects branch from 76ce0ea to e57f58c Compare August 10, 2026 04:56
@jdx
jdx force-pushed the agent/builtin-effects branch from e57f58c to d8a23e6 Compare August 10, 2026 05:07
@github-actions

github-actions Bot commented Aug 10, 2026 •

Copy link
Copy Markdown
Contributor

Instruction counts

benchmark trend instructions Ξ” wall (min) Ξ”
builtins β–β–β–β–β–β–β–β–β–β–β–β–β–β–ˆ 3,371,735 β†’ 3,628,840 +7.63% ⚠️ 2.85 β†’ 2.67ms -6.33%
check β€” β€” β€” 19.56 β†’ 20.08ms +2.65%
usage β–β–β–β–β–β–β–β–β–β–β–β–β–β–ˆ 23,189,004 β†’ 25,994,909 +12.10% ⚠️ 5.00 β†’ 5.25ms +4.90%
validate β€” β€” β€” 5.04 β†’ 4.72ms -6.28%

2 benchmark(s) above the 1% gate: builtins +7.63%, usage +12.10%

Only instruction counts gate. Wall clock is shown for context β€” on identical hardware it moves 4-20% run to run.

Measured by tak β€” instruction-counted CLI benchmarks, stored in this repository's git notes.

c77afc6e3737 vs 075e80902082 Β· measured on the runner, not pushed to the history.

@jdx
jdx force-pushed the agent/builtin-effects branch from d8a23e6 to d41d593 Compare August 10, 2026 05:30
Comment thread pkl/builtins/bundle_audit.pkl Resolved
@jdx
jdx force-pushed the agent/builtin-effects branch from d41d593 to f6ee638 Compare August 10, 2026 20:13

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit f6ee638. Configure here.

Comment thread pkl/builtins/err_check.pkl Resolved
@jdx
jdx merged commit f8914fe into main Aug 10, 2026
19 of 20 checks passed
@jdx
jdx deleted the agent/builtin-effects branch August 10, 2026 20:43
@jdx jdx mentioned this pull request Aug 10, 2026
jdx added a commit that referenced this pull request Aug 11, 2026
## Summary

- add normalized diagnostics with step, tool, severity, message, path,
range, rule, help URL, and optional fix data
- add first-party SARIF, Cargo JSON, ESLint JSON, and GCC-style parsers
with deduplication and parse warnings
- preserve raw step output alongside normalized diagnostics in JSON and
JSONL results
- add `--sarif <PATH>` export independently of the selected human or
structured output format

## Stack

Depends on #1167 (`feat(builtins): classify command effects`). This is
layer 5 of the agent-native hk stack.

## Test evidence

- `cargo fmt --check`
- `cargo test diagnostics::tests`
- `cargo test cli::tests::test_subcommands_are_sorted`
- `cargo clippy --quiet -- -D warnings` with local-only allowances for
unrelated Rust 1.97 lints already present on `main`
- `mise run build`
- `mise run test:bats test/diagnostics.bats`
- `mise run render`

## Rollout notes

Diagnostics are opt-in per step through `diagnostic_format`, so existing
output behavior is unchanged. A parser failure is represented as a
warning and never removes the raw command output. SARIF export writes a
standard 2.1.0 file and does not add an HTTP service or uploader.

*AI-assisted β€” Tool: Codex; model: unavailable/unavailable; version:
unavailable.*

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Touches hook execution, structured emit paths, and file I/O on every
run when `--sarif` is set; behavior is opt-in per step but SARIF write
failures can surface after hook failures.
> 
> **Overview**
> Adds **opt-in structured diagnostics** for hook steps and a **`--sarif
<PATH>`** export path on `check`, `fix`, `run`, and git-hook entrypoints
(documented in CLI help and usage specs).
> 
> Steps can set **`diagnostic_format`** (`sarif`, `cargo-json`,
`eslint-json`, `gcc`) and optional **`diagnostic_tool`** in `hk.pkl`.
Check output is parsed into a shared **`Diagnostic`** model (severity,
locations, rules, help URLs, fixes) with deduplication and **parse
warnings** when input is malformedβ€”raw command output is still kept in
JSON/JSONL results.
> 
> **`--sarif`** writes SARIF 2.1.0 from aggregated diagnostics
regardless of human vs machine output format, including empty reports on
no-op or preflight failures. A separate **`diagnostic_output_by_step`**
buffer keeps check-first and cancelled-run diagnostics available for
structured output and human summaries without letting a later successful
fixer hide the original failure text.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
f8a648e. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added `--sarif <PATH>` support to `check`, `fix`, and hook-running
commands.
* Export normalized diagnostics as SARIF, including locations, severity,
rules, help links, and fixes.
* Added support for SARIF, Cargo JSON, ESLint JSON, and GCC diagnostic
formats.
* Deduplicate diagnostics and report warnings for malformed or
unrecognized input.

* **Bug Fixes**
* Improved diagnostic preservation and error reporting during failed,
skipped, cancelled, and fixer runs.

* **Documentation**
  * Documented SARIF output support across applicable CLI commands.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
tmeijn pushed a commit to tmeijn/dotfiles that referenced this pull request Aug 12, 2026
This MR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [hk](https://github.com/jdx/hk) | tools | minor | `1.54.1` β†’ `1.55.0` |

MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot).

**Proposed changes to behavior should be submitted there as MRs.**

---

### Release Notes

<details>
<summary>jdx/hk (hk)</summary>

### [`v1.55.0`](https://github.com/jdx/hk/blob/HEAD/CHANGELOG.md#1550---2026-08-11)

[Compare Source](jdx/hk@v1.54.1...v1.55.0)

##### πŸš€ Features

- **(agent)** generate coding-agent integrations by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;1171](jdx/hk#1171)
- **(builtins)** add kingfisher secret scanner by [@&#8203;thoroc](https://github.com/thoroc) in [#&#8203;1160](jdx/hk#1160)
- **(builtins)** classify command effects by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;1167](jdx/hk#1167)
- **(check)** add normalized diagnostics and SARIF by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;1168](jdx/hk#1168)
- **(mcp)** expose hk to coding agents by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;1169](jdx/hk#1169)
- **(mcp)** add interactive hk dashboard by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;1170](jdx/hk#1170)
- **(run)** add agent-friendly execution scoping by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;1162](jdx/hk#1162)
- **(run)** add structured execution results by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;1163](jdx/hk#1163)
- **(step)** add command effect safety by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;1166](jdx/hk#1166)

##### πŸ› Bug Fixes

- **(check)** exclude deleted paths from ref checks by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;1177](jdx/hk#1177)
- **(docs)** keep search modal above site navigation by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;1188](jdx/hk#1188)
- **(step)** support argv prefixes for structured commands by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;1175](jdx/hk#1175)

##### πŸ§ͺ Testing

- **(init)** decouple agent docs assertion from releases by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;1187](jdx/hk#1187)

##### πŸ“¦οΈ Dependency Updates

- lock file maintenance by [@&#8203;renovate\[bot\]](https://github.com/renovate\[bot]) in [#&#8203;1151](jdx/hk#1151)
- simplify cargo version requirements by [@&#8203;jdx](https://github.com/jdx) in [#&#8203;1161](jdx/hk#1161)
- lock file maintenance by [@&#8203;renovate\[bot\]](https://github.com/renovate\[bot]) in [#&#8203;1173](jdx/hk#1173)
- update zizmorcore/zizmor-action action to v0.6.2 by [@&#8203;renovate\[bot\]](https://github.com/renovate\[bot]) in [#&#8203;1180](jdx/hk#1180)
- update jdx/mise-action action to v4.2.4 by [@&#8203;renovate\[bot\]](https://github.com/renovate\[bot]) in [#&#8203;1179](jdx/hk#1179)
- update anthropics/claude-code-action action to v1.0.184 by [@&#8203;renovate\[bot\]](https://github.com/renovate\[bot]) in [#&#8203;1178](jdx/hk#1178)
- update dependency [@&#8203;testing-library/jest-dom](https://github.com/testing-library/jest-dom) to v7 by [@&#8203;renovate\[bot\]](https://github.com/renovate\[bot]) in [#&#8203;1182](jdx/hk#1182)
- update dependency jsdom to v30 by [@&#8203;renovate\[bot\]](https://github.com/renovate\[bot]) in [#&#8203;1183](jdx/hk#1183)
- update dependency typescript to v7 by [@&#8203;renovate\[bot\]](https://github.com/renovate\[bot]) in [#&#8203;1184](jdx/hk#1184)
- update jdx/mise-action action to v4.2.4 by [@&#8203;renovate\[bot\]](https://github.com/renovate\[bot]) in [#&#8203;1181](jdx/hk#1181)

##### New Contributors

- [@&#8203;thoroc](https://github.com/thoroc) made their first contribution in [#&#8203;1160](jdx/hk#1160)

</details>

---

### Configuration

πŸ“… **Schedule**: (UTC)

- Branch creation
  - At any time (no schedule defined)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

β™» **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

πŸ”• **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yODguMCIsInVwZGF0ZWRJblZlciI6IjQzLjI4OC4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiLCJhdXRvbWF0aW9uOmJvdC1hdXRob3JlZCIsImRlcGVuZGVuY3ktdHlwZTo6bWlub3IiXX0=-->
stempler added a commit to wetransform/hk-config that referenced this pull request Aug 14, 2026
hk 1.55.0 (jdx/hk#1167) moved the builtins' argv under a CommandSpec
wrapper (command + effect), so the check/fix/check_list_files amends
that reached for `super.argv` failed with "Cannot find property argv
in object of type CommandSpec". Nest the mise-prefix argv override
under `command` to match the new schema.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
stempler added a commit to wetransform/hk-config that referenced this pull request Aug 14, 2026
hk 1.55.0 (jdx/hk#1167) moved the builtins' argv under a CommandSpec
wrapper (command + effect), so the check/fix/check_list_files amends
that reached for `super.argv` failed with "Cannot find property argv
in object of type CommandSpec". Nest the mise-prefix argv override
under `command` to match the new schema.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
stempler added a commit to wetransform/hk-config that referenced this pull request Aug 14, 2026
hk 1.55.0 (jdx/hk#1167) moved the builtins' argv under a CommandSpec
wrapper (command + effect), so the check/fix/check_list_files amends
that reached for `super.argv` failed with "Cannot find property argv
in object of type CommandSpec". Nest the mise-prefix argv override
under `command` to match the new schema.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant