Skip to content

fix(cli): count structured non-zero outcomes and skip account-disabled features in live dogfood - #5354

Merged
mergify[bot] merged 2 commits into
mvanhorn:mainfrom
cathrynlavery:fix/dogfood-structured-outcomes
Nov 6, 2026
Merged

mergify[bot] merged 2 commits into
mvanhorn:mainfrom
cathrynlavery:fix/dogfood-structured-outcomes

Conversation

@cathrynlavery

Copy link
Copy Markdown
Collaborator

Fixes #5351.

Live dogfood had two blind spots. A command whose contract is "exit non-zero with a structured verdict" (a go/no-go check that exits 2 on NO-GO) was always skipped, so the feature stayed hollow even though it returned a real result. And endpoints for a feature switched off in the account (403/404 "... is not enabled for this ...") were reported as failures, which pushed operators toward hand-editing rows.

Changes

  • liveDogfoodStructuredOutcome: a declared non-zero exit counts as a happy-path and JSON pass only when stdout is a non-empty JSON object (or results object) with no error and no success: false. Usage errors and silent exits still don't count.
  • 403/404 responses whose body says the feature is not enabled for the account are skipped, with the live response kept as evidence. An ordinary missing resource still fails.
  • live_dogfood_outcome_test.go covers NO-GO verdicts at the top level and under results, error envelopes in both places, a full run that counts a declared NO-GO exit, disabled-feature 403 and 404 bodies, and the cases that must stay failures: a plain 404, a 500 with the same message, the message without an HTTP status, and exit 0.
  • Phase 18 of the printing-press skill documents both rules and tells operators to inspect the evidence instead of editing rows.

Verification

  • go build ./..., go vet ./internal/... pass.
  • go test ./internal/pipeline: all pass except TestSnapshotLiveCheckBinaryStagesBesideSymlinkTarget, which fails the same way on main at 26d1e3f (macOS /var vs /private/var).
  • Found on a Circle Admin API print (reviews disabled in the account; a founding go/no-go command).

(cherry picked from commit fe403dec10e3b4acd7d519b3c1209737911b09e9)
@mergify

mergify Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 All 2 merge protections satisfied — ready to merge.

Show 2 satisfied protections

🟢 require-ready-label-and-ci

  • #changes-requested-reviews-by = 0
  • #review-threads-unresolved = 0
  • check-success = build-and-test
  • check-success = generated-test
  • check-success = go-lint
  • check-success = golden
  • check-success = pr-title
  • check-success = test
  • any of:
    • label = ready-to-merge
    • all of:
      • head = release-please--branches--main
      • title ~= ^chore\(main\): release
  • any of:
    • -files ~= ^(\.github/workflows/|\.github/scripts/|scripts/|\.github/CODEOWNERS$)
    • approved-reviews-by = mvanhorn
    • approved-reviews-by = tmchow
    • author = mvanhorn
    • author = tmchow
  • any of:
    • check-success = Greptile Review
    • label = queued
    • check-neutral = Greptile Review
    • check-skipped = Greptile Review
    • head ~= ^mergify/merge-queue/
    • all of:
      • head = release-please--branches--main
      • title ~= ^chore\(main\): release

🟢 🚦 Auto-queue

When all merge protections are satisfied and these conditions match, this pull request will be queued automatically.

  • label = ready-to-merge

@greptile-apps

greptile-apps Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; both previous findings are addressed and no new actionable issue was found.

Summary

The PR counts declared nonzero exits when they return a structured result and skips explicit account-disabled HTTP 403/404 responses.

  • Declared nonzero exits count as passes when commands return structured results.
  • Account-disabled feature replies are skipped with their live response kept as evidence.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Command finishes] --> B{Exit zero?}
  B -->|Yes| C[Existing success checks]
  B -->|No| D{Declared outcome code?}
  D -->|Yes| E{Valid nonempty object without an error?}
  E -->|Yes| F[Count as pass]
  E -->|No| G[Keep declared-exit skip]
  D -->|No| H{Explicit account-disabled 403 or 404?}
  H -->|Yes| I[Skip and retain response]
  H -->|No| J[Existing failure and access checks]
Loading

Reviews (2) · Last reviewed commit: "fix(cli): check account-disabled respons..." · Reviewed by Greptile

Comment thread internal/pipeline/live_dogfood.go Resolved
Comment thread internal/pipeline/live_dogfood.go Resolved
@cathrynlavery cathrynlavery added the ready-to-merge Queue when Mergify passes; scripts/.github PRs also need Approve from tmchow/mvanhorn. label Nov 6, 2026
@mergify mergify Bot added the queued PR is in the Mergify merge queue label Nov 6, 2026
@mergify
mergify Bot merged commit d10c27c into mvanhorn:main Nov 6, 2026
41 of 45 checks passed
@mergify

mergify Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

✅ Merged into main as d10c27c at 00:21 UTC

19 min in the queue, 18 min of it CI.

So far

13 merge conditions met
  • check-success = build-and-test · required by Merge Protections rule require-ready-label-and-ci
  • check-success = generated-test · required by Merge Protections rule require-ready-label-and-ci
  • check-success = go-lint · required by Merge Protections rule require-ready-label-and-ci
  • check-success = golden · required by Merge Protections rule require-ready-label-and-ci
  • check-success = test · required by Merge Protections rule require-ready-label-and-ci
  • github-review-approved · required by GitHub branch protection
  • #changes-requested-reviews-by = 0 · required by Merge Protections rule require-ready-label-and-ci
  • #review-threads-unresolved = 0 · required by Merge Protections rule require-ready-label-and-ci
  • check-success = pr-title · required by Merge Protections rule require-ready-label-and-ci
  • any of · required by Merge Protections rule require-ready-label-and-ci:
    • label = ready-to-merge
    • all of:
      • head = release-please--branches--main
      • title ~= ^chore\(main\): release
  • any of · required by Merge Protections rule require-ready-label-and-ci:
    • -files ~= ^(\.github/workflows/|\.github/scripts/|scripts/|\.github/CODEOWNERS$)
    • approved-reviews-by = mvanhorn
    • approved-reviews-by = tmchow
    • author = mvanhorn
    • author = tmchow
  • any of · required by Merge Protections rule require-ready-label-and-ci:
    • label = queued
    • check-neutral = Greptile Review
    • check-skipped = Greptile Review
    • check-success = Greptile Review
    • head ~= ^mergify/merge-queue/
    • all of:
      • head = release-please--branches--main
      • title ~= ^chore\(main\): release
  • any of · required by GitHub branch protection:
    • check-neutral = Mergify Merge Protections
    • check-skipped = Mergify Merge Protections
    • check-success = Mergify Merge Protections

Times in UTC · dashboard · how the merge queue works

@mergify mergify Bot removed the queued PR is in the Mergify merge queue label Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-merge Queue when Mergify passes; scripts/.github PRs also need Approve from tmchow/mvanhorn.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Live dogfood can't count documented non-zero outcomes or account-disabled features

1 participant