Skip to content

fix(cli): stop verification false positives on website-backed CLIs - #5256

Merged
mergify[bot] merged 3 commits into
mainfrom
tmchow/verify-false-positives-4907-0313
Oct 26, 2026
Merged

mergify[bot] merged 3 commits into
mainfrom
tmchow/verify-false-positives-4907-0313

Conversation

@tmchow

@tmchow tmchow commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Intent

Three press verification checks flagged correct code while printing a website-backed CLI. All three defects were still present on current main (no existing regression test pinned a fix). This change fixes all three.

Closes #4907

Approach

Traffic analysis. Vendor names and captcha words inside JS/CSS no longer outvote a clean document or XHR, and captcha words in an asset never set browser_required on their own. That suppression applies to body markers only. Header challenges (cf-mitigated, Vercel), vendor headers (X-DataDome, CF-RAY, X-Akamai-Transformed), and explicit 403/429 denial or login redirects still count on a static asset. A document or XHR that is itself a challenge still sets browser_required, including an HTML challenge whose URL ends in .js. A first-party AWS WAF challenge script still sets browser_clearance_http beside a clean API response; that existing case was left in place.

Novel-feature hosts. A declared registrable apex passes when research artifacts observed one of its subdomains (www.subito.it covers subito.it). Lookalikes do not: evilsubito.it, subito.it.evil.com, and an unobserved sibling such as shop.subito.it still fail. schema.org / www.schema.org are ignored only when every use of that name in the package is a string affix (TrimPrefix, HasPrefix, and the same family), or the name only initializes another exempt name. That includes a package-level const used from another file, local const and var, a const group that copies the previous expression, and an alias chain. A const spec that omits its expression list is walked as that previous list, at package level and inside a function, so const (base = "https://schema.org/"; trim = base; fetch) still fails when fetch is passed to http.Get. Fetching that host still fails, including through an alias, a helper in another file, or a use that still refers to the package constant before a local shadow.

verify-skill. Flag lookup walks helpers that receive cmd, nested up to 8 levels, and stops on cycles. A flag that is absent, or that lives past the depth guard, still errors.

Scope

Primary area: browsersniff traffic analysis, dogfood novel-host check, verify-skill flag resolution.

Why this belongs in this repo: the false positives come from the Printing Press checks, not from one printed CLI. The same shapes show up on any website-backed print.

Risk

A first-party challenge script body (awswaf / aws-waf) is still evidence even when a clean document exists, matching the existing hostless-target test. Vendor-name substrings in a static asset with no clean document or XHR still classify as browser_http; they do not set browser_required. A real vendor header or denial/login response on an asset now keeps that observation even beside a clean document.

No generator templates, manifests, or MCP schemas changed. Golden fixtures are unchanged.

Output Contract

N/A

Verification

  • go test -count=1 -timeout 45m ./... passed for every package except internal/generator, which was still running when the 45m package timeout fired. go test -count=1 -timeout 90m ./internal/generator/ then passed.
  • GOTOOLCHAIN=go1.26.10 bash scripts/golden.sh verify passed (fixtures request go 1.24, which this environment cannot download; the local 1.26.10 toolchain builds them)
  • GOOS=linux, GOOS=darwin, and GOOS=windows go vet of internal/browsersniff, internal/pipeline, and internal/cli passed
  • golangci-lint run (v2.11.4) on those three packages: 0 issues
  • Inherited const groups (327329e0): go test -count=1 -timeout 600s ./internal/pipeline/, go vet ./internal/pipeline/, and golangci-lint run ./internal/pipeline/ (v2.11.4) passed. No generator or golden fixture changes.

Self-review gate

Ran before pushing this update.

  • Removed no load-bearing comments; the comments that remain say why a marker or host literal is ignored, not what the next line does.
  • No generator template changed, so there is no generated-module compile to add.
  • Goldens were verified and did not need regeneration.
  • Cross-platform vet above found no Linux-only paths, permissions, or signals.
  • Each new branch has a test, including the negative case: a real document/XHR challenge and an HTML challenge at a script URL still require a browser; a challenge header on an asset still counts; a vendor header on an asset beside a clean document still counts while the same words in the asset body do not; a 403 asset without denial text and a redirect that is not a login do not count; a first-party WAF script still counts beside clean traffic; a genuinely unknown called host still fails; a fetch of schema.org still fails, including a local const/var, a const copied across files, an alias chain, an inherited const name after an alias (package-level and function-local), and a fetch that occurs before a local shadow; a missing flag, a cyclic helper chain, and a helper past the depth guard still error.
  • Bypass attempts that stay rejected: evilsubito.it, subito.it.evil.com, shop.subito.it, notschema.org, evil.schema.org, schema.org.evil.com, alias-then-fetch, grouped-const copy then fetch, inherited const alias then fetch, and a parameter that shadows the package const (the fetch host is the argument, not schema.org).

AI / Automation Disclosure

  • No AI or automation was used
  • Human-reviewed: AI or automation was used, and a human reviewed the work for intent, fit, and obvious issues before submission
  • AI-reviewed only: an AI agent reviewed the work, but no human reviewed it before submission
  • Fully automated: generated and submitted without human review for this specific change
Open in Web Open in Cursor 

Anti-bot names and captcha words inside JS/CSS no longer outvote a clean
document or XHR, and captcha words in an asset no longer force
browser_required. A first-party challenge script and a real document or
XHR challenge still do. Novel-host checks accept a registrable apex when
one of its subdomains was observed, and ignore schema.org only when it is
a string affix rather than a request. verify-skill follows nested flag
helpers, bounded by a depth and cycle guard.

Closes #4907

Co-authored-by: Trevin Chow <tmchow@users.noreply.github.com>
@tmchow
tmchow requested a review from mvanhorn as a code owner October 26, 2026 12:11
@mergify

mergify Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 All 2 merge protections satisfied — ready to merge.

Show 2 satisfied protections

🟢 require-ready-label-and-ci

  • #changes-requested-reviews-by = 0
  • #review-threads-unresolved = 0
  • check-success = build-and-test
  • check-success = generated-test
  • check-success = go-lint
  • check-success = golden
  • check-success = pr-title
  • check-success = test
  • any of:
    • label = ready-to-merge
    • all of:
      • head = release-please--branches--main
      • title ~= ^chore\(main\): release
  • any of:
    • author = tmchow
    • -files ~= ^(\.github/workflows/|\.github/scripts/|scripts/|\.github/CODEOWNERS$)
    • approved-reviews-by = mvanhorn
    • approved-reviews-by = tmchow
    • author = mvanhorn
  • any of:
    • check-success = Greptile Review
    • label = queued
    • check-neutral = Greptile Review
    • check-skipped = Greptile Review
    • head ~= ^mergify/merge-queue/
    • all of:
      • head = release-please--branches--main
      • title ~= ^chore\(main\): release

🟢 🚦 Auto-queue

When all merge protections are satisfied and these conditions match, this pull request will be queued automatically.

  • label = ready-to-merge

@greptile-apps

greptile-apps Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; the inherited-constant finding is fixed and no new blocking issue was found.

Summary

This PR reduces false positives in checks for website-backed CLIs.

  • Traffic analysis separates asset text from real challenge signals.
  • Novel-host checks recognize observed domains and vocabulary prefixes.
  • verify-skill finds flags declared by nested helpers.

The latest update fixes inherited constant expressions at package and local scope. The numbered previous finding is fully addressed. No new actionable issues or rule violations were found.

Reviews (3) · Last reviewed commit: "fix(cli): walk inherited const expressio..." · Reviewed by Greptile

Comment thread internal/browsersniff/analysis.go Resolved
Comment thread internal/pipeline/novel_host_check.go Resolved
Comment thread internal/pipeline/novel_host_check.go Resolved
Asset suppression now drops body markers only. Vendor headers and explicit
denial or login responses on static assets still count.

schema.org affix exemptions require every package use to be an affix,
including other files, local const and var, and aliases. A fetch keeps
the host checked.

Co-authored-by: Trevin Chow <tmchow@users.noreply.github.com>
Comment thread internal/pipeline/novel_host_affix.go Resolved
A const spec that omits its expression list repeats the previous list.
Skipping that spec left a fetch of the inherited name off the alias chain,
so a vocabulary URL could be exempted while still being fetched.

Co-authored-by: Trevin Chow <tmchow@users.noreply.github.com>
@tmchow tmchow added the ready-to-merge Queue when Mergify passes; scripts/.github PRs also need Approve from tmchow/mvanhorn. label Oct 26, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Mergify will not queue this PR yet. ready-to-merge is set, but this PR changes protected paths (scripts/, .github/workflows/, .github/scripts/, or CODEOWNERS) and still needs an explicit GitHub Approve from @tmchow or @mvanhorn.

The label signals intent to queue; Mergify's approved-reviews-by gate still requires a formal approval review on the PR. After approving, comment @Mergifyio refresh if Mergify Merge Protections does not update within a minute.

@mergify mergify Bot added the queued PR is in the Mergify merge queue label Oct 26, 2026
@mergify mergify Bot mentioned this pull request Oct 26, 2026
56 of 96 tasks
@mergify
mergify Bot merged commit bfcdf3c into main Oct 26, 2026
36 of 37 checks passed
@mergify

mergify Bot commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Merge Queue Status

This pull request spent 21 minutes 30 seconds in the queue, including 21 minutes 11 seconds running CI.

Required conditions to merge

@mergify
mergify Bot deleted the tmchow/verify-false-positives-4907-0313 branch October 26, 2026 16:13
@mergify mergify Bot removed the queued PR is in the Mergify merge queue label Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-merge Queue when Mergify passes; scripts/.github PRs also need Approve from tmchow/mvanhorn.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

verification false positives on a website-backed CLI (traffic analysis, dogfood host check, verify-skill helpers)

2 participants