Skip to content

fix(cli): skip unusable crowd-sniff base URL candidates and query scoped npm packages individually - #5199

Merged
mergify[bot] merged 3 commits into
mvanhorn:mainfrom
HawkForge:fix/crowd-sniff-base-url-candidates
Oct 23, 2026
Merged

mergify[bot] merged 3 commits into
mvanhorn:mainfrom
HawkForge:fix/crowd-sniff-base-url-candidates

Conversation

@HawkForge

Copy link
Copy Markdown
Contributor

Intent

Crowd-sniff fails fatally ("base URL must use HTTPS") when the first base URL harvested from SDK source is a local/templated one such as http://127.0.0.1:${port}, and npm download-count lookups silently lose the whole batch when it contains a scoped package.

Issue: Fixes #4244

Approach

  • ResolveBaseURL (internal/crowdsniff/specgen.go) still returns an explicit --base-url first. Otherwise it returns the first candidate that is https, parses with a host, is not localhost/loopback/private/link-local/unspecified, and has no {, }, or $ placeholder characters. Previously it returned the first non-blank candidate.
  • NPMSource.fetchDownloads (internal/crowdsniff/npm.go) sends unscoped packages in one bulk request as before and queries each @scope/name package on its own, since the bulk endpoint answers 400 for the whole batch when a scoped name is present. The request logic moved unchanged into fetchDownloadsBatch.
  • TestRunCrowdSniff_NonHTTPSBaseURL now passes the http URL via --base-url, since an http-only harvested candidate is now skipped (reported as "could not determine base URL") rather than rejected. The explicit-flag HTTPS check is unchanged.

Scope

Primary area: crowd-sniff (internal/crowdsniff, internal/cli/crowd_sniff.go tests)

Why this belongs in this repo: both defects are in the Printing Press crowd-sniff source aggregation and apply to any API, not to one printed CLI.

Risk

Candidates that were previously accepted and then rejected later (http, loopback) now fall through to the next candidate or to the existing "could not determine base URL; use --base-url" error. Valid https candidates behave as before. Scoped npm packages cost one extra request each (bounded by the existing 128-package cap); download failures remain non-fatal.

Output Contract

N/A (no templates, generated files, or output shape changes)

Verification

  • go test ./internal/crowdsniff/ ./internal/cli/ -run Crowd: pass
  • New table cases for TestResolveBaseURL and the scoped-package subtest of TestNPMSource_FetchDownloads fail with the source changes reverted and pass with them
  • gofmt -l internal: clean; go vet ./internal/crowdsniff/ ./internal/cli/: clean
  • scripts/golden.sh verify: not run, crowd-sniff output shape is unchanged
  • The scoped-package 400 is simulated in the test (as described in the issue); not checked against the live registry.

Follow-up, out of scope here: separating the exit code for "found nothing" from "could not look".

AI / Automation Disclosure

  • No AI or automation was used
  • Human-reviewed: AI or automation was used, and a human reviewed the work for intent, fit, and obvious issues before submission
  • AI-reviewed only: an AI agent reviewed the work, but no human reviewed it before submission
  • Fully automated: generated and submitted without human review for this specific change

…ped npm packages individually

ResolveBaseURL returned the first non-blank candidate, so an SDK-source match like http://127.0.0.1:${port} won and runCrowdSniff failed with "base URL must use HTTPS". It now skips candidates that are not https, are loopback/private/localhost, contain template placeholders, or do not parse. npm's bulk downloads endpoint rejects the whole request when it includes a scoped package, so scoped packages are now requested one at a time.

Fixes #4244
@mergify

mergify Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 All 2 merge protections satisfied — ready to merge.

Show 2 satisfied protections

🟢 require-ready-label-and-ci

  • #changes-requested-reviews-by = 0
  • #review-threads-unresolved = 0
  • check-success = build-and-test
  • check-success = generated-test
  • check-success = go-lint
  • check-success = golden
  • check-success = pr-title
  • check-success = test
  • any of:
    • label = ready-to-merge
    • all of:
      • head = release-please--branches--main
      • title ~= ^chore\(main\): release
  • any of:
    • -files ~= ^(\.github/workflows/|\.github/scripts/|scripts/|\.github/CODEOWNERS$)
    • approved-reviews-by = mvanhorn
    • approved-reviews-by = tmchow
    • author = mvanhorn
    • author = tmchow
  • any of:
    • check-success = Greptile Review
    • label = queued
    • check-neutral = Greptile Review
    • check-skipped = Greptile Review
    • head ~= ^mergify/merge-queue/
    • all of:
      • head = release-please--branches--main
      • title ~= ^chore\(main\): release

🟢 🚦 Auto-queue

When all merge protections are satisfied and these conditions match, this pull request will be queued automatically.

  • label = ready-to-merge

@greptile-apps

greptile-apps Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no blocking issues remain.

Summary

crowd-sniff skips unusable harvested base URLs and requests scoped npm download counts separately.

  • Crowd-sniff skips bad URL guesses when choosing an API base URL.
  • Scoped npm packages get their own download-count requests.

Reviews (3) · Last reviewed commit: "fix(cli): address lint findings in crowd..." · Reviewed by Greptile

Comment thread internal/crowdsniff/specgen.go Resolved
Comment thread internal/crowdsniff/npm.go Resolved
Comment thread internal/crowdsniff/npm.go Resolved
@tmchow tmchow added the ready-to-merge Queue when Mergify passes; scripts/.github PRs also need Approve from tmchow/mvanhorn. label Oct 23, 2026
@mergify mergify Bot added the queued PR is in the Mergify merge queue label Oct 23, 2026
@mergify mergify Bot mentioned this pull request Oct 23, 2026
56 of 96 tasks
@mergify
mergify Bot merged commit 5e3ca12 into mvanhorn:main Oct 23, 2026
35 of 36 checks passed
@mergify

mergify Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Merge Queue Status

This pull request spent 25 minutes 13 seconds in the queue, including 24 minutes 26 seconds running CI.

Required conditions to merge

@mergify mergify Bot removed the queued PR is in the Mergify merge queue label Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-merge Queue when Mergify passes; scripts/.github PRs also need Approve from tmchow/mvanhorn.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

crowd-sniff: npm downloads API returns 400 for every API, and a loopback base URL is fatal

2 participants