Repository navigation
Description
Filed by
/printing-press-retrofrom a run of Autom (scorecard 90/100, 100% verify pass).
Summary
Since the data-pipeline probe was scoped to one declared sync resource (#5364, first released in 4.35.3), verify fails every CLI that declares no sync resource. Query-only APIs (search/SERP, validation, screenshot and similar endpoints that need a query and have no collection endpoint) correctly get an empty defaultSyncResources(), so they now reach verify verdict FAIL at a 100% command pass rate and shipcheck cannot reach ship. The probe already has a branch for this shape, but it is no longer reachable.
Where to look
- Component: scorer
- Issue type:
bug - Likely area:
internal/pipeline/runtime.go(runDataPipelineTest) andinternal/pipeline/runtime_sync_probe.go(chooseDataPipelineProbeResource) - Triggered when: the printed CLI's
defaultSyncResources()returns[]string{}and neither the manifest (verify_sync_resource) nor research.json sets a verify sync hint, in mock or live mode.
What we observed
verify on a fresh 4.35.4 print of a query-only OpenAPI spec (22 GET endpoints, every search requires query; no list endpoint):
Pass Rate: 100% (53/53 passed, 0 critical)
Data Pipeline: FAIL: no safe sync resource for the data-pipeline probe; refusing unscoped sync
Verdict: FAIL
shipcheck then reports verify FAIL while every other leg passes.
In runDataPipelineTest, chooseDataPipelineProbeResource(cliDir) runs before any sync and returns "", "FAIL: no safe sync resource ..." when the declared list is empty (runtime_sync_probe.go, the len(safe) == 0 return). The function then returns false immediately. The branch further down that was written for this case,
if mode == "mock" && strings.TrimSpace(cliDir) != "" && !cliHasSyncableResources(cliDir) {
return annotateDataPipelineProbe(true, fmt.Sprintf("PASS: %d domain tables created (mock mode; no syncable resources declared)", len(tables)), resource)
}can no longer be reached. Both spots are unchanged on current main.
The same shape passed before 4.35.3. In the public library:
library/ai/serply(printed with 4.32.5):internal/cli/sync.gohasdefaultSyncResources()returning[]string{}, and its.printing-press.jsonrecords verifydata_pipeline: true,verdict: PASS.library/developer-tools/scrape-do(4.20.0):defaultSyncResources()also returns[]string{}.
A reprint of either on current main should hit the same FAIL.
The documented escape (source: local-sqlite, see #4065) does not fit: it also drops live API verification, which is wrong for a CLI that calls a live API.
Suspected root cause
High confidence: the early return added with the scoped probe treats "no declared resource" the same as "unsafe or undeclared hint". The refusal to run an unscoped sync is correct; turning it into a FAIL for CLIs that have nothing to sync is the regression.
Suggested direction
When the declared sync resources are empty and no verify sync hint is set, skip the sync step and return a passing detail (e.g. SKIP (no syncable resources declared)), instead of FAIL. Keep the existing FAILs when a hint names an undeclared or unsafe resource, and keep never running sync without --resources. Verify by running mock verify on a fixture CLI with an empty defaultSyncResources().
Acceptance criteria
- positive: a fixture CLI whose
defaultSyncResources()returns[]string{}, with no manifest or research hint, getsdata_pipeline: trueand verify PASS in mock mode, and nosyncsubprocess is spawned. - negative: a CLI that declares resources still probes exactly one scoped resource; a hint naming an undeclared resource still FAILs; no path runs
syncwithout--resources.
Frequency
subclass: query-only APIs (no collection endpoint)
Complexity
small
Scope boundary
Does not change sync generation, the local-sqlite classification, or probing for CLIs that declare resources.
Dependencies
None.
Related issues
- verify --live data-pipeline probe syncs every resource when the repos default fails #5360 / fix(cli): scope verify data-pipeline sync to one declared resource #5364 — prior fix (
extends): scoping the probe to one declared resource is right; this is the empty-declaration case it no longer handles. - skill: harvest/scrape-built corpora are not routed to the local-datastore classification, so verify Data Pipeline fails them in mock mode #4065 — prior retro (
extends): harvest-built corpora used thelocal-sqliteescape; networked query-only APIs cannot. - scorer: dogfood lacks a live stateful sync→analytics→search pass (per-command smoke misses populated-store bugs) #2488 — open issue (
related-area): dogfood lacks a live stateful sync→analytics→search pass.
Generated by /printing-press-retro · CLI Printing Press
Activity
tmchow commentedon Nov 12, 2026
Triage: confirmed on main.
chooseDataPipelineProbeResourcereturnsFAIL: no safe sync resourcewhensafeis empty (internal/pipeline/runtime_sync_probe.go:76-77), andrunDataPipelineTestreturns on it at internal/pipeline/runtime.go:712-714, so the no-syncable-resources PASS branch at runtime.go:778 can't be reached. Query-only CLIs fail verify. Labeled P2: it blocks shipcheck for that class, but the printed CLI itself works.josselinlbe commentedon Nov 12, 2026
Thanks for confirming and for the pointers.
One way to keep the safety check and reach the PASS branch: tell "the CLI declares no sync resource at all" apart from "it declares some, but none is safe".
chooseDataPipelineProbeResource, when the declared list is empty and there is no--sync-resourcehint, return an empty name and no failure.runDataPipelineTest, treat that as "no syncable resources" and fall through to the existing PASS at runtime.go:778.Happy to open a PR with that and a test for a query-only spec, if that helps. The Autom CLI I'm shipping is query-only and otherwise passes shipcheck, so this is the only thing holding it.
fix(cli): pass verify when a CLI declares no sync resource (mvanhorn#…