Skip to content

verify data-pipeline probe fails CLIs that declare no sync resource #5380

Description

@josselinlbe

Filed by /printing-press-retro from a run of Autom (scorecard 90/100, 100% verify pass).

Summary

Since the data-pipeline probe was scoped to one declared sync resource (#5364, first released in 4.35.3), verify fails every CLI that declares no sync resource. Query-only APIs (search/SERP, validation, screenshot and similar endpoints that need a query and have no collection endpoint) correctly get an empty defaultSyncResources(), so they now reach verify verdict FAIL at a 100% command pass rate and shipcheck cannot reach ship. The probe already has a branch for this shape, but it is no longer reachable.

Where to look

  • Component: scorer
  • Issue type: bug
  • Likely area: internal/pipeline/runtime.go (runDataPipelineTest) and internal/pipeline/runtime_sync_probe.go (chooseDataPipelineProbeResource)
  • Triggered when: the printed CLI's defaultSyncResources() returns []string{} and neither the manifest (verify_sync_resource) nor research.json sets a verify sync hint, in mock or live mode.

What we observed

verify on a fresh 4.35.4 print of a query-only OpenAPI spec (22 GET endpoints, every search requires query; no list endpoint):

Pass Rate: 100% (53/53 passed, 0 critical)
Data Pipeline: FAIL: no safe sync resource for the data-pipeline probe; refusing unscoped sync
Verdict: FAIL

shipcheck then reports verify FAIL while every other leg passes.

In runDataPipelineTest, chooseDataPipelineProbeResource(cliDir) runs before any sync and returns "", "FAIL: no safe sync resource ..." when the declared list is empty (runtime_sync_probe.go, the len(safe) == 0 return). The function then returns false immediately. The branch further down that was written for this case,

if mode == "mock" && strings.TrimSpace(cliDir) != "" && !cliHasSyncableResources(cliDir) {
    return annotateDataPipelineProbe(true, fmt.Sprintf("PASS: %d domain tables created (mock mode; no syncable resources declared)", len(tables)), resource)
}

can no longer be reached. Both spots are unchanged on current main.

The same shape passed before 4.35.3. In the public library:

  • library/ai/serply (printed with 4.32.5): internal/cli/sync.go has defaultSyncResources() returning []string{}, and its .printing-press.json records verify data_pipeline: true, verdict: PASS.
  • library/developer-tools/scrape-do (4.20.0): defaultSyncResources() also returns []string{}.

A reprint of either on current main should hit the same FAIL.

The documented escape (source: local-sqlite, see #4065) does not fit: it also drops live API verification, which is wrong for a CLI that calls a live API.

Suspected root cause

High confidence: the early return added with the scoped probe treats "no declared resource" the same as "unsafe or undeclared hint". The refusal to run an unscoped sync is correct; turning it into a FAIL for CLIs that have nothing to sync is the regression.

Suggested direction

When the declared sync resources are empty and no verify sync hint is set, skip the sync step and return a passing detail (e.g. SKIP (no syncable resources declared)), instead of FAIL. Keep the existing FAILs when a hint names an undeclared or unsafe resource, and keep never running sync without --resources. Verify by running mock verify on a fixture CLI with an empty defaultSyncResources().

Acceptance criteria

  • positive: a fixture CLI whose defaultSyncResources() returns []string{}, with no manifest or research hint, gets data_pipeline: true and verify PASS in mock mode, and no sync subprocess is spawned.
  • negative: a CLI that declares resources still probes exactly one scoped resource; a hint naming an undeclared resource still FAILs; no path runs sync without --resources.

Frequency

subclass: query-only APIs (no collection endpoint)

Complexity

small

Scope boundary

Does not change sync generation, the local-sqlite classification, or probing for CLIs that declare resources.

Dependencies

None.

Related issues


Generated by /printing-press-retro · CLI Printing Press

Activity

  1. added
    bugSomething isn't working
    priority:P2Medium priority: meaningful recurring defect or capability gap
    surface:verifyVerify, dogfood, scorecard, shipcheck, and validation gates
    on Nov 12, 2026
  2. tmchow commented on Nov 12, 2026

    @tmchow
    Owner

    Triage: confirmed on main. chooseDataPipelineProbeResource returns FAIL: no safe sync resource when safe is empty (internal/pipeline/runtime_sync_probe.go:76-77), and runDataPipelineTest returns on it at internal/pipeline/runtime.go:712-714, so the no-syncable-resources PASS branch at runtime.go:778 can't be reached. Query-only CLIs fail verify. Labeled P2: it blocks shipcheck for that class, but the printed CLI itself works.

  3. josselinlbe commented on Nov 12, 2026

    @josselinlbe
    Author

    Thanks for confirming and for the pointers.

    One way to keep the safety check and reach the PASS branch: tell "the CLI declares no sync resource at all" apart from "it declares some, but none is safe".

    • In chooseDataPipelineProbeResource, when the declared list is empty and there is no --sync-resource hint, return an empty name and no failure.
    • In runDataPipelineTest, treat that as "no syncable resources" and fall through to the existing PASS at runtime.go:778.
    • Keep the current FAIL when resources are declared but every one is excluded or unsafe.

    Happy to open a PR with that and a test for a query-only spec, if that helps. The Autom CLI I'm shipping is query-only and otherwise passes shipcheck, so this is the only thing holding it.

  4. added a commit that references this issue on Nov 12, 2026
    847b390
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcomp:scorerpriority:P2Medium priority: meaningful recurring defect or capability gapsurface:verifyVerify, dogfood, scorecard, shipcheck, and validation gates

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions