Skip to content

verify --live data-pipeline probe syncs every resource when the repos default fails #5360

Closed

Description

@cathrynlavery

verify --live data-pipeline probe syncs every resource when the GitHub-shaped default fails.

internal/pipeline/runtime.go (around lines 725-745) probes the data pipeline with sync --db <tmp> --resources repos --full. repos only exists on GitHub-shaped CLIs, so on any other API that first attempt fails and the probe falls back to sync --db <tmp> --full, then sync --db <tmp>, sync --full and sync. In live mode boundedSyncProbeArgs adds --max-pages 1, but the fallback is still unscoped: it reads one page of every syncable resource, each attempt capped at 30 seconds.

Two problems from a Tella reprint:

  1. Scope: the unscoped fallback reads resources the operator deliberately keeps out of live testing (Tella's social-account listings, publishing state). Reads only, but they're outside the agreed test surface, and nothing records which resources were touched.
  2. Time: a scoped sync --resources videos on that account took 98.5 s, so the 30 s attempts time out and the probe reports a pipeline failure on a working CLI.

Expected: pick the probe resource from the CLI itself (its declared sync resources, or a research/manifest hint such as a verify sync resource) instead of a hardcoded repos, let operators exclude resources from live probes, and size the timeout for one bounded page of one resource. If no safe resource is known, skip the live probe with a clear reason rather than syncing everything.

Activity

  1. added
    bugSomething isn't working
    priority:P2Medium priority: meaningful recurring defect or capability gap
    surface:verifyVerify, dogfood, scorecard, shipcheck, and validation gates
    on Nov 6, 2026
  2. tmchow commented on Nov 6, 2026

    @tmchow
    Owner

    Daily triage (taxonomy + merit)

    Validation (confirmed on main): internal/pipeline/runtime.go runDataPipelineTest (≈725–744) hardcodes sync --resources repos --full, then falls back to unscoped sync --db … --full / sync --db / sync --full / sync, each with a 30s cap. boundedSyncProbeArgs only adds --max-pages 1 in live mode — it does not pick a CLI-declared resource. Realistic non-GitHub sync CLIs hit the unscoped fallback (Tella-shaped repro in the report is reachable).

    Worth: P2. Real generalizing verify defect (over-broad live sync + false FAIL on slow single-resource accounts). Printed CLI still works; not a P1 safety/correctness break of generated commands.

    Taxonomy: bug · priority:P2 · comp:scorer · surface:verify (matches prior novel-host / dogfood-verify ownership). Keeping open for a fix wave.

  3. tmchow commented on Nov 6, 2026

    @tmchow
    Owner

    Fixed by #5364 (merged as 4e19151). verify --live data-pipeline probe now scopes sync to one declared safe resource instead of falling back to unscoped sync --full.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcomp:scorerpriority:P2Medium priority: meaningful recurring defect or capability gapsurface:verifyVerify, dogfood, scorecard, shipcheck, and validation gates

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions