Skip to content

Changelog

- `PIPELINE.md` — this pipeline spec

v0.4.0changelog

Files in this spec

  • PIPELINE.md — this pipeline spec
  • pipelines/code-refactor.yaml — reference implementation, updates with v0.2 fields
  • packages/pipeline-runtime/ — shared executor (server + CLI + Docker runner)
  • packages/cli/src/lib/pipeline-spec.ts — spec validator (extended in v0.2)
  • packages/server/src/routes/pipelines.ts — server-side execution + provenance signing

Changelog: v0.2 → v0.3

RID Feature Status Notes
R6 Explicit provider field BREAKING Decouples routing from model name. provider: openai + model: gpt-4o now mandatory on every LLM stage. v0.2 implicit provider resolution removed. Migration: see r6-migration.test.ts.
R7 ${VAR} / ${VAR:-default} env-var substitution New 12-factor convention. 3-layer defense: validate CLI, run CLI, server publish-pipeline. Substitution happens before yaml.load.
R10 Stage output refs (${stages.X.outputs.Y}) New Inline template syntax. Composes with env-var substitution (both use ${...}).
R10.1 Retry on transient errors New retry: { attempts, backoff, retryable_status }. No JS expressions. Throws RetryExhaustedError on budget exhaustion.
R10.1.b run_if fallback chains New run_if: "{{ stages.x.status == 'failed' }}" for compensation gates.
R10.2 Checkpoints (R2 storage) New Wall-clock + crash recovery for long pipelines. checkpoint: { on, storage: { type: r2, path }, retention }.
R10.3 Streaming event sink New pipeline.started / pipeline.completed / pipeline.failed / stage.started / stage.completed / stage.skipped / stage.failed. CallbackEventSink + NoopEventSink.
R10.4 Parallel DAG execution New Topological sort + Promise.all. parallel.of fan-out over collections. Partial-failure semantics: per-stage status reported in provenance, siblings continue.
R10.5 when DSL (conditional stages) New Pure-comparison DSL. Operators: ==, !=, <, >, <=, >=, in, not in, all, any. No JS sandbox. Skipped stages emit '' sentinel.
R10.6 Runtime constraints New allowed_providers, max_cost_usdc, max_tokens, allowed_skills. Validated by executor at runtime.
R11 Docker image multi-arch + cosign signing New linux/amd64 + linux/arm64. Image published to registry with cosign signature for supply-chain integrity.
R13 Composition (uses: pipeline:NAME@VERSION) New Stages reference published pipelines. MAX_COMPOSITION_DEPTH = 4, throws CompositionDepthError on cycle, throws PipelineFetchError on missing. Id-prefixing: parent.child.grandchild.

Behavioral-oracle tests proving these are shipped:

  • R6 → packages/pipeline-runtime/tests/pipeline-r6-schema.test.ts, provider-registry.test.ts, r6-migration.test.ts
  • R7 → packages/pipeline-runtime/tests/resolve-vars.test.ts, examples.test.ts, packages/cli/tests/validate.test.ts, run-registry.test.ts, packages/server/tests/publish-pipeline-r6.test.ts (env-var path), packages/cli/tests/r7-cli.test.ts, packages/server/tests/r7-server.test.ts, packages/pipeline-runtime/tests/r7-yaml-integration.test.ts
  • R10 (refs) → stage-output-refs.test.ts, integration-r10-x.test.ts
  • R10.1 (retry) → stages.test.ts, retry.test.ts
  • R10.1.b (fallback) → fallback.test.ts
  • R10.2 (checkpoints) → checkpoints.test.ts, packages/server/tests/checkpoint-r10-4.test.ts
  • R10.3 (streaming) → streaming-events.test.ts
  • R10.4 (parallel) → parallel.test.ts, integration-r10-x.test.ts
  • R10.5 (when) → when.test.ts, executor-when.test.ts
  • R10.6 (constraints) → runtime-constraints.test.ts
  • R11 (docker) → .github/workflows/publish-image.yml + Dockerfile
  • R13 (composition) → composition.test.ts, composition-expand.test.ts, integration-r10-x.test.ts

Open questions for v0.4+

  1. Tool-call pricing. Each MCP tool has different cost (Playwright navigate vs GitHub search vs Stripe refund). Pricing model TBD. Defer to v0.4 once MCP execution is wired (forward-compat declared in v0.2).
  2. Parallel DAG semantics — RESOLVED in v0.3. Per-stage status is recorded in provenance; siblings continue on a sibling’s failure; the dependency-merge stage that waits on the failed one decides abort vs partial. Buyer is refunded when the pipeline’s declared output is not produced. See parallel.test.ts + integration-r10-x.test.ts.
  3. TEE attestation. Phase 3 introduces Trusted Execution Environment (Intel TDX, AWS Nitro) for cryptographic execution proof beyond operator signature. Defer to v0.4+ — current provenance.ts uses EIP-191 operator signing only.
  4. agent_runner stage type. Deferred to v0.4+ until cost model + long-running subprocess runtime are figured out. Cloudflare Workers cannot host; needs Docker runner with resource limits + per-minute billing.
  5. Composition fetch policy. RESOLVED in v0.3: fetches happen at invoke-time via fetchPipeline callback injected into runPipelineV2. Stored YAML in D1 is not pre-expanded — original uses: pipeline:NAME@VERSION syntax is preserved verbatim in provenance. See Composition section above.

Versions

  • v0.1 (Oct 2025) — initial draft, model+prompt + uses stages
  • v0.2 (Sep 2026) — SKILL enforcement via uses@version + content_hash + system instruction, output_format: structured_json + fields: [...] conformance, mcp_servers/mcp_tools forward-compat, working_dir CLI-only. agent_runner and parallel DAG deferred to v0.3.
  • v0.3 (Sep 2026, this draft) — see Changelog: v0.2 → v0.3 above. Adds R6 provider≠model (BREAKING), R7 env-var substitution, R10/R10.1/R10.1.b/R10.2/R10.3/R10.4/R10.5/R10.6 (stage refs + retry + fallback + checkpoints + streaming + parallel DAG + when DSL + runtime constraints), R11 Docker multi-arch + cosign, R13 composition (uses: pipeline:NAME@VERSION).