Suites and artifacts

A suite run answers three questions: which spec ran, in which environment, and how it ended. The control plane retains comparable metrics; large files remain elsewhere and are linked.

The examples assume:

export DEMO_CONFIG="/path/to/project/chalupa.yml"

Run model

{
  "version": "1",
  "environmentSlug": "demo-harbor",
  "suiteName": "checkout-e2e",
  "runs": [
    {
      "sourceRunId": "run_demo_20260723_001",
      "specName": "create-order",
      "status": "passed",
      "startedAt": "2026-07-23T18:30:00.000Z",
      "durationMs": 18420,
      "metricName": "workflow-latency",
      "metricMs": 1220,
      "variant": "baseline",
      "artifacts": [
        {
          "$schema": "urn:filecheap.dev:artifact-ref:v1",
          "version": 1,
          "provider": "fcheap-local",
          "uri": "fcheap://stash/report_20260723_184500.123456789_0123456789abcdef01234567",
          "artifact_id": "report_20260723_184500.123456789_0123456789abcdef01234567",
          "kind": "cairntrace.run",
          "producer": {
            "tool": "cairntrace",
            "native_schema": "urn:cairntrace.dev:run:v1",
            "native_id": "run_demo_20260723_001",
            "entrypoint": "run.json"
          }
        },
        {
          "$schema": "urn:filecheap.dev:artifact-ref:v1",
          "version": 1,
          "provider": "link",
          "uri": "https://artifacts.example.com/traces/demo-001",
          "kind": "chalupa.report"
        }
      ]
    }
  ]
}

sourceRunId identifies the execution in its source system. It must remain stable and idempotent within the integration.

Statuses

Do not turn infrastructure errors into failed results if that distinction matters during investigation.

Duration and metric

durationMs always describes the total run duration. metricName and metricMs are optional and represent one comparable measurement within the spec.

The runs view calculates p50 only across matching suite, environment, spec, variant, and metric identities. A percentile with one sample is context, not a statistical trend.

Report results

When reporting is enabled, send your runner's JSON to the signed endpoint with chalupa report. Its inputs are read from the environment:

REPORT="$PWD/runs.json" \
ARTIFACTS="$PWD/artifact-sidecar.json" \
SUITE="checkout-e2e" \
  chalupa report --config "$DEMO_CONFIG"

The optional sidecar associates complete references with raw Cairn run IDs:

{
  "$schema": "urn:chalupa.run:artifact-sidecar:v1",
  "version": 1,
  "runs": {
    "run_demo_20260723_001": [
      {
        "$schema": "urn:filecheap.dev:artifact-ref:v1",
        "version": 1,
        "provider": "fcheap-local",
        "uri": "fcheap://stash/report_20260723_184500.123456789_0123456789abcdef01234567",
        "artifact_id": "report_20260723_184500.123456789_0123456789abcdef01234567",
        "kind": "cairntrace.run"
      }
    ]
  }
}

Create each reference from a completed stash instead of typing the envelope by hand:

fcheap artifact-ref <stash-id> \
  --kind cairntrace.run \
  --producer-tool cairntrace \
  --native-schema urn:cairntrace.dev:run:v1 \
  --native-id run_demo_20260723_001 \
  --entrypoint run.json \
  --json

The CLI prints one ArtifactRefV1 object. The MCP tool fcheap_artifact_ref returns the same object as structured content. Place that object under the matching raw run ID in the sidecar; do not add Chalupa fields inside the envelope.

Every sidecar key must match a run in the Cairn stats document. Orphan keys, duplicate provider:uri identities, unknown fields, more than 250 keys, or more than 20 references per run fail locally before the report request is sent. The CLI reads at most 4 MiB of raw Cairn JSON and 1 MiB of sidecar JSON, then sends the normalized rows in as many requests as the 256 KiB request limit requires (rows are independent and deduplicate by source run). Omitting ARTIFACTS keeps the artifact-free reporting flow unchanged.

The task handles signing and must not print the key. You can also implement a compatible sender by following API and schema.

Attach artifacts on the first ingest. A retry must use a fresh HMAC nonce while resending the exact same normalized run and sidecar. Identical content deduplicates; changing artifacts, producer metadata, status, metrics, or other run facts for the same source run returns 409 run_ingest_conflict and does not overwrite the stored run.

ArtifactRefV1

Chalupa stores and returns the complete envelope. It does not silently reduce it to a stash ID or URL. $schema, version, provider, uri, artifact_id, kind, producer routing metadata, and web_url all survive ingest, storage, and readback whenever they are present.

The validators keep transport locators free of embedded credentials, but they are not a data-loss-prevention scanner. Artifact IDs, kinds, and producer names, versions, commands, and entrypoints can reveal operational details. Producers must keep every metadata field non-sensitive.

Unknown fields are rejected. In particular, do not add an integrity field: file.cheap verifies restored bytes against its own manifest, and ArtifactRefV1 does not duplicate a legacy content hash.

New producers must send ArtifactRefV1. /api/ingest/runs temporarily accepts only the two exact historical locator shapes from older clients and upgrades them before normalization, fingerprinting, or storage. That compatibility path does not accept extra fields, unsafe links, invalid local IDs, or duplicates created by conversion. It is deprecated and must remain only until every deployed producer sends V1 and pre-fingerprint historical rows are remediated.

Conformance and versioning

The vendored schema.json defines the strict JSON shape and provider variants. Chalupa's runtime validators also enforce semantic rules that structural JSON Schema cannot express by itself:

• local and cloud URIs must identify the exact artifact_id;
• HTTP ports must be in the 0..65535 range;
• an opaque urn: value must contain a non-empty identifier;
• embedded credentials, query strings, and unsafe relative entrypoints are forbidden in transport locators.

The complete valid and invalid fixture corpus runs through both the report CLI validator and the Chalupa ingest validator. Its validation semantics are derived from the reviewed file.cheap commit. A provenance manifest records every intentional byte-level transformation and a deterministic SHA-256 checksum of the resulting schema plus fixtures. Chalupa renames two local fixtures, uses a reserved neutral example host, and updates the hosted-provider description to match the reviewed private service. This guards against accidental drift; it does not make the two runtime implementations a shared package.

ArtifactRefV1 is immutable. Any backward-incompatible shape or semantic change requires a new schema URI and contract version instead of silently changing urn:filecheap.dev:artifact-ref:v1.

Ordering and retries

Artifact references are an ordered list. Chalupa preserves their order and includes it in the run's idempotency fingerprint. JSON object key order does not matter, but reordering the artifact list is a semantic change. Resend the same list in the same order on retries; otherwise an existing source run returns 409 run_ingest_conflict.

fcheap-local provider

provider: fcheap-local identifies an opaque stash in a vault on the operator's machine. The uri must exactly match fcheap://stash/<artifact_id>. To retrieve it, the console presents:

fcheap restore report_20260723_184500.123456789_0123456789abcdef01234567

The file remains in the operator's local vault. Chalupa does not assume that a remote file.cheap service exists, does not turn the local URI into a browser link, and cannot determine whether another machine has that vault.

Link provider

provider: link accepts a stable HTTP or HTTPS uri. Credentials, query strings, and fragments are forbidden. Visibility and retention still depend on the external provider. Chalupa opens the link only after an administrator chooses it; it never crawls the URL server-side.

fcheap-cloud provider

provider: fcheap-cloud identifies a committed artifact in the private hosted file.cheap service. file.cheap v0.31.0 or newer can publish a bounded archive through an authenticated plan → PUT → commit exchange and returns a stable, credential-free ArtifactRefV1. Cairntrace v2.1.0 and Glyphrun v0.16.0 can use that publisher. Evidence from ci: suite runs is attached to the run automatically. Evidence from flows you run on your laptop stays in your local vault and is not attached to a run today; see What a run leaves behind.

Chalupa validates and preserves the reference but never receives the artifact bytes or storage credentials. It may open an optional stable HTTPS web_url; it never resolves the fcheap:// URI in the browser and rejects signed, query-bearing, or credential-bearing stable URLs. Private byte access remains a separate, short-lived, authenticated file.cheap capability.

Failure, steps, labels and runner

A run may also carry why it did not pass and where it ran. All four fields are optional, and a producer that sends none of them keeps working unchanged:

{
  "sourceRunId": "run_demo_20260723_002",
  "specName": "create-order",
  "status": "failed",
  "durationMs": 41000,
  "failure": {
    "stepId": "click-pay",
    "kind": "step",
    "message": "locator timed out: button[name=Pay]"
  },
  "steps": [
    { "id": "open-cart", "status": "passed", "durationMs": 4000 },
    { "id": "click-pay", "status": "failed", "durationMs": 30000 }
  ],
  "labels": { "round": "r7", "case": "cart", "sha": "abc1234" },
  "runner": "cairn"
}

• failure.message is redacted by the CLI and again by the server, and is cut to 1 KiB of UTF-8. A passed run cannot carry a failure.
• steps holds at most 200 entries with ids, verdicts and durations; the reason a step failed is the run's single failure.
• labels is an allowlist (round, variant, case, sha), not a free map. When a run has no explicit variant, the console shows its round label as the variant; the stored variant stays what the producer sent.
• These four fields explain a run rather than define it, so they are not part of its identity. Re-reporting a run that was stored without them fills them in; the first value stored wins and is never overwritten. A replay that carries a different value for one of them is still the same run, so it deduplicates and the stored value stays.
• The server attributes each run to the box it ran on when the run's start time falls between that box's launch and teardown. A run reported before the box's lifecycle events arrived is attributed when they do.

Each run row links to its own page, /app/runs/<id>, with the failing step, the steps timeline, the evidence below, the logs from 30 seconds either side of the run (when log collection is on), the containers that ran in the window with restart and OOM markers, and the Monitor issues of overlapping pipelines.

Run evidence

Evidence is attached after the run, because it arrives after the immutable run row. POST /api/ingest/runs/evidence is append-only and idempotent by SHA-256:

{
  "version": "1",
  "environmentSlug": "demo-harbor",
  "sourceRunId": "run_demo_20260723_002",
  "retentionDays": 7,
  "refs": [
    {
      "sha256": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08",
      "sizeBytes": 48211,
      "name": "screenshots/click-pay.png",
      "role": "screenshot",
      "failing": true,
      "ref": { "$schema": "urn:filecheap.dev:artifact-ref:v1", "version": 1, "provider": "fcheap-local", "uri": "fcheap://stash/report_20260723_184500.123456789_0123456789abcdef01234567", "artifact_id": "report_20260723_184500.123456789_0123456789abcdef01234567", "kind": "chalupa.run-evidence", "producer": { "tool": "chalupa-cli", "entrypoint": "screenshots/click-pay.png" } }
    }
  ]
}

A request carries at most 40 references and 16 MiB. Announcing the same bytes again changes nothing, and the run must already exist. Local references show a restore command on the run page; cloud references open through a short-lived signed URL that Chalupa issues on each click and never stores. chalupa report publishes local references only for the runs a report stores for the first time, so re-reporting a suite's history never stashes old failures again. Choose what is kept with the report.evidence block.

Artifact kinds

kind is a bounded lowercase namespaced token such as cairntrace.run, glyphrun.run, chalupa.report, or filecheap.stash. It helps the interface present the reference and does not change storage policy.

Responsibility boundaries

Chalupa:

• indexes references;
• associates them with runs;
• shows provenance, complete JSON, restore commands, or stable links;
• retains structured metrics.

The artifact store:

• stores bytes;
• defines retention;
• controls access;
• handles file sizes and transfers.