Declarative CI

An environment is only half of a test run. The ci: block declares what should execute inside it, so a suite can run without an operator watching a terminal for hours.

The schema, the offline plan preview, execution, secret delivery, and reporting all exist today. The offline surface was built and enforced first on purpose — a plan whose cost cannot be reviewed before it runs should not be started — and it stays mandatory even now that a suite can actually run: chalupa preview renders the CI plan, and the plan is part of the launch gate that up and data-up run.

Declaring a suite

ci:
  suite: demo-harbor-smoke
  budget:
    maxMinutes: 45
  steps:
    - id: stack-health
      run: curl -fsS http://127.0.0.1:8080/ >/dev/null
      timeoutMinutes: 10

    - id: api-migrations
      on: service:api
      needs: [stack-health]
      run: /app/bin/migrate --check
      timeoutMinutes: 10
      collect:
        - path: /app/var/migrate-report.json
          when: always

    - id: teardown-report
      needs: [api-migrations]
      when: always
      run: echo "collected"
      timeoutMinutes: 5

ci: requires cloud.url, because results and evidence have to outlive the compute that produced them.

The sibling test: block declares the local-runner suites (chalupa test) that execute on the operator machine against the same environment's tunnel ports. Its suites accept a sinceRef (a git ref shape: branch, tag or commit) so a suite can replay only the flows cairn considers affected since that ref — the ref is passed to the enumeration as --since, and cairn echoes back the since it used in the selection document.

Steps

Steps form a directed acyclic graph through needs:. With the default maxParallel: 1 execution is a deterministic topological order, so the graph costs nothing in complexity while still expressing two things a flat list cannot: that a step should be skipped when a dependency did not succeed, and that a reporting step should run even when one branch is red.

There is no expression language. when covers the real cases without putting a second language inside YAML strings.

Verifier steps

uses: glyphrun and uses: cairntrace are step types rather than plain shell commands, which lets Chalupa map their documented exit codes onto step status without parsing output, attach their standard evidence paths automatically, and report their results through the existing suite contract.

    - id: terminal-specs
      uses: glyphrun
      specs: [tests/glyph/navigation.yml]

    - id: browser-specs
      uses: cairntrace
      environment: chalupa
      specs: [flows/checkout.yml]
      labels: { path: temporal }

Both drive something that lives on the host — a pseudo-terminal in one case, a browser in the other — so they always run on the server and cannot target a service container.

An exit code neither tool documents is reported as errored, never as a test failure. "We do not understand what the tool did" and "your tests failed" are different statements and must not be conflated.

uses: cairntrace for cairn 3.x

The step runs cairn run on the server with these inputs. Declare exactly one of specs or suite; everything after labels is optional and absent from the plan when unset:

    - id: answer-change
      uses: cairntrace
      environment: onbox
      suite: answer-change
      backend: playwright
      artifactRoot: /opt/automations/runs
      labels: { env: remote }
      args: [--bail]
      timeoutMinutes: 60

One gap remains on cairn's side: it exits 1 for a command-line usage error, the same code as failed outcomes. The args check above catches the errors Chalupa can see; one it cannot (a flag the box's older cairn lacks) still reads as failed, which ci.requires with a minVersion prevents.

Exit codes follow cairn 3.x (cairn explain). Only 0 passes and only 1 means the outcomes failed; everything else is errored, because the suite did not answer the question or something it guards was not honoured:

args is checked against cairn run's own flags when the plan is built, and a bad entry fails chalupa preview and ci arm instead of the run: cairn exits 1 on a usage error (an unknown flag, a missing value), and 1 is the code this step reports as failed outcomes, so a typo would otherwise turn every run red with no spec started. Refused: a flag the step owns through a typed field (--env, --suite, --backend, --label, --artifact-root, --config), a flag Chalupa does not know, a value flag without its value (write ["--var", "k=v"] or "--var=k=v"), a positional argument (use specs), and --mock, --select-only and --services-dry-run, which exit 0 without testing anything. The list is cairn 3.x's; a flag newer than this Chalupa is refused until Chalupa learns it (run it from a uses: shell step meanwhile). The box's cairn must be new enough for the flags you use, which ci.requires checks before the first step:

  requires:
    - { tool: cairn, minVersion: "3.0.1" }

Each step's evidence is its own directory. The engine hands cairn --artifact-root <artifactRoot>/<runId>/<stepId> (default root /root/.cairntrace/runs; the engine runs as root), so a box that outlives one run (a warm reuse, a persistent volume) never mixes an earlier run's evidence into this one. The root is always passed, so an artifactRoot: inside cairntrace.config.yml does not apply to a CI step; declare the root here instead. The implicit globs are run.json, report.json, artifact-manifest.json, outcomes/** and, on failure, console/** under <artifactRoot>/{runId}/<stepId>/; add your own with the same {runId} placeholder. Earlier releases collected .cairntrace/runs relative to the step's directory, which a real cairn never wrote to.

The engine also stamps each cairn run with chalupa_ci_run and chalupa_ci_step labels. Its post-step cairn stats --include-runs reads the step's directory, filters by both labels and groups by chalupa_ci_step, a label every one of those runs carries (cairn stats skips a run without the group-by key), so the report names exactly what this step wrote.

Changing a cairntrace step changes its plan digest (the exit table, the default grace and the globs moved), so a plan built before and after this contract differ; nothing about arming or reuse depends on the old digest.

Host requirements

ci.requires names the host tools a suite depends on. The engine enforces them at host-prepare, before the first step runs — a missing tool is never a surprise two hours into a paid suite:

  requires:
    - { tool: docker, minVersion: "24.0.0" }

Presence on PATH is the gate. node, docker, tmux and yarn are apt-installed when missing (one bounded apt-get update, then one apt-get install per tool); cairn, glyph, bun and go have no Ubuntu package and must come from compute.image. A declared minVersion is checked against the first line of <tool> --version when the tool answers cheaply; an unreadable version counts as unmet. Anything still unmet after the install pass fails the run in the errored state with the missing tool, its floor version, and the fix (compute.image) named — never silently ignored. A dry run only inspects the host, so it reports a missing apt-installable tool as unmet without installing it.

Delivering the source

ci.source hands a workspace environment its code — the project being tested is not always a pre-built Compose image. The engine clones a shallow checkout into its workspace before the first step and exports two variables to every step:

  source:
    git: https://github.com/acme/widgets.git
    ref: main
    secret: GITHUB_DEPLOY_TOKEN   # optional; a ci.secrets name, never a value

• The URL must be https and credential-free; an ssh URL would depend on agent forwarding, and a token embedded in the URL would be an exfiltration primitive in a reviewed config.
• A deploy token, when declared, is delivered through the ci.secrets SSH channel and fed to git via a transient askpass script owned by the engine — it appears neither in ps, the journal, nor .git/config.
• Steps reach the checkout through $CHALUPA_WORKSPACE (absolute path) and $CHALUPA_SOURCE_COMMIT (the cloned HEAD). Nothing changes cwd semantics: a step that wants the source runs cd "$CHALUPA_WORKSPACE" itself.
• A clone failure ends the run in the errored state before any step, with the redacted git error as the reason; a dry run never clones.

Secrets

ci.secrets declares names only:

  secrets:
    - { name: APP_PASSWORD }
    - { name: OPTIONAL_TOKEN, required: false }

Values never appear in chalupa.yml, and Chalupa never learns where they are stored. Reading them stays the operator's job, outside Chalupa. Names carrying a reserved prefix — including CHALUPA_, DIGITALOCEAN_, TVAULT_ and FILECHEAP_ — are rejected so a project can never shadow a platform or provider credential.

A step only receives the names it lists in its own secrets: array, and every name it lists must be declared at suite level.

Declaring names is only half the setup. Once compute exists, deliver the actual values with chalupa ci secrets, piping them from stdin in NAME=value form:

printf 'APP_PASSWORD=%s\n' "$(tvault get --project demo-harbor APP_PASSWORD)" |
  chalupa ci secrets --config "$DEMO_CONFIG"

It needs no confirmation phrase (nothing is created and no spend starts), works without a terminal, and refuses a terminal on stdin outright: a secret typed at a prompt would echo.

The command validates the payload shape, computes its digest locally, and delivers it over the same SSH boundary as chalupa ci arm. Chalupa never learns where the values came from, and the delivered file is written 0600, readable only by the engine's root process. Deliver secrets before chalupa ci arm; the engine reads /etc/chalupa/ci/secrets.env at plan start and does not poll for a later delivery.

Budget

budget.maxMinutes is enforced offline. Validation sums every step's timeout multiplied by its attempts and rejects a configuration whose declared worst case does not fit:

declared worst case is 605 minutes; raise budget.maxMinutes or lower step timeouts

That check is the reason each step needs an explicit timeout. Optional budget.maxUsd bounds spend for the same reason.

The engine also enforces budget.maxMinutes as an absolute wall-clock deadline, independent of the per-step timeouts above: a daemon thread arms at maxMinutes plus a fixed 10-minute grace for host preparation, source fetch and result publication — the phases the budget's own step-by-step check cannot interrupt. If the deadline lands anywhere in that 10 minutes, the run aborts the same way it would for an ordinary budget breach. session's expiresAfterMinutes remains the console-side backstop: the reaper destroys the environment on schedule even if the server's agent has died and nothing inside it is watching a clock at all. Because of that, a sinking onFinish refuses a config offline when budget.maxMinutes + 10 > session.expiresAfterMinutes — a session that expires before the engine's own deadline could ever fire is not a backstop; with onFinish: keep the same shape is a warning instead of a refusal, since nothing sinks the environment automatically either way.

Cancelling, and the grace before SIGKILL

chalupa ci cancel stops the unit. The engine sends SIGTERM to each running step's process group, waits, SIGKILLs what is left, then packs and publishes the evidence. The wait is cancelGraceSeconds (5 to 300), per step or suite-wide:

ci:
  cancelGraceSeconds: 180   # every step; a step's own value wins

Unset, a shell or glyphrun step keeps the engine's 30 s on a cancel or abort (10 s after a step timeout) and a uses: cairntrace step gets 120 s. cairn's signal path is synchronous and bounded: suite after hooks and run.finally each get a 30 s window, the services teardown waits 5 s for a cancelled boot command and then 10 s per teardown command (CAIRN_SERVICES_SIGNAL_GRACE_MS, CAIRN_SERVICES_SIGNAL_TEARDOWN_TIMEOUT_MS, CAIRN_SIGNAL_HOOK_TIMEOUT_MS), and SIGKILL mid-teardown is how a billable resource gets left behind. Raise the grace together with those budgets for provisioners with slow down commands.

A cairn services.provisioner deserves a closer look. Its down defaults to onSignal: wait with a 10-minute timeout, and on SIGTERM during up cairn first forwards the signal and waits up to that same budget before down runs. The grace is capped at 300 s, so a provisioner that creates or destroys cloud machines can outlast it. Chalupa cannot read your cairntrace.config.yml when it builds the plan (it lives in the project on the box), so chalupa preview prints each cairntrace step's grace instead of checking it. Keep the provisioner's down timeout, suite after hooks and run.finally together under the step's grace with a margin, or tear the provisioned resource down from a step of its own (when: always) rather than relying on the signal path.

Two bounds keep a long grace from outliving the run:

• The grace is capped at 300 s, so an abort that begins at budget.maxMinutes finishes its grace, packing and publication inside the engine's 10-minute deadline pad.
• Past the absolute deadline the grace collapses to 30 s whatever the step declared, and a deadline that lands during a longer grace shortens it from the moment it fires.

The armed unit's TimeoutStopSec is derived from the plan it is armed with, in the order the engine spends the time after a systemctl stop: the failure hooks of a step that settled just before the stop (they run before the engine notices it: onFailure.record up to its seconds plus 30 s, or onFailure.investigate up to its timeoutSeconds plus 60 s of PID lookups; 80 s of Monitor issue queries when any Monitor feature is on; 30 s of cairn stats for a cairntrace step), then the longest step grace, then 185 s for evidence and summary publication, plus a 60 s margin. That is 275 s for shell steps with no declared grace and 395 s for a default cairntrace step. systemd's own default of 90 s was shorter than the engine's finalization alone, and systemctl stop SIGKILLs the whole unit when it elapses. ci arm reads the value from the plan file it delivers; bun src/cli/ci-plan.ts --stop-timeout --plan FILE (or --config PATH --stop-timeout) prints it.

Previewing offline

chalupa preview renders the resolved plan, including the ci: suite, without contacting any provider, and the plan is part of the launch gate up runs:

── ci suite: demo-harbor-smoke   maxParallel: 1   onFinish: keep
── plan (topological, 4 steps):
    1  stack-health      droplet                               10m
    2  api-migrations    service:api           needs: 1        10m
    3  gateway-smoke     service:gateway       needs: 1        10m
    4  teardown-report   droplet               needs: 2,3       5m  when: always
── compose services targeted: api gateway
── secret names required (values are never read here): (none)
── artifact globs: 1  budget 32 MiB  chunk 1024 KiB  retention 7d
── worst case: 35 min of 45 budgeted (78%)

Service targets are checked against the pruned Compose document rather than the services: list, because pruning can remove a service the list still names. Only the post-prune comparison catches a step that would otherwise fail hours into a paid run.

Running a suite

Once compute exists, the suite is operated through the CLI. The Taskfile names these actions ci:arm, ci:follow, ci:cancel, and ci:report; the CLI accepts the group and the member as two words so no command needs a colon:

chalupa ci arm --config "$DEMO_CONFIG"
chalupa ci follow --config "$DEMO_CONFIG"
chalupa ci cancel --config "$DEMO_CONFIG"
chalupa ci report --config "$DEMO_CONFIG"
chalupa ci secrets --config "$DEMO_CONFIG"   # NAME=value lines on stdin
chalupa ci collect --config "$DEMO_CONFIG"   # evidence home before a destroy

Arming is classified as billable: it starts a suite that holds the server open for its whole budget, so it requires an interactive terminal and the exact test <environment> phrase. Cancelling requires cancel <environment> and publishes what was already collected; it never sinks compute, which stays a separate chalupa down decision. ci report needs no confirmation phrase — the run is already a fact, and republishing it is inert because the control plane keys on sourceRunId.

chalupa ci follow needs a terminal. For an orchestrator or an agent, chalupa ci follow --json needs none: it prints one JSON object per engine event (urn:chalupa:ci-follow:v1, with the event's own fields: at, runId, event, stepId, status, exitCode, reason, output tails), replays a settled run once, and follows a live one until systemd reports the unit inactive (a cancelled unit stays "deactivating" through its grace and the evidence publication, and the follower stays with it). It follows the run ci arm armed last: arming records the run id on the box, the follower waits for the engine to point current at that run, and an event of any other run is never printed. The stream always ends with one follow.end record: reason: "run-finished" with the run's outcome; reason: "run-not-found" when the armed run recorded nothing (the engine never got as far as its lock; check journalctl -u chalupa-ci); or reason: "stream-closed" when the source ended unsettled (a lost connection). The exit code mirrors it: 0 for a settled run, whatever its outcome, 1 otherwise; read the outcome from the record. There is no phrase to confirm, and --confirm is refused as it is for any action without one.

A run the engine refuses before its first step (a required secret missing or too short to redact, budget.maxUsd with no hourly rate) is still a run: it settles errored with the reason in run.finished and in its summary, and ci report publishes it like any other. ci report publishes only the armed run's summary, never an earlier one left behind current.

chalupa ci status prints the engine's live run state once: the phase, the step tally and the declared budget. Add --json for the raw document, or open chalupa tui to watch it live. Right after arming, before the engine takes its lock and points current at the new run, the box still holds the previous run's state; status compares it with the run ci arm just recorded and, while the unit is live, reports the armed run with the phase starting (and no steps yet) instead of the previous run. starting is a status-view phase: the engine never writes it.

Arming never replaces a live run. ci arm asks systemd for the chalupa-ci unit before it installs anything, and refuses (exit non-zero, nothing delivered) while the unit is active, activating, deactivating or reloading, so the engine and plan of the run in flight are left alone. Wait for it to settle, or chalupa ci cancel it, then arm again.

Reporting results

The engine has no provider credential and cannot destroy compute, but it does reach the control plane on its own: it already signs artifact uploads with the server's ingest key, and now does the same for its own terminal result. Publication happens at up to four points, in order of how soon you learn the outcome:

• The host itself, the moment a run settles. The engine signs its own summary.json with the server's ingest key and posts it to the control plane directly, so a run's result is durable even if the operator's laptop is offline when the suite finishes. This is best effort with its own bounded retries; if the control plane cannot be reached, the summary stays on disk and one of the paths below (or the engine's own recover, next time it is armed against this run id) publishes it later.
• chalupa ci report, run explicitly once a suite settles;
• chalupa ci cancel, which reports for you after stopping the run;
• chalupa down, which reports once more immediately before destroying compute — the last moment summary.json still exists — so a finished run that nobody manually published is not lost by teardown.

Publishing is idempotent: the control plane keys on sourceRunId, so any overlap between these four simply republishes the same result. If the host already published, chalupa ci report (and by extension ci cancel and down) prints that the run was already published rather than "Reported" — nothing about the run's own record changes either way.

If the environment is already destroyed and a run is still shown as running (its box went away before any path above could publish), the settled summary.json that ci collect or an earlier run kept under ~/.local/state/chalupa/ci-runs/<runId>/ can be published without the box: the report helper takes --from-local <runId> in place of --summary, reads that file, refuses it if it was recorded for a different environment, and is idempotent like every other path.

Terminal specs run outside the ci: engine share the same reporting path:

REPORT="$PWD/cairn-stats.json" chalupa report --config "$DEMO_CONFIG"

The report action reads the runner's JSON file named by REPORT. Use absolute paths in these variables: relative ones resolve against the installed package, not your project. Terminal specs report through the glyphrun format, whose documents carry no suite name of their own; see Suites and artifacts.

Exporting an experiment run

A replay experiment (the experiment: block) writes its experiment-result.v1 under .chalupa-experiments/<name>/<stamp>/. Two export actions turn that run directory into a document you can move elsewhere — neither writes anywhere on its own, so the destination stays yours:

RUN_DIR="$PWD/.chalupa-experiments/<name>/<stamp>" OUT="$PWD/note.md" chalupa report-obsidian --config chalupa.yml
RUN_DIR="$PWD/.chalupa-experiments/<name>/<stamp>" chalupa report-cortex --config chalupa.yml   # OUT="$PWD/case.json" optional

report-obsidian renders the run as a Markdown note shaped for an Obsidian vault — YAML frontmatter (title, date, tags, verdict status and confidence), a results table with per-metric median (min–max), the metric deltas, the verdict rationale verbatim, and an evidence list of the run directory and its round journals. Without OUT the note goes to stdout, so it can be piped straight into a file you choose.

report-cortex writes a Cortex-shaped case file (default <run dir>/cortex-case.json) that holds references only: the experiment result document and its round journals (capped at 20, with the drop recorded), plus the verdict's status and confidence in Cortex's own verification vocabulary. No metric numbers and no secrets travel — the case points at the documents instead of copying them.

Both refuse anything that is not an experiment-result.v1, so a stale or hand-edited run directory fails loudly instead of exporting a guess.

Finishing the environment

onFinish declares what should happen to compute once a run settles:

ci:
  onFinish: sink-on-success

keep (the default), sink, and sink-on-success are the only values. The engine never executes this policy itself — it holds no provider credential — and the policy is not attached to the plan the engine receives either: chalupa ci arm registers it with the control plane at the moment it arms the run, alongside the plan's own sha256, as the run's running header. Nothing about what the host sees changes; a compromised or misbehaving host can at most provoke its own shutdown, never anyone else's, because the policy it would need to forge never travels to it.

Who actually executes the policy once the run settles depends on whether the deployment's own organization has an active credential for that deployment's own provider:

• DigitalOcean or Hetzner, with the console reaper armed for that provider — an organization provider token registered via chalupa setup --inference (DigitalOcean) or chalupa setup --provider hetzner (see Idle observation and the console reaper) — the control plane executes onFinish itself. Whichever of the four publication points above reports the terminal result first, the control plane checks it against the registered policy in the same transaction: if the reported plan hash matches the one the policy was armed with, the resolved deployment is this run's own, still active and unmanaged, and the outcome satisfies the policy (sink, or sink-on-success with a passed run), it marks the deployment for destruction. The same reaper cron that already enforces idle and session deadlines destroys it next tick, using the organization's own registered credential for that provider — with the operator's laptop closed, if the host itself was what published the result. A DigitalOcean credential never acts on a Hetzner deployment, or vice versa. The ingest response (and the deployment's event history) says which of requested, none, policy-mismatch, or no-credential applied, so a mismatched plan or a policy that simply did not authorize anything is never silently ignored.
• No console reaper armed for that deployment's provider — the control plane still records the decision (the deployment is marked with the pending intent, visible in its history) but reports no-credential and destroys nothing. The local teardown runner below is not optional here; it is the only thing that can actually act on onFinish.

The local, operator-run teardown runner remains a valid fallback on every provider, including as a backstop for the console-armed path above (a token that gets revoked between arming and settling, for instance): ci:report records the settled outcome against this deployment's registration, and the next teardown:run (or the armed, scheduled variant) destroys compute when the recorded outcome satisfies the declared policy. A sinking onFinish does nothing on its own without a registration:

chalupa teardown register --config "$DEMO_CONFIG"

Declaring onFinish: sink or sink-on-success without a session block is a validation error, because the local runner that can act on it only ever targets a registered, session-bound deployment. This has not changed: the console-armed path above lives entirely in the control plane and does not relax it.

Be plain about what session.expiresAfterMinutes actually is: control-plane state, not a hard infrastructure shutdown mechanism on its own — and not merely a hard spend cap either. Which path turns that deadline into a stopped host depends on whether this organization has armed the console reaper for the deployment's own provider (see Idle observation and the console reaper):

• With the reaper armed for that provider, the console holds the encrypted, per-organization credential, and the reaper destroys the reported compute at session expiry — even during a live CI run — at the idle deadline, and when a settled run's onFinish policy authorizes it (ci-finished), all without the operator's machine involved.
• Without an armed credential for that provider, nothing on the host or in Vercel holds one, so expiry cannot destroy anything by itself — it only marks a deployment as past its deadline for anything that later chooses to look. The only thing that then turns that deadline into a stopped host is a registered and armed local teardown runner (chalupa teardown register, then the armed teardown:run-armed target, or the installed launchd schedule) actually calling chalupa down on your machine. Register without arming, or never install the schedule, and a sinking onFinish policy is declared but inert: the environment keeps billing right through and past its expiry until a human runs chalupa down themselves.

Either way, a destroyed host's Pulumi local state is only reconciled the next time the operator runs chalupa down against it — the reaper deletes the remote server, not the local stack record. The validation warnings explain which path applies, every time.

Reporting from somewhere else

A project whose tests already run elsewhere — in its own pipeline, against a preview deployment — can publish results without a chalupa.yml and without provisioning anything. There is no chalupa verb for this yet; run the reporter that ships in the installed @thelacanians/chalupa-cli package with Bun, from the package directory:

bun src/cli/report-ci.ts \
  --summary summary.json \
  --environment my-project-preview \
  --cloud-url https://chalupa.run

Publishing reads two things: the environment slug and the control plane origin. It opens no Compose file, contacts no server and needs no SSH identity, so a Compose file the project does not have was never a real requirement. The control plane registers the environment on first report; it stays unknown because no agent ever described a deployment for it, which is the honest state for a run that had no compute of its own.

CHALUPA_INGEST_KEY is still required, and the slug and origin are validated against the same rules chalupa.yml uses.

Idle observation

Idle is measured from established SSH connections. A headless run has none, so declaring both ci: and idleShutdownMinutes raises a warning:

WARNING ci_idle_shutdown_conflict (idleShutdownMinutes): idle is observed from
established SSH connections, and a headless CI run has none

Treat it as a real conflict: an idle window shorter than the suite will report the environment as idle while the suite is still running.

Evidence size

artifacts.maxChunkBytes is capped at the artifact service limit and maxTotalBytes at the session limit, so an oversized policy fails offline instead of partway through a finished run.

Collecting trace or video output raises a warning, because failed browser runs with those enabled reach roughly a gigabyte each. Capture them while debugging a specific red run, not by default.

Collecting the evidence locally

The engine publishes each step's pack to the artifact service when the environment carries an ingest credential and the run's publication budget holds. When it does not, the packs — and the verifier reports under reports/ — exist only on the server, and chalupa down destroys the server. chalupa ci collect is the last copy: it reads the settled run's id from the artifact index and brings the reports, the index and the pack parts into ~/.local/state/chalupa/ci-runs/<runId>/, one directory per run, so a later run never overwrites an earlier one. Pack parts travel base64 and every name and size is re-validated on arrival; the server is not trusted. The ci:secrets and ci:collect Taskfile targets behind chalupa ci secrets and chalupa ci collect remain for anyone driving the bundled Taskfile directly.

down runs the same collection before destroying (best effort, like the summary publication it sits beside; CHALUPA_TASK_CI_COLLECT=off skips it), so an operator who never fetched anything still keeps whatever the engine never published. What this cannot reach is the autonomous case: a console reaper destroys the server from the cloud side, where no local collection can run. If a run must be readable after an unattended sink, configure the ingest credential so publication happens on the server, before the summary.

On-failure diagnose with Monitor

Every step already runs with three correlation variables in its environment, regardless of whether onFailure.investigate is declared:

CHALUPA_ENV=<the environment name>
CHALUPA_DEPLOYMENT_ID=<this droplet's numeric provider id, if registered>
CHALUPA_RUN_ID=<the pipeline run id>

These are the fallback names Monitor's own context-precedence table reads below CHALUPA_CI_ENVIRONMENT and CHALUPA_CI_RUN_ID, which the engine already set. CHALUPA_DEPLOYMENT_ID is absent, not empty, on a server whose environment predates registration.

Opt into automatic diagnosis by declaring ci.onFailure.investigate:

ci:
  onFailure:
    investigate:
      enabled: true
      pids: [api, top_cpu]
      timeoutSeconds: 30

pids names candidates to resolve a PID from, tried in order: a Compose service name (resolved via docker compose ps then docker inspect, falling back to monitor processes --filter when no such service exists), or the literal top_cpu, which asks Monitor's own process inventory for the server's hottest process instead of naming one. codebase is optional and defaults to the project directory; timeoutSeconds (default 30, max 120) bounds the whole monitor investigate call.

When a step fails or errors, the engine runs monitor investigate on the server (or tunnel host) with that PID, entirely best effort: a timeout, a missing Monitor install, or a malformed result never changes the step's own outcome and never blocks the run from finalizing. chalupa ci follow narrates either result:

[INCIDENT] deploy captured (api): fcheap://stash/inc_..., 1s
[INCIDENT] deploy not captured: pid-unresolved

A successful capture produces a monitor.incident ArtifactRefV1 — Monitor's own local file.cheap stash reference, never the profile or incident bytes themselves — and Chalupa attaches it to the run's own evidence, stored alongside the run's other artifacts and reported through ci report the same way step evidence already is. The console's pipeline run page renders it next to per-spec evidence links as View incident.

onFailure.investigate calls Monitor's own companion tools — fcheap, codemap, vecgrep — directly, never through ci.requires, whose tool enum is closed to what compute.image bakes in. They are not a gate: before the first step the engine records which of them the host has, and ci follow names any that are missing, so a later "not captured" explains itself. Captures that produce no reference also carry Monitor's own reason:

── onFailure.investigate uses on the droplet (reported at run start, never a gate): fcheap>=0.30.0 codemap>=0.40.0 vecgrep>=2.20.0
[INCIDENT] capture tools missing: fcheap
[INCIDENT] deploy not captured: no-artifact-ref (fcheap not found on PATH)

On-failure video capture

onFailure can arm one hook per run; the alternative to investigate is record, a vidtrace video capture of the failed step:

ci:
  onFailure:
    record:
      seconds: 120

Declaring the block arms it (seconds defaults to 120, max 600). When a step fails or errors, the engine runs timeout <seconds> vidtrace record --out <run-dir>/recordings/<step-id> on the server, and the recording's files ride the step's artifact pack next to anything collect gathered — the same retention-bound lane, no separate publication path. Best effort like every evidence feature: Chalupa never installs vidtrace for you, so on an image without it each failed step journals record.unavailable (binary-missing) and the run finishes unchanged. Bake the binary into a compute.image snapshot to use the hook; ci preview says so when nothing will provide it.

Crash capture with Monitor

ci.monitor turns a crashing step into a grouped issue with a culprit file:line, instead of a 4 KB log tail. It is opt-in:

observability:
  monitor:
    version: 2.2.0
    sha256: { amd64: "…", arm64: "…" }
    telemetry: false   # install the binary for diagnostics only

ci:
  monitor:
    wrap: true
    culpritInConsole: false

With wrap: true, each shell step that runs on the server is launched under monitor run --. The step's output and exit status pass through unchanged; Monitor reads the output as it streams and records crashes from Node, Deno, Bun, Python, Ruby and Go. It only ever attaches at launch, so service steps (on: service:…) and verifier steps are not wrapped. Each step gets its own issue store inside the run directory, and the names of the step's secrets are passed to Monitor so their values are redacted from what it records. Without the Monitor binary on the host, steps run unwrapped and the journal says so.

When a step fails, the engine gathers up to three of its issues:

• Each issue's full page (culprit, source snippet, causes, the commit that last touched the line) joins the step's artifact pack, so it outlives the server when the run sinks it. The commit author's email is removed first.
• The run summary gets a digest per issue: its fingerprint, kind and occurrence count. The fingerprint is stable across runs, so the same failure is recognisable next time.

culpritInConsole: true adds the culprit file, line and function, and the exception type, to that digest, and the pipeline run page shows them under Why it did not pass. It is off by default because it is source-code metadata leaving the host; the issue's message and snippet never go to the console either way. It requires something that records issues: wrap, replayContainerLogs or onFailure.investigate.

Container log replay

wrap cannot see inside compose services — the binary is not in their container and Monitor never attaches to a process that already started. replayContainerLogs: true covers that side: when a step fails, the engine captures that service's docker compose logs --timestamps --no-log-prefix since the step started and replays the file through monitor stacktrace parse --record into the step's own issue store. Crashes that only ever surfaced as container log lines become the same grouped issues, with the same digest and evidence pages a wrapped crash gets.

ci:
  monitor:
    replayContainerLogs: true

The step's own service is replayed when it ran inside one; otherwise every currently running service (bounded to five, alphabetical). Container paths are mapped back to the checkout through the service's bind mounts, with the image's working directory as a coarser fallback — frames outside any mount keep their container path and get no culprit, degraded rather than wrong. The replay needs a Monitor release carrying the replay flags (--line-timestamps, --path-map); on one without them the engine skips the replay and the journal says so, exactly like every other evidence feature: degraded, never a failed run.

Performance telemetry (see Performance telemetry) stays identity-free either way: it never contains a PID, a process name, or this run's incident evidence. Correlating a performance window with an incident is done by deployment_id and the run's time window, never by a shared process identifier — Monitor's own contract is explicit that the two lanes must never mix.