Troubleshooting

Start with the closest, least expensive layer: local configuration, preview, SSH, containers, and finally the hosted console. Do not delete data merely to “start from scratch.”

A readiness probe fails or the app is not ready

Three different checks can say “not ready”, and they run in different places:

• compute.health must target a port one of your services publishes. chalupa up and chalupa test both wait for it through a temporary tunnel and fail if it never answers (the server stays up and billing), so a green launch means the endpoint answered. Without compute.health, only the tunnel ports are checked and application readiness is unverified. chalupa exec and chalupa launch check it the same way before they run your command. The server runs the same check on its own at boot, and chalupa box timeline shows when it passed.
• compute.readiness probes run on your machine, through the tunnel. chalupa wait runs them, chalupa up --wait opens a supervised tunnel and waits for them, and chalupa test waits for them before its specs (see Make servers ready faster). With a plain chalupa wait, open the tunnel first, because every probe connects to localhost.
• Compose healthchecks decide when a service counts as healthy to Docker.

Check the order. Open chalupa tunnel, then try the service port yourself. An http probe passes on a 2xx or 3xx answer, so a 404 or an error page is not ready: list the codes your app legitimately returns in expect.status. A database that restores a large index, such as Elasticsearch recovering its shards, can take minutes after its container reports started; raise the timeoutSeconds instead of retrying by hand.

Missing CI evidence or crash capture

Evidence depends on how the run was made. Check which of these you did:

A crash was not captured when:

• the step is not wrapped: crash capture is ci.monitor.wrap: true on a ci: step, and it does not cover interactive servers or containers outside your Compose file;
• the Monitor version in observability.monitor does not match the one baked into a prebaked image. chalupa doctor --config warns about that mismatch;
• the host's telemetry is not reporting. Host metrics work on one provider today.

An empty issues page does not prove nothing crashed. See What a run leaves behind for what each piece covers today.

Missing GPU metrics, logs or turns

First open the correct organization and environment. Note its launch time and latest received update. Provisioning, telemetry delivery and model readiness are separate: a successful launch does not confirm all three.

For managed, Telemetry armed confirms bootstrap delivery, not that the console accepted a heartbeat. If retry succeeds but no update arrives, inspect host-agent errors. For BYOC, follow control plane setup and check cloud.url and the organization’s ingest credential.

When connected to the host through SSH, inspect bounded output:

systemctl is-active chalupa-agent.service chalupa-log-collector.service
journalctl -u chalupa-agent -u chalupa-log-collector --since "10 minutes ago" -n 80 --no-pager

A rejected payload (422) differs from a signature/authentication failure. Record the HTTP status and timestamp; share neither ingest keys nor request bodies. Old hosts may lack telemetry services and need an updated launch; bootstrap alone cannot install missing cloud-init units. Data never accepted by the console cannot be reconstructed after the host disappears.

See managed telemetry, log retention and agent reporting for the relevant setup.

The operator UI does not start

The UI requires interactive stdin and stdout. An explicit chalupa tui invocation fails in a pipe, CI job, or other non-TTY process; invoking chalupa without arguments there prints help instead.

Check the local executables without running them:

chalupa doctor

For CI or structured output, use inspect --json and, with an explicit config, status --json. Provider-changing and other foreground actions are deliberately refused outside an interactive terminal. See Local CLI and operator UI.

A local action failed or was interrupted

If data-up, up, or down returns non-zero or receives a signal, cloud state and billing may be unknown. Do not assume rollback and do not retry blindly.

Never interrupt chalupa down, or a data deletion, once it has started. The shell that runs the lifecycle only supports an EXIT trap, not INT/TERM/HUP, so a Ctrl-C mid-run skips whatever cleanup a signal handler would otherwise do. Deleting the data volume is the sharper case: it runs pulumi state unprotect --all before pulumi destroy, and a signal landing in that window can leave the protected <name>-data stack unprotected with the delete never completed. The CLI has no command for that deletion; it exists only as a deliberate, separately confirmed Taskfile operation described in Data and seeds. If it was ever interrupted, re-protect the stack immediately, before doing anything else:

pulumi stack select <name>-data
pulumi state protect --all --yes

That is the exact recovery the deletion's own failure trap runs automatically on an ordinary (non-signal) failure; a lost signal is the one case that trap cannot catch, so run it by hand.

On Hetzner, also re-enable the volume's provider-side delete protection: hcloud volume enable-protection $(pulumi stack output volumeId --stack <name>-data) delete (or in the Hetzner Console). The deletion lifts it before the destroy and normally restores it on failure, but an interrupted run may not have.

chalupa status --config "$DEMO_CONFIG"

That is the bounded local contract: it reads the exact local stacks and prints what Chalupa believes exists.

Then verify the exact resource in the DigitalOcean console. Local status reads bounded compute and protected-data contracts from exact local Pulumi stacks; it does not query DigitalOcean. A protected-data result can be not-configured, absent, present, or unknown, and its expected size comes from chalupa.yml rather than a live provider reading. The provider console and invoice remain authoritative. Never use data-nuke as an unknown-state recovery step.

The preview cannot find chalupa.yml

Confirm the path:

export DEMO_CONFIG="$PWD/chalupa.yml"
test -f "$DEMO_CONFIG"

Then run the free, offline preview against it:

chalupa preview --config "$DEMO_CONFIG"

If a path relative to the Compose file fails, check which directory your Chalupa version resolves it from.

A service is missing

Each entry in services must exactly match a key in Compose:

chalupa preview --config "$DEMO_CONFIG"

The output lists the included services. Do not add an exception to the engine; correct the declarative configuration.

A dependency is broken

Chalupa prunes depends_on references to unselected services. If a selected service still needs that dependency at runtime, include it explicitly in chalupa.yml.

The tunnel does not open

Check the known state first:

chalupa status --config "$DEMO_CONFIG"

If the box is already gone, or before relaunching, read what the box itself recorded. Relaunching blind costs a whole boot and fixes none of these:

chalupa box timeline --config "$DEMO_CONFIG"

Look at the infra stage first: if box-healthy passed, the box was fine and the problem is the path from your machine. The tunnel stage then names a class (when several apply, the first in this list):

• supervisor-crash: the tunnel supervisor itself exited (exit 69 is a failed start). Read its error output, then chalupa tunnel keep again.
• sshd-throttled: the box's SSH server refused new connections because too many opened at once (MaxStartups). Chalupa opens forwards in waves to stay under it; other SSH sessions against the same box at the same moment (a harness, a second tunnel) still count.
• target-not-listening: the forward reached the box, but nothing listened on the port, again and again (a few refusals while apps boot are normal and name nothing). Check that the service, or the app your harness starts after up, actually started.
• timeout: the supervised tunnel did not answer within its 30-second window.

Then run:

chalupa tunnel --config "$DEMO_CONFIG"

If the stack uses a dedicated identity, export the exact same absolute path:

export SSH_IDENTITY="$HOME/.ssh/id_ed25519_chalupa"
chalupa tunnel --config "$DEMO_CONFIG"

Chalupa rejects a relative path, symlink, non-private mode, wrong owner, encrypted key, or a file that the system OpenSSH implementation cannot parse before connecting. Use chmod 600 (or chmod 400) on the private key. Do not work around a failure by widening the firewall or copying the key into the repository.

Investigate:

• the selected stack;
• connectivity to port 22;
• the available SSH key;
• an unexpected host-key change;
• the inferred ports.

Do not expose database ports publicly as a temporary shortcut.

The seed reports that no tunnel is open

chalupa seed expects services on localhost. Keep chalupa tunnel running in another terminal and verify the service port before retrying.

If the seed stopped partway through, use the project's documented resume procedure. Do not delete the data volume to start over.

The environment appears stale

stale means the deployment appears active, but Chalupa has not received a recent signal.

Open a session and inspect the server without printing secrets:

chalupa ssh --config "$DEMO_CONFIG"
systemctl status chalupa-agent
journalctl -u chalupa-agent --since "10 minutes ago"

chalupa ssh resolves the address from Pulumi output, so you never need to look up or paste the server IP.

Look for network, system-clock, URL, signature, or payload-format errors. Redact headers and bodies before sharing logs.

The control plane rejects the signature

Common causes include:

• a timestamp outside the accepted window;
• a reused nonce;
• signing a different path from the one sent;
• serializing the body again after signing;
• a different key;
• a malformed v1= header.

Synchronize the clock and sign the exact body bytes. Do not print the HMAC key to compare it.

Bootstrap times out while compute remains present

SSH can become reachable before cloud-init has installed Docker and written the systemd units. Chalupa waits up to 15 minutes for cloud-init status --wait; a timeout intentionally leaves the billable server intact for diagnosis.

Check only bounded status output on the host:

chalupa ssh --config "$DEMO_CONFIG"
cloud-init status --long
systemctl is-active docker.service chalupa-compose.service

Repair package, network, or cloud-init failures, then rerun chalupa agent-bootstrap with the same config and SSH_IDENTITY. The retry does not run Pulumi or touch the protected data stack.

A log excerpt disappears while the download link still works

This is expected, not corruption. Redacted excerpts in the console are always removed after a fixed 24 hours; the complete chunk stays downloadable for the deployment's effective retention, which is the configured session.logs.retentionDays (7 days by default) capped by your plan: 7 days on Free and Solo, 30 days on Crew. That is a separate, usually longer clock. See Private log artifacts. Do not treat a missing excerpt as proof the underlying artifact was deleted; check the download link before escalating.

Teardown warns that logs were not finalized

The destroy path stops the isolated collector, retries durable spool delivery, and replays one stable finalization body within a bounded deadline. A warning means compute destruction continued but finalization was not confirmed. Every already committed file.cheap artifact remains private and durable; the final chunk may be incomplete. Check the environment's private Logs view and the collector state before the server disappears when possible. Retention later archives a stale collecting session, but it does not reconstruct bytes that were never committed.

A run does not appear

Validate:

• a valid slug in environmentSlug;
• a stable sourceRunId within the environment; identical retries reuse it;
• a passed, failed, or errored status;
• duration in milliseconds;
• an exact ArtifactRefV1 envelope for every artifact;
• stable HTTP(S) link and hosted web URLs without credentials, query strings, or fragments;
• sidecar keys that match raw Cairn run IDs with no orphan entries;
• the batch size.

The endpoint returns 202 when it accepts the ingest request.

409 run_ingest_conflict

The environment already contains that sourceRunId with different normalized content. Resend the original stats and sidecar exactly, or create a new source run. Chalupa does not overwrite an existing run to add or replace artifacts.

The countdown does not match exactly

The countdown is derived from discrete checks. Compare idleChecks, checkIntervalSeconds, limitMinutes, and the time of the latest signal. A small difference is expected; a frozen counter alongside fresh heartbeats is not.

A launch is refused before anything is created

chalupa up (and data-up, cache-up) runs a read-only capacity check first. A refusal means the provider would have refused too: the server size is not offered in that region right now, your account is at its server limit, or the declared memory limits do not fit the size. Nothing was created and nothing is billing. Pick another size or region in chalupa.yml, free a server in the provider account, or raise the limit with your provider. Declared alternatives are shown as suggestions only. If you are sure the check is wrong, CHALUPA_CAPACITY_PREFLIGHT=off chalupa up skips it.

The environment is sunk but still has cost

Review the breakdown. Compute should be zero, but a persistent volume can continue to incur estimated cost. The provider invoice remains authoritative.

Before escalating

Collect these details without secrets:

• output from chalupa preview --config "$DEMO_CONFIG" and chalupa doctor;
• Bun, Pulumi, and Chalupa versions (chalupa --version);
• a demo name or redacted identifier;
• UTC timestamps;
• agent status;
• the API error code;
• whether the problem affects compute, storage, or both.