Troubleshooting

Start with the closest, least expensive layer: local configuration, inference, Pulumi, SSH, containers, and finally the control plane. Do not delete data merely to “start from scratch.”

The operator UI does not start

The UI requires interactive stdin and stdout. An explicit chalupa tui invocation fails in a pipe, CI job, or other non-TTY process; invoking chalupa without arguments there prints help instead.

Check the local executables without running them:

chalupa doctor

If you are working from an unlinked checkout, use:

bun run chalupa -- doctor
bun run chalupa -- tui --config "$DEMO_CONFIG"

For CI or structured output, use inspect --json and, with an explicit config, status --json. Provider-changing and other foreground actions are deliberately refused outside an interactive terminal. See Local CLI and operator UI.

A local action failed or was interrupted

If data-up, up, or down returns non-zero or receives a signal, cloud state and billing may be unknown. Do not assume rollback and do not retry blindly.

Never interrupt task down or task data-nuke once they have started. Task's embedded shell only supports an EXIT trap, not INT/TERM/HUP, so a Ctrl-C mid-run skips whatever cleanup a signal handler would otherwise do. data-nuke is the sharper case: it runs pulumi state unprotect --all before pulumi destroy, and a signal landing in that window can leave the protected <name>-data stack unprotected with the delete never completed. If that happens, re-protect it immediately, before doing anything else:

pulumi stack select <name>-data
pulumi state protect --all --yes

That is the exact recovery _data-nuke's own restore_protection_on_failure trap runs automatically on an ordinary (non-signal) failure; a lost signal is the one case that trap cannot catch, so run it by hand. See BACKLOG.md ("Signal-time cleanup in lifecycle tasks") for why traps are EXIT-only today.

chalupa status --config "$DEMO_CONFIG"
task status CONFIG="$DEMO_CONFIG"

The first command is the bounded local contract. The second is the Taskfile's own human view and needs the checkout, so run it there when you want its extra diagnostics.

Then verify the exact resource in the DigitalOcean console. Local status reads bounded compute and protected-data contracts from exact local Pulumi stacks; it does not query DigitalOcean. A protected-data result can be not-configured, absent, present, or unknown, and its expected size comes from chalupa.yml rather than a live provider reading. The provider console and invoice remain authoritative. Never use data-nuke as an unknown-state recovery step.

The preview cannot find chalupa.yml

Confirm the path:

export DEMO_CONFIG="$PWD/chalupa.yml"
test -f "$DEMO_CONFIG"

Run validation from the Chalupa repository:

cd "$CHALUPA_HOME"
task validate CONFIG="$DEMO_CONFIG"

If a path relative to the Compose file fails, check which directory your Chalupa version resolves it from.

A service is missing

Each entry in services must exactly match a key in Compose:

task preview CONFIG="$DEMO_CONFIG"

The output lists the included services. Do not add an exception to the engine; correct the declarative configuration.

A dependency is broken

Chalupa prunes depends_on references to unselected services. If a selected service still needs that dependency at runtime, include it explicitly in chalupa.yml.

The tunnel does not open

Check the known state first:

chalupa status --config "$DEMO_CONFIG"

Then run:

chalupa tunnel --config "$DEMO_CONFIG"

If the stack uses a dedicated identity, export the exact same absolute path:

export SSH_IDENTITY="$HOME/.ssh/id_ed25519_chalupa"
chalupa tunnel --config "$DEMO_CONFIG"

Chalupa rejects a relative path, symlink, non-private mode, wrong owner, encrypted key, or a file that the system OpenSSH implementation cannot parse before connecting. Use chmod 600 (or chmod 400) on the private key. Do not work around a failure by widening the firewall or copying the key into the repository.

Investigate:

• the selected stack;
• connectivity to port 22;
• the available SSH key;
• an unexpected host-key change;
• the inferred ports.

Do not expose database ports publicly as a temporary shortcut.

The seed reports that no tunnel is open

chalupa seed expects services on localhost. Keep chalupa tunnel running in another terminal and verify the service port before retrying.

If the seed stopped partway through, use the project's documented resume procedure. Do not run task data-nuke.

The environment appears stale

stale means the deployment appears active, but Chalupa has not received a recent signal.

Open a session and inspect the droplet without printing secrets:

chalupa ssh --config "$DEMO_CONFIG"
systemctl status chalupa-agent
journalctl -u chalupa-agent --since "10 minutes ago"

chalupa ssh resolves the address from Pulumi output, so you never need to look up or paste the droplet IP.

Look for network, system-clock, URL, signature, or payload-format errors. Redact headers and bodies before sharing logs.

The control plane rejects the signature

Common causes include:

• a timestamp outside the accepted window;
• a reused nonce;
• signing a different path from the one sent;
• serializing the body again after signing;
• a different key;
• a malformed v1= header.

Synchronize the clock and sign the exact body bytes. Do not print the HMAC key to compare it.

Bootstrap times out while compute remains present

SSH can become reachable before cloud-init has installed Docker and written the systemd units. Chalupa waits up to 15 minutes for cloud-init status --wait; a timeout intentionally leaves the billable droplet intact for diagnosis.

Check only bounded status output on the host:

chalupa ssh --config "$DEMO_CONFIG"
cloud-init status --long
systemctl is-active docker.service chalupa-compose.service

Repair package, network, or cloud-init failures, then rerun chalupa agent-bootstrap with the same config and SSH_IDENTITY. The retry does not run Pulumi or touch the protected data stack.

A log excerpt disappears while the download link still works

This is expected, not corruption. Redacted excerpts in the console are always removed after a fixed 24 hours; the complete chunk stays downloadable for the deployment's configured session.logs.retentionDays (7 days by default, up to 30), which is a separate, usually longer clock. See Private log artifacts. Do not treat a missing excerpt as proof the underlying artifact was deleted; check the download link before escalating.

Teardown warns that logs were not finalized

The destroy path stops the isolated collector, retries durable spool delivery, and replays one stable finalization body within a bounded deadline. A warning means compute destruction continued but finalization was not confirmed. Every already committed file.cheap artifact remains private and durable; the final chunk may be incomplete. Check the environment's private Logs view and the collector state before the droplet disappears when possible. Retention later archives a stale collecting session, but it does not reconstruct bytes that were never committed.

A run does not appear

Validate:

• a valid slug in environmentSlug;
• a stable sourceRunId within the environment; identical retries reuse it;
• a passed, failed, or errored status;
• duration in milliseconds;
• an exact ArtifactRefV1 envelope for every artifact;
• stable HTTP(S) link and hosted web URLs without credentials, query strings, or fragments;
• sidecar keys that match raw Cairn run IDs with no orphan entries;
• the batch size.

The endpoint returns 202 when it accepts the ingest request.

409 run_ingest_conflict

The environment already contains that sourceRunId with different normalized content. Resend the original stats and sidecar exactly, or create a new source run. Chalupa does not overwrite an existing run to add or replace artifacts.

The countdown does not match exactly

The countdown is derived from discrete checks. Compare idleChecks, checkIntervalSeconds, limitMinutes, and the time of the latest signal. A small difference is expected; a frozen counter alongside fresh heartbeats is not.

The environment is sunk but still has cost

Review the breakdown. Compute should be zero, but a persistent volume can continue to incur estimated cost. The provider invoice remains authoritative.

Before escalating

Collect these details without secrets:

• output from task preview CONFIG="$DEMO_CONFIG" or bun preview-compose.ts "$DEMO_CONFIG";
• Bun, Pulumi, and Chalupa versions;
• a demo name or redacted identifier;
• UTC timestamps;
• agent status;
• the API error code;
• whether the problem affects compute, storage, or both.