Performance telemetry

Chalupa can install Monitor on an ephemeral environment and retain bounded performance summaries after the server is gone. Telemetry is optional. A stack without Monitor configured keeps the same provisioning, heartbeat, and shutdown behavior.

The design has a narrow privacy boundary: Monitor measures the host, a local adapter attaches the Chalupa deployment identity, and the control plane stores aggregated windows. The hosted application never receives a shell, a DigitalOcean write credential, or access to the workload's files.

Data flow

1. cloud-init verifies a pinned Monitor release checksum before installing it.
2. Monitor runs as an unprivileged, capability-free systemd service and writes versioned NDJSON windows through a bounded local FIFO. The service remains in the host metric namespace, but systemd denies every IP address and permits only Unix-domain sockets, so it cannot reach workload loopback or provider metadata.
3. A root-owned Chalupa adapter validates each line against the fixed schema, adds the environment and deployment identity, and signs the exact request body with the existing ingest key.
4. /api/ingest/telemetry verifies HMAC authentication, rejects replays, validates the payload again, and writes the window atomically. Source windows older than two hours or more than five minutes in the future are rejected.
5. The authenticated Performance view reads bounded deployment-scoped rollups. It never combines two deployments into one series.
6. Before compute shutdown, Chalupa makes a best-effort final flush. Shutdown still proceeds if the control plane is unavailable.

Hetzner limit. Step 1 runs from the machine's startup configuration (cloud-init user data), and Hetzner rejects user data above 32 KiB for the whole document, Monitor's installer included. A large Compose stack can leave no room for Monitor, in which case chalupa up stops with cloud_init_too_large while planning, before any server is created, instead of launching without it. chalupa preview is offline and does not check this limit. Telemetry for Hetzner environments is therefore not guaranteed today; DigitalOcean allows 64 KiB. Keep the selected Compose document small on Hetzner, or run the stack on DigitalOcean when you need telemetry.

The database rows belong to the deployment record, not to the server filesystem. Destroying compute therefore does not erase its performance history.

Export contract

monitor telemetry emits one JSON object per line. Defaults are a five-second sample interval and a 30-second rollup window:

monitor telemetry --interval 5s --window 30s

--once prewarms rate counters, emits one partial window, and exits. Every line is limited to 32 KiB, every window to 3,600 samples, and timestamps are UTC RFC 3339 values. A random session ID and increasing sequence number make retries deterministic.

Version 1 has these fixed metrics:

Each observed metric contains min, avg, nearest-rank p95, max, last, and the number of observations. Availability is reported separately for every metric as observed, partial, unsupported, or unavailable. This keeps a real zero distinct from a missing measurement.

Alerts are summaries, not log messages. The payload permits only a fixed rule, info, warning, or critical severity, and an occurrence count.

Privacy boundary

Telemetry never contains:

• hostnames, IP addresses, users, kernel versions, or machine IDs;
• PIDs, process names, command arguments, or environment variables;
• device names, mount points, filesystem paths, or application logs;
• raw collector errors, stack traces, profiles, or arbitrary labels;
• credentials, database contents, or artifact bytes.

The adapter supplies only the environment slug and deployment identity already known to Chalupa. Monitor itself knows nothing about Chalupa, its URL, HMAC, DigitalOcean, or file.cheap.

The adapter runs with the ingest key because it owns transport and retries. Monitor runs in a separate service without that key, cloud configuration, network file descriptors, capabilities, or IP networking. Process visibility is restricted, and /etc/chalupa, Compose configuration, persistent data, Docker and containerd state and runtime sockets, and /sys are inaccessible. Common host identity files, mount tables, interface and route tables, and socket tables are hidden as well. The closed collection plan reads only the aggregate /proc counters required for CPU, memory, load, network, and disk metrics. A bounded memory-backed spool absorbs short control-plane outages; it cannot become an unbounded durable log on the server.

Chalupa operators can erase the complete telemetry history for one environment on request: email hello@chalupa.run. The atomic deletion covers raw windows, hourly summaries, retry fingerprints, session summaries, and anomaly rows while retaining compute lifecycle and cost records. Repeating the same request is safe and returns zero deletion counts.

Reading the charts

The Performance view defaults to the latest deployment and offers fixed UTC ranges. Summary cards show the latest CPU average, peak window memory p95, estimated network transfer, collector samples, and anomaly counts. The transfer estimate integrates each observed average rate only over that metric's retained observation coverage; it is not a provider billing value. For a partial source window, coverage is the observed sample count multiplied by the declared sample interval and capped by the signed window duration. Hourly display buckets preserve the sum of that coverage, so one 30-second window in an hour is never treated as 3,600 seconds of transfer. Charts leave unavailable windows visibly blank instead of interpolating through them, and every chart has a keyboard-accessible data table.

The view distinguishes:

• telemetry not configured for this environment;
• configured but waiting for its first signed window;
• partial or unavailable metrics;
• gaps caused by collection or delivery failure;
• a selected deployment that has already been sunk;
• a range with no retained observations.

Use a specific deployment when comparing a past run. Reusing an environment slug does not make two servers the same measurement session.

Profiles are a separate capability

CPU and heap profiles may contain function names, paths, URLs, or application data. Chalupa does not capture or upload them automatically. The first release stores only privacy-safe numeric windows.

If profile support is added later, it must be explicit and time-bounded, delete local temporary files, publish bytes to the private artifact service, and keep only an ArtifactRefV1 reference in Chalupa. A profile must never be embedded in the telemetry endpoint or its database tables.

How the runtime reaches the box

Everything Chalupa runs on the box itself (the heartbeat agent, the Monitor forwarder, the optional crash-capture service and their systemd units) is delivered over SSH on every chalupa up, with the same pinned identity the tunnels use, whether the image is prebaked or not. The files travel as one archive, are checked against SHA-256 digests on the box before anything is touched, and are moved into place atomically. The Monitor binary is downloaded on the box from the pinned release and verified against the digest; if the box cannot reach the release host, your machine fetches the same archive, verifies the same digest and streams it over SSH. The box records what it runs in /etc/chalupa/runtime.json, and the heartbeat reports the runtime and Monitor versions, so the environment page shows what is actually installed rather than what the image was baked with.

chalupa box refresh is Beta: it is covered by offline tests and has not yet been run against a real box on every provider. To refresh a running box (after upgrading Chalupa, or changing the Monitor pin) without relaunching it:

chalupa box refresh --config /absolute/path/to/chalupa.yml

The refresh is not billable, needs no confirmation phrase and no terminal, and works without cloud.url; it never moves the ingest key (only the bootstrap step does). Running it twice changes nothing the second time.

Crash capture (Beta)

Crash capture is Beta: it is covered by offline tests with a fake Docker and a fake Monitor, and has not yet been exercised on a real box. Expect rough edges and check what it recorded before relying on it.

observability.crashCapture turns any container death in the listed Compose projects into a Monitor issue, outside ci::

observability:
  monitor:
    version: reviewed
  crashCapture:
    projects: [chalupa, my-other-stack]   # Compose project names; required
    maxPerHour: 20                        # 1-120, default 20
    culpritInConsole: false               # default; see below

A small on-box service listens for Docker die, oom and kill events. A container that Docker was asked to stop (compose restart, stop, down, or a recreate on up) is not a crash and is not reported, unless Docker says it ran out of memory. For any other non-zero exit it reads the container's exit code, OOMKilled and restart count, captures at most the newest 2,000 log lines of that run, and has Monitor parse the stack traces into the box's own issue store (readable with chalupa monitor issues --store /var/lib/chalupa/issues --json, because the service records into its own store rather than Monitor's default one). With cloud.url set it also sends a signed, closed digest to the control plane, which lists it next to CI issues on the console's Issues page. A container is called out of memory only when Docker says so, never because it exited with 137.

What leaves the box is fixed: the fingerprint, the issue kind, an occurrence count, and the project, service, exit code, OOM flag and restart count of the death. Log text, messages and stack frames stay on the box. The exception type and the culprit file:line are source-code metadata, so they are sent only with culpritInConsole: true. The service runs at low CPU and I/O priority, and maxPerHour bounds both its work and what it sends under a crash loop.

Reading Monitor from your terminal

With observability.monitor declared (telemetry: false is enough, because the binary is installed either way), chalupa monitor VERB [ARGS...] forwards Monitor's read subcommands to the running server over SSH and streams its output back, for example chalupa monitor issues --json --config ./chalupa.yml.

The pass-through is a read-only diagnostic front door. It forwards only these read verbs: snapshot, processes (ps), process, tree, issue, issues list|show, history list|query, baseline list, config show|get|path, analyze, doctor, diff, and hot (without --export). The Monitor subcommand must come first, so a leading Monitor flag is refused rather than guessed at. --config is read by Chalupa wherever it appears; after a bare --, everything goes to Monitor untouched (chalupa monitor -- issues --help). Any other verb, such as run, kill, config set, baseline save|delete, issues resolve|ignore|reopen, or hot --export, is refused locally with exit code 64 before any SSH connection is made. For agent use, run chalupa monitor serve instead (see below).

Forwarded arguments are quoted so values with spaces survive verbatim. The command refuses offline when the config declares no Monitor or no compute address resolves, and otherwise propagates Monitor's own exit code. Mutating Monitor verbs stay reachable only by SSHing in by hand.

Monitor v2.2.0 additionally recognizes the CHALUPA_CI_RUN_ID, CHALUPA_CI_STEP_ID, CHALUPA_CI_SUITE, CHALUPA_CI_ATTEMPT, and CHALUPA_CI_ENVIRONMENT values already injected into CI checks. Explicit monitor investigate/issue evidence can therefore be correlated with a Chalupa run without adding those identifiers to the automatic telemetry lane.

Monitor Desktop and other stdio clients

chalupa monitor serve is a read-only stdio door to the Monitor on an environment, for Monitor Desktop or any client that speaks monitor.app.v1 (JSON-RPC 2.0, one message per line on stdin and stdout; closing stdin ends the session). The client spawns the command locally and talks over the child's pipes; Chalupa supplies the address and the SSH identity (and, for managed environments, host-key pinning), so the client never reimplements them.

chalupa monitor serve --name <environment>      # a managed environment
chalupa monitor serve --config /path/to/chalupa.yml   # BYOC needs observability.monitor; a managed config resolves like --name

Host keys: --name (and a managed --config) verify the host key the console provided. A BYOC --config has no pin, so the box's host key must already be in ~/.ssh/known_hosts; run chalupa tunnel or chalupa ssh once (they accept a new key) or the connection fails with "Host key verification failed".

Exactly one of --name or --config is required. Read-only is always forced: Chalupa runs monitor serve --stdio --read-only on the host and there is no flag to relax it. Standard output carries only the remote protocol; diagnostics go to standard error, and the exit code is SSH's. Usage and precondition problems (unknown or ended environment, no observability.monitor) exit 64; a config that fails to load exits 69.

Monitor 2.2.0, the release this CLI pins, includes serve. Hosts launched with an older pin (2.1.0 or earlier) or from an image baked before 2.2.0 answer with Monitor's own "unknown command serve" message until they are reprovisioned or re-baked. chalupa ls --json reports, per managed environment, the Monitor release recorded when this CLI launched it (monitor.installed, monitor.version). Environments launched before that was recorded fall back to the release this CLI pins, which can be newer than what the host runs. It is a declaration, not a live probe of the host.

Failure behavior

Checksum mismatch stops Monitor installation. An invalid or oversized NDJSON line is rejected locally and never signed. Generic rejection notices are rate-limited to one summary per five minutes and never include producer bytes. The API rejects an unknown metric, unexpected field, non-finite number, impossible count, changed retry, a source window older than two hours, a timestamp more than five minutes ahead, a stale signature, or a reused nonce.

One deployment can retain at most 20,160 accepted source-window receipts, the seven-day maximum session lifetime at the default two windows per minute. Monitor restarts share that cap. After a deployment ends, Chalupa accepts new final windows for only 15 minutes and allows no window more than five minutes past the recorded end timestamp. Exact retries remain storage-free and deduplicate after teardown or quota exhaustion while they remain inside the signed timestamp limits; they do not reopen ingestion.

An unavailable control plane does not grant the adapter broader access. Transport failures, rate limits, server errors, and the short telemetry_deployment_unavailable launch race remain in the bounded spool. Permanent client errors, including telemetry_deployment_ended, telemetry_ingest_conflict, telemetry_monitor_version_mismatch, and telemetry_quota_exceeded, are discarded after a bounded response-code check so one poisoned retry cannot block later windows. No remote response body is written to the journal. Missing data remains a visible gap. The producer receives a bounded stop request so its final partial window can cross the FIFO before the adapter drains its spool. The remote helper waits at most 30 seconds for acknowledgement. The final shutdown flush remains best effort because telemetry must never make an operator keep billable compute alive.