Performance telemetry

Chalupa can install Monitor on an ephemeral environment and retain bounded performance summaries after the droplet is gone. Telemetry is optional. A stack without Monitor configured keeps the same provisioning, heartbeat, and shutdown behavior.

The design has a narrow privacy boundary: Monitor measures the host, a local adapter attaches the Chalupa deployment identity, and the control plane stores aggregated windows. The hosted application never receives a shell, a DigitalOcean write credential, or access to the workload's files.

Data flow

1. cloud-init verifies a pinned Monitor release checksum before installing it.
2. Monitor runs as an unprivileged, capability-free systemd service and writes versioned NDJSON windows through a bounded local FIFO. The service remains in the host metric namespace, but systemd denies every IP address and permits only Unix-domain sockets, so it cannot reach workload loopback or provider metadata.
3. A root-owned Chalupa adapter validates each line against the fixed schema, adds the environment and deployment identity, and signs the exact request body with the existing ingest key.
4. /api/ingest/telemetry verifies HMAC authentication, rejects replays, validates the payload again, and writes the window atomically. Source windows older than two hours or more than five minutes in the future are rejected.
5. The authenticated Performance view reads bounded deployment-scoped rollups. It never combines two deployments into one series.
6. Before compute shutdown, Chalupa makes a best-effort final flush. Shutdown still proceeds if the control plane is unavailable.

The database rows belong to the deployment record, not to the droplet filesystem. Destroying compute therefore does not erase its performance history.

Export contract

monitor telemetry emits one JSON object per line. Defaults are a five-second sample interval and a 30-second rollup window:

monitor telemetry --interval 5s --window 30s

--once prewarms rate counters, emits one partial window, and exits. Every line is limited to 32 KiB, every window to 3,600 samples, and timestamps are UTC RFC 3339 values. A random session ID and increasing sequence number make retries deterministic.

Version 1 has these fixed metrics:

Each observed metric contains min, avg, nearest-rank p95, max, last, and the number of observations. Availability is reported separately for every metric as observed, partial, unsupported, or unavailable. This keeps a real zero distinct from a missing measurement.

Alerts are summaries, not log messages. The payload permits only a fixed rule, info, warning, or critical severity, and an occurrence count.

Privacy boundary

Telemetry never contains:

• hostnames, IP addresses, users, kernel versions, or machine IDs;
• PIDs, process names, command arguments, or environment variables;
• device names, mount points, filesystem paths, or application logs;
• raw collector errors, stack traces, profiles, or arbitrary labels;
• credentials, database contents, or artifact bytes.

The adapter supplies only the environment slug and deployment identity already known to Chalupa. Monitor itself knows nothing about Chalupa, its URL, HMAC, DigitalOcean, or file.cheap.

The adapter runs with the ingest key because it owns transport and retries. Monitor runs in a separate service without that key, cloud configuration, network file descriptors, capabilities, or IP networking. Process visibility is restricted, and /etc/chalupa, Compose configuration, persistent data, Docker and containerd state and runtime sockets, and /sys are inaccessible. Common host identity files, mount tables, interface and route tables, and socket tables are hidden as well. The closed collection plan reads only the aggregate /proc counters required for CPU, memory, load, network, and disk metrics. A bounded memory-backed spool absorbs short control-plane outages; it cannot become an unbounded durable log on the droplet.

An administrator can erase the complete telemetry history for one environment through the bearer-only privacy endpoint documented in Private operations. The atomic deletion covers raw windows, hourly summaries, retry fingerprints, session summaries, and anomaly rows while retaining compute lifecycle and cost records. Repeating the same request is safe and returns zero deletion counts.

Reading the charts

The Performance view defaults to the latest deployment and offers fixed UTC ranges. Summary cards show the latest CPU average, peak window memory p95, estimated network transfer, collector samples, and anomaly counts. The transfer estimate integrates each observed average rate only over that metric's retained observation coverage; it is not a provider billing value. For a partial source window, coverage is the observed sample count multiplied by the declared sample interval and capped by the signed window duration. Hourly display buckets preserve the sum of that coverage, so one 30-second window in an hour is never treated as 3,600 seconds of transfer. Charts leave unavailable windows visibly blank instead of interpolating through them, and every chart has a keyboard-accessible data table.

The view distinguishes:

• telemetry not configured for this environment;
• configured but waiting for its first signed window;
• partial or unavailable metrics;
• gaps caused by collection or delivery failure;
• a selected deployment that has already been sunk;
• a range with no retained observations.

Use a specific deployment when comparing a past run. Reusing an environment slug does not make two droplets the same measurement session.

Profiles are a separate capability

CPU and heap profiles may contain function names, paths, URLs, or application data. Chalupa does not capture or upload them automatically. The first release stores only privacy-safe numeric windows.

If profile support is added later, it must be explicit and time-bounded, delete local temporary files, publish bytes to the private artifact service, and keep only an ArtifactRefV1 reference in Chalupa. A profile must never be embedded in the telemetry endpoint or its database tables.

Failure behavior

Checksum mismatch stops Monitor installation. An invalid or oversized NDJSON line is rejected locally and never signed. Generic rejection notices are rate-limited to one summary per five minutes and never include producer bytes. The API rejects an unknown metric, unexpected field, non-finite number, impossible count, changed retry, a source window older than two hours, a timestamp more than five minutes ahead, a stale signature, or a reused nonce.

One deployment can retain at most 20,160 accepted source-window receipts, the seven-day maximum session lifetime at the default two windows per minute. Monitor restarts share that cap. After a deployment ends, Chalupa accepts new final windows for only 15 minutes and allows no window more than five minutes past the recorded end timestamp. Exact retries remain storage-free and deduplicate after teardown or quota exhaustion while they remain inside the signed timestamp limits; they do not reopen ingestion.

An unavailable control plane does not grant the adapter broader access. Transport failures, rate limits, server errors, and the short telemetry_deployment_unavailable launch race remain in the bounded spool. Permanent client errors, including telemetry_deployment_ended, telemetry_ingest_conflict, telemetry_monitor_version_mismatch, and telemetry_quota_exceeded, are discarded after a bounded response-code check so one poisoned retry cannot block later windows. No remote response body is written to the journal. Missing data remains a visible gap. The producer receives a bounded stop request so its final partial window can cross the FIFO before the adapter drains its spool. The remote helper waits at most 30 seconds for acknowledgement. The final shutdown flush remains best effort because telemetry must never make an operator keep billable compute alive.