Private operations

A successful build proves that the product compiles. It does not prove that Neon, secrets, the scheduler, and provider inventory are coordinated. This guide defines the boundary between “verified locally” and “ready to receive a fleet.”

Confidence levels

Never present local verification as a production deployment.

Local preflight

Run every gate without contacting DigitalOcean:

bun run validate

cd cloud
bun run verify

The preview must contain only 127.0.0.1 published ports, prebuilt images, and redacted secrets. bun run verify includes an offline Drizzle schema check with a non-secret loopback placeholder URL; it does not connect to a database. Repeat both gates after changing chalupa.yml or the inference engine.

Vercel project settings

Configure the Vercel project's Root Directory as cloud. The root package is the Pulumi program and must never be selected as the web deployment.

Use the locked Bun workflow:

Scope secrets separately for Preview and Production. Run migrations as an explicit release step, not from the build command: builds may execute more than once and must remain free of database writes.

Inbound email is optional and Production-only. When enabled, configure RESEND_RECEIVE_API_KEY, RESEND_WEBHOOK_SECRET, and RESEND_FORWARD_TO together. Do not copy them to Preview. See Email delivery for the provider and privacy boundary.

Production release workflow

Vercel's Git Integration builds every pushed main commit for the Production environment. .github/workflows/production-release.yml supplies the independent database gate: it verifies the pushed commit, checks out that exact SHA again, and applies its migration. An in-flight migration is never cancelled. GitHub may supersede an older pending run with the newest commit; the cancelled required check keeps that older Vercel deployment from receiving the production alias, and the newest run applies the cumulative migration graph.

In the Vercel project's Production environment:

1. keep automatic Production aliasing enabled;
2. add the GitHub Actions checks named Production verification and Production migration gate;
3. mark both checks as required.

The job names are external contracts with Vercel. Keep them unique across workflows, and update the Deployment Checks before renaming either one. Both are required so a failed verification still blocks the alias when its dependent migration job is skipped. Vercel may build the production deployment while verification and migration run, but only its automatic deployment URL is available. Vercel assigns the branch and custom Production domains after both required checks pass. A pending, failed, or approval-blocked job therefore leaves the previous deployment current. Treat Force Promote as an emergency break-glass action, never as a routine path or a way to bypass required checks and skip a migration.

Create a GitHub Production Environment so deployments are recorded. On a plan that supports protection for private repositories, restrict it to main and move the encrypted secret into that environment. On the current plan, the workflow uses an encrypted repository Actions secret:

The application uses pooled DATABASE_URL at runtime. A provider-managed Neon integration may expose additional connection aliases in Vercel, but MIGRATIONS_DATABASE_URL remains confined to GitHub Actions or tvault and is never read by a pull request, Preview deployment, Vercel build, or Vercel runtime. The release validates only MIGRATIONS_DATABASE_URL; Vercel authenticates its own Git Integration, so GitHub Actions needs no VERCEL_TOKEN, VERCEL_ORG_ID, or VERCEL_PROJECT_ID.

Use this workflow only for migrations that are safe while the previous application remains live. Prefer an expand-contract sequence: add compatible schema first, deploy code that reads both shapes or tolerates the new shape, backfill in a bounded job, then remove obsolete schema in a later release. For an intentional incompatible cutover, take and verify a restore point, announce a maintenance window, and keep the previous deployment available.

Rolling plan prices

1. Merge the pricing change into preview. Apply its committed Drizzle migration with task cloud:db:migrate before promoting the matching deployment to production; then merge preview into main for history.
2. Immediately run bun run db:seed from cloud/ with the deployment's database and the same Stripe mode. It creates the $19 Solo and $79 Crew monthly prices, archives the old prices, and records their IDs as legacy. Rerun the seed if it fails; it recovers archived managed prices.
3. In test mode, verify hosted checkout shows $19 for Solo and $79 for Crew. Renew an existing test subscription on a legacy price and verify its invoice.paid delivery succeeds and grants 500 Solo or 2,500 Crew managed hours. Check that the subscription still uses its original price.

The window between deploy and seed is fail-closed for checkout by design. The error names the catalog amount, persisted Stripe price and amount, and seed command. Webhooks continue accepting trusted current and legacy prices through that window. Existing subscriptions keep paying their original amount; do not migrate them. Repeat the rollout with live-mode configuration only after the test-mode checks pass.

Staging configuration

Staging uses CHALUPA_DATA_MODE=database and requires:

• DATABASE_URL;
• CLOUD_ADMIN_TOKEN;
• CHALUPA_INGEST_KEY;
• CHALUPA_INGEST_MASTER_KEY;
• CRON_SECRET;
• DIGITALOCEAN_READ_TOKEN;
• DIGITALOCEAN_WRITE_TOKEN;
• NEXT_PUBLIC_CHALUPA_CLOUD_URL;
• DO_VOLUME_USD_PER_GIB_MONTH, reviewed against the current provider rate.

CHALUPA_INGEST_MASTER_KEY encrypts per-environment ingest key material at rest (AES-256-GCM) and is required in database mode alongside the other secrets; see Ingest key lifecycle.

The staging origin must use HTTPS. The HTTP loopback value in .env.example is accepted only for local development and is rejected by the staging check.

DIGITALOCEAN_WRITE_TOKEN is the only write credential the control plane holds against the provider. It is scoped by separation of duty only — the DigitalOcean API has no resource-scoped tokens — so it must be a dedicated token used solely by the enforce-idle-shutdown cron, never shared with the read token, and rotated with the same discipline as the ingest key. A leaked read token (cost snapshots, inventory) cannot destroy compute; a leaked write token can destroy every droplet it can name, so it belongs in tvault and is never printed by any task.

See Secrets with tvault for the injection workflow. Do not expose demo credentials in an internet-accessible deployment.

Migration and data

Review every committed SQL migration before running it:

task cloud:db:migrate

The task reads only MIGRATIONS_DATABASE_URL from the chalupa tvault project and passes that direct, non-pooled connection only to Drizzle. DATABASE_URL remains the pooled web-runtime connection. Do not store the direct migration credential in Vercel, and do not substitute the pooled URL when the release task fails closed.

The first test must use a separate staging database. Verify tables, indexes, constraints, exact telemetry retry behavior, and downsampling before sending real signals. Migration 0001 is expand-only: it adds a nullable suite-run fingerprint and leaves historical artifact JSON unchanged so the previous application can continue writing during the rollout. The replacement application reads both storage shapes. For a row without a fingerprint, it backfills only when every normalized run field matches and the incoming ArtifactRefV1 array has an exact, lossless historical projection. Unsafe, lossy, duplicate, or enriched references fail closed instead of discarding information to manufacture a match.

/api/ingest/runs temporarily accepts only the two exact historical artifact locator shapes and normalizes them immediately to ArtifactRefV1. Extra fields, unsafe URLs, invalid local IDs, and identities that collide after conversion remain rejected. This transport compatibility is deprecated. Remove it, and run any storage backfill or V1-only constraint migration, only after every deployed producer emits ArtifactRefV1 and the historical compatibility audit is clean.

Migration 0002 adds the Monitor deployment marker, durable telemetry sessions, source windows, hourly rollups, anomalies, and retry receipts. Migration 0003 adds bounded demo/test session policy and the private file.cheap log broker. Its PostgreSQL trigger serializes sequence reservations and enforces the service allowlist, expiry, chunk size, eight-session ceiling, and deployment-wide 256-chunk and byte quotas. Telemetry ingestion takes one deployment-scoped advisory lock, rejects source windows older than two hours, admits final windows for only 15 minutes after teardown, and caps immutable retry receipts at 20,160 per deployment. Test these fail-closed boundaries on the staging database before production migration. Each chunk also persists its complete immutable file.cheap plan before the first remote call, allowing exact upload-grant renewal and committed replay recovery after either side restarts.

Apply every committed Drizzle migration in order immediately before deploying the exact matching application commit. On a main push, Vercel builds that commit while the serialized production workflow verifies and migrates the same SHA. The required Production verification and Production migration gate checks prevent Vercel from assigning Production aliases until verification and migration both succeed. Never bypass the fail-closed checks or edit the schema manually from a provider console.

Before migration 0001, create and verify a restorable database point and retain the previous deployment. The migration is expand-only, so the previous application remains compatible during this release. Any later contract migration that rewrites or rejects historical artifact rows requires a separate tested restore and rollback plan.

Monitor release gate

Publish and verify the Monitor release before enabling it in a stack:

1. run Monitor's Go tests, race detector, vet, build, and Glyphrun specs;
2. publish one immutable semantic-version tag through the existing release workflow;
3. verify both Linux archives and checksums.txt from GitHub;
4. copy the exact amd64 and arm64 archive digests into chalupa.yml;
5. run task validate CONFIG=<path> and inspect the redacted preview;
6. enable one disposable synthetic environment before a broader rollout.

Never configure latest, a branch archive, a mutable URL, or a checksum copied from an untrusted channel. A checksum mismatch is a failed installation, not a reason to skip verification.

Database ownership boundary

The ArtifactRefV1 release stays on Chalupa's existing destination database. It does not require or authorize a provider move, and it must never reuse or connect file.cheap Blob or Neon resources. Any future database move needs a dedicated Chalupa-owned destination, a tested copy and rollback plan, and a separate release approval.

Low-cost Postgres baseline

Cost control starts with workload behavior, not only with a smaller compute ceiling. For a private single-operator installation:

1. enable provider-managed autosuspend when the plan supports it;
2. start at the provider's smallest supported compute allocation and set a conservative maximum based on measured query latency;
3. use the pooled DATABASE_URL for the web runtime;
4. keep MIGRATIONS_DATABASE_URL direct, release-only, and outside Vercel;
5. cap application connection concurrency and avoid retaining idle clients;
6. keep detailed retention bounded and verify that rollups complete before source rows are deleted;
7. configure provider spending alerts or limits where available.

An active fleet sends heartbeats and telemetry intentionally, so its control plane may not suspend. When no environment is active, do not add a database warmer or a short-interval polling loop. Prefer event-driven work and a low-frequency reconciliation interval that leaves enough quiet time for the configured autosuspend threshold.

The current cost-reconciliation and retention schedules are operational tradeoffs. Each invocation can wake Postgres even when the fleet is empty. Reduce a cadence only after confirming that provider drift, delayed retention, and recovery time remain acceptable. Increasing an endpoint's maximum compute does not repair a query or poller that prevents suspension.

Review a representative before-and-after window whenever database behavior changes. Record compute-active time, storage growth, connection count, query latency, scheduled-run failures, and backlog state. Roll back an optimization that loses lifecycle evidence or leaves retention work incomplete. Never lower cost by sharing Chalupa's database or credentials with file.cheap.

Ingest key lifecycle

Per-environment ingest keys replace fleet-wide write authority: a key bound to one environment can never forge lifecycle, run, or telemetry data for another. Raw key material is stored only as an AES-256-GCM envelope under CHALUPA_INGEST_MASTER_KEY; the plaintext appears exactly once, in the issuance response, and is delivered over the existing agent-bootstrap SSH channel (ingest.key on the droplet, keyId beside it). Never log or re-return key material — a re-issue is always a fresh key.

Issue a key for an environment:

curl --fail --silent --show-error \
  -H "Authorization: Bearer $CLOUD_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"environmentSlug":"demo-harbor"}' \
  "$CLOUD_ORIGIN/api/admin/ingest-keys"

The response carries keyId and keyMaterial once. The agent signs ingest requests with that material and presents x-chalupa-key-id: <keyId>; the verifier resolves the active key, verifies the v1 HMAC with its material, requires the body's environmentSlug to equal the key's binding (the refusal names the mismatch), and partitions the nonce replay window per key so one tenant can never exhaust another's.

Rotate when a key may be exposed or on a fixed cadence. Rotation keeps every current key for the environment valid (active → rotating) while issuing a fresh active key, so in-flight droplets keep writing during the overlap window; revoke the rotating keys once their producers have been re-delivered:

curl --fail --silent --show-error \
  -H "Authorization: Bearer $CLOUD_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"action":"rotate","environmentSlug":"demo-harbor"}' \
  "$CLOUD_ORIGIN/api/admin/ingest-keys"

curl --fail --silent --show-error \
  -H "Authorization: Bearer $CLOUD_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"action":"revoke","keyId":"env_demo-harbor_<16 hex>"}' \
  "$CLOUD_ORIGIN/api/admin/ingest-keys"

Revocation is immediate: rotating and revoked keys no longer sign. Producers that still omit x-chalupa-key-id continue on the shared CHALUPA_INGEST_KEY path for rotation overlap only; it is deprecated and must be retired after every producer has been migrated.

Health and readiness

Liveness answers whether the process can serve HTTP. Readiness answers whether required configuration is valid and critical dependencies respond. Neither response may include URLs, key names, SQL errors, or fleet data.

The exact probes are:

curl --fail --silent --show-error "$CLOUD_ORIGIN/api/health"
curl --fail --silent --show-error \
  -H "Authorization: Bearer $CRON_SECRET" \
  "$CLOUD_ORIGIN/api/ready"

Run the second command inside a tvault allowlisted subprocess and keep shell tracing disabled. Liveness is public and returns only {"status":"ok"}; readiness is protected and returns only {"status":"ready"} or a redacted 503.

After Production release succeeds, .github/workflows/production-smoke.yml waits for the production alias to settle and checks liveness, the landing page, the quickstart page, and the /app login redirect. Set the repository variable CHALUPA_PUBLIC_ORIGIN only when production does not use https://chalupa.run. If the GitHub secret CRON_SECRET is present, the same smoke also checks database-backed readiness; without it, the protected probe is intentionally skipped while all public contract checks still run.

Use these signals for the platform smoke test, then verify the real journey:

1. public landing page;
2. /app redirect to login;
3. administrator session;
4. correctly labeled empty or synthetic fleet;
5. HMAC ingestion with a fresh nonce;
6. exact telemetry retry deduplicated and changed retry rejected;
7. environment, performance window, run, and artifact visible to the administrator;
8. logout and renewed route protection.

Erase performance telemetry

POST /api/admin/telemetry/erase removes one environment's telemetry sessions, raw and hourly rollups, retry receipts, and anomaly summaries. It deliberately preserves the environment, deployments, heartbeats, lifecycle events, cost snapshots, suites, and artifacts, so proof of compute destruction remains. The operation is idempotent, accepts only the administrator bearer credential (never a browser cookie), and requires the exact phrase erase telemetry <environment-slug>.

Run it only from a shell with tracing disabled and inject the administrator credential from tvault. Keep the response as the deletion receipt; its counts contain no metric values or credentials. The endpoint body is:

{
  "environmentSlug": "demo-harbor",
  "confirmation": "erase telemetry demo-harbor"
}

Send that bounded JSON request from a tvault run --project chalupa --only CLOUD_ADMIN_TOKEN --strict -- ... subprocess and set its Authorization header to Bearer $CLOUD_ADMIN_TOKEN; do not export, print, or place the token in shell history.

For a whole-instance erasure, enumerate every environment in the platform organization, invoke this endpoint once per slug, verify every returned count, then run a database-side audit that all five telemetry tables contain zero rows. Do not delete lifecycle or cost history as part of the default privacy request; escalate that broader retention decision separately.

Edge abuse controls

Vercel's automatic DDoS mitigation protects the project without a custom rule. Chalupa still rejects every operational write at the application layer: ingest routes require all three timestamp, nonce, and HMAC headers; nonces are claimed durably; request bodies are bounded; and telemetry and log storage have deployment-level quotas. Administrator, cron, and download routes enforce their own authorization.

The private beta does not require a custom WAF rule to release. If public traffic grows, stage only narrow rules and observe them before enforcement:

1. log requests under /api/ingest that omit any required HMAC header for 24 hours, then deny only after the traffic view shows no valid agents;
2. a WAF-level login limit is optional defense in depth, never first line of defense: the application already rejects login past 10 failed attempts per source or 50 failed attempts fleet-wide within a ten-minute window (src/lib/auth/login-rate-limit.ts), and successful logins are never counted. A rotating-address attacker can make the operator wait out the window either way; that trade is accepted for a single-operator control plane.

Never compare an ingest key or administrator token inside a firewall rule. Do not apply a generic IP limit to signed ingest: one legitimate log flush may make a burst of plan and commit calls, and disposable droplets do not have stable addresses. Keep Attack Challenge Mode off during normal operation because non-browser HMAC agents cannot complete an interactive challenge. Follow Vercel's staged firewall rollout and retain the application-layer checks even after a WAF rule is enabled.

Scheduler and cost estimates

The DigitalOcean invoice is authoritative. Chalupa stores estimates by deployment and persistent storage:

• inventory reconciliation recovers still-observable resources that an agent did not report;
• launch-time rate capture preserves short-lived compute cycles;
• month-to-date totals retain already-sunk cycles;
• storage continues after compute reaches zero.

The production Vercel Pro configuration runs cost reconciliation every 15 minutes and retention at minute 43 of every hour. The offset avoids starting on the same minute as cost reconciliation. Hourly execution increases recovery capacity after missed runs and, with no backlog, limits scheduled eligibility drift for detailed telemetry to less than one hour. Retention first folds source windows older than 7 days into indefinite hourly rollups and removes detail only as part of the same database statement. It then finalizes ended Monitor sessions from the resulting detail and hourly levels in one snapshot. Retry identities remain durable so an old replay cannot be counted twice. The function repeats bounded 5,000-row batches for up to its internal drain deadline. Its response is operational evidence:

• telemetryMinuteRollups is the exact number of source rows removed;
• telemetryHourlyRollups counts completed hourly insert/update operations;
• telemetryBatches counts selected database batches;
• telemetryBacklogPending says whether any eligible source row remained at return time;
• telemetryDrainStopReason is drained, deadline, or batch-limit.

Treat telemetryBacklogPending: true as an incomplete maintenance run, even when some counters are non-zero. A later authenticated invocation resumes safely because each hourly group uses a nonblocking advisory lock, rechecks the candidate rows under row locks, and deletes the whole group or nothing before it can update the aggregate. Terminal summaries read detailed and hourly rows in one database snapshot, so an atomic move appears on exactly one side when cron invocations overlap. The summary write is also guarded by the session's precise update version. A delayed accepted window clears any prior terminal summary in the same transaction; a stale finalizer therefore writes nothing and a later authenticated run recomputes from the complete history. Vercel Hobby cron supports at most one daily invocation; deployments on that plan must reduce the schedule or use an external authenticated scheduler.

External idle reaper

Every heartbeat carries a reaper observation (idleChecks, limitMinutes, lastActivityAt, ciActive). The ingest path persists the derived absolute deadline (idle_deadline_at) on the deployment, so the deadline survives the agent dying. GET /api/cron/enforce-idle-shutdown runs every five minutes and for each active platform-organization deployment whose session expiry has passed, or whose idle deadline has passed without a live CI suite:

1. destroys the droplet with DIGITALOCEAN_WRITE_TOKEN (a 404 is success);
2. claims the deployment with an atomic active → ended transition;
3. records a reaper-destroyed event with reason idle or session-expired;
4. marks the environment sunk.

The claim is the lease: a concurrent tick loses it and records nothing. A failed destroy leaves the deployment active and the next tick retries. A live CI suite defers idle shutdown because a headless run produces no SSH activity. Session expiry overrides live CI and fresh activity. Customer organizations still rely on the local scheduler until per-organization credentials exist. The operator's task down remains the primary lifecycle owner; this cron is the backstop that does not need the laptop awake. After a reaper destruction the local Pulumi state converges on the next task down (the provider treats the missing droplet as already deleted) and the console shows "Destruction reported" from the event.

Private smoke test

Before accepting a real fleet:

health → 200
readiness → ready
login → HttpOnly session
signed heartbeat → 2xx
replayed nonce → 409
signed Monitor window → 202
exact window retry → deduplicated
changed window retry → 409
fleet → updated environment
performance → deployment-scoped charts
logout → /app redirects again

Use a synthetic or disposable environment. Creating or destroying a droplet requires explicit confirmation and is never part of a routine smoke test.

Bounded lifecycle proof

The private-beta infrastructure path completed one explicitly approved, disposable DigitalOcean smoke on 2026-07-25. The operator first ran the exact Compose preview and TypeScript gate, provisioned one stack without a protected data volume, connected through the dedicated Chalupa SSH identity, and reached the loopback-only demo service through an SSH tunnel. cloud-init installed the pinned Monitor release; Chalupa then received heartbeat, deployment, telemetry, and private-log metadata. Every log chunk was committed through file.cheap's private provider path, and an authenticated Chalupa download redirected to a short-lived private Blob grant without exposing bytes through Neon or the control-plane function.

task down removed the compute and firewall while leaving the absent data stack untouched, and the final provider inventory contained no resource from the smoke. That run exposed two shutdown races: the timer's boot-relative first trigger and a log session whose final response could be lost after systemd stopped the collector. The timer now starts relative to activation. Log finalization now persists one exact payload and replays it after deferred spool delivery under a bounded deadline; both regressions have deterministic offline coverage. A future routine disposable run may exercise that exact corrected shutdown path, but it is not necessary to keep billable infrastructure alive between releases.

This receipt proves the single-operator private-beta journey, not load, multi-tenant isolation, regional recovery, or invoice accuracy.

Rollback

Roll back when authentication, ingestion, or fleet reads stop working:

1. stop the rollout and retain evidence without secrets;
2. restore the previous deployment;
3. revert the migration only if that path was designed and tested;
4. verify health, login, and ingestion;
5. disable cron temporarily if it writes incompatible data;
6. document the affected estimate interval.

The agent uses a versioned contract. Keep the previous version compatible during rollout; do not replace the server and the entire fleet at the same instant.

Before any publication

The control plane remains private even when its landing page is public. Keep /app and operational APIs out of search indexes, add platform-level protection when available, and retain server-side auth.

The current tree excludes work material, but Git history may retain legacy names. Before making the repository public, create a clean history or perform a reviewed sanitization. Never publish local Pulumi state: it contains generated user data and must be handled as sensitive even though the ingest key is delivered separately and is absent from the state.

Customer console reaper credentials

Customers opt in through chalupa setup --inference or the account provider-token API. Each organization supplies its own DigitalOcean custom token with droplet:read and droplet:delete. These scopes permit reading and deleting droplets across that provider account; Chalupa only destroys Chalupa-tagged droplets belonging to that organization when session expiry or an idle deadline passes.

org_provider_tokens stores an AES-256-GCM envelope under CHALUPA_INGEST_MASTER_KEY. API responses and console pages never return the credential. Replacement rotates the stored token. DELETE marks it revoked; customers can also delete the token in DigitalOcean for instant provider-side revocation. In-flight requests can finish after console revocation.

The platform organization continues using DIGITALOCEAN_WRITE_TOKEN. Customer sweeps never use it, including after a rejected credential. DigitalOcean 401/403 marks the matching stored token invalid and records a reaper-warning event with reason provider; a concurrent rotation is preserved. Opt-in and rotation record reaper-armed, and revocation records reaper-warning with reason manual. Organization events exist even before the first environment launch.

At most five customer organizations are selected per tick, with a new shuffled order each minute and a 22-second provider-work budget. Monitor cron failures and retained billable compute; these deadlines are not a spend guarantee.

Managed GPU worker

Run the managed worker as a separate always-on Bun process. It needs CLOUD_ADMIN_TOKEN to call the internal managed job API and Chalupa's platform DigitalOcean token to run the existing Pulumi program. Keep both credentials in the worker's secret store. The console never runs Pulumi. Its managed reaper uses DIGITALOCEAN_WRITE_TOKEN, never an organization's customer credential.

The worker claims one queued job atomically, reports progress and completion, and posts availability heartbeats at least once every five minutes. Read its reported state using this read-only query in the operations database:

SELECT worker_id, version, last_seen_at, current_job_id
FROM managed_workers
ORDER BY last_seen_at DESC;

current_job_id is reported status, not an ownership grant. Ownership lives in managed_launch_jobs.locked_by and requires a running job.

The /api/cron/managed-jobs schedule runs every five minutes. A running job with a locked_at older than twenty minutes returns to queued if attempts are below three; otherwise it becomes failed with worker lost. Heartbeats and progress do not renew this lease. Reconcile provider resources before retrying an interrupted operation; use the organization and environment identity to avoid duplicate hosts.

If jobs are queued and no worker has checked in for five minutes, the cron writes a platform reaper-warning event with reason provider and message managed worker offline, suppressing repeats for thirty minutes.

A successful up result enters the lifecycle ingest path and is marked managed; down uses the existing sunk lifecycle path. Completion retries use stable lifecycle event IDs. The console reaper sweeps managed deployments across organizations, records destruction under the deployment's organization, and excludes managed deployments from customer-token sweeps. Session expiry takes precedence over live CI; ordinary idle shutdown still defers while CI is active.

Managed worker

The worker/ package claims managed GPU jobs with the console admin bearer token and runs the existing Pulumi provider program with the platform DigitalOcean token. Deploy its reviewed release tag to an Ubuntu s-2vcpu-4gb platform droplet using worker/deploy/cloud-init.yaml; install credentials separately in root-owned /etc/chalupa-worker.env with mode 0600, then start chalupa-worker.service. The worker requires no inbound ports. Its README lists all environment variables and the offline validation gates.

Before accepting traffic, verify host-key reachability: the rendered firewall includes the customer's /32 and the worker egress /32. A missing or incorrect egress address can leave provisioning billable while the host-key scan fails. A successful worker job does not check model readiness.

Apply 0033_managed_stack_states and deploy the console state API before the worker upgrade. managed_stack_states stores unchanged Pulumi exports per organization, environment, and stack; each upload increases its version. Only the admin bearer API and the running job owner can read/write checkpoints. POST /api/internal/managed/jobs/:id/state accepts { workerId, stack, checkpoint } with a 4 MiB request limit. GET takes workerId in the query string and returns checkpoints plus the original successful launch result. Never log these bodies. Protect the console database and its backups as sensitive state.

The default PULUMI_BACKEND_URL=file:///var/lib/chalupa-worker/pulumi is a local working backend. For a private Spaces bucket, use s3://<bucket>?endpoint=<region>.digitaloceanspaces.com&region=us-east-1 and put CHALUPA_SPACES_KEY / CHALUPA_SPACES_SECRET in the worker secret store. The worker maps them to AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY only for Pulumi children, never the console API client or other commands. Both modes use an empty Pulumi passphrase and require console checkpoint uploads after provider step groups and before completion. Spaces adds bucket cost and credentials but keeps Pulumi's intermediate backend writes off the worker disk.

Run one process and one job lane. Stop the old process before replacing it. The replacement reads console state before jobs, initializes/imports only missing stacks, and restores launch identity. It can resume down/allow with an empty backend and a new worker ID. Existing stacks are never overwritten. Before retiring an existing worker disk, ensure every managed environment has uploaded its compute and retained-data checkpoints. An upgrade alone does not backfill idle stacks. A hard crash before upload still requires reconciliation against provider inventory; uploads are not atomic with provider operations. Upload/import failure blocks completion and further claims. An interrupted import leaves a private checkpoint.json marker; finish and verify that import before removing the file. Its presence blocks uploads to protect the saved checkpoint. Keep private backups of local journals for interrupted jobs and ambiguous completion responses. Compute down retains the protected model cache. Never delete the -data stack during stuck-job cleanup.

To rotate credentials, stop the service so its active step can finish, update the console's admin token and the worker environment together, replace the platform provider token, then restart and verify claim/heartbeat access before revoking the old provider token. Do not print either token. Review journalctl -u chalupa-worker for redacted worker: events.

Read managed_workers in the private database to check worker_id, version, last_seen_at, and current_job_id. Heartbeats run every 30 seconds. Deployment writes the git short SHA to /opt/chalupa/.chalupa-version; the worker reports that version, falling back to the package version only when the marker is absent. Update the marker whenever the deployed checkout changes. Inspect managed_launch_jobs for job ownership, attempts, and progress. The managed-jobs cron recovers stale work every five minutes.

For a stuck job, compare the queue state, worker journal under jobs/, local Pulumi stack and provider inventory before retrying. A hard stop may leave a Pulumi update lock; only cancel it after proving the old process is gone. A completion HTTP failure preserves its exact outcome for replay. If replay returns 409, determine whether the console already committed it or reassigned ownership; archive the journal only after reconciling the job and resources. Never send a contradictory completion to bypass an error. The console state response restores result.json from the last successful launch, including after the reaper removes the droplet. Failed jobs can leave billable GPU hosts; reconcile those explicitly without touching retained caches.