Data and seeding

An ephemeral server is useful only if destroying it does not force you to rebuild expensive data every time. Chalupa solves this problem with an independent block storage stack.

The following commands assume:

export DEMO_CONFIG="/path/to/project/chalupa.yml"

Declare persistence

persist:
  sizeGb: 20

After validating Compose:

chalupa data-up --config "$DEMO_CONFIG"

The volume is created in the demo-harbor-data stack. The 20 GiB value is a synthetic example; choose capacity based on your dataset, operating margin, and current price.

How volumes are mounted

Block storage is mounted at /mnt/chalupa. Each named Compose volume gets its own subdirectory under /mnt/chalupa/volumes, named after the Compose volume itself:

/mnt/chalupa/volumes/
  postgres-data/
  redis-data/

Bind mounts are generated from the volumes actually used by the selected services. Chalupa does not add a volume that belongs only to a pruned service.

Restore a seed

Declare a local command:

seed: bun run seed:demo

Keep the tunnel open in one terminal:

chalupa tunnel --config "$DEMO_CONFIG"

Then run this command in another:

chalupa seed --config "$DEMO_CONFIG"

The command runs in your project, not inside the server. Your tools reach the remote databases through SSH at localhost.

To run the same command on the server instead, with no tunnel to hold open, declare seed: { command: ..., onBox: true }. See seed. For a large dataset, load it once rather than seeding through the tunnel every run.

Keep literal credentials and secret-store references out of the seed string: the command is stored as a local Pulumi output. Let the project-owned script inject any required credential at runtime and keep shell tracing disabled.

Load a dataset once

Streaming a large dump through the tunnel on every run is slow, and it repeats work you already did. A dataset is the other way: upload it to the protected volume once, then rebuild the databases from that copy on the server, with no bytes crossing SSH per run. It needs persist, because the dataset lives on the volume.

1. Load the archive

With the server running:

chalupa data load ./orders.archive.gz --dataset orders --config "$DEMO_CONFIG" \
  --confirm "load <environment>"

The archive's SHA-256 and size are computed on your machine first, and the server re-checks both before the file is moved into place. A mismatch installs nothing and leaves any earlier dataset of that name untouched. The file is stored at /mnt/chalupa/datasets/<name> with mode 0600, and the dataset id (orders@sha256:<12 hex>) is recorded in ~/.chalupa/datasets/<environment>.json. The load refuses before writing when the volume lacks room for the archive plus a safety margin.

Loading is not billable and creates no stack, but it replaces a dataset of the same name, which is why it asks for the load <environment> phrase. The older --name DATASET spelling still works here and names the dataset, not an environment.

2. Restore it on the server, per case

Declare a restore recipe in cleanupRecipes. It runs a throwaway container of your own database image on the server with the dataset mounted read-only at /dataset, so the restore tool is exactly the version your stack runs. A count gate must pass before the restore counts as done, which catches a restore that "succeeded" into an empty database.

cleanupRecipes:
  load-orders:
    description: Restore the orders dataset
    restore:
      dataset: orders
      service: mongo
      command: mongorestore --host mongo --archive=/dataset --gzip --drop
      gate:
        kind: count
        command: mongosh --quiet --host mongo --eval 'db.getSiblingDB("shop").orders.countDocuments()'
        min: 1000

Then restore by dataset name before each case:

CONFIRM_RESET="cleanup load-orders <environment>" chalupa reset --dataset orders --config "$DEMO_CONFIG"

chalupa reset --dataset finds the one recipe that restores that dataset. It refuses if none or several do. Unlike most actions, reset is authorized with the CONFIRM_RESET environment variable, not --confirm. A seed that runs on the server (seed: { command, onBox: true }) can also read the loaded files through CHALUPA_DATASETS_DIR.

3. Start another environment from a snapshot

A fresh environment does not have to load again. When another environment has data snapshots (see Advanced: back up the data volume), create the new protected volume from one:

chalupa data-up --from-snapshot other-env --config "$DEMO_CONFIG"
chalupa data-up --from-snapshot <snapshot id> --config "$DEMO_CONFIG"

--from-snapshot takes another environment's name, which picks its newest snapshot that has not failed verification, a snapshot id, or latest (alias self), which picks this environment's own newest snapshot as the provider lists it, across every earlier generation of its volume. It is DigitalOcean only. It creates a new protected volume and refuses, before changing anything, when <name>-data already exists: it never restores over a volume that holds data. The snapshot must be in the environment's region, and persist.sizeGb must be at least the snapshot's minimum disk size. The source is not modified. It is billable and asks for the persist <environment> phrase like any data-up.

Design a repeatable seed

A production seed should:

1. fail with a nonzero exit code;
2. support resuming or explain how to clean up a partial attempt;
3. validate that the tunnel and the correct service are available;
4. avoid printing credentials;
5. produce a clear completion signal.

For large datasets, prefer verifiable files and bounded retries for each restore unit. Avoid one fragile stream when the source tool supports partitioning.

Advanced: back up the data volume

Creating a snapshot has a chalupa verb, chalupa data-snapshot. Listing, pruning and restore checks have none yet, so they run through the Taskfile bundled in the installed @thelacanians/chalupa-cli package, with go-task: task --taskfile <package directory>/Taskfile.yml <target> CONFIG=<path>. Those commands are written as task <target> for brevity. Each is a known gap.

A volume that survives compute teardown is not a backup. A provider or operator mistake can still destroy the only copy. Backups are therefore opt-in DigitalOcean volume snapshots with an explicit retention policy:

persist:
  sizeGb: 20
  backup:
    retentionCount: 3
    verifyCommand: ls -la /mnt/chalupa-verify

retentionCount (1–52) is how many snapshots to keep. Snapshots are crash-consistent: they capture the disk as the provider sees it, not a database's application state. quiesced consistency is refused until Chalupa can declare per-service prepare/resume hooks.

Reviewing a plan is the same command as executing it — without --plan the command is read-only and prints the plan:

chalupa data-snapshot --rate 0.06 --config "$DEMO_CONFIG"

The provider publishes no snapshot rate through its API, and Chalupa hard codes none: you state the rate you verified against current pricing, the estimate is computed from it, and the rate becomes part of the reviewed plan. Without a rate, creation is refused (the preview tells you so).

The preview shows the volume, its provider-observed size, the region, the consistency mode, what is retained, what a prune would delete, the monthly estimate and a plan id. To create the billable snapshot, re-run with that exact plan id:

chalupa data-snapshot --rate 0.06 --plan <plan id> --config "$DEMO_CONFIG"

A copy of the environment (@COPY) seeds its own data volume from the newest snapshot of the primary; see Copies of one environment.

The plan is re-derived from live state at confirmation and hashed; an edited config, a changed rate or a new existing snapshot makes it a different plan, and confirmation refuses.

List what is retained, newest first:

task data-snapshots CONFIG="$DEMO_CONFIG"

Prune beyond retention

Deleting snapshots is irreversible and needs its own, different confirmation. Review, then execute exactly the reviewed plan:

task data-snapshot-prune CONFIG="$DEMO_CONFIG"
task data-snapshot-prune CONFIG="$DEMO_CONFIG" \
  CONFIRM_DATA_SNAPSHOT_PRUNE=<plan id>

Only the ids the review listed are deleted, each deletion is confirmed against the provider before it is recorded, and a partial run simply replans — what already succeeded is gone from the next review.

Verify a restore

A snapshot is only a backup once it has been proven to restore. Verification never touches the protected volume: it creates a new volume from the snapshot, attaches it read-only to disposable compute, runs the declared verifyCommand with a timeout (verifyTimeoutSeconds, default 120), then removes the verification resources:

task data-restore CONFIG="$DEMO_CONFIG" SNAPSHOT=<snapshot id>
task data-restore CONFIG="$DEMO_CONFIG" SNAPSHOT=<snapshot id> \
  CONFIRM_DATA_RESTORE=<plan id>

Verification creates billable compute for minutes; its preview prices that compute from the provider's own size rates and requires the separate confirmation shown above. A snapshot that has not passed verification is labeled unverified, not healthy — check task data-snapshots.

Snapshot records live in a local <stack>-backup Pulumi stack, separate from compute and data, so sinking either cannot affect them. They hold snapshot ids and verification outcomes only. Because of them, task data-nuke refuses while retained snapshots exist, unless you separately restate exactly what remains:

task data-nuke CONFIG="$DEMO_CONFIG" CONFIRM_DATA_NUKE=demo-harbor \
  RETAINED_SNAPSHOTS_ACK=<retained count>

The snapshots outlive the volume and keep billing until you prune them on purpose; data-nuke never deletes them.

Warm-day storage that exists only while you work

A protected volume bills around the clock, including the days nobody opens the project. Two opt-in declarations turn the volumes into a "warm day": the first chalupa up of a day may be slower (the data volume is restored from a snapshot, the cache is recreated), later launches that day reuse them, and after an idle period the storage is retired again.

persist:
  sizeGb: 20
  backup:
    retentionCount: 3
  onDemand:
    retireAfterIdleHours: 24   # data volume: snapshot, verify, destroy
cache:
  sizeGb: 50
  retireAfterIdleHours: 12     # disposable: just delete

persist.onDemand requires persist.backup (the snapshot is the only copy once the volume is gone), DigitalOcean, and no inference.cache. On Hetzner only cache.retireAfterIdleHours is available, because volume snapshots are a DigitalOcean capability in Chalupa. Omit both and nothing here ever runs.

Retiring. chalupa storage sweep reads this machine's session ledger (~/.chalupa/launch-history/<env>.ndjson) to find how long the environment has been unused, so run it on the machine you develop on. With no flags it is a read-only plan: it says what is idle, what would be snapshotted and destroyed, the volume sizes and monthly prices (an unknown price says so; it is never shown as $0), and changes nothing. It needs neither tvault nor a provider call until something is actually eligible.

chalupa storage sweep --config ./chalupa.yml
chalupa storage sweep --config ./chalupa.yml --rate 0.06 \
  --confirm "retire storage <environment>"

Applying does, in this order, and stops at the first failure:

1. Cache: deletes <environment>-cache, the same volume chalupa cache-nuke removes.
2. Data: takes a fresh snapshot, then reads it back from the provider and requires it to be listed as a snapshot of this volume; only then destroys the data volume, restating the retained-snapshot count. If the snapshot step fails, nothing is destroyed. Eligibility is checked again right before the destroy.

It never retires anything while a compute box is recorded for the environment (run chalupa down first), never without a ledger on this machine, and never when the local Pulumi state cannot be read. Retiring data takes a snapshot, and DigitalOcean publishes no snapshot rate through its API, so pass --rate (or export SNAPSHOT_RATE_USD_PER_GIB_MONTH) with the value you verified; without it the sweep refuses to retire the data volume. The snapshots outlive the volume and bill until pruned: retentionCount counts every generation of the volume, and the sweep never deletes backups: when a retire leaves more snapshots than retentionCount it says so, and pruning stays a separate, confirmed step (see Prune beyond retention). Each day of use adds one snapshot, so run it from time to time.

The phrase is retire storage <environment> (--confirm, needs --config on the same command line); --apply asks for it on a terminal instead. Applying takes one --config; run one command per environment.

Restoring. When an environment that opted in is launched and its data volume does not exist, chalupa up restores it from the newest snapshot of the same environment (chalupa data-up --from-snapshot latest does the same by hand) and recreates a retired cache, then launches compute as usual. The launch summary and the Taskfile print each volume's size and price before the phrase is asked for, and the launch phrase (launch <environment>) is what authorizes them: the environment opted in declaratively, what is created is a volume seeded from its own snapshot (never an empty one, never over an existing one) and an empty disposable cache. If there is no snapshot to restore the launch stops before creating anything; it never starts on an empty volume where you expect your data. The first volume of a new environment is still made with chalupa data-up.

Running it periodically. Chalupa installs nothing and the platform runs no schedule for this. Put one line per environment in the crontab (or a launchd job with StartInterval 3600) of the machine whose ledger it reads; TinyVault must be unlocked for a run that applies, and a run that finds nothing idle exits 0 without touching tvault:

17 * * * * SNAPSHOT_RATE_USD_PER_GIB_MONTH=0.06 chalupa storage sweep --config /abs/chalupa.yml --confirm "retire storage <environment>" >> ~/.chalupa/storage-sweep.log 2>&1

What happens when you sink compute

chalupa down --config "$DEMO_CONFIG"

This destroys the server. It does not destructively unmount or delete the demo-harbor-data stack. On the next chalupa up, the new compute mounts the volume again.

In the Chalupa console, a sunk environment can show:

• estimated compute cost of zero;
• estimated storage cost greater than zero;
• the latest observed capacity and usage;
• a seed-restored event, if one was reported.

Do not interpret “sunk” as “no remaining cost.”

Advanced: delete data

The chalupa CLI never exposes a data-deletion command, interactively or otherwise. The one way to do it by hand is the Taskfile-only data-nuke target (the only automated route is chalupa storage sweep, for an environment that opted in, and only after a snapshot is verified):

task data-nuke \
  CONFIG="$DEMO_CONFIG" \
  CONFIRM_DATA_NUKE=demo-harbor

This destroys the persistent volume. It is irreversible and not part of the normal lifecycle. Before you run it:

• confirm the exact stack;
• verify that no unique seed exists without a backup;
• retain any required artifacts or dumps;
• communicate the expected data loss.

The confirmation value must exactly match name in chalupa.yml. A missing or different value fails before Chalupa unprotects the data stack. Never turn this confirmation into a reusable shell default.

chalupa status --config "$DEMO_CONFIG" can distinguish a configured data stack that is absent, present, or unknown without selecting it or contacting the provider. Its expected size comes from chalupa.yml; it is not a live provider-capacity reading. See Local CLI and operator UI.

Artifacts are not volumes

Videos, reports, and screenshots from a suite must not accidentally live inside the database volume. Chalupa suite runs store ArtifactRefV1 metadata for a local stash, a stable external link, or a committed private hosted file.cheap artifact. Transport locators cannot contain embedded credentials, but metadata is not DLP-scanned and must remain non-sensitive. file.cheap can emit either local fcheap-local references or private hosted fcheap-cloud references. Chalupa stores the reference envelope only: it never stores artifact bytes or tries to resolve a local fcheap:// URI. See Suites and artifacts.