Run a suite with chalupa test

chalupa test <suite> runs one suite you declared in chalupa.yml. It is the whole loop in one command: it prepares the data volume, launches the server, opens the tunnel, runs the suite, collects the results and sinks the server.

Status: Beta. The command, its consent flow and its exit codes are covered by offline tests that use fake provider and runner processes. The only runner it supports is CairnTrace (runner: cairn), version 2.5.0 or newer, and the browser and the runner run on your machine, not on the server.

chalupa test is the laptop-side sibling of ci:. A test: suite runs on your machine against the server's ports over the tunnel. A ci: suite runs inside the server.

Declare a suite

Add a test: block next to the rest of the environment. The block is checked when the config loads, so a mistake is reported before anything bills.

test:
  runner: cairn
  config: ./cairntrace.config.yml
  environment: chalupa
  suitesRoot: ./flows
  secretsEnv: shop-test
  labels:
    team: checkout
  env:
    SHOP_BASE_URL: http://localhost:8080
  report: true
  onFinish:
    local: pkill -f agent-browser
    timeoutSeconds: 30
  suites:
    smoke: {}
    checkout:
      flows: ./flows/checkout-v2
    changed:
      sinceRef: main

A test: block needs compose and services: the suite reaches the server through the tunnelled Compose ports, so a workspace environment with no Compose stack is refused. chalupa test --list prints the declared suites and never runs anything. Without a test: block, chalupa test refuses and says so.

What a run does

chalupa test checkout --config ./chalupa.yml

Before anything bills, Chalupa resolves the suite offline: it loads the config, finds the flows directory, checks that CairnTrace is installed and new enough, and asks CairnTrace which specs the suite selects. Then it shows one summary (environment, specs, server size, estimated cost, data volume, what happens on exit) and asks for one confirmation.

The billable phases follow, each printed as one numbered line:

The run keeps a transcript of each phase on your machine. Read an earlier run's transcripts with chalupa test logs [--run ID] [--phase NAME]. The phase names are data-up, up, watchdog, tunnel, report, release and down.

The confirmation

chalupa test asks you to type one phrase, test <environment> <suite>, for example test shop-test checkout. That single phrase covers every billable phase of the run. It needs a terminal.

A script passes the same phrase with --confirm and an explicit --config:

chalupa test checkout --config ./chalupa.yml --confirm "test shop-test checkout"

When the run would adopt a server that is already running and create nothing, it needs no phrase and refuses --confirm: leave the flag out in that case. chalupa up with session.keepFor is the usual way to have a warm server waiting.

Options

--dry-run is the safest way to see what a suite will do:

chalupa test checkout --dry-run --config ./chalupa.yml

What happens when the run ends

By default the server is sunk and the data volume is left alone, so the next run starts from the same data.

• With --keep, nothing is sunk. The server bills until you run the printed chalupa down command.
• With session.keepFor declared, the run releases the server into a warm window instead of sinking it, shows the maximum cost of that window in the confirmation summary, and then the reaper sinks it. If the release fails or is refused, the run sinks the server. See Warm boxes and reuse.
• Ctrl-C interrupts the suite and sinks the server (exit 130). The results of an interrupted run stay on your machine and are not published.

The onFinish.local command, if declared, runs last, after the run finalizes. It receives CHALUPA_ENVIRONMENT and CHALUPA_TEST_SUITE.

Exit codes

A suite failure (1) and an infrastructure failure (69) are different on purpose, so a script can retry one and page someone for the other.

Related

• test: in the config reference
• ci: suites that run inside the server
• What a run leaves behind
• Warm boxes and reuse