Remote GPU inference

Chalupa can provision Ollama on a GPU server and tunnel it to 127.0.0.1:11435. Local embeddings can keep their own Ollama on 11434. This workflow is experimental: a supported configuration is not a measured compatibility or performance result.

Server capacity and provider support

The configured GPU tiers are gpu-small (20 GiB VRAM), gpu-large (48 GiB) and gpu-max (80 GiB). These are capacity presets, not a guarantee that a particular model or context fits. Use compute.size: auto for a model-based estimate, then inspect the resolved size and hourly estimate with chalupa preview. Available regions, account quotas and capacity are checked by the provider at launch.

GPU inference currently uses DigitalOcean. Hetzner supports CPU environments; Chalupa does not currently provision GPU inference there. Your selected provider remains visible in configuration, launch previews and deployment records.

Set up and connect

Managed GPU is the default. See Quickstart for account requirements and the current browser account-login limitation.

chalupa setup
chalupa balance --top-up 10 # only if empty
chalupa up
chalupa launch

Setup writes the default agent (opencode) and leaves context on auto. Use --config PATH, an explicit agent name, or --context only to override the project defaults.

DigitalOcean BYOC

chalupa setup --inference --provider digitalocean
chalupa preview
chalupa teardown schedule:install
chalupa teardown schedule:doctor
chalupa up

Setup registers your SSH key, links your Chalupa hosted account (reusing a valid stored session or asking for email verification), and writes a GPU starter with cloud.url: https://chalupa.run and 30-second heartbeats. These let the console observe the deployment for idle deadlines and session expiry once ingestion is configured. Use chalupa setup --inference --no-console to skip linking and omit cloud:; setup warns that expiry then depends on this machine staying awake and unlocked. --json does not prompt for account linking and omits cloud: when no valid account session is stored. Review the generated SSH allowlist before launching.

With a linked account, setup offers to arm your organization's console reaper. In DigitalOcean, open API → Generate New Token → Custom Scopes and select only droplet:read and droplet:delete. Paste this separate token into setup's hidden prompt. It is sent once to the linked console, stored encrypted there, and never kept on this machine. Reruns check whether the reaper is already armed. --no-reaper-token skips this step while keeping console linking. In --json mode, the step is skipped unless CHALUPA_SETUP_REAPER_TOKEN is supplied through the environment; a valid stored hosted account session is still required.

The preset is qwen3.8:27b, auto-negotiated context and a 24 GiB weight ceiling. The GPU selector estimates capacity from the model tag; actual memory use also includes the KV cache and runtime overhead.

From your working project, launch a supported agent harness:

chalupa launch opencode

Chalupa opens the tunnels, checks the configured endpoint and model metadata, and starts your agent in the current directory. It configures the endpoint for that process without changing your agent's settings files. The first turn can still load weights into GPU memory; observed model metadata is not a successful tool execution.

Unlisted tools, such as local-agent, can use chalupa exec -- <command>; see Agent harnesses for endpoint variables and integration steps.

The command inherits your terminal and returns the local command's exit code. On exit, Chalupa closes its tunnels. Compute remains provisioned until chalupa down succeeds. A lost SSH transport stops the local command rather than silently switching models. Startup checks reject occupied local ports.

The manual path remains available: chalupa tunnel holds the connection in a separate terminal and chalupa connect --format env prints endpoint exports for your shell. Neither command provisions compute. While connected, inspect chalupa status for model metadata and loaded GPU placement.

OpenCode and oh-my-pi

The same launcher serves OpenCode and oh-my-pi (omp). Both talk to Ollama's OpenAI-compatible /v1 routes on the tunnelled port. The shortest form uses inference.agents.default; the model comes from inference.model:

chalupa launch                      # inference.agents.default
chalupa launch opencode             # optional override
chalupa launch omp
chalupa launch omp -- -p --no-session "Summarize this repository"

launch works like a session-scoped ollama launch: the agent runs with the GPU endpoint configured for that process, and its global or project settings are never modified. OpenCode receives the configuration inline, omp receives a per-run --config overlay in a scratch directory that is deleted on exit. Per-project arguments, environment and config overrides live in inference.agents. The generic form stays available for any other command:

chalupa exec -- opencode
chalupa exec -- omp --model ollama/qwen3.8:27b

OpenCode receives OPENCODE_CONFIG_CONTENT, an inline opencode.json with a chalupa provider and chalupa/<model> as the default model; it merges over any project opencode.json. omp receives OLLAMA_BASE_URL and OLLAMA_CONTEXT_LENGTH for its implicit Ollama provider. Any other tool can read CHALUPA_INFERENCE_BASE_URL, CHALUPA_INFERENCE_OPENAI_BASE_URL, CHALUPA_INFERENCE_MODEL and CHALUPA_INFERENCE_CONTEXT_SIZE. OLLAMA_HOST moves to the tunnel only for omp and ollama; other commands keep it so laptop embeddings can continue using their own daemon.

When portless is installed, chalupa exec and chalupa launch also register a named loopback route for the session, http://chalupa-<environment>-ollama.localhost:<proxy port>, and remove it on exit; the child sees it as CHALUPA_INFERENCE_NAMED_URL. It is for you and the console, not for agents: they keep the 127.0.0.1:11435 address, which needs no proxy, DNS or TLS. An existing route with the same name is never taken over, a route that does not answer on loopback is withdrawn, and CHALUPA_PORTLESS=0 skips the step.

The OpenAI-compatible API cannot set num_ctx per request, so the context these values declare is the window the host daemon runs with; --context on chalupa exec re-renders them after negotiation. For the manual path, chalupa connect --format env prints shell exports for eval, --format opencode prints an opencode.json, and --format omp prints omp's models.yml and config.yml (an explicit omp provider must keep /v1 in its baseUrl). Verified with OpenCode 1.17.11 and omp 18.1.11 against Ollama 0.33.3. Ollama's OpenCode guide recommends a 64k window for tool calling; a larger window also grows the KV cache on the GPU.

Public IP and changing networks

Interactive setup looks up your IPv4 through ipify and proposes a single-address /32 SSH source. Press Enter to accept it or enter a different IPv4 CIDR. If detection fails, setup asks for manual input. CHALUPA_SETUP_SSH_CIDR supplies your own source and skips detection; non-interactive setup --json still requires that variable explicitly. Existing chalupa.yml files are left untouched.

To check your current IPv4 yourself:

curl -4 --fail --silent --show-error --connect-timeout 5 --max-time 10 https://api.ipify.org

Use the result followed by /32 in ssh.allowedCidrs. The laptop initiates the SSH connection, so you do not need router port forwarding or a static home IP. A VPN or HTTP proxy can change the observed address; confirm that the allowlist matches the route used by SSH.

If your IP changes, update ssh.allowedCidrs in the same config, review it with chalupa preview --config ./chalupa.yml, then apply it through chalupa up --config ./chalupa.yml before reconnecting the tunnel. Editing YAML alone does not update the live firewall. This uses the normal provisioning workflow, including its confirmation and expiry registration; it may apply other pending config changes and creates compute if the stack is absent. Setup does not watch for IP changes or update a running firewall automatically.

Collect GPU samples and report model turns

Managed hosts (provider: chalupa) get the same signed heartbeat and GPU telemetry, plus Ollama service logs, after automatic bootstrap at up. Retry with chalupa session bootstrap; tunnel and launch also retry incomplete delivery. The CLI streams a scoped key to the pinned host without saving its material locally. Compatible inference receipts remain explicit reports; OpenCode rounds alone do not populate the inference-runs table. See Managed GPU.

The following key configuration instructions describe BYOC hosts.

The environment detail page includes GPU & inference. GPU samples travel in the existing signed heartbeat; reported agent results require an explicit report command. Nothing uploads prompts, answers, raw tool arguments, tool output, error messages or workspace paths.

Interactive chalupa setup --inference now links the console by default and writes this block when the account is linked. For an older config, add it before launching a new host:

cloud:
  url: https://chalupa.run
  heartbeatSeconds: 30

--no-console opts out of linking and omits this block from the GPU starter. Configure the existing ingest credential before up; signing into the console alone does not issue an ingest key. For cloud.url environments, chalupa up loads the existing ingest key from TinyVault and delivers it over SSH through agent-bootstrap to /etc/chalupa/ingest.key. Platform operators can issue an environment key under Administration → Ingest keys. With an environment key, add its public identifier as cloud.ingestKeyId and inject its matching material as CHALUPA_INGEST_KEY. Keep the secret in TinyVault, never YAML. Omitting the identifier selects legacy platform ingestion, not a customer organization. Managed customer keys are issued automatically by the managed bootstrap workflow. Scoped-key configurations support heartbeat, Monitor, private logs and local reports; remote CI publishers still require legacy credentials.

New inference hosts with cloud configured collect NVIDIA GPU utilization, used and total VRAM, temperature and power through nvidia-smi. Unsupported sensors remain unknown. A failed probe reports unavailable without rejecting the rest of the heartbeat. Existing hosts need the updated agent and collector; changing your local config does not install them on a running host.

The page reads up to 240 heartbeats within the existing 48-hour retention window. Charts stop at missing readings. The history table shows the latest 20 GPU heartbeats. These snapshots do not detect every brief utilization spike or OOM event. There are no GPU threshold alerts or long-term GPU rollups yet.

Keep a JSON receipt with a unique run ID from a tool that supports Chalupa's receipt contract (defined in src/inference-report.ts). Launch support does not imply receipt support: arbitrary agent JSON, including OpenCode and oh-my-pi output, is not automatically accepted by inference report.

A failed turn can still produce a useful receipt. Publish it separately, so a reporting failure never reruns inference. Find the deployment provider ID using chalupa status --config ./chalupa.yml; replace 123456789 below with the ID that actually ran the turn. Do not substitute a newer deployment when reporting an older receipt.

Inspect the exact safe payload without credentials or network access:

chalupa inference report --config ./chalupa.yml \
  --file receipt.json --provider-id 123456789 --dry-run

Inject the matching ingest key to send that projection:

tvault run -p chalupa --only CHALUPA_INGEST_KEY -- \
  chalupa inference report --config ./chalupa.yml \
  --file receipt.json --provider-id 123456789

Add --publish to also store the same safe projection as an immutable artifact in file.cheap (kind chalupa.inference-receipt, 30-day retention): inject FILECHEAP_INGEST_TOKEN, the producer-bound credential file.cheap issues for chalupa-cli, the same way as the ingest key. The command prints the stable fcheap://cloud/... reference and saves it next to the receipt as <receipt>.artifact-ref.json (plus a small <receipt>.filecheap-plan.json with the plan facts a retry must repeat); the token, upload grant and commit receipt never reach the terminal or the sidecar. Re-publishing the same receipt replays the stored artifact instead of creating a second one. FILECHEAP_ARTIFACT_ORIGIN points a self-hosted broker.

Repeated reports with the same deployment and run ID deduplicate. Different content under that ID is refused. Use a new --run-id for each new turn. Reports can arrive after teardown without changing the deployment state. The interactive TUI does not automatically publish turns.

The dashboard shows model, context, reported tokens, generation speed, time to first token, tool error counts and estimated runtime cost. A zero or missing timing means unavailable; it does not mean instantaneous inference. Model digests appear only when the receipt includes one; remote receipts can omit them. A settled turn is not a verified quality result.

Runtime cost uses the recorded launch rate for that exact deployment multiplied by reported request duration. It excludes downloads, idle time, storage and provider billing minimums. Missing rates remain unknown. Compare equal tasks, model digests and context sizes before drawing performance conclusions. Inference summaries survive heartbeat expiry and are removed when their environment or deployment is deleted.

Model weights cache

inference.cache.sizeGb keeps the pulled weights on a retained block storage volume, chalupa-<environment>-models, mounted over Ollama's models directory on the GPU host. The first up still downloads the model; every later up in the same environment finds it and skips the download, so a 60-minute session does not start with a 10 to 20 minute pull on the GPU clock. The starter uses 60 GiB (about $6/month at list price).

The cache lives in the protected <environment>-data stack next to the seed volume, in the Chalupa project, tagged chalupa-models. chalupa up creates it once when it is missing (the up confirmation covers it); chalupa down leaves it; chalupa audit lists it as retained on purpose; only task data-nuke deletes it, together with the seed volume. The volume is region-bound: it follows the compute region of the environment. Without cache, weights are pulled on every boot.

The host runs with OLLAMA_NO_CLOUD=1, so no request is ever routed to ollama.com's hosted models from the droplet.

Expiry and billing

chalupa preview shows the three clocks offline: Managed hours — hours the control plane observed signed activity, Chalupa's plan clock, never GPU or droplet hours; Session — when Chalupa will try to stop compute, with session expiry, any idle window, and their enforcers; Provider estimate — estimated hourly rate × session length plus estimated retained storage per month; the provider invoice wins.

Extending a running session is an explicit lifecycle act:

chalupa session extend --hours 2 --config ./chalupa.yml
# Non-interactive authorization:
chalupa session extend --hours 2 --config ./chalupa.yml --confirm "extend <environment>"

Before confirmation, Chalupa prints the new UTC deadline and the estimated added provider cost using the running host's hourly rate (or says the cost is unknown). The provider invoice wins. Extending is never a Chalupa charge. Each extension adds 1–8 whole hours to the current deadline, with at most two extensions per registration. Total lifetime, counted from original registration, is capped at 12 hours for inference and 24 hours otherwise. Expired sessions cannot be extended. The local registration and the console's session clock both receive the new deadline through a signed session-extended lifecycle event.

If console delivery fails, the local extension still stands, but the console will stop the host at the original deadline unless reporting is retried. Run the printed retry command; it resends the same deadline without consuming another extension. Idle shutdown and ci.onFinish remain separate stop conditions.

Inference up checks the existing macOS teardown scheduler before spending and registers the provisioned deployment for the configured session expiry. If registration fails, it attempts compute teardown and returns an error. Editing YAML after registration does not cancel the registered deadline when the canonical config path and stack name still match. The runner notes the drift and checks the provider ID before teardown. A changed path or stack name, or an unreadable config, still blocks automatic teardown; ci.onFinish also requires the exact registered file digest. Without an armed console reaper, keep the operator Mac awake, online, and TinyVault unlocked. Failed provisioning or failed cleanup can leave billable resources; inspect your provider account.

For reported deployments (those with cloud.url and an ingest key), the configured console reaper also destroys compute at session expiry as well as at the idle deadline, even if the operator laptop is asleep. Customer organizations can opt in during chalupa setup --inference by supplying a DigitalOcean custom token with droplet:read and droplet:delete scopes. These scopes allow reading and deleting droplets across that DigitalOcean account. Chalupa only destroys Chalupa-tagged droplets registered to your organization when their session or idle deadline passes. The token is encrypted at rest with AES-256-GCM under CHALUPA_INGEST_MASTER_KEY and is never shown again.

Revoke console access with DELETE /api/account/provider-token?provider=digitalocean using your account session bearer, or revoke the token instantly in DigitalOcean. Without opt-in, your local scheduler remains responsible. Session expiry overrides fresh heartbeats, idle SSH tunnels, and live CI suites; live CI only defers idle shutdown.

This is not a hard spend cap. The deadline is not a provider-enforced budget: cron delays, missing reports, and provider or control-plane failures can leave compute billing past expiry. Persistent storage continues billing after teardown.

Weight downloads consume paid GPU time. Hosted plan managed hours are separate from provider GPU hours and buy no model tokens or GPU capacity. Finish explicitly and check that compute has disappeared:

chalupa down --config ./chalupa.yml
chalupa audit

If a down was interrupted, a laptop lost its Pulumi state, or a droplet was made by hand, chalupa gc shows every Chalupa droplet no local stack tracks, with its hourly price, and prints the exact command that reclaims it:

chalupa gc
chalupa gc --delete chalupa-gpu-ollama --confirm "reclaim chalupa-gpu-ollama"

Each call deletes one leftover and only reports success once DigitalOcean no longer lists it. Volumes and snapshots are never reclaimed this way; they are the seed data and the model cache, deleted only by task data-nuke.

Current limits

Only host Ollama on a single DigitalOcean GPU is managed here. Remote API aliases are rejected. Managed vLLM/SGLang, multi-GPU DeepSeek V4 Flash deployments, and provider-side hard spending limits are not implemented. The cloud's global provider credentials manage only its platform organization, never customer fleets; customer session expiry currently runs on the operator machine.

Choose context for each session

chalupa launch opencode --context auto
chalupa launch opencode --context 128k
chalupa launch opencode --context native

inference.contextSize is optional and defaults to auto, so chalupa launch and chalupa exec negotiate by default. A configured number preserves the fixed-window behavior. --context overrides either setting for one session. k means 1,024 tokens. auto starts at the model's reported context ceiling (up to the client's 1,048,576-token limit), warms the installed model, and checks its digest, loaded context and GPU placement. On an observed context or placement mismatch it halves the request, down to 1024 tokens for auto configs (or the configured numeric floor). It returns the first admitted size. Explicit token counts and native do not silently fall back. compute.accelerator.allowCpuOffload is still honored.

Negotiation uses an empty warm-up request on the existing host. It can reload the model and take several minutes; do not run it against another client's active session. Network failures, missing metadata, changed model identity and failed warm-ups stop the command. For an OOM during loading, retry with an explicit smaller context. A successful warm-up verifies the observed placement at that moment, not a guarantee that every full-length prompt fits.

The selected size reaches the agent profiles (OpenCode context limit, omp budgeting environment), per-run launch preparation and CHALUPA_CONTEXT_SIZE. Offline chalupa connect profile exports require a fixed context; use launch/exec for auto negotiation. Neither the project YAML nor the deployment's expiry identity changes. Context is selected when connecting; it does not grow during an existing conversation. Compaction remains the client's policy.

With auto, the host daemon starts with a native ceiling estimate: 262144 for qwen3.5/qwen3.8, 131072 for llama3.x, gemma3 and unknown families, always capped at 1,048,576. OLLAMA_NUM_PARALLEL=1 stays set. Launch uses observed metadata to choose the actual session size. Fixed configs keep their numeric daemon limit. Updating an existing host requires reprovisioning through the normal lifecycle; editing YAML alone does not change its daemon.

If omp reports “compaction freed too little context” after one message, a fixed 32768-token window may already be nearly full: its system prompt can consume roughly 28K tokens. Prefer auto to use available GPU context. If the GPU only admits a small window, reduce prompt/tool overhead or choose a model and GPU combination that admits a larger context.

maxModelBytes limits model weights, not total VRAM. GPU memory also holds the KV cache and runtime allocations. A model advertising 256k context does not prove that every GPU can serve it at 256k. See Ollama's context guide and generation API.

Inference console and billing

The console's Inference section lists the latest 100 reports received in the last 30 days for the current organization. Its totals cover that bounded list, not all inference since signup. It shows context, prompt/output tokens, generation speed, first-token time, completion state, truncation and tool errors, with links to each environment's GPU observations. No reports are invented in demo mode. The 30-day query window is not a plan-specific retention guarantee; raw inference expiry by subscription is not implemented yet.

The subscription pays for Chalupa's control plane. GPU uptime is billed separately by your provider, including downloads and idle time. Tokens are usage observations, not billable tokens sold by Chalupa. Context size has no plan gate. The per-turn runtime cost equivalent uses reported request time and the deployment's recorded rate; concurrent requests may overlap, so it must not be summed as an invoice or treated as allocated GPU cost.

Closing the client closes the tunnel, not compute. Finish with chalupa down; launch and sink now ask yes/no, defaulting to No. Non-interactive commands still require an explicit config and their exact --confirm phrase.