# AI Agent Workflows



# AI Agent Workflows

GPU CLI works well with coding agents that share your local workspace. The practical workflow today is:

1. install and authenticate GPU CLI locally
2. let the agent inspect your repo and generate or edit `gpu.jsonc`
3. use `gpu agent-docs` when the agent needs machine-oriented command and workflow reference
4. run `gpu run`, `gpu llm run`, `gpu use`, or `gpu serverless` from the same repo

## Start with `gpu agent-docs`

`gpu agent-docs` prints agent-focused reference material to stdout.

```bash
gpu agent-docs
gpu agent-docs | head -50
```

Use it when your agent needs a compact source of truth for:

* command names and examples
* workflow shape
* JSON output behavior
* common troubleshooting hints

## Good Agent Tasks

Agents are a good fit for:

* generating `gpu.jsonc` from a project description
* choosing GPU types and output patterns
* setting up `gpu llm` and template workflows
* debugging sync, auth, and startup issues
* adding provider or model-hub auth instructions to project docs

## Common Workflows

### Generate a project config

```text
Create a gpu.jsonc for fine-tuning on an RTX 4090, sync checkpoints/, and keep the pod warm for 20 minutes.
```

### Set up a routed LLM pod

```text
Configure this repo for gpu llm run with vLLM and a HuggingFace model.
```

### Turn a project into a reusable template

```text
Create a gpu.jsonc template with startup, ports, and hooks so I can run it with gpu use.
```

## Built-In GPU CLI Paths to Know

* `gpu run`: general remote command execution
* `gpu llm run`: routed Ollama or vLLM workflow
* `gpu use ollama` / `gpu use vllm`: official LLM templates
* `gpu comfyui ...`: curated ComfyUI workflows
* `gpu serverless ...`: RunPod Serverless deployment

## Official Templates Mentioned Often by Agents

* `hello-gpu`
* `ollama`
* `vllm`
* `invokeai`
* `crewai`
* `image-to-3d`
* `qwen`
* `chatterbox`

## Requirements

* [GPU CLI](/docs/quickstart) installed and authenticated
* a coding agent that can read and edit the same local workspace you use
* provider credentials configured with `gpu auth runpod` (or the relevant provider), or an exported provider API key such as `RUNPOD_API_KEY`; `gpu login` is not required

## Related Docs

* [LLM Inference](/docs/llm)
* [Configuration](/docs/configuration)
* [Commands Reference](/docs/commands)
* [Troubleshooting](/docs/troubleshooting)


---

# Commands Reference



# Commands Reference

This page tracks the public GPU CLI surface that ships today. Use `gpu --help` for a quick overview, `gpu --help-all` for advanced and hidden commands, and [LLM Inference](/docs/llm) for the routed LLM workflow.

## gpu run

Run any command on a remote GPU pod.

```bash
gpu run <COMMAND>
```

### Common examples

```bash
gpu run python train.py
gpu run -d python long_job.py
gpu run -p 8080:8080 python server.py
gpu run --gpu-type "RTX 4090" python train.py
gpu run --attach job_abc123
```

### Key flags

| Flag                             | Description                                          |
| -------------------------------- | ---------------------------------------------------- |
| `--attach, -a <JOB_ID>`          | Reattach to an existing job                          |
| `--detach, -d`                   | Submit the job and return immediately                |
| `--status, -s`                   | Show current pod state and recent jobs               |
| `--cancel <JOB_ID>`              | Cancel a running job                                 |
| `--tail, -n <N>`                 | Show last N lines when attaching to a completed job  |
| `--interactive, -i`              | Allocate a PTY and keep stdin open                   |
| `--gpu-type <TYPE>`              | Override GPU type for this run                       |
| `--gpu-count <N>`                | Request multiple GPUs (1-8)                          |
| `--min-vram <GB>`                | Set fallback VRAM floor                              |
| `--rebuild`                      | Force pod recreation when the Dockerfile has changed |
| `--output, -o <PATH>`            | Override synced output paths                         |
| `--no-output`                    | Disable output syncing                               |
| `--sync`                         | Wait for output sync before exiting                  |
| `--outputs`                      | Show live daemon-managed output sync events          |
| `--output-summary`               | Show a brief sync summary on completion              |
| `--no-summary`                   | Suppress the end-of-run summary banner               |
| `--show-sync`                    | Show detailed sync progress                          |
| `--force-sync`                   | Ignore change detection and do a full sync           |
| `--remote-path <PATH>`           | Override the remote workspace path                   |
| `--publish, -p <[LOCAL:]REMOTE>` | Publish remote ports to localhost                    |
| `--no-port-forward`              | Disable automatic port detection                     |
| `--no-persistent-proxy`          | Stop the local proxy when the pod stops              |
| `--env, -e <KEY=VALUE>`          | Set environment variables                            |
| `--json`                         | JSON output after submit; only works with `--detach` |

## gpu serve

Serve ML models on cloud GPUs with automatic configuration. Resolves the right GPU, engine, and quantization for your model.

```bash
gpu serve meta-llama/Llama-3.3-70B-Instruct --dry-run
gpu serve llama-70b --quantize int4 --budget 2.00 --dry-run
gpu serve deepseek-r1:70b --users 50 -y
gpu serve llama-8b whisper-large-v3 --dry-run
```

### Key flags

| Flag                  | Description                                           |
| --------------------- | ----------------------------------------------------- |
| `--dry-run`           | Show GPU recommendation without launching             |
| `-y, --yes`           | Skip confirmation and launch immediately              |
| `--quantize <FORMAT>` | Quantization: `fp16`, `int8`, `int4`, `gguf-q4`, etc. |
| `--gpu <TYPE>`        | Override GPU type (e.g., `h100`, `a100`)              |
| `--gpu-count <N>`     | Override GPU count (1-8)                              |
| `--engine <ENGINE>`   | Override engine: `vllm`, `ollama`, `llamacpp`         |
| `--context <TOKENS>`  | Override context window size                          |
| `--users <N>`         | Target concurrent users (affects GPU sizing)          |
| `--budget <DOLLARS>`  | Maximum hourly cost; shows comparison table           |
| `--optimize <TARGET>` | Optimization: `cost`, `latency`, `balanced`           |

Supports LLMs, diffusion models (FLUX, SDXL), STT (Whisper), and TTS (Kokoro). See the model catalog for supported models and aliases.

## gpu attach

Reattach to a job by ID. This is a focused alias for `gpu run --attach`.

```bash
gpu attach job_abc123
gpu attach job_abc123 --tail 50
```

## gpu wait

Block until a job reaches a terminal state, then exit with the job's exit code. Pairs with `gpu run -d` for detached workflows: submit, capture the job ID, and wait deterministically instead of polling.

```bash
gpu wait job_abc123
gpu wait job_abc123 --timeout 3600
gpu wait job_abc123 --json
```

Exits with the job's own exit code on completion, `12` if the job is unknown, and `14` on timeout. With `--json`, prints a final `{"job_id", "state", "exit_code"}` object.

## gpu cp

Copy files to or from the remote workspace.

```bash
gpu cp model.pt gpu:/workspace/
gpu cp gpu:/workspace/results/ ./results
gpu cp -r ./data gpu:/workspace/data/
```

Use the `gpu:` prefix, or the shorthand `:` prefix, for the remote side.

## gpu transfer

Inspect the durable transfer receipts the local daemon retains for `gpu cp`.

```bash
gpu transfer list
gpu transfer status <transfer-id>
gpu transfer status <transfer-id> --remote
gpu transfer cleanup <transfer-id>
```

`gpu cp` prints a transfer ID. `list` shows receipts newest first, `status`
reports one of them, and `cleanup` removes a retained hidden partial without
touching the final destination. `--remote` queries the partial on its exact pod
and volume instead of reading the local receipt. Every subcommand takes `--json`.

## gpu login

Authenticate with GPU CLI in your browser. Optional: `gpu login` adds a free GPU CLI account (portal identity and owned-node/fleet features). It is not required to connect a provider, rent GPUs, or run jobs. GPU CLI is free forever, no account required. Bring your own GPU or a provider API key. You pay the provider directly at their rates, zero markup.

```bash
gpu login
```

| Flag                  | Description                         |
| --------------------- | ----------------------------------- |
| `--timeout <SECONDS>` | Browser auth timeout (default: 300) |

## gpu logout

Remove the stored browser session.

```bash
gpu logout
```

| Flag                 | Description       |
| -------------------- | ----------------- |
| `--yes, -y, --force` | Skip confirmation |

## gpu auth

Manage provider and model-hub credentials. Connecting a provider does not require `gpu login` first.

### auth `<provider>`

Connect a cloud provider by name. Enter the API key when prompted, and an SSH key for pod access is generated automatically.

```bash
gpu auth runpod
gpu auth vast
gpu auth thunder
gpu auth ionet
```

You can also skip the prompt by exporting the provider API key, which GPU CLI picks up automatically (`RUNPOD_API_KEY`, `VAST_API_KEY`, `THUNDER_API_KEY`, or `IONET_API_KEY`; the `GPU_`-prefixed form wins if both are set).

### auth login

Authenticate with your cloud provider profile.

```bash
gpu auth login
gpu auth login --profile dev
gpu auth login --generate-ssh-keys
```

### auth logout

Remove provider credentials.

```bash
gpu auth logout
```

### auth status

Show current auth state.

```bash
gpu auth status
```

### auth add

Add HuggingFace or Civitai credentials for model downloads.

```bash
gpu auth add hf
gpu auth add civitai --token <VALUE>
```

### auth remove

Remove model-hub credentials.

```bash
gpu auth remove hf
```

### auth hubs

List configured model-hub credentials.

```bash
gpu auth hubs
```

## gpu init

Create a `gpu.jsonc` file for the current project.

```bash
gpu init
```

| Flag                               | Description                     |
| ---------------------------------- | ------------------------------- |
| `--gpu-type <TYPE>`                | Default GPU type                |
| `--max-price <PRICE>`              | Maximum hourly price            |
| `--profile <NAME>`                 | Profile to use                  |
| `--encryption` / `--no-encryption` | Toggle volume encryption        |
| `--force, -f`                      | Reinitialize an existing config |

## gpu prime

Print gpu-cli conventions for a coding agent's context. Emits the project's agent guidance (use `uv` not `pip`, forward secrets via `--env`, `gpu run -p` not raw SSH) plus a compact summary of this project's config. Intended for a Claude Code SessionStart/PreCompact hook (installed by `gpu init`) so the conventions stay in context across compaction. Override the output with a `.gpu/PRIME.md` file.

```bash
gpu prime
```

| Flag         | Description                                                                           |
| ------------ | ------------------------------------------------------------------------------------- |
| `--defaults` | Print the built-in conventions only, ignoring `.gpu/PRIME.md` and the project summary |

## gpu inventory

Compare GPU pricing and availability across every cloud you're authenticated with.

```bash
gpu inventory                              # Cross-provider matrix (default)
gpu inventory --provider runpod            # Detailed view for one provider
gpu inventory --gpu-type H100 --min-vram 80
gpu inventory --available                  # Only in-stock GPUs
gpu inventory --show-datacenters           # Show DC under each cell
gpu inventory --show-unmapped              # Include "Unknown GPU" listings
gpu inventory --json                       # Structured output for scripts
```

### Cross-provider matrix (default)

Without `--provider`, `gpu inventory` queries every authenticated cloud
(RunPod, Vast.ai, Thunder Compute, io.net) in parallel and renders a
comparison matrix. Rows are canonical GPU types; columns are providers;
cells show price + stock indicator.

```
GPU                    VRAM    RunPod          Vast.ai         io.net
─────────────────────────────────────────────────────────────────────
NVIDIA H100 80GB HBM3  80GB    $ 2.99 ███      $ 1.87 ██░      $ 1.85 ██░ *
NVIDIA H100 PCIe       80GB    $ 2.39 █░░      ── ─            ── ─
NVIDIA RTX 4090        24GB    $ 0.69 ███      $ 0.45 ███ ◆    $ 0.60 ██░

  * = best price in row     ◆ = best price + best stock     ── = not offered
```

A footer summary highlights the winner across all rows: most availability,
lowest average $/hr, and the intersection (or `split` when no single
provider wins both).

### Single-provider mode

Pass `--provider <name>` (`runpod`, `vastai`, `thunder`, `ionet`) for the
original detailed table, with every offer, datacenter, and stock indicator for
just that cloud. The flat JSON shape is preserved here for backward
compatibility with existing scripts.

### Filters

| Flag                  | Effect                                                                           |
| --------------------- | -------------------------------------------------------------------------------- |
| `--available` / `-a`  | Only show GPUs with stock                                                        |
| `--min-vram <GB>`     | Minimum VRAM                                                                     |
| `--max-price <USD>`   | Maximum hourly price                                                             |
| `--region <id>`       | Filter by datacenter region                                                      |
| `--gpu-type <name>`   | Fuzzy match on GPU name                                                          |
| `--cloud-type <type>` | RunPod-specific: `secure`, `community`, `all`                                    |
| `--show-datacenters`  | Append the winning datacenter under each cell                                    |
| `--show-unmapped`     | Include rows where the provider couldn't identify the GPU model (mainly Vast.ai) |

### Authentication

The matrix only includes providers you've authenticated with. Run
`gpu auth status` to see which are configured, and `gpu auth <provider>`
to add one. Providers that error during the fetch (rate limit, bad key,
network) render as an `ERR` column with the message in the footer. Other
providers still surface.

### JSON output

`--json` emits a structured object designed for scripts:

```json
{
  "providers": ["runpod", "vastai", "ionet"],
  "rows": [
    {
      "canonical_type": "h100_sxm",
      "display_name": "NVIDIA H100 80GB HBM3",
      "vram_gb": 80,
      "tier": "Flagship",
      "cells": {
        "runpod": { "price_per_hour": 2.99, "stock_level": "high",   "datacenter_id": "us-ca",   ... },
        "vastai": { "price_per_hour": 1.87, "stock_level": "medium", "datacenter_id": "us-west", ... }
      },
      "best_price_provider": "vastai",
      "best_stock_provider": "runpod"
    }
  ],
  "summary": {
    "most_availability": "runpod",
    "lowest_average_price": "vastai",
    "best_overall": null
  },
  "errors": {},
  "filters": { "min_vram": null, ... }
}
```

When `--provider <name>` is set, `--json` returns the legacy flat array
of GPU choices instead; the matrix shape is opt-out.

## gpu enroll

Register this Linux/WSL machine as a local-fleet beta node.

```bash
gpu fleet pair-node --name @james-5090
# then run the printed command on the GPU host:
gpu enroll --auth-key=tskey-... --name @james-5090
```

Local Fleet is a beta feature. Keep the trusted controller and every GPU host
on the current GPU CLI release while this flow stabilizes.

`gpu enroll` is intentionally target-side: run it inside the Linux/WSL
environment that owns the GPU. The portal never SSHes into your machine.
Use the full command printed by `gpu fleet pair-node`; it includes pairing
metadata that lets the node pin your fleet root locally. After enrollment, the
node uses its own credentials, portal JWT rotation, relay control, and WebRTC
data channels for local fleet jobs.

| Flag                 | Description                                                                      |
| -------------------- | -------------------------------------------------------------------------------- |
| `--auth-key <KEY>`   | One-time enrollment key printed by `gpu fleet pair-node`                         |
| `--portal-url <URL>` | Portal URL override for development or staging                                   |
| `--relay-url <URL>`  | Relay WebSocket control URL override                                             |
| `--name <NAME>`      | Fleet handle for this node, with or without leading `@`; alias for `--node-name` |

If the portal shows a node as needing pairing, it registered without a
controller-signed trust binding. Run `gpu fleet pair-node --name @<name>` from
a trusted controller, then run the printed `gpu enroll` command on the GPU host.

## gpu node

Manage beta nodes registered to your local fleet.

```bash
gpu node retire @james-5090
gpu node retire node-abc123def456 --no-wait
```

`retire` revokes the portal-side node record. The agent treats a 401
heartbeat as a revocation signal and exits on its next heartbeat cycle.

| Subcommand        | Description                                       |
| ----------------- | ------------------------------------------------- |
| `retire <TARGET>` | Revoke a node by `@name` or literal `node-...` id |

## gpu fleet

Manage local fleet beta trust and controller pairing.

```bash
gpu fleet init
gpu fleet request-access --device-name mbp
gpu fleet approve-device ctrl_abc123
gpu fleet pair-node --name @james-5090
```

Browser sessions can store pending requests, but approval is signed by a
trusted local controller key. The portal cannot mint a controller grant on
its own.

| Subcommand                | Description                                                    |
| ------------------------- | -------------------------------------------------------------- |
| `init`                    | Create the fleet root and approve this machine as a controller |
| `request-access`          | Request controller access from another machine                 |
| `approve-device <CODE>`   | Approve a pending controller request locally                   |
| `pair-node --name <NAME>` | Create a pairing session and enrollment token for `@<name>`    |

## gpu placement

Explain how the cross-provider placement decider would rank candidates
for your project, without creating a pod.

```bash
gpu placement explain                                  # Human table, current project config
gpu placement explain --json                           # Machine-readable output
gpu placement explain --providers runpod,vastai        # Limit to an allow-list
gpu placement explain --exclude-provider ionet         # Repeatable exclusion
gpu placement explain --min-vram 40 --max-price 2.50   # Extra hard filters
gpu placement explain --gpu-type H100 --gpu-type A100  # GPU-type filter (repeatable)
```

Read-only diagnostic for `provider = "auto"` / `providers = [...]` configs.
The decider runs against live inventory plus the project's deployment
footprint (volume-affinity, cache-affinity, per-provider session count)
and surfaces both the ranked candidate list and any hard-filter
rejections.

### Human output

A ranked table with one row per `(provider, gpu, datacenter)` candidate:

```
Rank  Provider  GPU                         DC        $/hr   Stock  Score   Reasons
──────────────────────────────────────────────────────────────────────────────────────
 1    vastai    NVIDIA H100 80GB HBM3       us-west   1.87   ██░    0.812   price-win, stock-boost
 2    runpod    NVIDIA H100 80GB HBM3       us-ca     2.99   ███    0.741   stock-boost
 3    ionet     NVIDIA H100 PCIe            sfo-1     1.85   ██░    0.698   price-win

Rejections (hard filter):
  - runpod / NVIDIA A100: volume_locked (volume datacenter us-ca, candidate in us-east)
  - ionet  / NVIDIA L40S: price_cap (2.89 > 2.50 max_price)
```

### Flags

| Flag                               | Effect                                                                                           |
| ---------------------------------- | ------------------------------------------------------------------------------------------------ |
| `--providers <csv>`                | Ordered allow-list; empty = all authenticated                                                    |
| `--exclude-provider <name>` / `-x` | Exclude a provider (repeatable)                                                                  |
| `--gpu-type <name>` / `-g`         | Filter to these GPU types (repeatable)                                                           |
| `--min-vram <GB>`                  | Hard VRAM floor                                                                                  |
| `--max-price <USD>`                | Hard price ceiling per hour                                                                      |
| `--json`                           | Machine output: `{ candidates: [...], rejections: [...], inventory_fetched_at_unix_ms: number }` |

### When the decider runs for real

`gpu placement explain` is purely diagnostic. The same decider dispatches
automatically during `gpu start` / `gpu run` whenever you haven't pinned a
specific provider, which is the default. Pin a provider (e.g. `provider = "runpod"`) to bypass the decider. See
[`crates/gpu/docs/providers/placement.md`](https://github.com/gpu-cli/platform/blob/main/crates/gpu/docs/providers/placement.md)
for the full scoring model and how to tune `PlacementWeights`.

## gpu status

Show project status, active pods, recent jobs, and cost information.

```bash
gpu status
gpu status --json
```

## gpu dashboard

Open the TUI dashboard for pods and jobs.

```bash
gpu dashboard
```

Keybindings: `j`/`k` to move, `Tab` to switch panels, `Enter` to expand, `a` to attach, `c` to cancel, `s` to stop, `e` for events, `?` for help, `q` to quit.

## gpu logs

Stream or inspect unified logs for jobs, sync events, hooks, and agent output.

```bash
gpu logs
gpu logs --job job_abc123 --tail 50
gpu logs --follow --type lifecycle
gpu logs --json
```

## gpu last

Render the most recent run from the run-history index (mesh plane). With `--failed`, shows the most recent run that failed, failure-first (kind, cause, attempts, last node, traceback ref).

```bash
gpu last             # newest run summary
gpu last --failed    # newest failed run, failure detail
```

| Flag       | Description                                         |
| ---------- | --------------------------------------------------- |
| `--failed` | Show the most recent failed run with failure detail |
| `--json`   | Output JSON                                         |

## gpu wait

Block until a job reaches a terminal state, then exit with the job's own exit code. Pairs with `gpu run -d`: submit detached, then wait on the printed job ID instead of polling logs.

```bash
gpu wait job_abc123
gpu wait job_abc123 --timeout 3600
gpu wait job_abc123 --json
```

| Flag               | Description                                                                   |
| ------------------ | ----------------------------------------------------------------------------- |
| `--timeout <SECS>` | Give up after this many seconds (exit code 14); waits indefinitely without it |
| `--json`           | Emit the final state as JSON: `{"job_id", "state", "exit_code"}`              |

Exit codes: the job's exit code on completion, `12` when the job is unknown, `13` when the job exists but its state can no longer be followed, `14` on `--timeout` expiry.

## gpu rent

Rent GPU capacity through the mesh rentals resource — a thin client over the daemon's `/v1/rentals` surface: list offers, create a rental, inspect status, release. A freshly-ready rental enrolls as a node and starts matching queued constraints.

```bash
gpu rent offers --constraint 'vram>=48'
gpu rent create of-123 --keep-alive 30 --max-spend 12.50
gpu rent list
gpu rent status r-01
gpu rent release r-01
```

| Subcommand            | Description                                               |
| --------------------- | --------------------------------------------------------- |
| `offers`              | List available rental offers (filter with `--constraint`) |
| `create <OFFER_ID>`   | Create a rental (`--keep-alive`, `--max-spend`)           |
| `list`                | List active rentals                                       |
| `status <RENTAL_ID>`  | Inspect a rental's status                                 |
| `release <RENTAL_ID>` | Release a rental                                          |

## gpu stop

Stop the active pod immediately.

```bash
gpu stop
gpu stop pod_abc123 --no-sync
```

| Flag                 | Description            |
| -------------------- | ---------------------- |
| `[POD_ID]`           | Optional pod ID        |
| `--yes, -y, --force` | Skip confirmation      |
| `--no-sync`          | Skip final output sync |
| `--json`             | Output JSON            |

## gpu use

Run a template directly or resume a template session.

```bash
gpu use ollama
gpu use vllm
gpu use owner/repo@v1.0.0
gpu use
```

### Flags

| Flag                  | Description                          |
| --------------------- | ------------------------------------ |
| `--name <NAME>`       | Override the session or project name |
| `--yes`               | Skip interactive prompts             |
| `--dry-run`           | Show what would be created           |
| `--input <KEY=VALUE>` | Provide template input values        |

See [LLM Inference](/docs/llm) for the built-in Ollama, vLLM, and llama.cpp flows.

## gpu llm

Launch the routed LLM workflow for Ollama, vLLM, or llama.cpp, with either local proxy registration or hosted publish.

```bash
gpu llm run
gpu llm run --ollama --model deepseek-r1:8b -y
gpu llm run --vllm --url meta-llama/Llama-3.1-8B-Instruct -y
gpu llm run --vllm --model Qwen/Qwen2.5-0.5B-Instruct --publish hosted --name staging-qwen -y
gpu llm info deepseek-r1:70b
gpu llm info --url meta-llama/Llama-3.1-8B-Instruct --json
```

Use [LLM Inference](/docs/llm) for setup, port layout, and wake-on-request routing behavior.

## gpu comfyui

Run ComfyUI workflows from the curated workflow catalog.

```bash
gpu comfyui list
gpu comfyui info flux_schnell
gpu comfyui validate flux_schnell --gpu-type "RTX 4090"
gpu comfyui run flux_schnell --volume-id vol_abc123
gpu comfyui generate "a cat astronaut on mars"
gpu comfyui stop --all
```

## gpu notebook

Run Marimo notebooks on GPU pods.

```bash
gpu notebook
gpu notebook train.py
gpu notebook train.py --run
gpu notebook --new analysis
```

### Common flags

| Flag                | Description                   |
| ------------------- | ----------------------------- |
| `--run, -r`         | Serve in read-only app mode   |
| `--new <NAME>`      | Create a new notebook         |
| `--gpu-type <TYPE>` | Override GPU selection        |
| `--volume-id <ID>`  | Use a specific network volume |
| `--no-volume`       | Use ephemeral storage         |
| `--yes, -y`         | Skip confirmation             |

## gpu volume

Manage network volumes.

### volume list

```bash
gpu volume list
gpu volume list --detailed
gpu volume list --json
```

### volume create

```bash
gpu volume create --name my-models --size 200
gpu volume create --datacenter US-TX-3 --set-global
```

### volume delete

```bash
gpu volume delete my-models --force
```

### volume extend

```bash
gpu volume extend my-models --size 300
```

### volume set-global

```bash
gpu volume set-global my-models
```

### volume status

```bash
gpu volume status
gpu volume status --volume my-models --json
```

### volume migrate

```bash
gpu volume migrate my-models --to US-OR-1
```

### volume sync

```bash
gpu volume sync source-volume dest-volume
gpu volume sync source-volume dest-volume --method rsync
```

### volume cancel-migration

```bash
gpu volume cancel-migration transfer_abc123
```

## gpu vault

Manage encrypted vault storage for sensitive outputs.

### vault list

```bash
gpu vault list
gpu vault list --project my-project --json
```

### vault export

```bash
gpu vault export checkpoints/model.pt ./model.pt
gpu vault export generated/image.png ~/Desktop/ --force
```

### vault stats

```bash
gpu vault stats
gpu vault stats --project my-project --json
```

## gpu proxy

Manage the local proxy router for OpenAI-compatible model access.

### proxy start

```bash
gpu proxy start
```

Starts the local proxy listeners. By default the LLM proxy listens on `127.0.0.1:4000` and the diffusion proxy listens on `127.0.0.1:4001`.

### proxy stop

```bash
gpu proxy stop
```

Stops all proxy listeners and health checks.

### proxy status

```bash
gpu proxy status
gpu proxy status --json
```

Shows whether the proxy is running, the configured listen addresses, and registered model counts.

### proxy models

```bash
gpu proxy models
gpu proxy models --json
```

Lists registered models, backend counts, service types, and healthy backend counts.

### proxy register

```bash
gpu proxy register --pod-id llm-ollama-qwen3-5-9b --model qwen3.5:9b --port 11434 --type ollama
gpu proxy register --pod-id abc123 --model meta-llama/Llama-3-8B --port 8000 --type vllm
```

Manually registers a backend when you need to attach an existing service to the local proxy router.

### proxy deregister

```bash
gpu proxy deregister --pod-id llm-ollama-qwen3-5-9b --model qwen3.5:9b
gpu proxy deregister --pod-id abc123 --model meta-llama/Llama-3-8B
```

Removes a backend from the proxy router.

## gpu proxy hosted

Manage the org-scoped hosted proxy for the active organization.

### proxy create

```bash
gpu proxy create
gpu proxy create --backend runpod
gpu proxy create --backend vastai
```

Creates the org-scoped hosted proxy deployment. Use `--backend` to pin a provider-hosted backend when you do not want the portal-managed Railway path.

### proxy hosted status

```bash
gpu proxy hosted status
gpu proxy hosted status --json
```

Shows hosted deployment status, the public connection URL, and version drift details.

### proxy hosted connect-info

```bash
gpu proxy hosted connect-info
gpu proxy hosted connect-info --json
```

Prints the connection details for the active org's hosted proxy.

### proxy hosted upgrade

```bash
gpu proxy hosted upgrade --yes
```

Requests a hosted-proxy upgrade when drift is detected. Railway remains portal-managed end to end, while provider-hosted backends reconcile the requested `gpu-cli` version through the running hosted daemon after initial provisioning.

### proxy hosted destroy

```bash
gpu proxy hosted destroy
```

Destroys the org-scoped hosted proxy deployment.

### proxy keys

```bash
gpu proxy keys create --name "alice"
gpu proxy keys list
gpu proxy keys revoke --name "alice"
```

Manage org-scoped hosted-proxy API keys for the active organization.

## gpu config

Inspect or validate configuration.

```bash
gpu config show
gpu config validate
gpu config schema
gpu config get default_profile
gpu config set updates.auto_update false
```

## gpu template

Browse official templates or clear the local template cache.

```bash
gpu template list
gpu template list --json
gpu template clear-cache -y
```

## gpu org

Manage organizations, membership, sub-accounts, and service accounts.

```bash
gpu org list
gpu org create "My Team"
gpu org delete my-team
gpu org switch my-team
gpu org invite alice@example.com --role admin
gpu org service-account create --name ci
gpu org service-account revoke sa_abc123
```

See [Organizations & Service Accounts](/docs/organizations) for the current model, including headless automation.

## gpu serverless

Deploy and manage RunPod Serverless endpoints.

```bash
gpu serverless deploy
gpu serverless list --json
gpu serverless status <ENDPOINT_ID>
gpu serverless delete
gpu serverless warm <ENDPOINT_ID> --gpu
gpu serverless logs <ENDPOINT_ID>
gpu serverless template delete
```

### serverless status

```bash
gpu serverless status <ENDPOINT_ID>
gpu serverless status <ENDPOINT_ID> --json
```

### serverless logs

```bash
gpu serverless logs <ENDPOINT_ID>
gpu serverless logs <ENDPOINT_ID> --tail 100
gpu serverless logs <ENDPOINT_ID> --status error
```

`gpu serverless logs` currently points you to the RunPod dashboard rather than streaming or filtering logs inside the CLI, but these are still the public CLI flags.

### serverless warm

```bash
gpu serverless warm <ENDPOINT_ID> --gpu
gpu serverless warm <ENDPOINT_ID> --cpu
gpu serverless warm <ENDPOINT_ID> --timeout 900
```

### Current behavior

* `gpu serverless status` and `gpu serverless warm` should be treated as endpoint-ID-driven commands.
* `gpu serverless delete` supports name lookup and interactive selection when you omit the endpoint.
* `gpu serverless warm --gpu` is the supported warmup path today.
* `gpu serverless warm --cpu`, deploy-time `--warm`, and deploy-time `--write-ids` are not wired through the current runtime.
* `gpu serverless logs` currently points you to the RunPod dashboard rather than streaming filtered logs in the CLI.

Use [Serverless Endpoints](/docs/serverless) for templates, config shape, and request examples.

## gpu start

Start a new GPU pod explicitly instead of letting `gpu run` manage pod lifecycle for you.

```bash
gpu start
gpu start --gpu-type "NVIDIA RTX 6000 Ada"
gpu start --min-vram 24 --max-price 1.50
```

### Key flags

| Flag                     | Description                            |
| ------------------------ | -------------------------------------- |
| `--gpu-type <TYPE>`      | Pin a specific GPU type                |
| `--gpu-count <N>`        | Request multiple GPUs                  |
| `--max-price <PRICE>`    | Limit hourly spend during selection    |
| `--min-vram <GB>`        | Set a VRAM floor                       |
| `--cloud-type <TYPE>`    | Choose `secure`, `community`, or `all` |
| `--region <REGION>`      | Restrict to a region or datacenter     |
| `--docker-image <IMAGE>` | Override the image                     |
| `--force`                | Skip reuse checks and create a new pod |

## gpu instances

Inspect or terminate provider-side instances directly.

```bash
gpu instances list
gpu instances list --project my-project
gpu instances show pod_abc123
gpu instances terminate pod_abc123 -y
```

## gpu rename

Rename an instance's memorable name. Works for both rentals and owned-fleet
nodes; the leading `@` is optional, and running jobs are unaffected. The pod
(or node) id is stable, only the human-readable name changes.

```bash
gpu rename bold-falcon h100-bench    # rename by bare names
gpu rename @bold-falcon @h100-bench  # leading @ is optional
```

## gpu desktop

Manage the desktop app.

```bash
gpu desktop install
gpu desktop install --channel beta
gpu desktop uninstall -y
```

## gpu update

Update GPU CLI to the latest version or inspect update availability.

```bash
gpu update
gpu update --check
gpu update --target-version 1.2.3
gpu update --dismiss
```

## gpu changelog

View release notes by version or version range.

```bash
gpu changelog
gpu changelog 0.8.0
gpu changelog --from 0.7.0 --to 0.8.0
```

## gpu daemon

Manage the background daemon.

```bash
gpu daemon status
gpu daemon start
gpu daemon restart
gpu daemon logs --follow
```

## gpu doctor

Run a setup diagnostic.

```bash
gpu doctor
gpu doctor --json
```

## gpu issue

Submit an issue report with optional daemon logs.

```bash
gpu issue
gpu issue "sync not working"
gpu issue --no-logs -y "bug report"
```

## gpu agent-docs

Print the agent-focused CLI reference to stdout.

```bash
gpu agent-docs
gpu agent-docs | head -50
```

## gpu support

Open the GPU CLI community support link in your browser.

```bash
gpu support
```

## Global flags

These flags work with all commands:

| Flag                       | Description                                                       |
| -------------------------- | ----------------------------------------------------------------- |
| `-v, --verbose`            | Increase logging verbosity (`-v`, `-vv`, `-vvv`)                  |
| `-q, --quiet`              | Minimal output                                                    |
| `--progress-style <STYLE>` | Progress display style: `panel`, `pipeline`, `minimal`, `verbose` |
| `--help-all`               | Show all commands including hidden ones                           |
| `--no-auto-update`         | Disable the update check for this invocation                      |


---

# Configuration



# Configuration

GPU CLI uses a `gpu.jsonc` file in your project root. `gpu init` creates a starting config, and the checked-in JSON schema powers autocomplete and validation.

## JSON Schema

```json
{
  "$schema": "https://gpu-cli.sh/schema/v1/gpu.json"
}
```

## Core Project Settings

### project\_id, provider, profile

```json
{
  "project_id": "my-ml-project",
  "provider": "runpod",
  "profile": "default"
}
```

* `project_id` stabilizes the project identity across machines.
* `provider` chooses the provider implementation.
* `profile` selects credential isolation for auth and SSH keys.

### gpu\_types

Specify preferred GPU types with optional counts:

```json
{
  "gpu_types": [
    { "type": "RTX 4090" },
    { "type": "A100", "count": 4 }
  ]
}
```

If omitted, GPU CLI falls back to intelligent selection.

### min\_vram, max\_price, regions, cloud\_type

```json
{
  "min_vram": 24,
  "max_price": 1.5,
  "regions": ["US-TX-1", "US-CA-1"],
  "cloud_type": "secure"
}
```

Use these to constrain fallback GPU selection.

### encryption

`encryption` controls volume encryption-at-rest behavior for the selected storage strategy.

```json
{
  "encryption": true
}
```

* On RunPod built-in storage, this uses the provider-native encrypted volume behavior.
* On providers that support GPU CLI-managed LUKS, this flag enables that path.
* This setting does not mean every provider uses the same zero-trust storage model, so keep the provider-specific security model in mind.

### storage\_mode

`storage_mode` chooses the primary storage strategy for the workspace.

```json
{
  "storage_mode": "built-in"
}
```

Valid values:

* `built-in`: use the provider's default workspace storage
* `network`: prefer an attached network volume
* `managed`: use provider-managed persistent storage when available

Use this with `network_volume_id` or `volume_mode` when you want explicit network-volume behavior.

## Sync and Output Control

### include

`include` force-includes gitignored files when syncing to the pod.

```json
{
  "include": ["data/weights.bin", "models/", "*.onnx"]
}
```

### outputs and exclude\_outputs

`outputs` controls what syncs back from the pod. `exclude_outputs` removes paths from that set.

```json
{
  "outputs": ["outputs/", "checkpoints/", "*.safetensors"],
  "exclude_outputs": ["*.tmp", "*.log"]
}
```

### outputs\_enabled, outputs\_root, outputs\_backfill\_seconds

```json
{
  "outputs_enabled": true,
  "outputs_root": "outputs",
  "outputs_backfill_seconds": 3600
}
```

* `outputs_enabled` disables daemon-managed output watching entirely.
* `outputs_root` scopes the watch subtree.
* `outputs_backfill_seconds` limits how far back the daemon rescans after a restart.

### extra\_outputs

`extra_outputs` lets you sync files from absolute paths outside the workspace on the pod.

```json
{
  "extra_outputs": [
    {
      "remote": "/gpu-cli-workspaces/cache/checkpoints",
      "local": "checkpoints/"
    }
  ]
}
```

### vault

Use `vault` for outputs that should stay encrypted at rest on your local machine instead of syncing into the workspace.

```json
{
  "outputs": ["logs/**"],
  "vault": {
    "patterns": ["checkpoints/**", "generated/**"]
  }
}
```

## Downloads

GPU CLI can pre-stage models, repos, and assets before your command runs.

### HuggingFace

```json
{
  "download": [
    {
      "strategy": "hf",
      "source": "black-forest-labs/FLUX.1-dev"
    }
  ]
}
```

### HTTP

```json
{
  "download": [
    {
      "strategy": "http",
      "source": "https://example.com/model.bin",
      "target": "models/model.bin"
    }
  ]
}
```

### Git

```json
{
  "download": [
    {
      "strategy": "git",
      "source": "https://github.com/comfyanonymous/ComfyUI",
      "target": "ComfyUI",
      "tag": "v0.3.7"
    }
  ]
}
```

### Civitai

```json
{
  "download": [
    {
      "strategy": "civitai",
      "source": "4384"
    }
  ]
}
```

For persistent model storage, prefer `${workspace_base}` in targets so the path stays aligned with the provider's mounted workspace.

## Pod Runtime Settings

### keep\_alive\_minutes

Cooldown in minutes after the last meaningful activity.

```json
{
  "keep_alive_minutes": 10
}
```

### persistent\_proxy, resume\_timeout\_secs, max\_queued\_requests, health\_check\_paths

These settings control wake-on-request behavior for long-lived services.

```json
{
  "persistent_proxy": true,
  "resume_timeout_secs": 180,
  "max_queued_requests": 100,
  "health_check_paths": ["/health", "/ready"]
}
```

* `persistent_proxy`: keep the local proxy listening after the pod stops so incoming requests can resume it.
* `resume_timeout_secs`: how long to wait for the pod to come back.
* `max_queued_requests`: cap queued requests while the pod is resuming.
* `health_check_paths`: paths that should never count as meaningful activity.

### docker\_image, dockerfile, workspace\_size\_gb, target\_architecture

```json
{
  "docker_image": "ghcr.io/gpu-cli/base:latest",
  "dockerfile": "Dockerfile",
  "workspace_size_gb": 50,
  "target_architecture": "linux/amd64"
}
```

Use `target_architecture` to override the provider default when you need ARM builds.

### startup

`startup` sets the default command for `gpu run` when you do not pass an explicit command. It is also the main entrypoint for many template workflows.

```json
{
  "startup": "python app.py --host 0.0.0.0 --port 8080"
}
```

### inputs

`inputs` defines the prompts shown when someone runs `gpu use <template>`.

```json
{
  "inputs": [
    {
      "type": "select",
      "key": "model_variant",
      "label": "Model Variant",
      "options": [
        { "value": "small", "label": "Small" },
        { "value": "large", "label": "Large" }
      ],
      "default": "small",
      "required": true
    },
    {
      "type": "text",
      "key": "system_prompt",
      "label": "System Prompt",
      "placeholder": "You are a helpful assistant",
      "required": false
    },
    {
      "type": "boolean",
      "key": "enable_auth",
      "label": "Enable Auth",
      "default": true
    }
  ]
}
```

* Use `key` names that match the values your template expects to substitute.
* `type` controls the prompt UI. Common choices are `select`, `text`, `number`, `boolean`, `model`, and `secret`.
* Callers can answer interactively or prefill values with `gpu use --input key=value`.

## Ports and Activity Routing

`ports` supports both simple port numbers and rich port objects.

### Simple ports

```json
{
  "ports": [8000, 8080]
}
```

### Rich ports

```json
{
  "ports": [
    {
      "port": 8080,
      "description": "ui",
      "http": {
        "activity_paths": ["/api/chat", "/api/generate"],
        "ignore_paths": ["/health", "/queue"],
        "ignore_methods": ["OPTIONS", "HEAD"]
      },
      "websocket": {
        "data_frames_are_activity": true,
        "ping_pong_is_activity": false
      }
    }
  ]
}
```

### How activity routing works

* If a port has no HTTP rules, all HTTP requests count as activity.
* `ignore_methods` and `ignore_paths` block known background traffic.
* If `activity_paths` is set, only those paths reset the cooldown timer.
* WebSocket data frames keep a connection warm, but the primary cooldown signal is still HTTP activity.

Use this for apps that poll frequently, such as queues, health checks, chat UIs, or dashboard frontends.

## Hooks

Hooks run scripts during pod lifecycle events.

```json
{
  "hooks": {
    "readiness": {
      "type": "command",
      "name": "service-ready",
      "run": ["curl", "-sf", "http://localhost:8000/health"],
      "retry_count": 30,
      "retry_delay_secs": 2,
      "timeout_secs": 10
    }
  }
}
```

The most common use is a readiness hook that waits for a service to become available before GPU CLI marks the pod ready.

## Network Volumes

### network\_volume\_id

Attach a specific network volume by ID or unique friendly name.

```json
{
  "network_volume_id": "vol_abc123xyz"
}
```

### volume\_mode

```json
{
  "volume_mode": "global"
}
```

Options:

* `"global"`: use the shared global volume
* `"dedicated"`: use a project-specific volume
* `"none"`: use ephemeral storage only

### dedicated\_volume\_id and dedicated\_volume\_name

```json
{
  "volume_mode": "dedicated",
  "dedicated_volume_name": "my-project-models"
}
```

`dedicated_volume_id` wins if both are set.

## Environment

```json
{
  "environment": {
    "python": {
      "requirements": "requirements.txt"
    },
    "system": {
      "apt": [{ "name": "ffmpeg" }]
    },
    "shell": {
      "steps": [{ "run": "pip install -e ." }]
    }
  }
}
```

Use this to declare Python packages, system packages, and shell setup steps.

## Serverless

The `serverless` block configures RunPod Serverless deployment. For the workflow and request examples, see [Serverless Endpoints](/docs/serverless).

### Main fields

```json
{
  "serverless": {
    "template": "vllm",
    "gpu_type": "NVIDIA A100 80GB PCIe",
    "gpu_types": ["NVIDIA L4"],
    "scaling": {
      "min_workers": 0,
      "max_workers": 3,
      "idle_timeout": 5
    },
    "volume": {
      "name": "my-project-vol",
      "size_gb": 200,
      "mount_path": "/runpod-volume"
    },
    "runpod": {
      "flashboot": true,
      "cached_model": "meta-llama/Llama-3.1-8B-Instruct",
      "env": {
        "MODEL_NAME": "meta-llama/Llama-3.1-8B-Instruct"
      }
    }
  }
}
```

### Current caveats

* `serverless.prewarm` exists in the schema but is not wired in the current runtime.
* Deploy-time `--warm` and `--write-ids` are accepted by the CLI but not wired through today.
* `serverless.runpod.ids` can exist in config, but GPU CLI does not currently auto-populate it during deploy.

## Full Schema

See the complete checked-in schema at [gpu-cli.sh/schema/v1/gpu.json](https://gpu-cli.sh/schema/v1/gpu.json).


---

# Hosted Proxy Quickstart



# Hosted Proxy Quickstart

This is the shortest happy path for getting an org-scoped hosted proxy online.

Use this guide when you want:

* one shared OpenAI-compatible base URL for your team
* API keys that look like `sk-gpu-*`
* a portal-managed deployment instead of a laptop-local proxy

This flow assumes you are the org owner or an org admin.

## What You Need

* GPU CLI installed
* `gpu login` completed
* permission to create or manage an organization

If you have not created an organization yet, do that first.

## Supported Client APIs

The hosted proxy gives your org one shared public URL and supports these client-facing APIs:

* OpenAI `GET /v1/models`
* OpenAI `POST /v1/chat/completions`
* OpenAI `POST /v1/responses`
* Anthropic `POST /v1/messages`

Authentication is accepted as either:

* `Authorization: Bearer sk-gpu-...`
* `x-api-key: sk-gpu-...`

That means the same hosted proxy can work with OpenAI-style clients and Anthropic-style clients.

Current Responses support is a subset:

* background responses are not supported
* multimodal inputs are not supported
* unsupported `tool_choice` values are rejected

## 1. Create or Switch to the Organization

Create a new org:

```bash
gpu org create "My Team"
```

Or switch to an existing org:

```bash
gpu org switch my-team
```

All hosted-proxy commands act on the active org context.

## 2. Create the Hosted Proxy

```bash
gpu proxy create
```

This asks the portal control plane to provision a shared hosted proxy for the active organization.

## 3. Wait Until It Is Ready

Check deployment status:

```bash
gpu proxy hosted status
```

Once it reports `ready`, print the connection details:

```bash
gpu proxy hosted connect-info
```

You should end up with an HTTPS base URL that looks like:

```bash
https://<your-org-hosted-proxy>/v1
```

## 4. Create a Hosted-Proxy API Key

Generate an org-scoped hosted-proxy API key for yourself or a teammate:

```bash
gpu proxy keys create --name "alice"
```

The CLI prints the new `sk-gpu-*` key once. Copy it into your password manager or secret store right away.

## 5. Point a Client at the Hosted Proxy

Export the URL from `connect-info` and the key you just created:

```bash
export OPENAI_BASE_URL=https://<your-org-hosted-proxy>/v1
export OPENAI_API_KEY=sk-gpu-xxxxxxxx
```

### OpenAI compatibility

Quick connectivity check:

```bash
curl -sS -i \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  "$OPENAI_BASE_URL/models"
```

If your org has not attached any models yet, the request can still succeed with an empty model list.

OpenAI Chat Completions example:

```bash
curl -sS \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  "$OPENAI_BASE_URL/chat/completions" \
  -d '{
    "model": "qwen3.5:9b",
    "messages": [{"role": "user", "content": "Say hello"}]
  }'
```

OpenAI Responses example:

```bash
curl -sS \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  "$OPENAI_BASE_URL/responses" \
  -d '{
    "model": "qwen3.5:9b",
    "input": "Say hello"
  }'
```

### Anthropic compatibility

Anthropic-style clients can point at the same deployment:

```bash
export ANTHROPIC_BASE_URL=https://<your-org-hosted-proxy>/v1
export ANTHROPIC_API_KEY=sk-gpu-xxxxxxxx
```

Anthropic Messages example:

```bash
curl -sS \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  "$ANTHROPIC_BASE_URL/messages" \
  -d '{
    "model": "qwen3.5:9b",
    "max_tokens": 128,
    "messages": [{"role": "user", "content": "Say hello"}]
  }'
```

## 6. Share With the Org

For each teammate or automation identity in the org:

* create a separate API key with `gpu proxy keys create`
* give them the shared `OPENAI_BASE_URL`
* give them only their own `sk-gpu-*` key

That keeps access revocable per user or workflow.

## Common Follow-Ups

Check current deployment state:

```bash
gpu proxy hosted status --json
```

List existing keys:

```bash
gpu proxy keys list
```

Revoke a key:

```bash
gpu proxy keys revoke --name "alice"
```

If you are using a Railway-backed hosted proxy, upgrade the hosted daemon to your installed GPU CLI version:

```bash
gpu proxy hosted upgrade --yes
```

Provider-hosted backends do not support in-place upgrades today. Reprovision them instead:

```bash
gpu proxy hosted destroy
gpu proxy create --backend runpod
```

Destroy the hosted proxy when you no longer need it:

```bash
gpu proxy hosted destroy
```

## Related Docs

* [Organizations & Service Accounts](/docs/organizations)
* [Commands Reference](/docs/commands)
* [Troubleshooting](/docs/troubleshooting)


---

# GPU CLI Documentation



# GPU CLI

GPU CLI makes remote GPU execution feel like local development. Just prefix any command with `gpu run` to execute it on a cloud GPU.

```bash
# Local development
uv run python train.py

# Remote GPU: just add 'gpu run'
gpu run uv run python train.py
```

## Quick Install

```bash
curl -fsSL https://gpu-cli.sh/install.sh | sh
```

## Key Features

* **Fast Connection**: Connect in under 6 seconds via relay (no waiting for public IPs)
* **Auto-Stop**: Pods automatically stop after 5 minutes of idle time
* **Smart Sync**: Files sync using `.gitignore` rules, outputs stream back in real-time
* **BYOK**: Bring your own API keys. You pay your cloud provider directly, no markup

## How Billing Works

GPU CLI is an orchestration layer. You connect your own RunPod (or other provider) API key and pay them directly at their standard rates. GPU CLI never touches your billing.

<Cards>
  <Card title="Quickstart" href="/docs/quickstart" />

  <Card title="LLM Inference" href="/docs/llm" />

  <Card title="Configuration" href="/docs/configuration" />

  <Card title="Hosted Proxy Quickstart" href="/docs/hosted-proxy" />

  <Card title="Commands" href="/docs/commands" />

  <Card title="Organizations & Service Accounts" href="/docs/organizations" />

  <Card title="AI Agent Workflows" href="/docs/ai-agent-skill" />

  <Card title="Troubleshooting" href="/docs/troubleshooting" />
</Cards>


---

# LLM Inference



# LLM Inference

GPU CLI ships a pod-based LLM workflow for Ollama, vLLM, and llama.cpp.

Use it when you want:

* a local Web UI backed by a remote GPU pod
* a local forwarded API port
* an org-scoped hosted model deployment on an existing hosted proxy
* wake-on-request behavior after the pod cools down
* a reusable template/session flow without building a serverless endpoint

If you want a deployed RunPod endpoint instead, use [Serverless Endpoints](/docs/serverless).

## Choose a Workflow

### `gpu llm run`

The guided path. It launches the LLM wizard, writes a local project directory, and starts the pod-backed service.

```bash
gpu llm run
```

Use `--publish local` to register the launched service with your local proxy router, or `--publish hosted --name <NAME>` to create an org-scoped hosted model deployment through the active hosted proxy.

### `gpu use ollama` / `gpu use vllm`

The direct template path. Use this when you want to work with the official templates yourself.

```bash
gpu use ollama
gpu use vllm
```

## Quick Start

### Interactive wizard

```bash
gpu llm run
```

The wizard walks you through:

* choosing Ollama or vLLM
* selecting a model
* reviewing the generated template files and launch settings

### Direct launch

Skip the wizard when you already know the engine and model.

```bash
gpu llm run --ollama --model deepseek-r1:8b -y
gpu llm run --vllm --url meta-llama/Llama-3.1-8B-Instruct -y
gpu llm run --vllm --model Qwen/Qwen2.5-0.5B-Instruct --publish hosted --name staging-qwen -y
```

Hosted publish currently creates one hosted deployment per model. If you want multiple hosted models, run the command once per model name.

### Compatibility aliases

The preferred publish surface is:

* `--publish local`
* `--publish hosted --name <NAME>`

The older hidden flags remain as temporary compatibility aliases through `0.20.x` only:

* `--register-proxy` -> `--publish local`
* `--deployment-name <NAME>` -> `--publish hosted --name <NAME>`

Use the target-based flags in scripts and operator runbooks. The aliases are scheduled for removal in `0.21.0`.

### Model lookup

```bash
gpu llm info deepseek-r1:70b
gpu llm info --url meta-llama/Llama-3.1-8B-Instruct
gpu llm info --url meta-llama/Llama-3.1-8B-Instruct --json
```

## Engine Differences

| Engine | Best for                               | API surface                                         | Notes                                          |
| ------ | -------------------------------------- | --------------------------------------------------- | ---------------------------------------------- |
| Ollama | multi-model experimentation            | Ollama API plus OpenAI-compatible `/v1/*` endpoints | supports multiple pulled models in one session |
| vLLM   | single-model high-throughput inference | OpenAI-compatible API                               | optimized for one loaded model at a time       |

## Port Layout

### Ollama

| Port    | Purpose                              |
| ------- | ------------------------------------ |
| `8080`  | Web UI                               |
| `11434` | Ollama API and OpenAI-compatible API |

### vLLM

| Port   | Purpose               |
| ------ | --------------------- |
| `8080` | Web UI                |
| `8000` | OpenAI-compatible API |

For local runs, these ports are forwarded to localhost, so your client code talks to local URLs while the model runs on the remote pod.

Hosted publishes use the active hosted proxy's public `/v1` base URL instead of a localhost port.

## Generated Files and Session State

The LLM workflow generates a local project directory such as `llm-ollama` or `llm-vllm` and stores template session state in `.gpu/template.json`.

That gives you:

* a reusable generated template
* normal `gpu use` resume behavior
* persistent project-local config and startup files

## Wake-on-Request and Persistent Proxy

By default, GPU CLI keeps the local proxy listening after the pod stops. When a new request arrives, GPU CLI resumes the pod and shows a loading page while the service comes back.

### Default behavior

* pod cools down after `keep_alive_minutes`
* local forwarded port stays bound
* a new request resumes the pod
* requests in the normal stopped/resuming path get a loading page and should retry while the pod comes back

### Disable it

For one-off runs, you can opt out:

```bash
gpu run --no-persistent-proxy python app.py
```

Or in config:

```json
{
  "persistent_proxy": false
}
```

## Activity Routing for Long-Lived Apps

Polling-heavy apps can stay awake forever if every request counts as activity. Use rich `ports` rules to define which requests reset the cooldown timer.

```json
{
  "keep_alive_minutes": 20,
  "persistent_proxy": true,
  "ports": [
    {
      "port": 8080,
      "description": "ui",
      "http": {
        "activity_paths": ["/api/chat", "/api/generate"],
        "ignore_paths": ["/health", "/queue", "/metrics"],
        "ignore_methods": ["OPTIONS", "HEAD"]
      },
      "websocket": {
        "data_frames_are_activity": true,
        "ping_pong_is_activity": false
      }
    }
  ]
}
```

### Rules of thumb

* Use `activity_paths` for the requests that represent real user work.
* Put health checks, queue polling, and metrics in `ignore_paths`.
* Leave WebSocket ping/pong as non-activity.
* Keep `health_check_paths` aligned with any service-level health routes.

## Storage Notes

LLM workflows are built around persistent storage so large model downloads survive restarts. That is the main reason the generated flows feel faster after the first run.

Use [Configuration](/docs/configuration#network-volumes) if you need to override volume behavior.

## Calling the APIs

### Local Ollama example

```bash
curl http://127.0.0.1:11434/api/tags
```

### Local vLLM example

```bash
curl http://127.0.0.1:8000/v1/models
```

### OpenAI SDK against local vLLM

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: "not-used-locally",
  baseURL: "http://127.0.0.1:8000/v1",
});
```

### OpenAI SDK against a hosted publish

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.GPU_PROXY_KEY!,
  baseURL: "https://your-hosted-proxy.gpu-cli.sh/v1",
});
```

## When to Use Serverless Instead

Choose [Serverless Endpoints](/docs/serverless) when you need:

* a deployed RunPod endpoint URL
* endpoint autoscaling instead of a pod-backed local proxy
* a production-facing remote API surface

Stay with `gpu llm` when you want interactive work, local forwarding, and wake-on-request behavior.

## Related Docs

* [Commands Reference](/docs/commands#gpu-llm)
* [Configuration](/docs/configuration#ports-and-activity-routing)
* [Serverless Endpoints](/docs/serverless)
* [Troubleshooting](/docs/troubleshooting)


---

# Organizations & Service Accounts



# Organizations & Service Accounts

Organizations let multiple people share billing context, ownership, and automation credentials while using the same public GPU CLI workflow.

> Organization features depend on account enablement. If `gpu org create` is rejected for your account, contact support.

If you want a shared OpenAI-compatible endpoint for your team, use [Hosted Proxy Quickstart](/docs/hosted-proxy) after your org is ready.

## Concepts

* **Organization**: A named account container with members, roles, billing ownership, and audit history.
* **Active org context**: The CLI targets one org at a time. Switch context with `gpu org switch <SLUG>`. Clear the slug to go back to your personal context.
* **Sub-account**: A child organization created with `--parent`. This is useful when one parent account manages multiple teams or clients.
* **Service account**: A non-human identity for CI/CD and other headless workflows.

## Roles

* **OWNER**: Full control over the org, including ownership transfer.
* **ADMIN**: Can manage members and service accounts.
* **MEMBER**: Can run workloads inside the org context.

## Getting Started

### Create an organization

```bash
gpu org create "My Team"
```

This generates a slug, such as `my-team`, for future commands.

### Switch to the org context

```bash
gpu org switch my-team
```

Once active, normal commands such as `gpu run`, `gpu llm run`, and `gpu serverless` run in that org context.

### Invite members

```bash
gpu org invite alice@example.com --role admin
gpu org invite bob@example.com
```

### Transfer ownership

```bash
gpu org transfer my-team --to alice@example.com
```

The recipient must already be a member.

### Delete an organization

```bash
gpu org delete my-team
```

Only the org owner can delete an organization, and deletion is only allowed when no other members remain. If the org still has admins or members, remove them through org member management in the portal before retrying deletion.

If you delete the currently active org, the CLI clears the active org context automatically so later commands do not keep pointing at a removed slug.

## Sub-Accounts

Sub-accounts are child organizations created under a parent org.

```bash
gpu org create "Client Project" --parent my-team
```

Use sub-accounts when one parent account needs separate org boundaries for multiple teams, projects, or clients.

## Service Accounts for CI/CD

Service accounts let automation run GPU CLI non-interactively.

### Create a token

```bash
gpu org service-account create --name "github-actions"
```

The CLI prints a token with the prefix `gpt_sat_*`. It is shown once, so store it in your secret manager immediately.

### Use it in automation

```bash
export GPU_SERVICE_TOKEN=gpt_sat_xxxxxxxxxxxx
export GPU_RUNPOD_API_KEY=rpa_xxxxxxxxxxxx   # or RUNPOD_API_KEY
export USER=railway-image-generator

# Optional: pin a specific SSH keypair (e.g. fully-hermetic CI).
# Omit both and GPU CLI generates and caches a stable SSH identity for you.
# export GPU_SSH_PRIVATE_KEY=...
# export GPU_SSH_PUBLIC_KEY=...

gpu run python train.py
```

When `GPU_SERVICE_TOKEN` is set, the CLI authenticates as the service account instead of opening a browser.

### Why `USER` matters

In CI, containers, and other headless environments, `USER` is often unset. GPU CLI uses it for project identity and workspace naming in automation flows.

Use a stable non-human value:

* service account name
* service name
* infrastructure workload name

Example:

```bash
export USER=railway-image-generator
```

## Manage Service Accounts

```bash
gpu org service-account list
gpu org service-account revoke sa_abc123
```

## Security Notes

* Tokens are shown once at creation.
* Revoked tokens stop working immediately.
* Each service account has its own identity in audit logs.
* Use separate service accounts per pipeline so you can revoke access cleanly.

## Example: GitHub Actions

```yaml
name: Train on GPU
on:
  push:
    branches: [main]

jobs:
  train:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install GPU CLI
        run: curl -fsSL https://gpu-cli.sh/install.sh | sh

      - name: Run training
        env:
          GPU_SERVICE_TOKEN: ${{ secrets.GPU_SERVICE_TOKEN }}
          GPU_RUNPOD_API_KEY: ${{ secrets.GPU_RUNPOD_API_KEY }}
          # Optional: pin a specific SSH keypair. Omit both to auto-generate a stable identity.
          # GPU_SSH_PRIVATE_KEY: ${{ secrets.GPU_SSH_PRIVATE_KEY }}
          # GPU_SSH_PUBLIC_KEY: ${{ secrets.GPU_SSH_PUBLIC_KEY }}
          USER: railway-image-generator
        run: gpu run python train.py
```

## Next Step: Shared Team Inference

Once the org exists, an org admin can provision the shared hosted proxy with:

```bash
gpu proxy create
```

See [Hosted Proxy Quickstart](/docs/hosted-proxy) for the simplest setup flow.


---

# Quickstart



# Quickstart

This guide walks you through installing GPU CLI, authenticating, and running your first workload on a cloud GPU.

## 1. Install GPU CLI

```bash
curl -fsSL https://gpu-cli.sh/install.sh | sh
```

This installs the `gpu` command to `~/.gpu-cli/bin` and adds it to your PATH.

## 2. Connect Your GPU Provider

GPU CLI uses your provider API key to provision GPUs. The default hosted path today is RunPod, and you pay your provider directly. No `gpu login` is required first.

1. Get your API key from [RunPod Settings](https://www.runpod.io/console/user/settings)
2. Add it to GPU CLI:

```bash
gpu auth runpod
```

Enter your API key when prompted. An SSH key for pod access is generated automatically. Use `gpu auth vast`, `gpu auth thunder`, or `gpu auth ionet` for the other providers.

Or skip the prompt entirely by exporting the provider API key, which GPU CLI picks up automatically:

```bash
export RUNPOD_API_KEY=rpa_xxxxxxxxxxxx
```

## 3. Optional: GPU CLI Account

```bash
gpu login
```

`gpu login` is optional. It adds a free GPU CLI account (portal identity and owned-node/fleet features). It is not required to rent GPUs or run jobs, so you can skip it and come back later. GPU CLI is free forever, no account required. Bring your own GPU or a provider API key. You pay the provider directly at their rates, zero markup.

## 4. Check Available GPUs

See what GPUs are available and their prices:

```bash
gpu inventory
```

Filter to show only available GPUs:

```bash
gpu inventory --available
```

## 5. Run Your First Command

Test that everything works:

```bash
gpu run python -c "import torch; print(f'CUDA: {torch.cuda.is_available()}, Device: {torch.cuda.get_device_name(0) if torch.cuda.is_available() else \"N/A\"}')"
```

GPU CLI will:

1. Provision a GPU pod (auto-selects best available)
2. Run your command
3. Stream output back to your terminal
4. Auto-stop the pod after 5 minutes of idle

## 6. Initialize a Project

For a real project, initialize GPU CLI in your project directory:

```bash
cd my-ml-project
gpu init
```

This creates a `gpu.jsonc` configuration file. You can customize:

* GPU type
* Output patterns to sync back
* Port forwarding
* And more

See [Configuration](/docs/configuration) for all options.

## 7. Run a Training Script

```bash
gpu run python train.py
```

Your code syncs to the pod, runs on the GPU, and outputs sync back automatically.

## 8. Try the LLM Workflow

GPU CLI also ships a routed LLM workflow for Ollama and vLLM:

```bash
gpu llm run
```

Use this for a pod-based chat UI and local API surface that auto-resumes on request. See [LLM Inference](/docs/llm) for the full guide.

## 9. Check Status

See what's running:

```bash
gpu status
```

Or open the interactive dashboard:

```bash
gpu dashboard
```

## Next Steps

* [LLM Inference](/docs/llm): Run Ollama or vLLM with wake-on-request routing
* [Configuration Reference](/docs/configuration): Customize your `gpu.jsonc`
* [Commands Reference](/docs/commands): All CLI commands
* [Organizations & Service Accounts](/docs/organizations): Organizations, service accounts, and CI/CD tokens
* [Hosted Proxy Quickstart](/docs/hosted-proxy): Create a shared OpenAI-compatible endpoint for your org
* [Troubleshooting](/docs/troubleshooting): Common issues and solutions


---

# Serverless Endpoints



# Serverless Endpoints

GPU CLI wraps RunPod Serverless with a config-first workflow for ComfyUI, vLLM, Whisper, and custom worker images.

Use this guide when you want a deployed RunPod endpoint. If you want a pod-based LLM workflow with a local Web UI, local API ports, and wake-on-request routing, use [LLM Inference](/docs/llm) instead.

## Prerequisites

* [GPU CLI installed and authenticated](/docs/quickstart)
* RunPod API key configured with `gpu auth runpod` (or set `RUNPOD_API_KEY`)

## Quick Start

### 1. Create a `serverless` block

```json
{
  "$schema": "https://gpu-cli.sh/schema/v1/gpu.json",
  "serverless": {
    "template": "vllm",
    "gpu_type": "NVIDIA A100 80GB PCIe",
    "scaling": {
      "min_workers": 0,
      "max_workers": 3,
      "idle_timeout": 5
    }
  }
}
```

### 2. Deploy

```bash
gpu serverless deploy
```

GPU CLI resolves the official template, applies your config, and creates or updates the RunPod endpoint.

### 3. Call the endpoint

```bash
curl https://api.runpod.ai/v2/{endpoint-id}/runsync \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"input": {"prompt": "a cat in space"}}'
```

## Templates

| Template       | Use case                        |
| -------------- | ------------------------------- |
| `auto`         | Auto-detect from project files  |
| `comfyui`      | Image generation workflows      |
| `vllm`         | OpenAI-compatible LLM inference |
| `whisper`      | Audio transcription             |
| `custom-image` | Custom worker image             |

## Configuration

The full schema lives in [Configuration](/docs/configuration#serverless). The most important fields are:

* `serverless.template`
* `serverless.gpu_type`
* `serverless.gpu_types`
* `serverless.scaling`
* `serverless.volume`
* `serverless.runpod`

### Example: ComfyUI endpoint

```json
{
  "$schema": "https://gpu-cli.sh/schema/v1/gpu.json",
  "serverless": {
    "template": "comfyui",
    "gpu_type": "NVIDIA GeForce RTX 4090",
    "scaling": {
      "min_workers": 0,
      "max_workers": 5,
      "idle_timeout": 10
    },
    "volume": {
      "name": "comfyui-models",
      "size_gb": 200
    },
    "runpod": {
      "flashboot": true
    }
  }
}
```

### Example: vLLM endpoint

```json
{
  "$schema": "https://gpu-cli.sh/schema/v1/gpu.json",
  "serverless": {
    "template": "vllm",
    "gpu_type": "NVIDIA A100 80GB PCIe",
    "gpu_types": ["NVIDIA L4"],
    "scaling": {
      "min_workers": 1,
      "max_workers": 3,
      "idle_timeout": 30
    },
    "runpod": {
      "cached_model": "meta-llama/Llama-3.1-8B-Instruct",
      "env": {
        "MODEL_NAME": "meta-llama/Llama-3.1-8B-Instruct"
      }
    }
  }
}
```

## Managing Endpoints

### List

```bash
gpu serverless list
gpu serverless list --all
gpu serverless list --json
```

### Status

Treat status as endpoint-ID-driven today:

```bash
gpu serverless status <ENDPOINT_ID>
gpu serverless status <ENDPOINT_ID> --json
```

### Delete

Delete accepts an endpoint ID or name. If you omit it, the CLI shows an interactive picker.

```bash
gpu serverless delete
gpu serverless delete my-endpoint --force
```

### Warm

GPU warmup is the supported path today:

```bash
gpu serverless warm <ENDPOINT_ID> --gpu
gpu serverless warm <ENDPOINT_ID> --timeout 900
```

### Template delete

```bash
gpu serverless template delete
gpu serverless template delete tmpl_123 --force
```

### Logs

```bash
gpu serverless logs <ENDPOINT_ID>
```

`gpu serverless logs` currently points you to the RunPod dashboard rather than streaming filtered logs inside the CLI.

## Current Limitations

* `gpu serverless warm --cpu` is not implemented in the current runtime.
* Deploy-time `gpu serverless deploy --warm ...` is accepted by the CLI but not wired through the runtime.
* Deploy-time `gpu serverless deploy --write-ids ...` is accepted by the CLI but not wired through the runtime.
* `serverless.prewarm` exists in the schema but should be treated as reserved configuration today.

## Calling a vLLM Endpoint

### TypeScript

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.RUNPOD_API_KEY,
  baseURL: `https://api.runpod.ai/v2/${process.env.ENDPOINT_ID}/openai/v1`,
});

const response = await client.chat.completions.create({
  model: "meta-llama/Llama-3.1-8B-Instruct",
  messages: [{ role: "user", content: "Hello!" }],
  stream: true,
});

for await (const chunk of response) {
  process.stdout.write(chunk.choices[0]?.delta?.content || "");
}
```

### Python

```python
from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ["RUNPOD_API_KEY"],
    base_url=f"https://api.runpod.ai/v2/{os.environ['ENDPOINT_ID']}/openai/v1",
)
```

## Next Steps

* [LLM Inference](/docs/llm): Pod-based Ollama and vLLM with wake-on-request routing
* [Configuration Reference](/docs/configuration#serverless): Full serverless config
* [Commands Reference](/docs/commands#gpu-serverless): CLI flags and limitations
* [Troubleshooting](/docs/troubleshooting): Current caveats and recovery steps


---

# Troubleshooting



# Troubleshooting

Use this page for the issues that show up most often in the current GPU CLI workflow.

## Daemon and Connection Issues

### Daemon not running

**Symptoms**

* `Failed to connect to daemon`
* `Connection refused`
* commands hang before job submission

**Fix**

```bash
gpu daemon start
gpu daemon status
gpu daemon logs --tail 50
gpu daemon restart
```

### SSH connection failures

**Symptoms**

* `Host key verification failed`
* `Permission denied (publickey)`
* repeated SSH retries during pod setup

**Fix**

```bash
gpu auth status
gpu auth runpod
gpu auth login --generate-ssh-keys
```

`gpu auth runpod` (or `gpu auth vast` / `gpu auth thunder` / `gpu auth ionet`) is the lighter fix: it reconnects the provider and auto-generates an SSH key for pod access. GPU CLI now generates and caches a stable SSH identity for you, so `gpu auth login --generate-ssh-keys` is only needed when you want to force a fresh keypair.

Also verify the provider API key you entered is valid and that the pod actually reached a running state.

## Headless and CI Issues

### `USER` is missing

**Symptoms**

* `Failed to detect project: Cannot detect username`
* `gpu run` fails early in CI, Railway, or containers

**Fix**

Set `USER` explicitly to a stable service identity:

```bash
export USER=railway-image-generator
```

Good values:

* service account name
* service name
* infrastructure workload name

## Sync Issues

### Files are missing on the pod

GPU CLI excludes files that match `.gitignore` unless you explicitly include them.

**Fix**

* check `.gitignore`
* use `include` in `gpu.jsonc` for gitignored files you still want synced
* use `gpu run --force-sync ...` when you need a clean full sync
* use `gpu run --show-sync ...` for detailed sync progress

### Outputs are not syncing back

Make sure the output path is covered by `outputs`, not ignored by `exclude_outputs`, and still exists inside the remote workspace.

Useful commands:

```bash
gpu run --outputs python train.py
gpu logs --type sync
```

## Wake-on-Request and Port Routing

### A web UI or API keeps the pod awake forever

By default, all HTTP requests on a forwarded port count as activity. That means polling paths such as `/health`, `/queue`, or `/metrics` can keep the cooldown timer alive.

**Fix**

Use rich `ports` rules:

```json
{
  "ports": [
    {
      "port": 8080,
      "http": {
        "activity_paths": ["/api/chat"],
        "ignore_paths": ["/health", "/queue"],
        "ignore_methods": ["OPTIONS", "HEAD"]
      }
    }
  ]
}
```

See [LLM Inference](/docs/llm) and [Configuration](/docs/configuration#ports-and-activity-routing) for the full model.

### My forwarded app does not wake back up

Check these points:

* make sure you did not use `gpu run --no-persistent-proxy`
* make sure `persistent_proxy` is not disabled in config
* if you use `activity_paths`, confirm the incoming request path actually matches
* if the app relies on WebSocket traffic only, remember that WebSocket frames keep the connection warm but do not reset the HTTP cooldown timer by themselves

## Serverless Limitations

### CPU warmup is not working

`gpu serverless warm --cpu` is not implemented in the current runtime.

Use GPU warmup instead:

```bash
gpu serverless warm <ENDPOINT_ID> --gpu
```

### Deploy-time `--warm` or `--write-ids` has no effect

Those flags are accepted by the CLI but are not wired through the current runtime yet.

### `gpu serverless status` or `gpu serverless warm` fails for a name

Treat both commands as endpoint-ID-driven today:

```bash
gpu serverless status <ENDPOINT_ID>
gpu serverless warm <ENDPOINT_ID> --gpu
```

### I expected CLI log streaming for serverless

`gpu serverless logs` currently points you to the RunPod dashboard instead of streaming filtered endpoint logs in the CLI.

## Storage and Keychain

### Keychain corruption (`aead::Error`)

If every `gpu` command fails with `Decryption failed: aead::Error`, the encrypted keychain file is corrupted.

```bash
rm ~/.gpu-cli-dev/keychain.enc
gpu auth runpod
```

For production mode, use `~/.gpu-cli/keychain.enc` instead.

Then restart the daemon:

```bash
pkill -f gpud
```

## Still Stuck?

* `gpu doctor`
* `gpu agent-docs`
* `gpu issue`
* `gpu support`