llama.cpp

Source: https://pi.dev/docs/latest/llama-cpp

Pi supports the llama.cpp router server, which discovers multiple GGUF models and loads or unloads them on demand. Use a current llama.cpp build with router support.

Start the Router

Start llama-server without --model, -m, or -hf. Passing a model starts single-model mode instead of router mode.

llama-server \
  --models-dir ~/models \
  --no-models-autoload \
  --jinja \
  --host 127.0.0.1 \
  --port 8080 \
  -ngl 999 \
  -c 32768

• --models-dir discovers local GGUF files.
• --no-models-autoload keeps loading explicit through /llama.
• --jinja enables compatible chat templates and tool calling.
• -ngl 999 offloads as many layers as possible to the GPU.
• -c 32768 sets the context window per loaded model; omit to use the model's native context (may need much more memory).

Single-file models sit directly in the models directory. Multimodal and multi-shard models go in their own subdirectories with all components included. Restart the router after manually adding files; use llama.cpp model presets for per-model context sizes.

Configure Pi

/login llama.cpp

Enter the router URL (default http://127.0.0.1:8080) and optional API key. Environment variables configure the same values without /login:

export LLAMA_BASE_URL=http://127.0.0.1:8080
export LLAMA_API_KEY=optional-secret
pi

If the server uses an API key, start llama-server with the matching --api-key. Keep --host 127.0.0.1 for local-only access.

Manage Models with /llama

• Select an unloaded model to load it; select a loaded model to unload it.
• Select Download model… to search Hugging Face, then choose a repository and quantization. Exact owner/repository[:quant] values also work.
• Escape during a load or download prompts to confirm cancellation.
• If the router disconnects, /llama offers Retry and Close; Retry reconnects and refreshes state without replaying the interrupted operation.

Hugging Face search uses HF_TOKEN when set, then $HF_TOKEN_PATH, $HF_HOME/token, $XDG_CACHE_HOME/huggingface/token, and ~/.cache/huggingface/token. Search works unauthenticated with lower rate limits. Pi warns before downloading gated repositories. The llama.cpp server performs the download, so its process also needs HF_TOKEN for gated repositories.

Pi never silently unloads models and never deletes model files; it asks whether to unload others first. The router may be shared with other clients, so /llama always displays the router's current state.

Only loaded models appear in /model. Load with /llama, then select the model with /model.

Troubleshooting

curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8080/models

• No models in /llama: check --models-dir, the directory layout, and restart the router.
• Model missing from /model: load it with /llama first.
• Load fails or uses too much memory: lower -c or unload another model.
• Server not in router mode: start it without --model, -m, or -hf.