Running a 70-billion-parameter model locally cost $0 in API fees and kept every token on your hardware — if you had 48 GB of VRAM. Most developers don't. RunPod's per-second GPU rentals solve that: spin up an A100 80 GB for $1.19/hr, pull a quantized Llama-3-70B-GGUF, and serve it through Ollama or vLLM in under ten minutes. This guide walks you through the exact commands, model choices, and hardening steps so you own the stack end-to-end.
Quick Answer: Create a RunPod pod with an NVIDIA GPU, SSH in, install Docker, pull the Ollama or vLLM image, download a GGUF or AWQ model from Hugging Face, expose port 11434 or 8000, and query the OpenAI-compatible endpoint — all with open-source tooling and no vendor lock-in.
Why RunPod for Local LLM Deployment
Cost Control Beats Cloud APIs
OpenAI's GPT-4o charges $5 per million input tokens. A single 70B model on RunPod's A100 80 GB costs $1.19/hr and processes unlimited tokens. At 2,000 tokens/second sustained throughput, you break even after 420 million tokens — roughly 600 novel-length conversations. For teams running batch evals, nightly fine-tunes, or latency-sensitive prototypes, the math favors rented metal every time.
Full Weight Access Enables Customization
API endpoints hide logits, attention weights, and embedding layers. Local deployment lets you extract steering vectors, run activation patching, or quantize to 4-bit GPTQ for edge devices. Meta's Llama 3.1 405B released July 2024 under a permissive community license — you can download the 405B FP8 weights, quantize to AWQ 4-bit, and serve 128k context on a single H100 80 GB pod.
Open-Source Toolchain Maturity
Ollama reached 1.0 in July 2024 with native GPU scheduling, model libraries, and a REST API compatible with OpenAI SDKs. vLLM 0.6 added PagedAttention v2, speculative decoding, and multi-LoRA serving — all Apache 2.0. These tools run identically on your laptop, a RunPod pod, or an on-prem DGX, eliminating environment drift.
Prerequisites and Pod Configuration
Choose the Right GPU Template
RunPod's template library includes prebuilt images for Ollama, vLLM, Text-Generation-Inference, and llama.cpp. For a first deploy, select the "Ollama" community template — it bakes in CUDA 12.4, drivers, and the Ollama binary. Match GPU to model size: RTX 3090/4090 24 GB runs Llama-3-8B-Q4_K_M at 45 tok/s; A100 40 GB runs Llama-3-70B-Q4_K_M at 35 tok/s; H100 80 GB runs Llama-3.1-405B-FP8 at 18 tok/s with 128k context.
Set Persistent Volume and Ports
Attach a 200 GB network volume at /workspace — model weights survive pod restarts. Expose TCP 11434 for Ollama's API and 8000 for vLLM's OpenAI-compatible endpoint. Enable "Start on creation" and "Stop on idle" (30 min default) to avoid runaway charges. SSH key injection at launch lets you `ssh root@
Verify Driver and CUDA Stack
- Run `nvidia-smi` — confirm driver 550+ and GPU UUID matches pod spec.
- Run `nvcc --version` — verify CUDA 12.4 toolkit for vLLM compilation.
- Run `docker run --rm --gpus all nvidia/cuda:12.4-base nvidia-smi` — validates Docker GPU passthrough.
Deploying with Ollama — Simplest Path to Production
Pull and Quantize Models in One Command
Ollama's model library hosts 200+ GGUF variants tagged by quantization (Q4_K_M, Q5_K_S, Q8_0). `ollama pull llama3.1:70b-instruct-q4_K_M` downloads the 39 GB quantized weight, verifies SHA256, and registers it in `~/.ollama/models`. First token latency on A100 40 GB: 280 ms; sustained throughput: 38 tok/s. For custom quantizations, `ollama create my-llama3-70b-q3 -f Modelfile` where Modelfile specifies `FROM ./llama-3-70b-q3_k_m.gguf` and `TEMPLATE llama3`.
Serve Behind Nginx with TLS Termination
Ollama binds to 127.0.0.1:11434 by default. Create `/etc/nginx/sites-available/ollama` with `proxy_pass http://127.0.0.1:11434;` and `proxy_set_header X-Forwarded-Proto https;`. Obtain a Let's Encrypt cert via `certbot --nginx -d llm.yourdomain.com`. Restart nginx. Your OpenAI-compatible endpoint is now `https://llm.yourdomain.com/v1/chat/completions` — drop-in replacement for `api.openai.com`.
Enable Structured Output and Tool Calling
Llama 3.1 and Nemotron 3 Ultra support JSON schema enforcement. Add `format: json` to the request body or use the `tools` parameter with function definitions. Example: `curl -X POST https://llm.yourdomain.com/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"llama3.1:70b-instruct-q4_K_M","messages":[{"role":"user","content":"Extract name, email, company as JSON"}],"format":"json"}'`. Ollama 0.3.12+ validates against the schema before returning.
Deploying with vLLM — Maximum Throughput for Batch Workloads
Install vLLM in a Dedicated Container
RunPod's vLLM template uses the official `vllm/vllm-openai:latest` image. For custom builds, `docker build -t vllm-custom -f Dockerfile .` where Dockerfile pins `torch==2.4.0` and `vllm==0.6.3`. Mount your model volume: `docker run -d --gpus all -v /workspace/models:/models -p 8000:8000 vllm/vllm-openai:latest --model /models/Meta-Llama-3.1-70B-Instruct-AWQ --quantization awq --max-model-len 131072 --tensor-parallel-size 1`.
Leverage PagedAttention and Continuous Batching
vLLM's PagedAttention partitions KV cache into 16-token blocks, eliminating fragmentation. With continuous batching, new requests join the running batch without waiting for completion. On A100 80 GB serving Llama-3-70B-AWQ at 4k context: 2,200 tok/s aggregate throughput vs. 800 tok/s for naive batching. Enable `--enable-chunked-prefill --max-num-batched-tokens 8192` for further 15% gains on mixed-length prompts.
Multi-LoRA Serving for Fine-Tuned Variants
Load base model once, swap LoRA adapters per request. `--enable-lora --lora-modules sql-lora=/models/sql-lora,code-lora=/models/code-lora` registers adapters. Request with `model=meta-llama/Meta-Llama-3.1-70B-Instruct-lora-sql`. Each adapter adds ~0.5% VRAM. Ideal for multi-tenant SaaS where each customer gets a specialized adapter without dedicated GPU.
Model Selection and Quantization Strategy
Match Quantization to Quality Budget
GGUF Q4_K_M (4-bit + k-quant) retains 99% of FP16 MMLU score for Llama-3-70B at 39 GB. Q3_K_L drops to 96% at 31 GB — acceptable for classification, risky for coding. AWQ 4-bit (activation-aware) preserves 99.5% on GSM8K for 70B models. GPTQ 4-bit with `group_size=128` matches AWQ quality but requires calibration data. For 405B, FP8 (8-bit float) is the only practical option — 405 GB FP16 vs 203 GB FP8.
Context Length vs. VRAM Trade-offs
KV cache scales linearly with context: 2 bytes/token/layer (FP16) × 80 layers × 128k tokens = 20.5 GB for Llama-3.1-70B. At 4-bit KV (vLLM `--kv-cache-dtype fp8`), halve that to 10.2 GB. Sliding window attention (SWA) in Mistral-Nemo-12B caps effective context at 128k while using fixed 4k KV — enables 1M token logical context on 24 GB VRAM.
Real-World Example: Legal Document Analysis Pipeline
A legal tech startup processes 500 200-page contracts nightly. They deploy Llama-3.1-70B-AWQ on RunPod H100 80 GB ($2.69/hr) with vLLM, 128k context, 4-bit KV. Each contract averages 48k tokens. Batch size 8 saturates GPU at 1,800 tok/s. Full run: 2.2 hours, $5.92 GPU cost. API equivalent (GPT-4o-mini batch): $180. Annual savings: $64,000.
Comparison: Ollama vs. vLLM vs. TGI vs. llama.cpp Server
Choose Ollama for fastest time-to-first-token and simplest ops. Choose vLLM for highest throughput, multi-LoRA, and OpenAI API parity. Choose Text-Generation-Inference (TGI) for enterprise features (watermarking, guided decoding) and Hugging Face Hub integration. Choose llama.cpp server for CPU-only or Apple Silicon targets.
| Feature | Ollama | vLLM | TGI | llama.cpp |
|---|---|---|---|---|
| OpenAI API Compatible | Yes (v0.1.40+) | Native | Native | Via `--openai-endpoint` |
| Continuous Batching | No | Yes (PagedAttention) | Yes | No |
| Multi-LoRA | No | Yes | Yes | No |
| Quantization Formats | GGUF only | AWQ, GPTQ, FP8, GGUF | AWQ, GPTQ, FP8 | GGUF, EXL2, HQQ |
| Max Context (70B, 80 GB) | 131k | 131k (FP8 KV: 200k+) | 131k | 131k |
| Throughput (70B-Q4, A100 40 GB) | 38 tok/s | 2,200 tok/s (batched) | 1,900 tok/s | 42 tok/s |
| Setup Complexity | Low (single binary) | Medium (Docker, args) | Medium (Docker, YAML) | Low (single binary) |
Common Mistakes and Pro Fixes
Mistake: Exposing Ollama Port 11434 to Public Internet
Why It Hurts: Unauthenticated API allows model extraction, prompt injection, and GPU burn. SentinelOne found 1,200+ exposed Ollama instances in January 2024.
Fix: Bind to 127.0.0.1 only (`OLLAMA_HOST=127.0.0.1:11434`). Terminate TLS at nginx with client cert auth or Cloudflare Access. Never bind 0.0.0.0.
Mistake: Ignoring KV Cache Memory Pressure
Why It Hurts: 128k context at FP16 KV consumes 20+ GB on 70B models — OOM kills pod mid-request.
Fix: Enable FP8 KV (`--kv-cache-dtype fp8` in vLLM) or sliding window attention. Monitor `nvidia-smi dmon -s pucvmt` — keep VRAM < 90%.
Mistake: Using Default Quantization Without Evaluation
Why It Hurts: Q4_K_M fails on function calling benchmarks (BFCL score drops 12 pts vs FP16).
Fix: Run `lm-eval --model vllm --model_args pretrained=/model --tasks bfcl` on each quantization. Pick highest compression passing your eval threshold.
Mistake: No Request Queue or Rate Limiting
Why It Hurts: Burst traffic OOMs GPU; single user monopolizes throughput.
Fix: Deploy Redis + Celery queue in front of vLLM. Set `--max-num-seqs 256 --max-model-len 8192` per pod. Horizontal pod autoscaler on queue depth.
Pro Tips
- Warm-up runs: Send 3 dummy requests at pod start to compile CUDA graphs — cuts first-token latency 40%.
- Model offloading: For 405B on 2×H100, use `--pipeline-parallel-size 2 --tensor-parallel-size 1` in vLLM 0.6+.
- Speculative decoding: Pair Llama-3.1-70B with Llama-3.1-8B draft model (`--speculative-model meta-llama/Llama-3.1-8B-Instruct`) — 2.3× speedup.
- Log structured JSON: `--log-format json` in vLLM feeds Datadog/ELK directly — correlate latency spikes with batch sizes.
- Snapshot volumes: RunPod's snapshot API (`POST /pods/{id}/snapshot`) creates point-in-time volume backups before major updates.
FAQ
What is the minimum GPU VRAM to run a 70B parameter model?
39 GB VRAM runs Llama-3-70B at Q4_K_M (4-bit GGUF) with 4k context. 24 GB VRAM requires Q3_K_L (3-bit) or offloading layers to CPU via llama.cpp `--split-mode layer` — expect 3-5 tok/s. For production throughput, budget 40 GB (A100 40 GB) or 80 GB (A100 80 GB / H100 80 GB).
How does RunPod pricing compare to AWS EC2 P4/P5 instances?
RunPod A100 80 GB: $1.19/hr on-demand, $0.79/hr spot. AWS p4d.24xlarge (8×A100 40 GB): $32.77/hr on-demand, $9.83/hr spot. RunPod wins for single-GPU workloads; AWS wins for multi-GPU clusters with InfiniBand and EFA. RunPod bills per-second with 60-second minimum; AWS bills per-second with 1-minute minimum.
Can I use my existing OpenAI SDK code with a local vLLM endpoint?
Yes. vLLM implements `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, and `/v1/models` endpoints. Change `base_url` in your client: `openai.OpenAI(base_url="https://llm.yourdomain.com/v1", api_key="sk-local")`. Streaming, function calling, and logprobs work identically. Test with `curl -H "Authorization: Bearer sk-local" https://llm.yourdomain.com/v1/models`.
Why does my pod OOM when context exceeds 32k tokens?
KV cache grows linearly: 2 bytes × layers × context_tokens. Llama-3-70B (80 layers) at 32k context = 5.1 GB FP16 KV. At 128k context = 20.5 GB. Enable FP8 KV (`--kv-cache-dtype fp8`) to halve usage, or reduce `--max-model-len`. Monitor with `watch -n 1 nvidia-smi` during load test.
What happens to my model weights when the pod terminates?
Network volumes mounted at `/workspace` persist across pod stop/start/delete cycles. Only the compute container is ephemeral. RunPod snapshots create immutable volume copies for rollback. For multi-region DR, replicate volumes via `rclone sync /workspace s3:bucket/models --progress` nightly.
Conclusion
Deploying open-source LLMs on RunPod gives you API-grade latency, full weight access, and 90% cost reduction versus closed models — without surrendering control. Start with Ollama for single-model prototypes; graduate to vLLM when batch throughput or multi-LoRA tenancy matters. Quantize aggressively but validate against your eval suite. Harden the network path before exposing any endpoint. The toolchain matured in 2024: what took a distributed systems team six months now takes an afternoon. Your next model ships on your terms.
- RunPod + Ollama = fastest path to OpenAI-compatible local endpoint
- vLLM + PagedAttention = highest throughput for batch and multi-tenant workloads
- Quantization choice (GGUF Q4_K_M vs AWQ 4-bit) determines quality/cost frontier
- Persistent volumes + snapshots = reproducible, recoverable model deployments
0 comments:
Post a Comment