Running open-source large language models locally on RunPod slashes inference costs by up to 90% compared to closed APIs while giving you full data privacy and model control. As of 2025, over 500,000 developers deploy models like Llama 3.1 405B, Qwen 2.5, and Nemotron 3 Ultra on RunPod's GPU cloud — paying as little as $0.17/hour for H100s versus $30/hour on major cloud providers. This masterclass walks you through every decision from GPU selection to production hardening so you ship a reliable LLM endpoint in under an hour.
Quick Answer: Create a RunPod account, select a GPU template (H100 for 70B+ models, A100 80GB for 30B-70B, RTX 4090/3090 for 7B-13B), deploy a vLLM or Ollama pod with your chosen model from Hugging Face, expose port 8000, and test with curl or OpenAI-compatible client — total setup takes 15-30 minutes and costs $0.17-$2.69/hour depending on GPU.
Why RunPod for Local LLM Deployment
Cost Advantage Over Major Cloud Providers
RunPod's community cloud pricing beats AWS, GCP, and Azure by 3-5x for equivalent GPUs. An H100 80GB SXM runs $2.69/hour on RunPod versus $10-15/hour on AWS p5 instances. A100 80GB is $1.19/hour versus $3.50-4.00/hour. For a team running a 70B parameter model 24/7, that's $876/month on RunPod versus $2,500+ on AWS — annual savings exceed $19,000 per GPU.
Instant Spin-Up and Template Ecosystem
RunPod's template system eliminates the 30-60 minute environment setup tax. Pre-built templates for vLLM, Ollama, TGI (Text Generation Inference), and llama.cpp launch with optimized Docker images, CUDA drivers, and model caching pre-configured. The vLLM template includes PagedAttention and continuous batching out of the box — features that deliver 2-4x throughput gains over naive implementations.
Persistent Storage and Network Flexibility
Each pod gets 50GB free container disk plus optional network volumes (starting at $0.10/GB/month) that survive pod restarts. This matters for model weights: Llama 3.1 405B BF16 weighs 810GB — you download once to a network volume, then attach to any pod in seconds. RunPod also provides static IPs and custom domains for production endpoints.
GPU Selection Guide for Open Source LLMs
Model Size to VRAM Mapping (BF16/FP16)
Quantization dramatically changes GPU requirements. The table below shows minimum VRAM for common model sizes at different quantization levels, assuming 1.2x overhead for KV cache and activation memory:
| Model (Params) | FP16/BF16 VRAM | 4-bit (GPTQ/AWQ) VRAM | Recommended GPU |
|---|---|---|---|
| 7B (Llama 3.1, Qwen 2.5) | 16 GB | 6 GB | RTX 3090/4090 24GB, A10G 24GB |
| 13B-14B | 28 GB | 10 GB | RTX 4090 24GB (4-bit only), A100 40GB |
| 32B-34B (Qwen 2.5 32B) | 68 GB | 22 GB | A100 80GB, H100 80GB |
| 70B-72B (Llama 3.1 70B, Qwen 2.5 72B) | 148 GB | 48 GB | 2x A100 80GB, H100 80GB (4-bit) |
| 405B (Llama 3.1 405B) | 810 GB | 260 GB | 4x H100 80GB (4-bit), 8x H100 (BF16) |
RunPod GPU Pricing and Availability (2025)
RunPod's secure cloud (data center grade) vs community cloud (consumer GPUs) trade-off: Secure cloud H100s at $2.69/hour guarantee ECC memory and NVLink; community cloud RTX 4090s at $0.34/hour lack ECC but work fine for inference. A100 80GB secure cloud at $1.19/hour is the sweet spot for 70B 4-bit models. Spot/interruptible instances save 40-60% but can be reclaimed — use only for batch workloads with checkpointing.
Multi-GPU vs Single-GPU Decision
vLLM tensor parallelism splits model weights across GPUs with near-linear scaling up to 4 GPUs. For Llama 3.1 70B 4-bit (48 GB), a single H100 80GB works. For BF16 70B (148 GB), you need 2x A100 80GB or 2x H100 80GB with tensor parallel size 2. Beyond 4 GPUs, pipeline parallelism adds latency — consider TGI with sharding instead. Real example: A 32B model on 2x A100 80GB with tensor parallel 2 achieves 2,800 tokens/sec vs 1,400 on 1x A100.
Step-by-Step Deployment: vLLM on RunPod
1. Create Account and Add Payment Method
Sign up at runpod.io with GitHub or email. Add a credit card — RunPod bills per-second with a $10 minimum deposit. Enable auto-refill to avoid pod termination mid-inference. Verify email to unlock community cloud GPUs (secure cloud requires manual approval, typically <24 hours).
2. Create a Network Volume for Model Weights
In the RunPod console, go to Storage → Network Volumes → Create. Name it "llm-models", select a data center region matching your GPU (e.g., US-EAST-1 for H100s), allocate 200GB minimum for 70B+ models. Cost: $0.10/GB/month = $20/month. This volume persists across pod restarts and lets you swap GPUs without re-downloading weights.
3. Deploy vLLM Pod from Template
Go to Pods → Deploy → Template → Search "vllm". Select "vLLM OpenAI Compatible" (official template). Configure: GPU type (H100 80GB for 70B 4-bit), GPU count (1 for 7B-32B, 2-4 for 70B+), Container Disk 50GB, Volume Mount: attach "llm-models" to /workspace. Environment variables: MODEL_ID=meta-llama/Llama-3.1-70B-Instruct (or your HF model), TENSOR_PARALLEL_SIZE=1 (match GPU count), DTYPE=auto, MAX_MODEL_LEN=8192, GPU_MEMORY_UTILIZATION=0.9. Click Deploy — pod spins up in 30-90 seconds.
4. Download Model Weights to Network Volume
Once pod is running, open the Web Terminal (or SSH). Run: cd /workspace && huggingface-cli download meta-llama/Llama-3.1-70B-Instruct --local-dir Llama-3.1-70B-Instruct. First download pulls ~40GB for 4-bit quantized weights. Subsequent pod starts skip download — weights stay on network volume. For private/gated models, add --token $HF_TOKEN after setting HF_TOKEN in pod env vars.
5. Start vLLM Server and Test
In terminal: python -m vllm.entrypoints.openai.api_server --model /workspace/Llama-3.1-70B-Instruct --tensor-parallel-size 1 --port 8000 --host 0.0.0.0. vLLM loads model into VRAM (watch nvidia-smi — 70B 4-bit uses ~48GB). Once "Uvicorn running on http://0.0.0.0:8000" appears, test from your local machine: curl -X POST https://YOUR_POD_ID-8000.proxy.runpod.net/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "meta-llama/Llama-3.1-70B-Instruct", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 100}'. Expect 50-200 tokens/sec on H100.
Alternative: Ollama for Simpler Workflows
When to Choose Ollama Over vLLM
Ollama excels at developer experience: single binary, automatic model management, built-in quantization (GGUF), and OpenAI-compatible API. Choose Ollama if you're prototyping, switching models frequently, or need CPU offloading for models larger than VRAM. vLLM wins for production throughput: PagedAttention, continuous batching, and tensor parallelism deliver 3-5x higher concurrent request handling.
Deploy Ollama on RunPod
Template: "Ollama" (official). Same GPU selection logic. After deploy, SSH and run: ollama run llama3.1:70b — Ollama auto-downloads GGUF quantized weights (Q4_K_M ~40GB) to /root/.ollama. To persist, mount network volume to /root/.ollama. Expose port 11434. Test: curl http://POD_URL:11434/api/generate -d '{"model": "llama3.1:70b", "prompt": "Hello"}'. Ollama's default context is 4096 — add ollama create mymodel -f Modelfile with PARAMETER num_ctx 8192 for longer context.
Quantization Choice: AWQ/GPTQ vs GGUF
vLLM uses AWQ/GPTQ (4-bit weight-only quantization, runs on GPU). Ollama uses GGUF (4-bit with CPU offloading support). For pure GPU inference, AWQ/GPTQ is faster — vLLM's kernels are optimized for it. GGUF shines when model exceeds VRAM: Ollama offloads layers to CPU/RAM seamlessly. Example: Llama 3.1 70B Q4_K_M on 24GB VRAM + 32GB RAM runs at 8 tokens/sec on Ollama vs OOM on vLLM without tensor parallel.
Production Hardening and Optimization
Autoscaling and Load Balancing
RunPod's Serverless (beta) auto-scales vLLM workers from 0 to N based on queue depth — cold start ~30s for 7B, ~3min for 70B. For consistent traffic, keep 1-2 warm pods and use RunPod's load balancer (TCP/HTTP) in front. Configure health checks on /health endpoint. Set max concurrency per worker: vLLM handles 128-256 concurrent requests on H100 with continuous batching.
Monitoring, Logging, and Observability
Add Prometheus metrics: vLLM exposes /metrics with request latency, token throughput, GPU utilization, KV cache usage. Ship to Grafana Cloud (free tier 10k series) or self-hosted. Log structured JSON to stdout — RunPod captures pod logs. Key alerts: GPU memory >95%, request latency p99 >5s, error rate >1%. Example: A 70B model at 200 req/min shows 15% KV cache utilization at 4K context — scale at 70%.
Security: API Keys, TLS, and Network Isolation
vLLM supports API key auth via --api-key flag (comma-separated keys). For TLS, terminate at RunPod load balancer (free Let's Encrypt) or use Caddy/Traefik sidecar in pod. Restrict pod network: RunPod's "Secure Cloud" pods are VPC-isolated; community cloud pods share host network — use firewall rules to allow only your IP CIDR. Never expose pod ports publicly without auth — RunPod proxy URLs are guessable.
Comparison: RunPod vs Alternatives for LLM Deployment
Choosing a GPU cloud depends on workload pattern, budget, and operational maturity. The table below compares RunPod against the top alternatives for open-source LLM inference as of Q1 2025.
Pricing reflects on-demand secure cloud rates; community/spot prices can be 40-60% lower. Throughput numbers are for Llama 3.1 70B 4-bit on specified GPU with vLLM continuous batching.
| Platform | H100 80GB Hourly | A100 80GB Hourly | RTX 4090 Hourly | Template Ecosystem | Persistent Volumes | Serverless/Autoscale | Llama 3.1 70B 4-bit Throughput (tok/s) |
|---|---|---|---|---|---|---|---|
| RunPod | $2.69 | $1.19 | $0.34 (community) | Excellent (vLLM, Ollama, TGI, llama.cpp) | $0.10/GB/mo, multi-attach | Beta (cold start 30s-3min) | ~2,200 on 1x H100 |
| Lambda Labs | $2.49 | $1.10 | N/A | Good (Docker Hub images) | $0.12/GB/mo | No | ~2,100 on 1x H100 |
| AWS p5 (H100) | $9.83 (p5.48xlarge) | $3.50 (p4d.24xlarge) | N/A | Manual (Deep Learning AMIs) | EBS $0.08/GB/mo | SageMaker (complex setup) | ~2,300 on 1x H100 |
| GCP A3 (H100) | $11.06 (a3-highgpu-8g) | $3.67 (a2-highgpu-8g) | N/A | Manual (Container images) | PD $0.10/GB/mo | Vertex AI (expensive) | ~2,250 on 1x H100 |
| Together AI | $3.20 (dedicated) | $1.80 (dedicated) | N/A | API only (no raw GPU) | Included | Native serverless | API rate-limited |
| Fireworks AI | API only | API only | API only | API only | Included | Native serverless | API rate-limited |
Common Mistakes and Pro Tips
Mistake 1: Underestimating VRAM for Context Length
Why It Hurts: KV cache scales linearly with context length and batch size. A 70B model at 8K context needs ~8GB extra VRAM per concurrent request. At 32K context, that's 32GB/request — OOM crashes production.
Fix: Set MAX_MODEL_LEN to your actual max (not model max). Use GPU_MEMORY_UTILIZATION=0.85 headroom. Calculate: VRAM_needed = model_weights + (2 * num_layers * hidden_size * max_seq_len * batch_size * 2 bytes). Monitor gpu_cache_usage_perc metric.
Mistake 2: Ignoring Quantization Quality Trade-offs
Why It Hurts: 4-bit GPTQ/AWQ loses 1-3% MMLU vs FP16; 3-bit loses 5-8%; 2-bit is unusable for reasoning. GGUF Q4_K_M preserves quality better than GPTQ-4bit but runs slower on GPU.
Fix: Benchmark your specific task. For coding: use 4-bit AWQ (Phind-CodeLlama-34B-v2-AWQ). For chat: Q4_K_M GGUF on Ollama. For reasoning-heavy: stay at BF16 or 8-bit. Never guess — run lm-eval-harness on your eval set.
Mistake 3: No Persistent Volume = Re-downloading 80GB Every Deploy
Why It Hurts: Pod restarts (maintenance, preemption, crashes) wipe container disk. Re-downloading Llama 3.1 405B takes 45+ minutes on RunPod's network — $2/hour wasted.
Fix: Always create network volume first. Mount to /workspace (vLLM) or /root/.ollama (Ollama). Verify with ls -la /workspace after pod restart. Cost: $20/month for 200GB vs hours of downtime.
Mistake 4: Running Without Tensor Parallel for Large Models
Why It Hurts: A 70B BF16 model (148GB) OOMs on single 80GB GPU. Even 4-bit (48GB) leaves <10GB for KV cache — supports ~1 concurrent request at 4K context.
Fix: Set TENSOR_PARALLEL_SIZE=GPU_COUNT. For 70B BF16: 2x A100 80GB with TP=2. For 405B 4-bit: 4x H100 80GB with TP=4. vLLM's tensor parallel is near-linear — 2x GPU ≈ 2x throughput.
Mistake 5: Exposing Pods Publicly Without Authentication
Why It Hurts: RunPod proxy URLs (pod-id-8000.proxy.runpod.net) are enumerable. Unauthenticated endpoints get scraped, racking up GPU bills and leaking proprietary prompts.
Fix: Always pass --api-key sk-xxx,sk-yyy to vLLM. Use RunPod load balancer with header-based auth. Rotate keys monthly. Monitor /metrics for unknown client IPs.
Pro Tips
- Pre-warm models on pod start: Add a startup script that runs a dummy inference (
curl localhost:8000/v1/completions -d '{"prompt": "warmup", "max_tokens": 1}') so first real request isn't slowed by CUDA graph capture. - Use flashinfer backend: vLLM 0.6+ supports
--attention-backend flashinfer— 10-20% faster decode on Hopper (H100) and Blackwell. - Enable prefix caching: vLLM's
--enable-prefix-cachingreuses KV cache for shared prompts (system prompts, few-shot examples) — 30-50% latency reduction for RAG workloads. - Spot instances for batch inference: RunPod spot H100s at ~$1.00/hour (60% off). Use with checkpointing: save vLLM engine state every 100 requests, resume on reclaimed pod.
- Benchmark with your data: Run
python -m vllm.benchmarks.benchmark_serving --model /workspace/model --dataset-path your_data.jsonl --num-prompts 100before committing to GPU config.
FAQ
What is the cheapest GPU on RunPod that can run Llama 3.1 70B?
The cheapest option is a community cloud RTX 4090 24GB at $0.34/hour running 4-bit quantized (Q4_K_M GGUF on Ollama or AWQ on vLLM) with CPU offloading. Expect 8-15 tokens/sec. For production throughput (>100 tokens/sec), you need at least an A100 80GB ($1.19/hour secure cloud) or H100 80GB ($2.69/hour) for 4-bit without offloading.
How does RunPod compare to Together AI or Fireworks for open-source LLM hosting?
RunPod gives you raw GPU access with full control — you choose model, quantization, vLLM config, and pay per-second for compute. Together AI and Fireworks offer serverless APIs with per-token pricing ($0.10-0.90/M tokens) — easier to start but 5-10x costlier at scale, no custom quantization, and rate limits. RunPod wins for sustained workloads; serverless APIs win for sporadic traffic.
Can I run multiple different models on one RunPod GPU?
Yes, but with caveats. vLLM supports multi-model serving via --model list, but all models must fit in VRAM simultaneously. A 7B + 13B 4-bit fits on 24GB (6GB + 10GB + KV cache). For larger combos, use Ollama which loads/unloads models on demand (slow switch). Best practice: one model per pod, use RunPod load balancer to route by model name.
Why is my vLLM throughput lower than benchmark numbers?
Common causes: (1) Context length too long — KV cache eats VRAM, reducing batch size. (2) GPU_MEMORY_UTILIZATION too low — default 0.9 leaves 10% headroom; drop to 0.85 for stability. (3) Missing flashinfer/flash-attn — install flash-attn and use --attention-backend flashinfer. (4) Network bottleneck — RunPod proxy adds ~50ms latency; test locally via SSH port-forward.
Will RunPod support Blackwell (B200) GPUs for LLM inference?
RunPod typically adds new NVIDIA architectures within 60-90 days of general availability. Blackwell B200 (announced March 2024, shipping late 2024) should appear on RunPod Q1-Q2 2025. Expect 2.5x inference throughput over H100 for FP8/FP4 workloads. vLLM 0.6+ already has FP8 kernel support — migration will be a template update.
Conclusion
Deploying open-source LLMs on RunPod delivers the best price-performance-control triangle for teams moving off closed APIs. The workflow — create network volume, deploy vLLM/Ollama template, download weights once, start server — takes 15 minutes for 7B-32B models and 30-45 minutes for 70B+ with tensor parallelism. At $0.34/hour (RTX 4090) to $2.69/hour (H100), you pay 3-5x less than AWS/GCP/Azure for identical hardware. The key discipline: persistent volumes for model weights, tensor parallelism for large models, API keys from day one, and monitoring KV cache utilization. Master these and you'll run production LLM endpoints at a fraction of API costs with zero vendor lock-in.
- Start with vLLM template on A100 80GB for 70B 4-bit — best balance of cost ($1.19/hr) and throughput (1,800+ tok/s)
- Always use network volumes ($0.10/GB/mo) — eliminates re-downloads on pod restarts or GPU swaps
- Enable API keys, prefix caching, and flashinfer backend before going live
- Benchmark with your actual workload before committing to GPU count or quantization level
0 comments:
Post a Comment