Deploy Local Open Source LLMs on RunPod: Beginner Guide

Running open source large language models locally gives you full data privacy and zero per-token costs, but most laptops lack the VRAM for models above 7B parameters. RunPod solves this by renting NVIDIA GPUs by the second — starting at $0.17 per hour for an RTX 3090 — so you can spin up a 70B model in minutes without buying hardware. This guide walks you through every step: account setup, template selection, model download, and serving with Ollama or vLLM, so you go from zero to a working API endpoint in under 30 minutes.

Quick Answer: Create a RunPod account, launch a GPU pod using the Ollama or vLLM template, SSH in, pull your model from Hugging Face (for example, ollama pull llama3.1:70b), then expose the API port (11434 for Ollama, 8000 for vLLM) to start prompting your private LLM instantly.

Why RunPod for Local LLMs Instead of Colab or Local Hardware

Cost per Hour Beats Buying a GPU

An NVIDIA RTX 4090 24 GB costs roughly $1,800 retail. At RunPod's $0.44 per hour for that same card, you would need 4,090 hours — about 170 days of continuous use — to break even. For intermittent experimentation, per-second billing wins. A 70B parameter model quantized to 4-bit fits in 48 GB VRAM, meaning dual 3090s at $0.34 per hour total can serve it with room for context.

No Time Limits or Disconnects Like Colab

Google Colab free tier kills sessions after 12 hours and often disconnects idle notebooks at 90 minutes. RunPod pods persist until you terminate them, keeping model weights in VRAM across sessions. You pay only while the pod runs; stopping it preserves the container disk for pennies per month.

Full Root Access and Custom Containers

Colab restricts sudo and blocks many system packages. RunPod gives you a root shell inside a Docker container — you can install any driver, compile llama.cpp with custom flags, or run vLLM with PagedAttention for 2–4× throughput gains over naive decoding.

Step-by-Step: Launch Your First Pod

1. Create Account and Add Credits

1. Go to runpod.io and sign up with GitHub or email.
2. Navigate to Billing → Add Credits. Minimum deposit is $10; $25 covers roughly 70 hours on an RTX 3090.
3. Enable auto-refill if you plan long runs — it prevents mid-training termination.

2. Choose a Template Optimized for LLM Inference

1. Click "Deploy" → "Pods" → "Community Templates".
2. Search "Ollama" or "vLLM". The official runpod/ollama template includes CUDA 12.1, Python 3.10, and Ollama pre-installed.
3. For vLLM, select runpod/vllm which bundles the inference engine with OpenAI-compatible API routes.

3. Pick GPU and Configure Storage

1. GPU type: RTX 3090 (24 GB) at $0.17/hr for models up to 13B 4-bit; RTX 4090 (24 GB) at $0.44/hr for faster inference; A100 80 GB at $1.64/hr for 70B+ unquantized.
2. GPU count: Start with 1. Add a second only after confirming single-GPU performance.
3. Container disk: 50 GB minimum for one 70B 4-bit model (~40 GB). Set 100 GB if you plan multiple models.
4. Volume disk (optional): Attach a network volume ($0.10/GB/mo) to persist models across pod recreations.

4. Deploy and Connect via SSH

1. Click "Deploy". Provisioning takes 30–90 seconds.
2. Once "Running", click "Connect" → "SSH". Copy the command, e.g., ssh root@123.45.67.89 -p 22888.
3. First login drops you into /workspace — this directory persists across stops if you used a volume.

Download and Serve Models with Ollama

Pull a Quantized Model from Hugging Face

Ollama's library mirrors popular Hugging Face repos with GGUF quantization. Run ollama pull llama3.1:70b-instruct-q4_K_M — the tag breaks down as 70B parameters, instruct-tuned, 4-bit quantization with K-means medium quality. Download takes 3–5 minutes on RunPod's 1 Gbps internal network. Verify with ollama list.

Run the Built-in API Server

Execute ollama serve in a tmux or screen session so it survives disconnect. The server binds to 0.0.0.0:11434 by default. Test locally: curl -X POST http://localhost:11434/api/generate -d '{"model":"llama3.1:70b-instruct-q4_K_M","prompt":"Why is the sky blue?"}'. You should see streaming JSON tokens within 200 ms per token on a 3090.

Expose the Port Securely

In the RunPod dashboard, click the pod name → "Ports" → add TCP 11434. RunPod generates a proxy URL like https://abcd1234-11434.proxy.runpod.net. Add a Bearer token via OLLAMA_API_KEY env var in the pod settings to prevent unauthorized access. Your OpenAI-compatible endpoint is now https://abcd1234-11434.proxy.runpod.net/v1/chat/completions.

High-Throughput Serving with vLLM and PagedAttention

Why vLLM Outperforms Ollama for Concurrent Requests

vLLM implements PagedAttention, a memory-management technique from UC Berkeley's Sky Computing Lab (2023) that reduces KV-cache fragmentation by up to 96%. In benchmarks, vLLM serves 2.7× more requests per second than Hugging Face Text Generation Inference at equal latency. For a 70B model on dual A100s, vLLM achieves ~120 tokens/sec/user with 8 concurrent users versus ~45 for Ollama.

Launch vLLM with an OpenAI-Compatible API

Inside the vLLM template pod, run:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --dtype auto \
  --quantization awq \
  --max-model-len 8192 \
  --port 8000

--tensor-parallel-size 2 splits the model across two GPUs. --quantization awq loads a 4-bit AWQ quantized variant from Hugging Face, cutting VRAM from 140 GB to ~40 GB. The server exposes /v1/chat/completions and /v1/completions matching OpenAI's schema.

Benchmark Your Deployment

Use hey or locust to measure throughput. Example: hey -n 100 -c 8 -m POST -H "Content-Type: application/json" -d '{"model":"meta-llama/Meta-Llama-3.1-70B-Instruct","messages":[{"role":"user","content":"Hello"}],"max_tokens":100}' https://your-proxy-url/v1/chat/completions. Record requests/sec, latency p50/p99, and GPU utilization via nvidia-smi dmon. Target >80% SM utilization for good efficiency.

RunPod vs. Alternatives: Cost and Capability Comparison

Choosing a GPU cloud provider depends on model size, concurrency needs, and budget. The table below reflects on-demand pricing as of January 2025 for the most common LLM inference cards.

All prices are per GPU per hour; multi-GPU pods multiply linearly. Network egress is free on RunPod and Lambda; vast.ai charges per GB.

Provider	RTX 3090 (24 GB)	RTX 4090 (24 GB)	A100 80 GB
RunPod	$0.17	$0.44	$1.64
Lambda Labs	$0.18	$0.50	$1.80
vast.ai (spot)	$0.08–0.15	$0.25–0.40	$0.90–1.40
CoreWeave	N/A	N/A	$2.21
Google Cloud (on-demand)	N/A	N/A	$3.67

Common Mistakes and Pro Tips

Mistake: Underestimating VRAM for Context Length

Why It Hurts: KV-cache scales linearly with context. A 70B 4-bit model needs ~40 GB for weights; 8K context adds ~8 GB; 32K adds ~32 GB. Exceeding VRAM triggers CPU offload, dropping throughput 10–50×.

Fix: Calculate: VRAM_GB = model_size_GB + (context_tokens × 2 × num_layers × hidden_size × 2_bytes) / 1e9. For Llama-3.1-70B (80 layers, 8192 hidden), 8K context ≈ 7.5 GB. Leave 20% headroom.

Mistake: Using FP16 Weights on Consumer GPUs

Why It Hurts: FP16 doubles VRAM vs. 4-bit quantization with negligible quality loss for inference. A 70B FP16 model needs 140 GB — impossible on dual 3090s.

Fix: Always request quantized variants: AWQ, GPTQ, or GGUF 4-bit. In vLLM, add --quantization awq; in Ollama, pull tags ending in q4_K_M or q5_K_M.

Mistake: Forgetting to Stop Pods After Testing

Why It Hurts: Idle pods still bill GPU time. A forgotten A100 costs $39/day.

Fix: Enable "Auto-stop on idle" in pod settings (default 30 min). Set a billing alert at $5/day in RunPod's dashboard.

Mistake: Skipping Health Checks Before Production Traffic

Why It Hurts: Cold starts add 10–30 sec for model load. First request after scale-up times out if client expects sub-second latency.

Fix: Add a readiness probe hitting /health (vLLM) or /api/tags (Ollama). Keep one warm replica if using RunPod's serverless autoscaling.

Pro Tips

• Use a network volume ($0.10/GB/mo) mounted at /workspace/models — models survive pod recreation, saving re-download time and bandwidth.
• Enable Flash Attention 2 in vLLM with --enforce-eager disabled (default) for 1.3–1.5× speedup on Ampere/Ada GPUs.
• For multi-user apps, put nginx in front of vLLM with rate limiting (e.g., 60 req/min per IP) and request buffering to smooth bursts.
• Monitor GPU memory with nvidia-smi --query-gpu=memory.used,memory.total --format=csv -l 1; alert if usage >90% sustained.
• Quantize your own models with llama.cpp or autoawq if a variant doesn't exist — 4-bit GPTQ preserves 99%+ quality on MMLU for most instruct models.

FAQ

What is the minimum GPU VRAM to run a 7B parameter model?

A 7B model quantized to 4-bit GGUF requires approximately 4.5 GB VRAM for weights plus 1–2 GB for KV-cache at 4K context. An RTX 3060 12 GB or RTX 4060 8 GB handles this comfortably. For 8-bit quantization, budget 8 GB VRAM minimum.

How does RunPod pricing compare to Lambda Labs for LLM inference?

RunPod is 5–10% cheaper on consumer GPUs (RTX 3090 $0.17 vs $0.18; RTX 4090 $0.44 vs $0.50) and offers per-second billing with no minimum. Lambda Labs requires a $20 minimum deposit and bills per minute. RunPod's template library is larger; Lambda's Ubuntu base images are more bare-metal.

Can I run multiple different models on one pod simultaneously?

Yes, but each loaded model consumes VRAM. Two 7B 4-bit models need ~10 GB total. With vLLM, you can serve multiple LoRA adapters on a single base model with minimal overhead (<1 GB per adapter). For distinct full models, use Ollama's ollama run in separate terminals — only the active model occupies VRAM; others stay in CPU RAM.

Why is my vLLM throughput lower than benchmark numbers?

Common causes: (1) Tensor parallel size doesn't match GPU count — verify nvidia-smi -L shows expected GPUs. (2) Context length too long — KV-cache eats VRAM, forcing CPU offload. (3) Missing --quantization flag loading FP16 weights. (4) Network bottleneck — RunPod's proxy adds ~20 ms latency; test locally via SSH port-forward first.

Will RunPod support Blackwell (GB200) GPUs for LLM inference?

RunPod typically adds new NVIDIA architectures within 4–8 weeks of general availability. CoreWeave received GB200 NVL72 shipments in November 2024; RunPod announced a waitlist in December 2024. Expect on-demand GB200 pods by Q2 2025 at an estimated $3.50–4.00/hr. Blackwell's 192 GB HBM3e per GPU will fit 400B+ parameter models in 4-bit on a single card.

Conclusion

RunPod turns LLM experimentation from a capital expenditure into an operational one — spin up a 70B model on dual 3090s for $0.34/hour, pay only while prompting, and tear it down when done. The key decisions are template choice (Ollama for simplicity, vLLM for throughput), quantization (always 4-bit AWQ/GPTQ/GGUF for consumer GPUs), and VRAM budgeting (model weights + KV-cache + 20% headroom). With a network volume, your model library persists across sessions, making each subsequent launch a 30-second affair. Start small: deploy an Ollama pod, pull llama3.1:8b, verify the API works, then scale up model size and GPU count as your use case demands.

• Use Ollama + 4-bit GGUF for fastest setup; switch to vLLM + AWQ when you need >5 concurrent users.
• Budget VRAM: model_size_GB + (context_tokens × 0.0009) GB for Llama-3 architecture.
• Persist models on a $0.10/GB/mo network volume to avoid re-downloads.
• Set auto-stop and billing alerts — a forgotten A100 costs $1,200/month.

Sources

• vLLM - Wikipedia
• Hugging Face - Wikipedia
• CoreWeave - Wikipedia