Deploy Local Open Source LLMs on RunPod: Complete Step-by-Step Guide

Open-source AI adoption surged 67% in 2023 as developers flee vendor lock-in and $0.06/1K token API fees. Running Llama 3 or Mistral locally sounds appealing until your RTX 3090 hits VRAM limits and thermal throttling kills throughput. RunPod solves this: spin up A100s at $1.19/hr, pay per second, shut down when done. This guide walks you from zero to production inference endpoint in under 30 minutes — no Kubernetes, no Terraform, just Docker and a few CLI commands.

Quick Answer: Create a RunPod account, launch a GPU pod with the "RunPod PyTorch" template, SSH in, pull your model from Hugging Face using `huggingface-cli`, quantize with `llama.cpp` or `AutoGPTQ`, then serve via `vLLM` or `text-generation-inference` on port 8000. Total setup: 15 minutes, ~$0.50 for a test run on an A10G.

Why RunPod for Local LLM Deployment

Cost-Performance Beats Colab and Vast.ai

Google Colab Pro+ caps at 52GB VRAM on A100s with frequent preemption. Vast.ai offers cheaper spot instances ($0.20/hr for 3090s) but reliability drops below 85% uptime. RunPod's secure cloud tier guarantees 99.9% uptime on A100 80GB at $1.19/hr — 40% cheaper than AWS p4d.24xlarge on-demand. You pay per second with no minimum, so a 20-minute quantization job costs $0.40.

Template System Eliminates Dependency Hell

RunPod's "RunPod PyTorch 2.1.0 + CUDA 12.1" template ships with Python 3.10, PyTorch 2.1, CUDA 12.1, cuDNN 8.9, and `bitsandbytes` precompiled. Docker (released March 2013 per Wikipedia) handles containerization so your environment reproduces identically across pods. No more debugging `ninja` build errors on `flash-attn` at 2 AM.

Persistent Volumes Survive Pod Restarts

Attach a 50GB network volume ($0.10/GB/month) to `/workspace`. Models, quantized weights, and LoRA adapters persist across pod terminations. Spin down Friday, spin up Monday — zero re-download time. A 70B 4-bit model (39GB) loads in 90 seconds from volume vs 18 minutes from Hugging Face Hub.

Prerequisites and Account Setup

Create RunPod Account and Add Credits

1. Sign up at runpod.io with GitHub or email — no credit card for $10 free tier.
2. Add $25 via credit card or crypto to unlock A100/H100 access (free tier limited to RTX 3090/4090).
3. Generate API key in Settings → API Keys for CLI automation later.

Install Local Tooling

1. Install `runpodctl`: `brew install runpodctl` (macOS) or download Linux binary from GitHub releases.
2. Configure: `runpodctl config set api-key YOUR_KEY`.
3. Verify: `runpodctl get gpus` — should list A100 80GB at $1.19/hr.

SSH Key Setup for Passwordless Access

1. Generate key: `ssh-keygen -t ed25519 -C "runpod-llm" -f ~/.ssh/runpod_ed25519`.
2. Add public key to RunPod: Settings → SSH Keys → Paste `cat ~/.ssh/runpod_ed25519.pub`.
3. Test: `ssh -i ~/.ssh/runpod_ed25519 root@POD_IP` after first pod launch.

Launch Your First GPU Pod

Select the Right Template and GPU

1. Dashboard → Pods → Deploy → "RunPod PyTorch 2.1.0 + CUDA 12.1" (Ubuntu 22.04 base).
2. GPU: A100 80GB PCIe ($1.19/hr) for 70B models, A10G 24GB ($0.49/hr) for 7B-13B quantized.
3. Container Disk: 20GB (ephemeral, for OS + temp). Volume: 100GB network volume mounted at `/workspace` ($10/mo).
4. Ports: Expose 8000 (vLLM), 8080 (TGI), 8888 (Jupyter), 22 (SSH).
5. Deploy — pod boots in 30-60 seconds.

SSH In and Verify Environment

1. Copy pod IP from dashboard: `ssh -i ~/.ssh/runpod_ed25519 root@`.
2. Run verification: `nvidia-smi` (shows GPU), `python -c "import torch; print(torch.cuda.is_available())"` → True.
3. Check disk: `df -h /workspace` — confirms volume mount.

Install Inference Stack

1. Update and install: `apt update && apt install -y git wget curl`.
2. Install vLLM (fastest throughput): `pip install vllm --no-cache-dir`.
3. Install llama.cpp for quantization: `pip install llama-cpp-python[server] --no-cache-dir`.
4. Install Hugging Face CLI: `pip install huggingface_hub[cli] --no-cache-dir`.

Download, Quantize, and Serve Your Model

Pull Model from Hugging Face

Hugging Face (founded 2016 per Wikipedia) hosts 500K+ models. Use their CLI for authenticated downloads:

1. Login: `huggingface-cli login` — paste token from hf.co/settings/tokens.
2. Download 7B model: `huggingface-cli download meta-llama/Meta-Llama-3-8B-Instruct --local-dir /workspace/models/llama-3-8b-instruct`.
3. Download 70B model: `huggingface-cli download meta-llama/Meta-Llama-3-70B-Instruct --local-dir /workspace/models/llama-3-70b-instruct`.

Quantize to 4-bit with AWQ or GPTQ

4-bit quantization cuts VRAM 75% with <1% perplexity loss. AWQ preferred for vLLM:

1. Install AutoAWQ: `pip install autoawq --no-cache-dir`.
2. Quantize 8B: `python -m awq.entry --model_path /workspace/models/llama-3-8b-instruct --quant_path /workspace/models/llama-3-8b-instruct-awq --w_bit 4 --q_group_size 128 --version GEMM`.
3. Quantize 70B (needs A100 80GB): `python -m awq.entry --model_path /workspace/models/llama-3-70b-instruct --quant_path /workspace/models/llama-3-70b-instruct-awq --w_bit 4 --q_group_size 128 --version GEMM`.
4. Verify: `ls -lh /workspace/models/llama-3-70b-instruct-awq/` — should show ~39GB vs 140GB FP16.

Launch vLLM OpenAI-Compatible API Server

1. Start server: `python -m vllm.entrypoints.openai.api_server --model /workspace/models/llama-3-70b-instruct-awq --tensor-parallel-size 1 --gpu-memory-utilization 0.9 --port 8000 --host 0.0.0.0`.
2. Test locally: `curl -X POST http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "llama-3-70b", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 50}'`.
3. Expect 45-60 tokens/sec on A100 80GB for 70B 4-bit.

Production Hardening and Automation

Secure the Endpoint with API Key

1. Generate key: `openssl rand -hex 32` → save as `VLLM_API_KEY`.
2. Restart vLLM with auth: `--api-key $VLLM_API_KEY`.
3. Client usage: `Authorization: Bearer $VLLM_API_KEY` header.

Auto-Start on Pod Boot via Systemd

1. Create service: `cat > /etc/systemd/system/vllm.service << 'EOF'\n[Unit]\nDescription=vLLM API Server\nAfter=network.target\n[Service]\nType=simple\nUser=root\nWorkingDirectory=/workspace\nExecStart=/usr/local/bin/python -m vllm.entrypoints.openai.api_server --model /workspace/models/llama-3-70b-instruct-awq --tensor-parallel-size 1 --gpu-memory-utilization 0.9 --port 8000 --host 0.0.0.0 --api-key $VLLM_API_KEY\nRestart=always\nRestartSec=10\nEnvironment=VLLM_API_KEY=your_key_here\n[Install]\nWantedBy=multi-user.target\nEOF`
2. Enable: `systemctl daemon-reload && systemctl enable vllm && systemctl start vllm`.

Automate Pod Lifecycle with runpodctl

1. Start pod: `runpodctl start pod POD_ID`.
2. Wait for ready: `runpodctl wait pod POD_ID --timeout 120`.
3. Stop when idle: `runpodctl stop pod POD_ID` — saves $28/day on A100.
4. Cron job example: stop at 2 AM UTC, start at 6 AM UTC for dev workloads.

Comparison: RunPod vs Alternatives for LLM Inference

Choosing the right GPU cloud depends on model size, budget, and reliability needs. The table below compares real pricing and specs as of 2024.

All prices are on-demand per hour; spot/preemptible prices shown where available.

Platform	A100 80GB Price/hr	Key Differentiator
RunPod Secure Cloud	$1.19	Per-second billing, 99.9% uptime, persistent volumes, template system
AWS p4d.24xlarge	$3.06	Enterprise SLAs, integrated ecosystem, 8x A100 per instance
Lambda Labs	$1.10	Slightly cheaper, no per-second billing (1-hr minimum), fewer templates
Vast.ai (spot)	$0.65	Cheapest, but ~85% reliability, no guaranteed uptime, manual Docker setup
Google Colab Pro+	~$0.48*	*Effective rate; preemption frequent, 52GB VRAM cap, 12-hr session limit
Hugging Face Inference Endpoints	$1.50	Managed, auto-scaling, but vendor lock-in, higher cost at scale

Common Mistakes and Pro Tips

Mistake 1: Using FP16 Instead of 4-bit Quantization

Why It Hurts: 70B FP16 needs 140GB VRAM — requires 2x A100 80GB ($2.38/hr) with tensor parallelism. 4-bit AWQ fits on one A100 80GB at $1.19/hr with negligible quality loss.

Fix: Always quantize to 4-bit AWQ for vLLM or GPTQ for TGI. Use `--w_bit 4 --q_group_size 128` for best quality/size tradeoff.

Mistake 2: Skipping Persistent Volumes

Why It Hurts: Re-downloading 70B model (140GB) takes 18 minutes on 1 Gbps link. At $1.19/hr, that's $0.36 per boot — $10.80/month wasted.

Fix: Attach 100GB network volume ($10/mo) at `/workspace`. Models survive pod termination.

Mistake 3: Ignoring GPU Memory Utilization Tuning

Why It Hurts: Default `--gpu-memory-utilization 0.9` leaves 8GB headroom on 80GB. For 70B 4-bit (39GB), you can safely push to 0.95, enabling longer context (32K vs 8K).

Fix: Test incremental increases: `--gpu-memory-utilization 0.93` → 0.95. Monitor for OOM in logs.

Mistake 4: Running Without API Key Authentication

Why It Hurts: Exposed port 8000 on public IP invites abuse. Crypto miners and prompt injectors scan cloud ranges hourly.

Fix: Always use `--api-key` in vLLM. Rotate keys monthly via systemd environment variable.

Pro Tips

• Use FlashAttention-2: Add `--enable-prefix-caching --enable-chunked-prefill` to vLLM for 2-3x throughput on long contexts.
• Enable Prometheus Metrics: `--enable-metrics` exposes `/metrics` for Grafana dashboards — track tokens/sec, queue depth, KV cache usage.
• Batch Inference for Throughput: Send 8-16 requests concurrently. vLLM's continuous batching yields 45 tok/s/request vs 12 tok/s sequential.
• Warm Up on Boot: Add `curl -X POST ...` in `ExecStartPost` to trigger model load before first real request avoids 30s cold start.
• Monitor VRAM with nvitop: `pip install nvitop` → run `nvitop` in tmux. Watch for memory leaks during long-running servers.

FAQ

What is the minimum GPU VRAM for running Llama 3 8B?

8B 4-bit quantized needs 6GB VRAM. An RTX 3060 12GB ($0.20/hr on RunPod community cloud) runs it comfortably with 4K context. For 8K+ context or concurrent requests, step up to 24GB (A10G at $0.49/hr).

How does RunPod compare to Vast.ai for reliability?

RunPod secure cloud offers 99.9% uptime SLA with enterprise-grade data centers. Vast.ai spot instances average 85% reliability with frequent preemption. For production workloads, RunPod's $0.54/hr premium over Vast.ai spot pays for itself in avoided downtime.

Can I run multiple models on one A100 80GB?

Yes. 70B 4-bit (39GB) + 8B 4-bit (5GB) + 3B 4-bit (2GB) = 46GB, leaving headroom for KV cache. Use vLLM's `--model` flag with multiple paths or run separate vLLM instances on ports 8000, 8001, 8002 with `--gpu-memory-utilization 0.3` each.

Why does my vLLM server OOM after 2 hours?

Likely KV cache memory leak from unbounded context growth. Fix: set `--max-model-len 8192` (or your max context), enable `--enable-chunked-prefill --max-num-batched-tokens 8192`, and restart daily via systemd `RestartSec=86400`.

Will RunPod support H100 and Blackwell GPUs?

RunPod launched H100 80GB at $2.69/hr in Q1 2024. Blackwell (B100/B200) support typically follows NVIDIA general availability by 4-6 weeks. Check runpod.io/gpus for latest offerings — they add new GPUs faster than AWS/GCP.

Conclusion

Deploying open-source LLMs on RunPod transforms a weekend project into a production API in 30 minutes. The template system eliminates dependency hell, persistent volumes survive pod cycles, and per-second billing on A100 80GB at $1.19/hr makes 70B inference cheaper than any managed endpoint. Quantize to 4-bit AWQ, serve with vLLM, secure with API keys, automate with runpodctl — that's the entire stack. No Kubernetes, no YAML, no surprise bills. Start with an 8B model on A10G ($0.49/hr), validate your pipeline, then scale to 70B on A100 when traffic demands it.

• Key takeaway: 4-bit quantization + A100 80GB = 70B models on single GPU at $1.19/hr
• Key takeaway: Persistent volumes at $10/mo eliminate 18-minute model downloads per boot
• Key takeaway: vLLM + systemd + runpodctl = production-grade API with zero DevOps overhead
• Key takeaway: Always secure with API keys and monitor VRAM — exposure costs far exceed GPU spend

Sources

• Open-source artificial intelligence - Wikipedia
• Docker (software) - Wikipedia
• Hugging Face - Wikipedia