Deploy Local Open Source LLMs on RunPod: Python Step-by-Step Guide

Running open-source large language models locally used to require expensive hardware — NVIDIA A100 GPUs cost $10,000+ each as of 2024, putting serious LLM development out of reach for most developers. RunPod changed this by offering per-second GPU rentals starting at $0.17/hour for RTX 3090 instances, with no long-term contracts. This guide walks you through deploying any Hugging Face model on RunPod using Python, from account setup to production-ready inference endpoints, with real commands you can copy-paste.

Quick Answer: Create a RunPod account, install the runpod Python SDK, configure your API key, create a pod with your chosen GPU and Docker image (like pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime), SSH in, install vLLM or TGI, load your model from Hugging Face, and expose an API endpoint — all in under 30 minutes for roughly $0.50–$2.00/hour depending on GPU choice.

Why RunPod for Local LLM Deployment

Cost Comparison: Own vs Rent

Buying an RTX 4090 (24 GB VRAM) costs $1,600–$2,000 upfront plus electricity (~$0.12/kWh). At 450W sustained load, that's $0.054/hour just in power. RunPod's RTX 4090 pods cost $0.69/hour — you'd need 30,000+ hours (3.4 years continuous) to break even on hardware alone. For intermittent workloads like fine-tuning or batch inference, renting wins decisively. RunPod also offers A100 80GB at $1.64/hour and H100 at $4.69/hour (2024 pricing), covering every model size from 7B to 70B+ parameters.

No Infrastructure Maintenance

RunPod handles GPU driver updates, CUDA version compatibility, Docker runtime, and network configuration. You get a fresh Ubuntu 22.04 environment with NVIDIA drivers pre-installed. The platform supports both secure cloud (shared tenancy) and community cloud (lower cost, preemptible) tiers. Community cloud RTX 3090 pods drop to $0.17/hour — ideal for experimentation.

Python-Native Workflow

The runpod Python SDK (pip install runpod) lets you manage pods programmatically: create, start, stop, terminate, and retrieve logs. Combined with SSH tunneling and vLLM's OpenAI-compatible API server, you can treat a remote GPU exactly like a local inference engine — your Python code barely changes.

Prerequisites and Account Setup

Create RunPod Account and API Key

1. Sign up at runpod.io using GitHub, Google, or email
2. Navigate to Settings > API Keys > Create New Key
3. Name it (e.g., "llm-deployment") and copy the key — you won't see it again
4. Add billing: minimum $10 credit via card or crypto

Local Python Environment

Python 3.10+ recommended. Create a virtual environment and install dependencies:

python -m venv runpod-llm
source runpod-llm/bin/activate
pip install runpod requests python-dotenv

SSH Key Configuration

RunPod requires SSH keys for pod access. Generate if needed:

ssh-keygen -t ed25519 -C "runpod-llm"
cat ~/.ssh/id_ed25519.pub

Copy the output, then in RunPod dashboard: Settings > SSH Keys > Add Key. Paste the public key.

Creating and Configuring Your Pod

Choose the Right GPU for Your Model

Model Size	Minimum VRAM (4-bit)	Recommended RunPod GPU	Hourly Cost
7B params	6 GB	RTX 3090 / 4090 (24 GB)	$0.17–$0.69
13B params	10 GB	RTX 4090 (24 GB) / A10G (24 GB)	$0.69–$0.76
30B–34B params	20 GB	A100 40GB / A100 80GB	$1.19–$1.64
70B params	40 GB	A100 80GB / H100 80GB	$1.64–$4.69
Multiple 7B concurrent	24 GB+	RTX 4090 / A100 40GB	$0.69–$1.19

VRAM estimates assume 4-bit quantization (GGUF/GPTQ/AWQ). FP16 needs 2x VRAM. Community cloud prices shown first; secure cloud ~30% higher.

Select Base Docker Image

Use official PyTorch images with CUDA pre-installed. For most LLMs in 2024:

• pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime (PyTorch 2.1, CUDA 12.1)
• pytorch/pytorch:2.2.0-cuda12.1-cudnn8-runtime (PyTorch 2.2, CUDA 12.1)
• nvidia/cuda:12.1-runtime-ubuntu22.04 (CUDA only, install PyTorch manually)

Avoid "devel" images — they're 5–10 GB larger with build tools you won't need.

Create Pod via Python SDK

import runpod
import os

runpod.api_key = os.getenv("RUNPOD_API_KEY")

pod = runpod.create_pod(
    name="llm-inference",
    image_name="pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime",
    gpu_type_id="NVIDIA RTX A6000",  # or "NVIDIA RTX 4090", "NVIDIA A100 80GB"
    cloud_type="COMMUNITY",           # or "SECURE"
    gpu_count=1,
    volume_in_gb=50,                  # model storage
    container_disk_in_gb=20,          # OS + packages
    ports="8000/http,22/tcp",         # vLLM API + SSH
    env={"HF_TOKEN": os.getenv("HF_TOKEN")}  # for gated models
)

print(f"Pod ID: {pod['id']}")
print(f"Status: {pod['desiredStatus']}")

Run this script. Pod spins up in 30–90 seconds. Save the pod ID for later management.

Setting Up the Inference Server

SSH Into the Pod

Once pod status shows "RUNNING", get connection details:

pod = runpod.get_pod(pod_id)
ssh_cmd = f"ssh root@{pod['runtime']['ip']} -p {pod['runtime']['ports'][0]['publicPort']}"
print(ssh_cmd)

Copy-paste the printed command. First connection prompts for host verification — type "yes".

Install vLLM (Recommended for Production)

vLLM delivers 2–5x throughput over Hugging Face's generate() via PagedAttention and continuous batching. Install inside the pod:

pip install vllm==0.3.3  # pinned version for stability
# or for latest: pip install vllm

vLLM 0.3.3 requires CUDA 12.1+ and Python 3.10–3.11. If you hit wheel errors, install flash-attn first: pip install flash-attn --no-build-isolation.

Alternative: Text Generation Inference (TGI)

Hugging Face's TGI excels at quantization (bitsandbytes, GPTQ, AWQ) and streaming. Run via Docker:

docker run --gpus all --shm-size 1g -p 8000:80 \
  -v $PWD/data:/data ghcr.io/huggingface/text-generation-inference:1.4 \
  --model-id meta-llama/Meta-Llama-3-8B-Instruct \
  --quantize bitsandbytes-nf4

TGI 1.4 supports Llama 3, Qwen 2, Phi-3, Gemma, and Mistral families out of the box.

Launch the Model Server

For vLLM with a 7B model (example: Mistral-7B-Instruct-v0.2):

python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.2 \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype auto \
  --quantization awq \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.9

Flags explained: --quantization awq loads 4-bit AWQ weights (saves 75% VRAM), --max-model-len 8192 sets context window, --gpu-memory-utilization 0.9 leaves 10% headroom for KV cache. Server starts in 10–30 seconds depending on model size.

Testing and Production Hardening

Verify Inference Works

From your local machine (not the pod), test the OpenAI-compatible endpoint:

import requests

POD_IP = "your-pod-ip"
POD_PORT = "mapped-port"  # from runpod.get_pod() runtime.ports

response = requests.post(
    f"http://{POD_IP}:{POD_PORT}/v1/chat/completions",
    json={
        "model": "mistralai/Mistral-7B-Instruct-v0.2",
        "messages": [{"role": "user", "content": "Explain quantum computing in 3 sentences."}],
        "temperature": 0.7,
        "max_tokens": 200
    }
)
print(response.json()["choices"][0]["message"]["content"])

Expect 20–50 tokens/second on RTX 4090 for 7B models. First request slower (model load + KV cache warmup).

Add Authentication and Rate Limiting

vLLM supports API keys via --api-key YOUR_SECRET. For production, put nginx in front:

# /etc/nginx/sites-available/vllm
server {
    listen 80;
    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Authorization "Bearer YOUR_API_KEY";
        limit_req zone=api burst=20 nodelay;
    }
}
# Run: nginx -t && systemctl reload nginx

Create limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s; in http block.

Monitor GPU Utilization

Inside pod: watch -n 1 nvidia-smi. Target 85–95% GPU compute utilization. If below 50%, increase batch size or concurrent requests. vLLM's --max-num-batched-tokens and --max-num-seqs tune this.

Common Mistakes and Pro Tips

Mistake: Underestimating Volume Storage Needs

Why it hurts: Default 10–20 GB container disk fills fast — model weights (7B AWQ = ~4 GB), Docker layers, pip cache, and logs. Pod crashes with "no space left on device."

Fix: Allocate 50 GB volume for models, 20 GB container disk minimum. Mount volume at /workspace and set HF_HOME=/workspace/hf_cache.

Mistake: Using FP16 Instead of Quantized Weights

Why it hurts: Llama-3-8B FP16 needs 16 GB VRAM — barely fits on 24 GB GPU with zero headroom for KV cache. OOM crashes on first real request.

Fix: Always use 4-bit quantization (AWQ, GPTQ, or bitsandbytes NF4). AWQ models from TheBloke/ or casperhansen/ on Hugging Face drop VRAM to 6 GB with <1% quality loss.

Mistake: Forgetting to Stop Pods

Why it hurts: Stopped pods still incur storage costs ($0.05/GB/month). Forgotten $0.69/hour pods = $500+/month surprise bills.

Fix: Add cleanup script with cron or use runpod.terminate_pod(pod_id) in your workflow. Set billing alerts at $25/$50/$100 in dashboard.

Mistake: Ignoring CUDA Version Mismatch

Why it hurts: PyTorch 2.1 wheels target CUDA 11.8 or 12.1. RunPod's A100s ship with 535 drivers (CUDA 12.2). Mismatch = "CUDA driver version insufficient" errors.

Fix: Pin pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime exactly. Verify inside pod: python -c "import torch; print(torch.version.cuda)" must match driver.

Pro Tips

• Pre-download models to volume: Run huggingface-cli download mistralai/Mistral-7B-Instruct-v0.2 --local-dir /workspace/models/mistral-7b once, then point vLLM to local path — avoids re-download on pod restart.
• Use community cloud for dev, secure for prod: Community pods can be preempted (30s notice). Run long fine-tunes on secure; use community for inference testing at 60% discount.
• Enable vLLM prefix caching: Add --enable-prefix-caching for repeated system prompts — 30–50% latency reduction on multi-turn conversations.
• Batch requests client-side: Send arrays of prompts in one HTTP call; vLLM continuous batching processes them in one forward pass — 3–5x throughput vs sequential calls.
• Snapshot volumes for reproducibility: RunPod volumes persist across pod termination. Create a "golden image" volume with all models + deps, then clone for new pods in seconds.

FAQ

What's the cheapest GPU that runs a 7B parameter model?

Community cloud RTX 3090 (24 GB VRAM) at $0.17/hour runs any 7B model at 4-bit quantization with 8K context. RTX 4090 at $0.69/hour delivers 2.5x faster tokens/second. For pure cost-per-token, 3090 wins; for latency-sensitive apps, 4090 pays for itself.

RunPod vs Lambda Labs vs vast.ai — which is best for LLMs?

RunPod: best Python SDK, per-second billing, largest GPU selection (including H100), reliable networking. Lambda Labs: cheaper A100 40GB ($0.90/hour), but limited stock, no Python SDK, hourly minimums. vast.ai: cheapest raw metal ($0.10–$0.30/hour for 3090), but consumer-grade hardware, no SLA, manual Docker setup. RunPod wins for developer experience.

How do I deploy a fine-tuned LoRA adapter on RunPod?

Save LoRA weights to your persistent volume during training. At inference, pass --lora-modules lora_name=/workspace/adapters/my-lora to vLLM, or load via model.load_adapter() in TGI. LoRA adds ~1% VRAM overhead. Merge before serving for production: peft merge creates standalone model.

Why does my pod show "STARTING" for over 5 minutes?

Usually Docker image pull timeout on large images (>15 GB). Check pod logs in dashboard — "pulling image" stuck means registry rate limit. Fix: use smaller base image (CUDA runtime not devel), or pre-bake custom image on Docker Hub with models baked in. Community cloud pods also queue during high demand.

Will RunPod support AMD MI300X or Intel Gaudi GPUs?

As of 2024, RunPod offers NVIDIA only. AMD MI300X support announced for H2 2024 via ROCm 6.0+ partnership. Intel Gaudi 2/3 not on roadmap. For now, NVIDIA CUDA ecosystem (vLLM, TGI, TensorRT-LLM) remains the most mature inference stack — sticking with NVIDIA avoids framework porting pain.

Conclusion

Deploying open-source LLMs on RunPod with Python gives you production-grade GPU inference without hardware capital expenditure. The workflow — create pod via SDK, SSH in, launch vLLM or TGI, hit the OpenAI-compatible endpoint — takes 30 minutes end-to-end and costs pennies per hour. Key success factors: pick quantized models (AWQ/GPTQ) to fit VRAM, allocate sufficient persistent volume storage, and terminate pods when idle. With vLLM's continuous batching and RunPod's per-second billing, you can serve thousands of requests/day for under $10/month — a fraction of API provider costs.

• Use 4-bit quantized models (AWQ/GPTQ) — they fit 7B–13B on $0.17–$0.69/hour GPUs with minimal quality loss
• Prefer vLLM for throughput, TGI for quantization flexibility — both expose OpenAI-compatible APIs
• Always mount a 50 GB+ persistent volume at /workspace for models and caches
• Automate pod lifecycle (create/terminate) via runpod Python SDK to avoid surprise bills

Sources

• Large language model - Wikipedia
• Python (programming language) - Wikipedia
• RunPod Official Documentation
• vLLM Official Documentation
• Text Generation Inference - Hugging Face