Deploy Local Open Source LLMs on RunPod: Step-by-Step Guide

Running open source large language models locally gives you full data control and zero per-token costs, but consumer GPUs often lack the VRAM for larger models like Llama 3 70B. RunPod solves this by renting NVIDIA GPUs — A100, H100, RTX 4090 — by the second, starting at $0.44/hr for an RTX 3090. This guide walks you from account creation to a production-ready Ollama + Open WebUI stack in under 20 minutes, using only official templates and SSH tunneling so your model never touches the public internet.

Quick Answer: Create a RunPod account, launch a GPU pod using the official Ollama template, SSH into the pod, pull your model with ollama pull llama3:70b, expose the API via SSH tunnel to localhost:11434, and optionally add Open WebUI for a ChatGPT-style interface — all without writing Dockerfiles or managing Kubernetes.

Why RunPod for Local LLM Deployment

Cost Comparison: Cloud GPU vs. Hardware Purchase

An NVIDIA RTX 4090 (24 GB VRAM) retails around $1,800 and runs Llama 3 70B at 4-bit quantization. The same GPU on RunPod costs $0.69/hr. At 40 hours per month, you break even in 6.5 years — ignoring electricity, cooling, and hardware depreciation. For sporadic workloads, per-second billing wins. CoreWeave and Lambda Labs offer similar GPUs but require reserved instances or higher minimums; RunPod's community cloud starts at $0.17/hr for RTX 3090 spot instances.

Data Privacy and Network Isolation

RunPod pods run in your selected region (US-East, EU-West, APAC) behind a private network. You access them via SSH keys — no public IP, no open ports. The Ollama API binds to 127.0.0.1:11434 inside the container; an SSH tunnel forwards localhost:11434 on your machine to that port. Your prompts, embeddings, and model weights never traverse the public internet. This matches the threat model of on-premise deployments while keeping GPU elasticity.

Template Ecosystem Eliminates DevOps

RunPod's template library includes official images for Ollama, vLLM, Text Generation Inference, and ComfyUI. The Ollama template pulls ollama/ollama:latest, exposes port 11434, and mounts /workspace for model persistence across pod restarts. You skip Dockerfile authoring, CUDA version matching, and llama.cpp compile flags — the template maintainers handle those.

Prerequisites and Account Setup

Create RunPod Account and Add Credits

1. Sign up at runpod.io with GitHub or email.
2. Navigate to Settings → Billing → Add Credits. Minimum deposit is $10 via card or crypto.
3. Enable auto-refill at $5 threshold to avoid pod termination mid-inference.

Generate and Upload SSH Key

1. Run ssh-keygen -t ed25519 -C "runpod-llm" locally. Accept defaults.
2. Copy public key: cat ~/.ssh/id_ed25519.pub.
3. In RunPod console, go to SSH Keys → Add Key. Paste and save.

Choose Region and GPU Type

For Llama 3 70B 4-bit (��40 GB), you need 48 GB VRAM minimum — dual RTX 3090 (24 GB × 2) or single A100 80 GB. RTX 4090 (24 GB) runs 7B–13B models comfortably. Spot instances are 40–60% cheaper but can be preempted; use secure cloud for production. Select a region near you for lower SSH latency.

Launch and Configure the Ollama Pod

Deploy from Official Template

1. In RunPod console, click Pods → Deploy → Community Cloud.
2. Search "ollama" and select the template by runpod (verified badge).
3. Choose GPU: RTX 3090 (spot) for testing, A100 80 GB for 70B models.
4. Set container disk to 50 GB (models + quantized weights). Volume disk: 100 GB for /workspace persistence.
5. Attach your SSH key. Click Deploy.

Connect via SSH and Verify Ollama

1. Wait for "Running" status (30–90 seconds). Copy the SSH command from the pod's Connect button: ssh root@ -p -i ~/.ssh/id_ed25519.
2. Inside pod, run ollama --version — expect 0.1.47+ (July 2024).
3. Run ollama list — empty initially. Models download to /root/.ollama/models which is ephemeral; we'll fix persistence next.

Persist Models to Volume Disk

1. Stop pod. Edit pod → Volumes → Mount path: /workspace (already configured by template).
2. Restart pod. SSH in and run:

mkdir -p /workspace/ollama/models
ln -sfn /workspace/ollama/models /root/.ollama/models

Now ollama pull llama3:70b stores weights in /workspace, surviving pod recreation.

Pull, Quantize, and Serve Models

Pull Popular Open Models

1. ollama pull llama3:8b — 4.7 GB, runs on RTX 3090/4090.
2. ollama pull llama3:70b — 40 GB, needs A100 80 GB or dual 3090.
3. ollama pull mistral:7b — 4.1 GB, strong coding benchmarks.
4. ollama pull codellama:13b — 7.3 GB, specialized for code.

Create Custom Quantized Modelfile

Ollama supports GGUF quantization via Modelfile. Example for 4-bit Llama 3 70B:

FROM llama3:70b
QUANTIZE q4_k_m
PARAMETER temperature 0.7
PARAMETER num_ctx 8192

Save as Modelfile, then ollama create llama3-70b-q4 -f Modelfile. The q4_k_m quantization reduces VRAM to ~40 GB with minimal quality loss versus f16.

Benchmark and Optimize Context Length

1. Run ollama run llama3:70b "benchmark prompt" and note tokens/second.
2. Increase num_ctx in Modelfile for longer context (8K, 32K, 128K). Each doubling adds ~VRAM proportional to batch size.
3. For production throughput, consider vLLM template instead — it supports continuous batching and PagedAttention, yielding 2–4× higher tokens/sec on same hardware.

Expose Securely and Add Open WebUI

SSH Tunnel for Localhost-Only Access

1. Local machine: ssh -L 11434:localhost:11434 root@ -p -i ~/.ssh/id_ed25519 -N
2. Keep terminal open. Test: curl http://localhost:11434/api/tags returns model list.
3. Configure your IDE (Continue, Cody) or CLI to use http://localhost:11434 as base URL.

Deploy Open WebUI for Chat Interface

1. In pod: docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main
2. SSH tunnel second port: ssh -L 3000:localhost:3000 ...
3. Open http://localhost:3000, create admin account, select Ollama models from dropdown.

API Authentication and Rate Limiting

Ollama has no built-in auth. For multi-user access, place Open WebUI behind OAuth2-Proxy (Google/GitHub SSO) or use RunPod's new pod-level firewall rules (beta 2024) to restrict SSH source IPs. Rate limit at nginx layer if exposing beyond localhost.

Comparison: RunPod vs. Alternatives for LLM Hosting

Below compares GPU cloud providers on price, model persistence, template ecosystem, and network isolation for open source LLM workloads.

All prices are on-demand USD/hr for single GPU as of July 2024; spot/preemptible rates in parentheses.

Provider	RTX 3090 (24 GB)	A100 80 GB	Model Persistence	Templates	Private Network
RunPod	$0.44 ($0.17 spot)	$1.89 ($1.19 spot)	Volume mount /workspace	Ollama, vLLM, TGI, ComfyUI	Yes, SSH only
Lambda Labs	$0.50 (no spot)	$2.49 (no spot)	Persistent home dir	Custom Docker only	Yes, VPN option
CoreWeave	N/A	$2.21 (reserved)	Kubernetes PVC	vLLM, TGI via K8s	VPC peering
Vast.ai	$0.35 ($0.12 spot)	$1.65 ($0.98 spot)	Manual volume mgmt	Community templates	SSH tunnel
Google Cloud (A2)	N/A	$3.67 (preemptible $1.10)	GCS FUSE / Filestore	Vertex AI / GKE	VPC

Common Mistakes and Pro Tips

Mistake: Using Ephemeral Container Disk for Models

Why It Hurts: Pod restart or GPU swap wipes /root/.ollama, forcing re-download of 40 GB weights.

Fix: Symlink /root/.ollama/models → /workspace/ollama/models before first pull. Verify with ls -la /workspace/ollama/models after download.

Mistake: Under-Provisioning VRAM for Quantization

Why It Hurts: Llama 3 70B at 4-bit needs ~40 GB VRAM + overhead. RTX 4090 (24 GB) OOMs at load time.

Fix: Use ollama run --verbose to see actual VRAM allocation. For 70B, rent A100 80 GB or dual 3090 pod. For 8B/13B, single 24 GB GPU suffices.

Mistake: Exposing Ollama Port Publicly

Why It Hurts: Shodan scans find open port 11434; attackers pull proprietary models or run up your GPU bill.

Fix: Never check "HTTP" exposure in RunPod pod settings. Use SSH tunnel exclusively. Verify with nmap -p 11434 — must show closed/filtered.

Mistake: Ignoring Spot Preemption Mid-Inference

Why It Hurts: Spot pods can be reclaimed with 30-second warning. Long generations (128K context) get killed.

Fix: Use secure cloud for production. For spot, checkpoint long conversations via Open WebUI export; script auto-relaunch with runpodctl on preemption webhook.

Pro Tips

• Pre-warm models: Add ollama run llama3:70b "" to pod start script (/etc/rc.local) so first inference isn't cold.
• Use aria2 for faster pulls: aria2c -x 16 -s 16 $(ollama show llama3:70b --format '{{.Layers}}' | grep -o 'https://[^ ]*') downloads layers in parallel.
• Monitor GPU utilization: watch -n 1 nvidia-smi in second SSH session. Target 90%+ compute utilization; below 50% means batch size too small.
• Automate pod lifecycle: runpodctl create pod --gpu-type "RTX 3090" --template ollama --name llm-prod in CI/CD for reproducible environments.
• Combine with RAG: Mount /workspace/data for PDFs, run ollama create rag-model -f Modelfile with PARAMETER num_ctx 32768 for local retrieval-augmented generation.

FAQ

What is the minimum GPU VRAM to run Llama 3 8B?

Llama 3 8B at 4-bit quantization requires approximately 6 GB VRAM. An RTX 3060 12 GB or RTX 4060 8 GB handles it comfortably with headroom for context window. On RunPod, the cheapest option is RTX 3090 spot at $0.17/hr, which provides 24 GB — overkill but cheapest per hour.

How does RunPod pricing compare to buying an RTX 4090 for home use?

An RTX 4090 costs ~$1,800 plus ~$200/yr electricity at $0.12/kWh (300W × 24/7). RunPod RTX 4090 at $0.69/hr equals $6,043/yr if run continuously. For workloads under 300 hours/month, RunPod is cheaper; above that, buying wins. Spot instances shift breakeven to ~500 hours/month.

Can I run multiple models simultaneously on one pod?

Yes, but they share VRAM. Two 8B models (6 GB each) fit on a 24 GB GPU with careful num_ctx tuning. Ollama loads models on-demand and unloads after 5 minutes of inactivity (configurable via OLLAMA_KEEP_ALIVE). For concurrent serving, vLLM's PagedAttention handles multi-model better.

What happens to my data if RunPod has an outage?

Pod data on /workspace (volume disk) persists across host failures. RunPod's SLA offers 99.9% uptime for secure cloud; community cloud has no SLA. For critical workloads, replicate /workspace to S3-compatible storage (MinIO, Backblaze B2) via cron job: rclone sync /workspace b2:runpod-backup.

Will RunPod support NVIDIA Blackwell (GB200) GPUs?

RunPod typically adds new GPU generations within 30–60 days of general availability. CoreWeave received first GB200 NVL72 shipments November 2024. Expect RunPod GB200 availability by Q1 2025. Blackwell's 192 GB HBM3e per GPU will enable single-GPU Llama 3 405B inference.

Conclusion

RunPod turns open source LLM deployment from a hardware procurement project into a 20-minute operational task. The official Ollama template, per-second billing, and SSH-only networking give you a private, scalable inference endpoint without Kubernetes expertise. Persist models to /workspace, tunnel API via SSH, and add Open WebUI for a polished interface. For teams, script pod creation with runpodctl and version-control Modelfiles alongside application code. The same workflow scales from 7B prototypes to 70B production by swapping GPU type — no code changes required.

• Start with RTX 3090 spot ($0.17/hr) for 7B–13B models; upgrade to A100 80 GB for 70B+.
• Always symlink /root/.ollama/models to /workspace before first pull.
• SSH tunnel is the only secure exposure method — never open port 11434 publicly.
• Automate with runpodctl and Modelfile version control for reproducible deployments.

Sources

• Ollama - Wikipedia
• CoreWeave - Wikipedia
• Cloud computing service models - Wikipedia
• Intel Capital investments (RunPod) - Wikipedia