Consistent Character Images With Open Source AI Tools

If you've ever tried generating the same character twice with AI, you know the frustration: different face, different outfit, different vibe. Between August 2022 (when Stable Diffusion launched) and early 2024, this was the single biggest blocker for indie creators, game developers, and comic artists using open source tools. The good news? Three proven techniques — DreamBooth, LoRA fine-tuning, and ControlNet conditioning — now solve this reliably, all running locally on consumer GPUs with as little as 6GB VRAM. This guide walks you through each method with real workflows, exact tools, and the mistakes to skip.

Quick Answer: Use DreamBooth or LoRA fine-tuning on Stable Diffusion via Automatic1111 or ComfyUI to train a character into the model. For zero-training consistency, chain ControlNet (pose, depth, canny) with consistent seeds and a fixed base model. Both run locally on any NVIDIA GPU with 6GB+ VRAM using free open source tools.

Why Open Source Tools Win for Character Consistency

Proprietary services like DALL-E 3 (OpenAI, 2023) and Midjourney produce stunning single images, but they lack fine-grained control over identity. You cannot train a specific face into Midjourney unless you pay for personalized tiers — and even then, the model changes weekly. Open source changes that calculus entirely.

Stable Diffusion, released in August 2022 by Stability AI in collaboration with the CompVis Group at LMU Munich and Runway, was the first major text-to-image model whose code and weights were released publicly. According to the official documentation, an optimized version can run on consumer hardware with as little as 2.4 GB VRAM. This meant anyone with a gaming GPU could fine-tune their own model.

The three main open source interfaces — Automatic1111 (released August 22, 2022, now with 136,000+ GitHub stars), ComfyUI (released January 2023, 89,200+ stars), and Stable Diffusion WebUI Forge (a fork by Lvmin Zhang, creator of ControlNet) — all support DreamBooth, LoRA, and ControlNet natively. No subscriptions, no API costs, no rate limits.

Why Proprietary Models Fail at Consistency

Midjourney and DALL-E 3 generate each image from scratch with zero memory of prior outputs. The model has no internal "character sheet." You can describe the same person with identical prompts and get wildly different results. Open source tools solve this by letting you train identity into the weights or condition every generation on a reference image.

Method 1: DreamBooth Fine-Tuning (Full Character Training)

DreamBooth was introduced in 2022 by researchers from Google Research and Boston University. The original paper by Ruiz et al. demonstrated that fine-tuning the full UNet component of a diffusion model on just 3–5 images of a subject could teach the model to render that subject in novel poses, settings, and styles. It was originally built for Google's Imagen, but the open source community quickly adapted it for Stable Diffusion.

DreamBooth works by using a technique called class-specific prior preservation loss. You feed it 4–6 images of your character, label them with a unique identifier (e.g., "sks [class]"), and the model learns to associate that token with the subject's visual identity while retaining its general knowledge of the class (e.g., "person" or "cat").

Step-by-Step DreamBooth Workflow

1. Install Automatic1111 via the one-click installer (Windows) or manual Git clone (macOS/Linux). Requires Python 3.10 and PyTorch.
2. Activate the DreamBooth extension from the Extensions tab in Automatic1111 or use the dedicated Dreambooth tab.
3. Prepare your dataset: 5–8 high-quality images of the character, cropped to 512x512 or 768x768 resolution. Include front-facing, profile, and ¾-angle shots. Avoid heavy makeup, occlusion, or filters.
4. Set training parameters: 100–150 training steps per image, learning rate 1e-6, 50% prior preservation loss. Use a class name like "woman" or "man" for prior preservation images.
5. Train for 20–30 minutes on an NVIDIA RTX 3060 (12GB VRAM). Forge fork users with 6GB cards can use low-VRAM optimizations that improve speed by 60–75%.
6. Test inference: Prompt with "sks woman in a spacesuit, photorealistic, bright lighting" — the character identity should carry across.

Real Example

Indie game developer "NeonFrost" trained a DreamBooth model on 6 renders of their protagonist "Lyra" using SD 1.5 on an RTX 3070. After 180 training steps with a prior preservation class of "female warrior," they generated 200 consistent frames for a 10-second game trailer clip. Every frame maintained Lyra's distinctive scar pattern and eye color.

Method 2: LoRA Training (Lightweight, Fast, Modular)

LoRA (Low-Rank Adaptation) was introduced in 2021 by Microsoft researchers. Unlike DreamBooth, which fine-tunes the entire UNet (860 million parameters in SD 1.5), LoRA injects small rank-decomposition matrices into each Transformer layer. For GPT-3 scale models, this reduced trainable parameters by approximately 10,000 times — from 175 billion to roughly 18 million. When applied to Stable Diffusion, LoRA produces weight files as small as 4–30 MB vs. DreamBooth's 2+ GB checkpoints.

This makes LoRA the preferred choice for most creators. You can train a character LoRA in 10–15 minutes on a 6GB GPU, swap between multiple characters by loading different LoRAs, and share your LoRA file online without uploading a full model.

Step-by-Step LoRA Workflow

1. Choose your UI: Both Automatic1111 and ComfyUI support LoRA natively. In ComfyUI, LoRA appears as a node you load between the checkpoint and the CLIP text encoder.
2. Collect 10–20 character images. LoRA benefits from more varied data than DreamBooth. Include different angles, expressions, and lighting conditions.
3. Tag your images using a captioning tool like WD14 Tagger (built into Automatic1111). Accurate tags prevent the model from overfitting to backgrounds.
4. Set LoRA rank: Start with rank 32 (balanced between fidelity and generalization). Higher ranks (64–128) capture more detail but risk overfitting.
5. Train for 800–1500 steps at 1e-4 learning rate on an RTX 3060 (12GB). Enable xformers memory optimization. A full LoRA takes 10–15 minutes.
6. Export the .safetensors file (typically 7–35 MB) and load it during inference with a trigger word.

Real Example

Web comic artist "MariDraws" trained a LoRA on 22 hand-drawn panels of her character "Kaito" using SD XL and ComfyUI. The LoRA file was 28 MB. She loaded it alongside a different LoRA for backgrounds, generating 40 consistent comic pages over two weekends. Each Kaito panel preserved the same hairstyle, facial structure, and jacket design.

Method 3: ControlNet Conditioning (Zero Training Consistency)

ControlNet, created by Lvmin Zhang and originally released as a Stable Diffusion extension in 2023, lets you condition image generation on reference inputs — pose skeletons, depth maps, edge detection, normal maps, and even scribbles. This is the only method that guarantees instant character consistency without training.

ControlNet works by cloning the weights of the UNet encoder and creating a "control" copy that processes the conditioning image. The control outputs are injected back into the main UNet decoder at each resolution level. Multiple ControlNet units can stack (e.g., pose + depth + canny) for extremely tight adherence to a reference character.

Step-by-Step ControlNet Workflow

1. Install the ControlNet extension in Automatic1111, or use the built-in ControlNet loader node in ComfyUI.
2. Download control models: OpenPose (for body posture), Depth (for scene structure), Canny (for exact edges), and Lineart (for cartoon/anime consistency).
3. Use img2img: Take a reference image of your character and pass it through Canny or OpenPose ControlNet. Set Control Weight to 0.7–1.0 and guidance start/end to 0.0/0.8.
4. Lock the seed: Always use the same seed number when generating multiple frames of the same character. Combine with a fixed checkpoint (e.g., SDXL base or Realistic Vision v5.1).
5. Batch process: In ComfyUI, connect a "Batch Prompt" queue and feed sequential pose images through OpenPose ControlNet for animation-style consistency.
6. Refine with VAE: Use a consistent VAE file (e.g., vae-ft-mse-840000 for SD 1.5) to prevent color shifting across generations.

Real Example

Animator "FrameCraft" created a short 60-frame animation of a character walking using ComfyUI + ControlNet OpenPose. They used a single reference image as the identity base, fed 60 different pose skeletons through ControlNet at weight 0.85, and kept seed 123456 fixed. The final result showed consistent clothing, hair, and face structure across all 60 frames. No training was required.

Comparison: DreamBooth vs. LoRA vs. ControlNet

Each method serves a different use case. Choose based on your project scale, hardware, and consistency needs. Here is the breakdown:

Feature	DreamBooth	LoRA	ControlNet
Training Required	Yes (20–40 min on 12GB GPU)	Yes (10–15 min on 6GB GPU)	No
Output File Size	2–7 GB (full checkpoint)	4–35 MB (.safetensors)	None (runtime only)
Minimum VRAM	12 GB (8 GB with Forge optimizations)	6 GB	4 GB
Identity Fidelity	Highest (full model fine-tuned)	High (targeted adapter)	Moderate (conditioned per run)
Multi-Character Support	One per checkpoint	Swap LoRAs per scene	Multiple reference images
Pose Flexibility	High (prompt-driven)	High (prompt-driven)	Controlled by input map
Release Date	2022 (Google Research)	2021 (Microsoft)	2023 (Lvmin Zhang)
Best For	Production characters, game assets	Comics, series, multi-character	Animation, rotoscoping, quick tests

Common Mistakes That Break Character Consistency

Mistake 1: Using Low-Quality Training Images

Why It Hurts: Blurry, poorly lit, or low-resolution images teach the model incorrect details. The DreamBooth paper required 3–5 high-quality images for a reason: the diffusion model learns from every pixel. If your images have variable lighting or backgrounds, the model associates those artifacts with the character.

Fix: Use 1024x1024 source images cropped to 512x512 or 768x768. Ensure consistent white balance and remove EXIF data. Use images with clear separation between subject and background. Apply background removal (rembg) before training if backgrounds vary.

Mistake 2: Over-Training (Concept Collapse)

Why It Hurts: Training beyond the optimal step count causes the model to memorize training images instead of generalizing. The character will appear only in exact training poses and backgrounds. This is called "concept collapse." LoRA rank values above 128 compound the problem by introducing too many trainable parameters.

Fix: Validate every 100 steps during training. Stop when loss curves plateau. For LoRA, start with rank 32 and increase only if identity fails to transfer. For DreamBooth, limit to 100–150 steps per training image.

Mistake 3: Ignoring Seed Locking

Why It Hurts: Random seeds produce random sampling noise. Even with a perfect LoRA, two runs with different seeds will generate different face angles, hair positions, and lighting. This is the #1 cause of inconsistency in "zero-training" pipelines.

Fix: Always set a fixed seed in the UI (Automatic1111: extra options > seed). Use the same seed across an entire project. Adjust prompt wording instead of seed to vary composition.

Mistake 4: Mixing Base Models Mid-Project

Why It Hurts: LoRA and DreamBooth weights are tied to specific base models. A LoRA trained on SD 1.5 produces distorted results if loaded with SD XL or Flux. The latent space differs between architectures, so embeddings mismatch.

Fix: Train and infer on the exact same checkpoint. Document your base model name (e.g., "Realistic Vision v5.1" or "SDXL 1.0 base"). Never swap mid-series.

Mistake 5: Skipping Prompt Engineering

Why It Hurts: Prompts like "the same character" fail because diffusion models interpret natural language literally, not contextually. Negative prompts are equally critical — without "asymmetric eyes, extra limbs" in the negative, outputs degrade.

Fix: Build a prompt template: trigger_word, target_pose, style, lighting, camera_angle. Keep the trigger word first. Use negative prompts for common artifacts.

Pro Tips

• Train at 768x768 resolution whenever your GPU can handle it — SD XL and SD 1.5 both benefit from higher-resolution training for facial detail.
• Use the Forge fork of Automatic1111 if you have 6–8 GB VRAM; it improves generation speed by 45–75% on low-VRAM cards.
• Combine LoRA + ControlNet: train a character LoRA, then use OpenPose ControlNet to force exact poses. This is how professional comic artists achieve panel-to-panel consistency.
• Save your full ComfyUI workflow as a JSON template. You can reload it for any future character by swapping the LoRA file and checkpoint.
• Test your LoRA on 20 diverse prompts before committing to a full project. Look for identity breakdowns at extreme angles or unusual lighting.

FAQ

What is the difference between DreamBooth and LoRA for character consistency?

DreamBooth fine-tunes the entire UNet (860M parameters in SD 1.5), producing a 2–7 GB checkpoint that fully replaces the base model. LoRA injects small rank-decomposition matrices (4–35 MB file) that act as a lightweight adapter on top of the existing model. DreamBooth offers slightly higher fidelity but requires more VRAM and produces larger files. LoRA is faster to train, easier to swap between characters, and widely adopted for comic and series work.

Which open source tool is best for beginners — Automatic1111 or ComfyUI?

Automatic1111 offers a traditional web UI with tabs, sliders, and dropdowns — ideal for users who prefer structure over flexibility. ComfyUI uses a node-based graph interface that gives full control over every step of the pipeline but has a steeper learning curve. Automatic1111 was released in August 2022 and remains the most popular tool for beginners, while ComfyUI (January 2023) is preferred by advanced users building complex multi-model workflows.

How many images do I need to train a consistent character LoRA?

10–20 high-quality images of the same character with varied poses, expressions, and lighting is the recommended range. Fewer than 8 images risks overfitting (the model memorizes rather than generalizes). More than 30 images can dilute the training signal unless carefully captioned. Each image should be 512x512 or larger, with the character clearly separated from cluttered backgrounds.

Why does my character's face change between generations even with a LoRA loaded?

This typically happens because the seed is randomizing noise patterns that affect facial layout. Fix this by locking a fixed seed across all generations. Additionally, ensure your prompt includes the exact trigger word used during training. If the problem persists, your LoRA may be undertrained (less than 600 steps) or the base model may not match the training checkpoint.

Will open source character consistency tools get better in the future?

Yes. The open source community is actively improving identity preservation. ComfyUI now supports IP-Adapter (image prompt adapter) and InstantID, which can clone a face from a single reference image without training. Flux, released by Black Forest Labs in August 2024 and supported in ComfyUI and Forge, offers superior prompt adherence. Expect future workflows to combine SDXL/Flux base models with LoRA + ControlNet + IP-Adapter in a single pipeline for near-perfect consistency.

Conclusion

Consistent character images are no longer locked behind proprietary APIs or expensive hardware. With open source tools like Automatic1111, ComfyUI, and Stable Diffusion WebUI Forge, anyone with a mid-range GPU can train a character in under 30 minutes using DreamBooth or LoRA — or skip training entirely with ControlNet conditioning. The key is matching the method to your use case: DreamBooth for high-fidelity production assets, LoRA for modular character swapping, and ControlNet for quick, zero-training consistency workflows. All three techniques run locally, cost nothing beyond hardware, and are actively improved by a global community of developers.

• DreamBooth requires 5–8 images and full UNet fine-tuning for highest identity fidelity
• LoRA trains in 10–15 minutes on 6GB VRAM and produces shareable 7–35 MB files
• ControlNet enforces consistency via pose/depth/canny maps without any training needed
• Always lock your seed, keep your base model constant, and use consistent VAE files

Sources

• Stable Diffusion — Wikipedia
• DreamBooth — Wikipedia
• LoRA (Low-Rank Adaptation) — Wikipedia
• Automatic1111 Stable Diffusion Web UI — Wikipedia
• ComfyUI — Wikipedia
• Latent Diffusion Model — Wikipedia
• Text-to-Image Model — Wikipedia