Quick Answer
ControlNet — the neural network that adds precise composition control to AI image generation. Install in ComfyUI, all 14 control modes explained with use cases, when to combine with LoRA, and the AI influencer production pipeline using ControlNet for consistent poses across content.
Quick Answer
ControlNet adds conditional control to AI image generation — pose, depth, edges, sketches. Instead of relying on text alone, you feed a control signal (skeleton, depth map, edge map) that guides composition. Install via ComfyUI Manager. The 2026 standard for precise pose control + character LoRA combinations.
What ControlNet Is
ControlNet (introduced in 2023, evolved through 2024-2026) is a neural network that adds an extra input pathway to diffusion models. Without ControlNet, you describe an image in text and hope the model produces what you want. With ControlNet, you provide a second input — a control image — that constrains the generation.
- Text prompt: describes content (“woman in red dress, Tokyo street”)
- Control image: specifies composition (pose skeleton, depth map, edge outline)
- Output: image matching BOTH text and control
Install ControlNet in ComfyUI
- Install ComfyUI Manager (see our guide).
- Install custom nodes via Manager:
ComfyUI ControlNet Aux— preprocessors (turn images into pose maps, depth, etc.)ComfyUI Advanced ControlNet— extended control modes
- Download ControlNet weights matching your base model:
- SDXL:
diffusers/controlnet-canny-sdxl-1.0,controlnet-depth-sdxl-1.0, etc. - Flux:
XLabs-AI/flux-controlnet-canny,flux-controlnet-depth - SD 1.5:
lllyasviel/control_v11p_sd15_*series (legacy, still works)
- SDXL:
- Place weights in
ComfyUI/models/controlnet/ - Restart ComfyUI. Load a ControlNet example workflow from Templates → ControlNet.
The 14 ControlNet Modes (2026)
Composition control
- Canny: edge detection. Best for preserving outlines from a reference photo. Most-used mode.
- Depth: 3D depth map. Preserves spatial layout — foreground/midground/background.
- Lineart: clean line drawings. Best for illustration and concept art.
- Soft Edge / HED: smoother edge maps than Canny. Better for organic subjects.
- Scribble: rough sketches. Forgives loose drawing, generates polished output.
Pose / human
- OpenPose: human pose skeleton. Forces specific body pose, hand position, facial keypoints.
- DensePose: dense pose mapping. More precise than OpenPose for body coverage.
Structural
- MLSD: straight lines. Best for architecture, interior design, geometric layouts.
- Normal map: surface normals. 3D-style lighting and texture preservation.
- Segmentation: regional masks (sky, building, person, etc). Color-coded layout control.
Identity / style
- Reference-only: pulls style/identity from a reference image without explicit control map.
- IP-Adapter: stronger than Reference-only — uses image as a prompt-equivalent. Pair with face encoder for consistent characters.
- Shuffle: rearranges color/style from a reference.
Editing
- Tile: upscaling and detailing — adds details when generating at higher resolution.
- Inpaint: mask-based editing inside ControlNet (vs separate inpaint workflow).
ControlNet in AI Influencer Pipelines
The standard 2026 production workflow combines ControlNet + character LoRA + base model:
- Train a character LoRA on your AI character (face, identity).
- Find a reference pose — could be a real photo, an OpenPose example, or a 3D-rendered pose.
- Extract pose using ControlNet preprocessors (OpenPose preprocessor) into a skeleton image.
- Generate with both: base model + character LoRA + OpenPose ControlNet → character with the exact pose you specified.
- Result: consistent character in any pose you can find a reference for.
See our AI face swap guide for the full character pipeline.
When to Use Which Mode
- Replicating a pose from a reference photo: OpenPose
- Preserving exact composition (architecture, layout): Canny or MLSD
- Style transfer with structural preservation: Lineart + style prompt, or IP-Adapter
- Realistic 3D-style scenes: Depth + Normal map combined
- Concept art from sketches: Scribble (forgiving) or Lineart (precise)
- Color-by-region control: Segmentation
- Upscaling with detail: Tile
- Editing specific regions: Inpaint
- Maintaining character identity: Reference-only or IP-Adapter (often combined with LoRA)
ControlNet Weight Tuning
Each ControlNet has a strength parameter (0.0 to 2.0):
- 0.5-0.8: loose guidance — model has freedom to interpret. Use for stylistic outputs.
- 0.9-1.0: standard guidance. Most use cases.
- 1.1-1.5: strong guidance. Use when control image must be respected exactly.
- 1.6-2.0: very strong. Often produces artifacts; use sparingly for testing.
Combining multiple ControlNets (e.g. OpenPose 1.0 + Canny 0.6) gives layered control without over-constraining.
ControlNet vs LoRA: Different Tools
| Aspect | ControlNet | LoRA |
|---|---|---|
| Controls | Composition (pose, depth, edges) | Identity / style (face, art style) |
| Input format | Per-generation control image | Pre-trained weights file |
| Setup time | Instant (just provide control image) | 2-8 hours training initially |
| Use case | Generation-specific composition | Recurring character/style |
They're complementary, not alternatives. Production AI creators use both.
Build Production-Grade Character Pipelines
ControlNet is one of the most powerful tools in the AI creator stack. Combined with character LoRAs, it gives you precise control over both identity AND composition. Our AI Influencers course teaches the full pipeline — LoRA training, ControlNet workflows, video animation — and the monetization playbook scaling synthetic creators to $5K-$50K/month.
AI Influencers: ControlNet + LoRA + ComfyUI Stack
Character LoRA training, ControlNet pose control, video animation — full production pipeline scaling AI creators past $50K/month.
Get AI Influencers →