ControlNet is an add-on network that makes an AI image follow the structure of a control image, such as an OpenPose skeleton, a depth map or a Canny edge map, while the prompt sets the content. In ComfyUI you load the control model, prepare the control image with a preprocessor, and pass your conditioning through Apply ControlNet. Each control model works with one base model family only, and ControlNet does not hold a face: that is the job of a LoRA or a reference method.
ControlNet is an add-on network that makes an AI image model follow the structure of a second image, such as a pose skeleton, a depth map or an edge map, while your prompt still decides what is drawn. In ComfyUI you load a control model that matches your base model, prepare the control image, and pass your prompt conditioning through the Apply ControlNet node.
Checked 13 October 2026 against the ControlNet paper, the original repository and its licence file, and ControlNet 1.1; ComfyUI's ControlNet, pose, depth, mixing and FLUX.1 tutorials, its ControlNet examples and node source; comfyui_controlnet_aux; the model cards from xinsir, Stability AI, Diffusers, SD 3.5, Alibaba PAI and InstantX; and Black Forest Labs' FLUX.1 Tools page. The settings quoted are the publishers' own, not results of a test of ours.
This guide is part of our AI image generation for influencers workflow. It assumes you have ComfyUI running and a checkpoint loaded.
What is ControlNet?
ControlNet was introduced in February 2023 in the paper "Adding Conditional Control to Text-to-Image Diffusion Models". Its repository describes the method in two lines: it copies the weights of the network's blocks into a "locked" copy and a "trainable" copy, where the trainable one "learns your condition" and the locked one "preserves your model". The two are joined by layers that start at zero, so before training the add-on changes nothing.
The practical result is a separate file you load beside a Stable Diffusion checkpoint or a newer model. It is often confused with a LoRA. A LoRA teaches the model a face, a style or a concept. A ControlNet does not teach anything: it steers the layout of one image from a picture you supply.
- 01Reference
A photo you own, a render or a sketch.
- 02Preprocessor
Turns it into a skeleton, depth map or edge map.
- 03Control image
The picture the ControlNet was trained to read.
- 04Apply ControlNet
Adds that structure to the prompt conditioning.
- 05KSampler
Generates with your checkpoint and any LoRA.
ControlNet types: which control for which job
| Control type | What the control image is | Use it for |
|---|---|---|
| OpenPose | A stick-figure skeleton of body, hand and face keypoints | Repeating a pose without copying the reference person's shape or clothes |
| Depth | A greyscale map: brighter areas are closer, darker areas are farther away | Keeping a room, a framing or the distance between subject and background |
| Canny | Hard edges, as white lines on black | Locking an exact outline, such as a product in a shot |
| Soft edge and lineart | Softer line drawings of the same picture | Shape guidance with more freedom than Canny |
| Scribble | A rough sketch | Turning a hand-drawn layout into an image |
| MLSD | Straight lines | Scenes built from straight edges |
| Tile | The image itself | Replacing bad detail with new detail in a finished picture |
Three details from the official docs help when choosing. ComfyUI's pose tutorial says an OpenPose skeleton can carry 18 body keypoints, 70 facial keypoints and 21 hand keypoints. Its depth tutorial explains that brighter areas of a depth map are closer to the camera. And the ControlNet 1.1 notes say the Tile model is "not a super resolution model": it ignores the detail in an image and generates new detail.
A lighter relative exists too. ComfyUI's examples page says a ControlNet "is run once every iteration", while a T2I-Adapter "runs once in total", so adapters have almost no effect on generation speed. They load through the same nodes.
How to use ControlNet in ComfyUI
- 1Match the control model to the base model
Find your checkpoint's family in the compatibility table below and pick a control model from its row.
- 2Place the file
Most go in ComfyUI/models/controlnet. A few newer ones go in models/model_patches or models/loras.
- 3Make the control image
Use the core Canny node for edges, or a preprocessor pack for pose and depth. Save the result so you can reuse it.
- 4Load ControlNet Model
Select the file. Restart or refresh if it is not listed.
- 5Apply ControlNet
Connect positive and negative conditioning, the control model and the control image. Its two outputs go to the KSampler.
- 6Queue, then adjust
Start at the defaults and change one setting per run.
ComfyUI's tutorial notes that its core nodes "do not include all types of preprocessors". The core Canny node covers edges, with a low threshold of 0.4 and a high threshold of 0.8 by default. For pose and depth the docs point to comfyui_controlnet_aux, whose README lists nodes such as OpenPose Estimator, DWPose Estimator and Depth Anything V2, plus an AIO Aux Preprocessor node that wraps them. Install it through ComfyUI Manager.
Use the current Apply ControlNet node, with separate positive and negative inputs. The tutorial says the older single-input version is deprecated and hidden by default. For a union model, the SetUnionControlNetType node sets which control type the file should act as, with auto as its default.
ControlNet strength, start and end
Apply ControlNet has three settings. In the node source, strength defaults to 1.0, start_percent to 0 and end_percent to 1. The tutorial explains the last two with examples: a start of 0.2 means guidance begins when 20% of the diffusion is complete, and an end of 0.8 means it stops at 80%.
- Stability AI, SD 3.5 Large ControlNets: "We recommend starting with a ControlNet strength of 0.7-0.8".
- Diffusers, SDXL Canny: the example sets the scale to 0.5, marked "recommended for good generalization".
- Alibaba PAI, Z-Image-Turbo union: an optimal
control_context_scalefrom 0.65 to 0.80, with a detailed prompt. - ComfyUI, mixing two ControlNets: similar strengths for both, for example 1.0 each, so that one does not suppress the other.
ComfyUI's pose tutorial also shows a two-pass graph: the first sampler uses the ControlNet, then a second sampler refines an upscaled latent at a lower denoise, typically 0.4 to 0.6. That second pass is img2img, and it is where skin and fabric detail usually improve.
| Problem | Likely cause | Fix |
|---|---|---|
| The model is missing from the loader | Wrong folder, or ComfyUI has not re-read it | Move it to the folder in the table above, then restart or refresh |
| The control has no effect | A photo was passed in where a control image belongs, or the control model is for another family | Preprocess the photo first, and match the control model to the checkpoint |
| The body looks stiff | Control is strong for the whole run | Lower strength, or set end_percent below 1 |
| Two ControlNets fight | One is much stronger than the other | Use similar strengths and describe both regions in the prompt |
ControlNet model compatibility by base model
Every control model is trained for one base model family. Choose the family your checkpoint and LoRA belong to, then take a control model from its row.
| Base model | Control models | ComfyUI folder | Notes |
|---|---|---|---|
| SD 1.5 | ControlNet 1.1 by lllyasviel: 14 models, 11 marked production-ready and 3 experimental | models/controlnet | The files ComfyUI's own ControlNet tutorials use |
| SDXL | xinsir OpenPose and ControlNet++ union; Stability AI Control-LoRAs for Canny and Depth; Diffusers Canny | models/controlnet | A Control-LoRA is about 738MB at rank 256, against 4.7GB for a full ControlNet |
| SD 3.5 Large | Stability AI Canny, Depth and Blur | models/controlnet | The card says they work only with SD 3.5 Large. It lists no pose model |
| FLUX.1 [dev] | Black Forest Labs Canny and Depth, as full models or LoRAs; community models from InstantX and XLabs | diffusion_models for full models, loras for the LoRA versions, controlnet for community files | Black Forest Labs now marks its Canny and Depth models as deprecated; the weights stay on Hugging Face |
| Z-Image-Turbo | Fun ControlNet Union from Alibaba PAI: Canny, HED, Depth, Pose and MLSD | models/model_patches | Apache 2.0 on the model card |
| Qwen-Image | InstantX union (canny, soft edge, depth, pose); DiffSynth patches (canny, depth, inpaint) | controlnet for InstantX, model_patches for DiffSynth | Apache 2.0 on the InstantX card |
For SDXL pose work, xinsir's model card warns that results can be unstable with the default skeleton, because the model was trained on thicker pose lines. Its ControlNet++ union model covers more than ten control types in one file. If you are still choosing a base, our SDXL guide sets it beside SD 1.5, SD 3.5 and FLUX.
A control model does not change the base model's licence. The FLUX.1 Canny and Depth files are under the FLUX.1 [dev] non-commercial licence, and Black Forest Labs' page now says the two are deprecated and no longer supported in its API. The SD 3.5 ControlNets are under the Stability AI Community License. Read the card before you sell work made with one.
ControlNet for a persona: poses yes, identity no
ControlNet sets structure for one image. Run the same skeleton twice and you get the same pose on two different faces. For a persona, the face has to come from somewhere else: a character LoRA trained on the same base model, or a reference method such as InstantID. The LoRA goes on the model path and the ControlNet on the conditioning, so they work together. If you plan to train one, the free LoRA training steps calculator works out the run.
The useful habit is a pose battery: a folder of saved skeletons that you run the persona through for every new outfit or location. Because the skeletons never change, each batch comes out as a set.
- The control model, the checkpoint and the persona LoRA are all from one model family
- Each skeleton was checked by eye: limbs separated, hands present where needed
- Skeletons are cropped to the output aspect ratio
- Pose references are your own photos, licensed images or figures you posed yourself
- Checkpoint, LoRA strength and the identity words in the prompt stay fixed across the batch
- Strength and end_percent were tested on three skeletons before queuing all of them
One rule sits above the settings. A persona is a fictional adult. Do not use ControlNet, with or without a LoRA, to recreate a real person who has not agreed in writing, and do not lift poses from photos of identifiable people you have no rights to. The AI Influencers program covers the full persona graph, from the trained face to pose control and the disclosure rules for publishing. What ComfyUI itself needs to run is on our ComfyUI statistics page.
ControlNet FAQ
What is ControlNet?
ControlNet is an add-on network that makes a diffusion model follow the structure of a second image, such as a pose skeleton, a depth map or an edge map, while the prompt still decides what is drawn. It comes from the 2023 paper Adding Conditional Control to Text-to-Image Diffusion Models by Lvmin Zhang, Anyi Rao and Maneesh Agrawala, and the original code is Apache 2.0.
What are the ControlNet types?
Each type reads a different control image. OpenPose reads a skeleton, Depth reads a distance map, Canny reads hard edges, soft edge and lineart read softer lines, Scribble reads a rough sketch, MLSD reads straight lines and Tile reads the image itself. ControlNet 1.1 for Stable Diffusion 1.5 ships 14 models, and union models put several types in one file.
How do I use ControlNet in ComfyUI?
Put the control model in ComfyUI/models/controlnet, load it with Load ControlNet Model, and pass your positive and negative conditioning through Apply ControlNet together with a prepared control image. ComfyUI does not convert a photo into a pose or depth map for you, so make that image first with the core Canny node or a preprocessor pack such as comfyui_controlnet_aux.
Which ControlNet works with SDXL?
One trained for SDXL. Options read for this guide are xinsir's OpenPose model and its ControlNet++ union model, both Apache 2.0, Stability AI's Control-LoRAs for Canny and Depth, and the Diffusers team's Canny model. An SD 1.5 control model will not work on an SDXL checkpoint, and an SDXL one will not work on FLUX.
Does ControlNet work with FLUX?
Yes, with FLUX-specific files. Black Forest Labs released FLUX.1 Canny and Depth as full models and as LoRAs, and InstantX and XLabs published community ControlNets. Black Forest Labs now marks its Canny and Depth models as deprecated in its API, with the open weights still on Hugging Face under the FLUX.1 [dev] non-commercial licence.
What ControlNet strength should I use?
Start from the default. ComfyUI's Apply ControlNet node defaults to a strength of 1.0, applied from 0 to 1 of the sampling run. Model makers publish their own starting points: Stability AI recommends 0.7 to 0.8 for its SD 3.5 Large ControlNets, and the Diffusers SDXL Canny example uses 0.5. Lower the strength or end control earlier if the result looks stiff.
Does ControlNet keep a face the same?
No. ControlNet fixes structure, one image at a time, so the same skeleton gives a different face on each run. Identity comes from a character LoRA or a reference method such as InstantID. Use both together: one holds the face and ControlNet holds the pose. Never use either to reproduce a real person without their written permission.
Free · no card needed
Start with the free Starter Kit.
An AI character bible from AI Influencers, plus four more working files.
Poses are solved. Now build the persona around them.
AI Influencers, included in All Access, covers persona LoRA training, ComfyUI pose and batch workflows, video and the disclosure rules for publishing. The other three programs, live coaching and the private community come with the same subscription.
Plan the face before the poses
A pose only helps once the face holds. Work out the training steps for your image set with the free calculator, and follow the Telegram channel for ComfyUI updates.