LoRA Training Guide for Consistent AI Influencer Faces

LoRA can improve character consistency, but no training recipe guarantees the same face in every generation. Results depend on the base model, dataset, captions, training exposure, checkpoint, and inference settings. The reliable workflow is to control those variables, save intermediate checkpoints, and compare them with a repeatable test set.

This guide uses the sd-scripts project (version 0.12.0, released September 24, 2026) as the command-line reference and Kohya SS as its graphical interface. It also covers AI Toolkit, OneTrainer and musubi-tuner, which train the newer FLUX.2, Qwen-Image and Z-Image models that sd-scripts does not. These tools change often, so match every command to your installed version and your base model.

Checked against the trainers' READMEs, the model cards and the hosted trainers' pricing pages on October 1, 2026. This update adds FLUX.2, Qwen-Image and Z-Image options, current trainer versions and hosted-trainer prices to the July 2026 version.

What a character LoRA actually trains

Low-Rank Adaptation freezes a pretrained model and learns smaller injected matrices instead of updating the full set of model weights. The original LoRA paper introduced the method for language models; diffusion tooling later adapted the approach to image generation.

For a character, the adapter learns patterns associated with the training images and captions. It does not store a formal identity model, and it cannot repair conflicting references. A clean adapter can still drift when you change the base checkpoint, sampler, resolution, prompt, seed, VAE, or LoRA strength.

Choose the base model before the trainer

Pick the model you will use for production before preparing a run. An adapter is architecture-specific: an SDXL LoRA will not work on FLUX, and a FLUX.1 LoRA is not a FLUX.2 LoRA. Check the license of the weights as well, because it decides whether you may use the model commercially.

Sources: model cards for SDXL, FLUX.1 [dev], FLUX.2 [dev], klein Base 4B, klein Base 9B, Qwen-Image-2512, Qwen-Image-2.1, Z-Image and Chroma1-Base, plus each trainer's README.

In sd-scripts, the entry point follows the architecture: train_network.py for SD 1.x and 2.x, sdxl_train_network.py, sd3_train_network.py, and flux_train_network.py for FLUX.1 and Chroma, plus separate scripts for Lumina, HunyuanImage-2.1 and Anima. The repository lists its currently supported model families. Read the model-specific document shipped with your version before copying a command from another architecture.

Pick a trainer

If you already work in Kohya SS or train SDXL, stay with sd-scripts. For FLUX.2, Qwen-Image or Z-Image, use AI Toolkit, OneTrainer or musubi-tuner, because sd-scripts has no entry point for them.

Hosted trainers if you have no suitable GPU

Prices as listed on each service's page on October 1, 2026, in US dollars:

Hosted trainers choose most settings for you and bill per run or per step, so a failed experiment still costs money. Start small, keep every checkpoint the service lets you download, and run the same comparison described below. fal lists commercial use for these trainers, but the base model's license still applies, and Civitai has not allowed models of real people since May 22, 2025 (its rules cover celebrities, influencers and private individuals alike).

Build an identity-consistent dataset

Dataset quality matters more than hitting a universal image count. Replicate's FLUX fine-tuning guide says training can work with as few as two images but recommends at least 10 for best results, and BFL's klein training example, written for a style LoRA, gives 20 to 40 images at 1,024 px or higher as the optimal dataset size (checked October 2026). For a face, coverage of angles, expressions and light matters more than the total. Start with images you are permitted to use, then keep only references that look like the same subject and are clear enough to teach the intended facial details.

Keep references that add useful coverage

• Sharp facial features without heavy compression, motion blur, or masking.
• Front, three-quarter, and profile views represented naturally.
• Useful variation in expression, lighting, framing, clothing, and background.
• A coherent visual domain for the production style you intend to generate.
• Captions that distinguish identity from changeable attributes.

Reject references that teach the wrong identity

• Another person, blended faces, or images where the subject is ambiguous.
• Distorted eyes, teeth, hands near the face, strong beauty filters, or generated artifacts.
• Near-duplicates that increase exposure without adding a new view.
• A permanent outfit, location, or color grade repeated across most references.
• Images without clear consent, provenance, or sufficient usage rights.

Use a side-by-side contact sheet for the final review. Ask one question: would a neutral reviewer identify every reference as the same subject? Remove outliers before training. Keep a separate evaluation set out of training so the adapter is tested on views it did not memorize.

Building references for a character who does not exist

A fully synthetic character has no photos, so you make the reference set first. Write a fixed description of the face, generate many candidate portraits on your base model, and choose one hero image. Then create the other views from that hero with a reference-based editor, such as Qwen-Image-Edit or FLUX.2's multi-reference editing: front, three-quarter and profile, several expressions, different light and outfits. Curate hard, because small drifts in this set become the LoRA's idea of the face. Log the prompts, seeds and models you used so you can show where the character came from.

Caption identity and variables separately

Give the character a distinctive trigger token that is unlikely to collide with ordinary prompt words. BFL suggests invented tokens that are not real words, and Replicate warns against existing words such as "dog". Put the token consistently in each caption, then describe variable attributes such as angle, expression, clothing, framing, and setting. When captions omit a repeated attribute, the adapter is more likely to bind it to the character. BFL's guide explains the same mechanism from the other side: with a small dataset, whatever the captions leave out is learned from the images.

Match the caption style to the model. FLUX.1 reads captions through a T5-XXL encoder alongside CLIP-L, and FLUX.2 [dev] uses a single Mistral Small 3.1 text encoder, according to Hugging Face's FLUX.2 announcement, so plain descriptive sentences suit them. Tag-style captions suit checkpoints that were trained on tags; check the model card. The WD14 tagger ships with sd-scripts, and OneTrainer includes BLIP, BLIP-2 and WD-1.4 captioning.

Caption shuffling and keep_tokens can preserve a trigger at the front while varying the remaining tags. Inspect the final text files rather than trusting an automatic tagger blindly; a wrong eye color or hairstyle label creates conflicting supervision.

Use an explicit TOML dataset configuration

Current sd-scripts accepts a TOML file through --dataset_config. Images should sit directly inside image_dir. Set repeats explicitly in the subset instead of relying on directory naming conventions.

[general]
caption_extension = ".txt"
shuffle_caption = true
keep_tokens = 1

[[datasets]]
resolution = 1024
batch_size = 1
enable_bucket = true
bucket_no_upscale = true

  [[datasets.subsets]]
  image_dir = "C:/datasets/creator_subject/images"
  num_repeats = 1

This is an SDXL-oriented example, not a cross-model prescription. Change resolution, batch size, repeats, captions, and augmentation only after checking the base model requirements and your available memory. The official dataset configuration reference explains option precedence and the supported TOML scopes. musubi-tuner uses a similar TOML dataset file, while AI Toolkit uses YAML job configs and a web UI for setting up jobs.

class_tokens is only used as a fallback for an image that has no matching caption file. Because this workflow supplies a .txt caption for each image, the trainer reads those captions instead.

Start with a controlled pilot run

The command below shows the shape of an SDXL LoRA pilot. The rank, alpha, learning rate, epoch count, precision, optimizer, and caching choices are experiment inputs. They are not promises about likeness quality.

accelerate launch sdxl_train_network.py ^
  --pretrained_model_name_or_path="<sdxl-base-model>" ^
  --dataset_config="dataset.toml" ^
  --output_dir="output" ^
  --output_name="creator_subject" ^
  --save_model_as=safetensors ^
  --network_module=networks.lora ^
  --network_dim=16 ^
  --network_alpha=8 ^
  --learning_rate=1e-4 ^
  --optimizer_type=AdamW8bit ^
  --max_train_epochs=10 ^
  --save_every_n_epochs=1 ^
  --mixed_precision=bf16 ^
  --gradient_checkpointing

Windows uses ^ for line continuation in Command Prompt. PowerShell and other shells use different syntax. The sd-scripts training overview documents the common arguments, while the model-specific files define the options that apply to each architecture.

FLUX.1 on a smaller card

The sd-scripts FLUX.1 guide needs four files (the FLUX.1 [dev] model, CLIP-L, T5-XXL and the autoencoder) and the networks.lora_flux module. With --fp8_base, it recommends these settings by VRAM:

Higher block-swap values save memory but slow training. For FLUX.2, Qwen-Image and Z-Image, start from AI Toolkit's example configs, OneTrainer's presets or musubi-tuner's per-model documents, and keep the same discipline: one dataset definition, explicit repeats and saved checkpoints.

Treat parameters as hypotheses

The official examples are anchors, not recipes. The sd-scripts SDXL guide uses a network dimension of 32, alpha 16 and a 1e-4 learning rate; its FLUX.1 guide uses dimension 16, alpha 1 and 1e-4; BFL's klein guide trains at 1e-4 in AI Toolkit and saves a checkpoint every 250 steps. Change one variable per run and write down what changed.

Compare saved checkpoints under identical conditions

Save intermediate adapters with --save_every_n_epochs or --save_every_n_steps. Then run a fixed prompt matrix against each checkpoint. Hold these inference variables constant:

• Base model, VAE, sampler, scheduler, steps, and image dimensions.
• Positive and negative prompts, including the trigger token.
• Seed and batch size.
• LoRA strength and any other adapters in the graph.

Include neutral portraits, profiles, full-body compositions, varied lighting, expressions, and clothing not present in the training set. Score identity separately from prompt adherence and artifact rate. A useful checkpoint preserves identity while still responding to those changes.

Training and validation loss can expose a bad run, but the lowest number is not automatically the best image checkpoint. BFL's klein training guide tells you to monitor sample outputs to avoid overfitting, so judge checkpoints by the images they produce, since loss can keep falling after the images start to overfit. Review the images and held-out prompts together. The Diffusers LoRA guide also shows how adapter weights are saved and loaded in a reproducible training pipeline.

Troubleshoot the failure you can observe

The advanced sd-scripts guide documents memory controls and their tradeoffs. For example, gradient checkpointing reduces memory at a speed cost. Latent caching disables image augmentation, while text-encoder output caching disables caption augmentation and prevents text-encoder LoRA training in the same run, so it needs --network_train_unet_only.

Load the adapter in a compatible inference UI

Load the adapter with the same base model family it was trained on. The Automatic1111 feature guide documents placing compatible adapters under models/Lora and applying them with <lora:filename:multiplier>, and it points users to Kohya SS for training. Its latest release is still v1.10.1, so newer architectures usually need another UI.

In ComfyUI, put the file in ComfyUI/models/loras and use the Load LoRA node, whose strength_model and strength_clip inputs set how strongly the adapter changes the model and the text encoder (ComfyUI LoRA tutorial). Connect it to the same model and text-encoding path used by the workflow. If you are new to the graph, start with the ComfyUI overview and installation guide.

When reference images may be enough

Training is not the only route. These tools keep a character recognizable from reference images without a training run:

• Qwen-Image-2.1 accepts up to 10 reference images and aims to preserve identity for people and products, under a non-commercial research license.
• FLUX.2 [dev] and [klein] support multi-reference editing, per their model cards.
• Midjourney's Edit model for V8.1 and V8.2 takes up to 4 reference images and replaces Omni Reference and Character Reference (Midjourney docs). See our Midjourney prompts guide.
• Kling Elements bind a reference-built character to video shots; our Kling tutorial covers them.

Test references first. Train a LoRA when they cannot hold the identity across your shot list, or when you need the character inside your own workflow with weights that do not change when a hosted model updates.

Consent, licensing, and safe sharing

• Use a real person's likeness only with informed permission that covers training and intended distribution. Many services refuse likeness models even with permission; Civitai bans them outright.
• Verify the licenses for the base model, dataset images, trainer, and hosting destination. FLUX.1 [dev], FLUX.2 [dev] and klein 9B weights are non-commercial, and Qwen-Image-2.1 is research-only.
• Do not train on minors or on private, intimate, stolen, or deceptive source material.
• Document the character's provenance and disclose synthetic media where the platform, campaign, or audience context requires it. Our face swap guide summarizes the likeness laws and platform rules.
• Before publishing adapter weights, assume recipients can generate outputs outside your planned use case.

A repeatable character LoRA workflow

1. Choose the production model family, check its license, and pick a trainer that supports it.
2. Collect consented, licensed references and remove identity outliers.
3. Caption the trigger token, identity, and changeable attributes deliberately.
4. Declare the dataset explicitly in TOML (or the trainer's own config).
5. Run a small pilot and save intermediate checkpoints.
6. Compare checkpoints with the same held-out prompts and inference settings.
7. Change one training variable, record the result, and repeat only when the evidence supports it.

LoRA training FAQ

How many images do I need for a character LoRA?

There is no universal number. Replicate's FLUX guide says as few as two images can work but recommends at least 10, and BFL's klein example gives 20 to 40 as optimal for a style LoRA. For a face, coverage matters more than count: sharp front, three-quarter and profile views in varied light, with near-duplicates removed.

Which base model should I train on in 2026?

The one you will generate with, because a LoRA only works on its own architecture. For commercial work, check the weights license first: FLUX.2 [klein] 4B, Qwen-Image, Qwen-Image-2512, Z-Image and Chroma are Apache 2.0, SDXL uses CreativeML Open RAIL++-M, and FLUX.1 [dev], FLUX.2 [dev] and klein 9B use non-commercial licenses, although the two [dev] model cards say generated outputs can be used commercially. Qwen-Image-2.1 is limited to research and evaluation.

Can I train a LoRA without my own GPU?

Yes. fal, Replicate and Civitai run the job for you and bill per run, per step or in Buzz, as listed in the table above. You still need to test the checkpoints yourself.

What rank and learning rate should I start with?

Start from the trainer's documented example for your model, then change one variable at a time. sd-scripts' SDXL example uses dimension 32 and alpha 16, its FLUX.1 example uses dimension 16 and alpha 1, and both use a 1e-4 learning rate.

Can I train a LoRA of a real person?

Only with that person's informed permission for training and every intended use, and many platforms refuse such models anyway. Sexual or deceptive use of someone's likeness is illegal in many places regardless of how the model was made.

Primary sources reviewed

• LoRA: Low-Rank Adaptation of Large Language Models
• kohya-ss/sd-scripts repository and supported models
• sd-scripts LoRA training overview
• sd-scripts dataset configuration
• sd-scripts advanced training options
• sd-scripts FLUX.1 training guide and SDXL training guide
• Kohya SS GUI
• AI Toolkit, OneTrainer, musubi-tuner and FluxGym READMEs
• BFL: FLUX.2 [klein] LoRA guide and klein training example
• Hugging Face Diffusers LoRA training guide
• Automatic1111 LoRA loading documentation and ComfyUI LoRA tutorial
• Pricing and policy pages: fal trainers, Replicate pricing, Civitai trainer and Civitai real-person policy

Technical instructions, licenses, prices and source links were reviewed on October 1, 2026. Recheck the installed tool version and the pricing page before starting a training run.

Continue the AI influencer workflow

Building the image set first? The 40-shot character LoRA dataset checklist covers angles, lighting, outfits and culling before you train.

Browse the AI Influencers topic hub, compare current ComfyUI model families, try the Z-Image Turbo tutorial, estimate the cost to create an AI influencer, and plan the next stage with the AI influencer monetization guide. The AI Influencers course covers character design, LoRA-based face consistency in ComfyUI, and the disclosure rules for publishing.