LoRA Training Guide for Consistent AI Influencer Faces

LoRA can improve character consistency, but no training recipe guarantees the same face in every generation. Results depend on the base model, dataset, captions, training exposure, checkpoint, and inference settings. The reliable workflow is to control those variables, save intermediate checkpoints, and compare them with a repeatable test set.

This guide uses the current sd-scripts project as the command-line reference and Kohya SS as its graphical interface. Both change over time, so match the instructions to the installed release and the architecture of your base model.

What a character LoRA actually trains

Low-Rank Adaptation freezes a pretrained model and learns smaller injected matrices instead of updating the full set of model weights. The original LoRA paper introduced the method for language models; diffusion tooling later adapted the approach to image generation.

For a character, the adapter learns patterns associated with the training images and captions. It does not store a formal identity model, and it cannot repair conflicting references. A clean adapter can still drift when you change the base checkpoint, sampler, resolution, prompt, seed, VAE, or LoRA strength.

Choose the base architecture before the trainer

Pick the model family you will use for production before preparing a run. An adapter is architecture-specific; do not assume an SDXL LoRA will behave correctly on SD 1.x, SD3, or FLUX.

The project repository lists its currently supported model families. Read the model-specific document shipped with your version before copying a command from another architecture.

Build an identity-consistent dataset

Dataset quality matters more than hitting a universal image count. Start with images you are permitted to use, then keep only references that look like the same subject and are clear enough to teach the intended facial details.

Keep references that add useful coverage

• Sharp facial features without heavy compression, motion blur, or masking.
• Front, three-quarter, and profile views represented naturally.
• Useful variation in expression, lighting, framing, clothing, and background.
• A coherent visual domain for the production style you intend to generate.
• Captions that distinguish identity from changeable attributes.

Reject references that teach the wrong identity

• Another person, blended faces, or images where the subject is ambiguous.
• Distorted eyes, teeth, hands near the face, strong beauty filters, or generated artifacts.
• Near-duplicates that increase exposure without adding a new view.
• A permanent outfit, location, or color grade repeated across most references.
• Images without clear consent, provenance, or sufficient usage rights.

Use a side-by-side contact sheet for the final review. Ask one question: would a neutral reviewer identify every reference as the same subject? Remove outliers before training. Keep a separate evaluation set out of training so the adapter is tested on views it did not memorize.

Caption identity and variables separately

Give the character a distinctive trigger token that is unlikely to collide with ordinary prompt words. Put the token consistently in each caption, then describe variable attributes such as angle, expression, clothing, framing, and setting. When captions omit a repeated attribute, the adapter is more likely to bind it to the character.

Caption shuffling and keep_tokens can preserve a trigger at the front while varying the remaining tags. Inspect the final text files rather than trusting an automatic tagger blindly; a wrong eye color or hairstyle label creates conflicting supervision.

Use an explicit TOML dataset configuration

Current sd-scripts accepts a TOML file through --dataset_config. Images should sit directly inside image_dir. Set repeats explicitly in the subset instead of relying on directory naming conventions.

[general]
caption_extension = ".txt"
shuffle_caption = true
keep_tokens = 1

[[datasets]]
resolution = 1024
batch_size = 1
enable_bucket = true
bucket_no_upscale = true

  [[datasets.subsets]]
  image_dir = "C:/datasets/creator_subject/images"
  num_repeats = 1

This is an SDXL-oriented example, not a cross-model prescription. Change resolution, batch size, repeats, captions, and augmentation only after checking the base model requirements and your available memory. The official dataset configuration reference explains option precedence and the supported TOML scopes.

class_tokens is only used as a fallback for an image that has no matching caption file. Because this workflow supplies a .txt caption for each image, the trainer reads those captions instead.

Start with a controlled pilot run

The command below shows the shape of an SDXL LoRA pilot. The rank, alpha, learning rate, epoch count, precision, optimizer, and caching choices are experiment inputs. They are not promises about likeness quality.

accelerate launch sdxl_train_network.py ^
  --pretrained_model_name_or_path="<sdxl-base-model>" ^
  --dataset_config="dataset.toml" ^
  --output_dir="output" ^
  --output_name="creator_subject" ^
  --save_model_as=safetensors ^
  --network_module=networks.lora ^
  --network_dim=16 ^
  --network_alpha=8 ^
  --learning_rate=1e-4 ^
  --optimizer_type=AdamW8bit ^
  --max_train_epochs=10 ^
  --save_every_n_epochs=1 ^
  --mixed_precision=bf16 ^
  --gradient_checkpointing

Windows uses ^ for line continuation in Command Prompt. PowerShell and other shells use different syntax. The sd-scripts training overview documents the common arguments, while the model-specific files define the options that apply to each architecture.

Treat parameters as hypotheses

Compare saved checkpoints under identical conditions

Save intermediate adapters with --save_every_n_epochs or --save_every_n_steps. Then run a fixed prompt matrix against each checkpoint. Hold these inference variables constant:

• Base model, VAE, sampler, scheduler, steps, and image dimensions.
• Positive and negative prompts, including the trigger token.
• Seed and batch size.
• LoRA strength and any other adapters in the graph.

Include neutral portraits, profiles, full-body compositions, varied lighting, expressions, and clothing not present in the training set. Score identity separately from prompt adherence and artifact rate. A useful checkpoint preserves identity while still responding to those changes.

Training and validation loss can expose a bad run, but the lowest number is not automatically the best image checkpoint. Review the images and held-out prompts together. The Diffusers LoRA guide also shows how adapter weights are saved and loaded in a reproducible training pipeline.

Troubleshoot the failure you can observe

The advanced sd-scripts guide documents memory controls and their tradeoffs. For example, gradient checkpointing reduces memory at a speed cost. Latent caching disables image augmentation, while text-encoder output caching disables caption augmentation and prevents text-encoder LoRA training in the same run.

Load the adapter in a compatible inference UI

Train with sd-scripts or Kohya SS, then load the resulting adapter in a UI that supports its base architecture. The Automatic1111 feature guide documents placing compatible adapters under models/Lora and applying them with <lora:filename:multiplier>. It points users to Kohya SS for training.

In ComfyUI, use a loader node that matches the model family and connect it to the same model and text-encoding path used by the workflow. If you are new to the graph, start with the ComfyUI overview and installation guide.

Consent, licensing, and safe sharing

• Use a real person's likeness only with informed permission that covers training and intended distribution.
• Verify the licenses for the base model, dataset images, trainer, and hosting destination.
• Do not train on minors or on private, intimate, stolen, or deceptive source material.
• Document the character's provenance and disclose synthetic media where the platform, campaign, or audience context requires it.
• Before publishing adapter weights, assume recipients can generate outputs outside your planned use case.

A repeatable character LoRA workflow

1. Choose the production model family and its matching trainer.
2. Collect consented, licensed references and remove identity outliers.
3. Caption the trigger token, identity, and changeable attributes deliberately.
4. Declare the dataset explicitly in TOML.
5. Run a small pilot and save intermediate checkpoints.
6. Compare checkpoints with the same held-out prompts and inference settings.
7. Change one training variable, record the result, and repeat only when the evidence supports it.

Primary sources reviewed

• LoRA: Low-Rank Adaptation of Large Language Models
• kohya-ss/sd-scripts repository and supported models
• sd-scripts LoRA training overview
• sd-scripts dataset configuration
• sd-scripts advanced training options
• Kohya SS GUI
• Hugging Face Diffusers LoRA training guide
• Automatic1111 LoRA loading documentation

Technical instructions and source links were reviewed on July 12, 2026. Recheck the installed tool version before starting a training run.

Continue the AI influencer workflow

Browse the AI Influencers topic hub, compare current ComfyUI model families, estimate the cost to create an AI influencer, and plan the next stage with the AI influencer monetization guide.