October 24, 2026-12 min read

Listen to this briefing

Speech is not supported in this browser. Try Chrome or Edge.

Infographic comparing LoRA adapters, QLoRA quantization, and DoRA magnitude-direction fine-tuning on a frozen LLM — PEFT methods freeze most of the model and train small adapters. QLoRA adds 4-bit quantization; DoRA splits magnitude and direction.

LoRA vs QLoRA vs DoRA: Parameter-Efficient Fine-Tuning Compared

A builder briefing comparing LoRA, QLoRA, and DoRA: how each adapts LLMs with far fewer trainable parameters, memory trade-offs, when to pick which, tools, and how Compile Vision fine-tunes models for production.

Full fine-tuning an LLM means updating every weight. That is expensive, slow to store per customer, and awkward to ship. Parameter-efficient fine-tuning (PEFT) freezes the base model and trains a thin set of adapters. LoRA, QLoRA, and DoRA are the three names teams type into search when they want specialization without a second full model copy.

LoRA injects low-rank matrices. QLoRA keeps the base in 4-bit so big models fit on one GPU. DoRA decomposes weights into magnitude and direction to close more of the gap to full fine-tuning, still without extra inference cost after merge.

Compile Vision uses these methods in applied AI delivery: style and schema adapters, domain copilots, and multimodal instruction tunes. This briefing compares them so you can pick a default before the first training run.

LoRA: the default adapter

LoRA (Low-Rank Adaptation) freezes pretrained weights W and learns an update ΔW = BA where B and A are low-rank. Only A and B train. After training you can merge BA into W so inference latency matches the base model. That merge property is why LoRA beat many older adapters for production.

Typical practice: attach LoRA to attention projections (and sometimes MLP), choose rank r (4-64 common), train on a clean instruction or domain set, evaluate with held-out tasks. Start here unless you are GPU-memory constrained on a 30B+ model.

QLoRA: LoRA when VRAM is the bottleneck

QLoRA fine-tunes LoRA adapters while the base model stays frozen in 4-bit (NF4) with double quantization and paged optimizers. The famous claim: fine-tune models on the order of 65B parameters on a single ~48GB GPU while aiming for 16-bit fine-tune quality on many tasks.

Use QLoRA when the base does not fit in memory for normal LoRA, or when you want cheaper experimentation. Watch for slightly finickier training dynamics and make sure your serving path matches how you quantized. For many Compile Vision MVPs on 7B-13B models, plain LoRA on a 24GB GPU is enough; QLoRA shines as models grow.

DoRA: closer to full fine-tuning behavior

DoRA (Weight-Decomposed Low-Rank Adaptation) analyzes how full fine-tuning changes magnitude vs direction of weights, then fine-tunes those components separately: magnitude as a learnable vector, direction updated with a LoRA-style low-rank path. The goal is better learning capacity and stability than LoRA at similar parameter counts, with no added inference overhead after merge.

Reach for DoRA when LoRA underfits on hard instruction or multimodal tunes and you still refuse full FT. QDoRA-style stacks (DoRA on a quantized backbone) appear when you need both memory savings and stronger adaptation.

Key takeaways

LoRA is the default PEFT: small trainable ranks, mergeable, no inference tax.

QLoRA adds 4-bit bases so larger models fine-tune on fewer GPUs.

DoRA targets LoRA quality gaps by separating magnitude and direction.

Prefer RAG/prompts first; use PEFT when behavior must stick in the weights.

Compile Vision designs datasets, trains adapters, and ships serving + evals.

LoRA vs QLoRA vs DoRA at a glance

Same family of PEFT ideas; different constraints. Use this as a decision table, not a purity contest.

LoRA

How it works: Frozen full-precision (or bf16) base + trainable low-rank matrices; mergeable at deploy.
Best for: Most product adapters on 7B-13B (and many 70B with multi-GPU).
How it differs: Simplest mental model and widest tooling support.

QLoRA

How it works: 4-bit quantized frozen base + LoRA adapters; NF4, double quant, paged optimizers.
Best for: Large models on limited VRAM; cheap iteration on 30B-70B-class bases.
How it differs: Memory king; training stack is heavier than vanilla LoRA.

DoRA

How it works: Decompose weight into magnitude + direction; LoRA-style updates on direction.
Best for: When LoRA quality plateaus but full FT is still too costly.
How it differs: Often stronger than LoRA at similar ranks; still merge-friendly.

QDoRA / DoRA + quantization

How it works: DoRA adapters on a quantized backbone (community and research variants).
Best for: Hard tasks on big models with one-GPU budgets.
How it differs: Combines QLoRA memory wins with DoRA learning capacity.

Full fine-tuning

How it works: Update all (or most) weights; store a full checkpoint per variant.
Best for: When you own the model line and need maximum plasticity.
How it differs: Highest cost and storage; still the quality ceiling for some tasks.

Prompt / RAG only (no PEFT)

How it works: No weight updates; specialize with retrieval, tools, and prompts.
Best for: Fast MVPs and knowledge that changes weekly.
How it differs: Zero training ops; weaker for style/schema that must be permanent.

Tools & stack options

Practical pieces teams use to prototype and ship.

Hugging Face PEFT + Transformers
LoRA/DoRA configs and training loops
bitsandbytes
4-bit loaders used by QLoRA-style training
TRL / Axolotl / LLaMA-Factory
Opinionated fine-tune pipelines for instruction data
Unsloth / efficient trainers
Faster LoRA/QLoRA runs on consumer and cloud GPUs
Weights & Biases / eval suites
Track loss, win rates, and regressions per adapter
vLLM / TGI
Serve merged or adapter-stacked models in production

What you can build

Brand-voice support copilot adapters per customer
Schema-constrained JSON generators via LoRA
Domain medical/legal style tunes with strict eval gates
Multimodal instruction tunes with DoRA when LoRA stalls
QLoRA experiments on 70B bases before buying multi-GPU FT

Architecture & diagrams

LoRA update path

Frozen base weights plus low-rank trainable path.

Choosing PEFT method

Practical decision tree for product teams.

References & further reading

LoRA: Low-Rank Adaptation of Large Language Models (Hu et al.)
Original LoRA paper: frozen base + low-rank adapters, mergeable at inference.
QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al.)
4-bit NF4 fine-tuning with LoRA adapters on large models.
DoRA: Weight-Decomposed Low-Rank Adaptation (Liu et al.)
Magnitude/direction decomposition to improve on LoRA without inference overhead.
Hugging Face PEFT documentation
Practical APIs for LoRA and related PEFT methods.
Parameter-Efficient Fine-Tuning Methods: A Critical Review
Broader survey context for PEFT families including quantization adapters.

Need LoRA, QLoRA, or DoRA fine-tuning for your product?

Contact Compile Vision. We curate data, train adapters, run evals, and deploy merged models your users can trust. Start at /contact or email contact@compilevision.com.

Contact Compile Vision