LoRA from Scratch: Why LLM Fine-Tuning Got 1000x Cheaper

LoRA and QLoRA in pure PyTorch. Why low-rank adapters need 1000x fewer params, and how 4-bit base quantisation lets you fine-tune 65B models on one GPU.

Cover Image for LoRA from Scratch: Why LLM Fine-Tuning Got 1000x Cheaper

You have a trained model and want to adapt it to a new task, but updating every weight needs more training memory than you can spare. Much of the original model may already be useful for the job. LoRA keeps those weights fixed and learns a smaller adjustment alongside them, giving you a way to adapt the model with fewer trainable parameters.

Those smaller updates reduce gradient and optimiser-state storage. The LoRA paper reports up to 10,000 times fewer trainable parameters for GPT-3 175B. QLoRA fitted a 65-billion-parameter model on one 48 GB GPU. We build the low-rank adjustment and compressed base weights in PyTorch to compare storage and accuracy.

On our toy task, rank-2 LoRA matches full fine-tuning with 22x fewer trainable parameters. QLoRA's 4-bit quantisation gives 8x base-storage compression for a 1.4-percentage-point accuracy cost. We implement each in about 30 lines of PyTorch; the same operations apply to larger LLMs.

A working example: LoRA in 20 lines of PyTorch

import math
import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    """Wraps a frozen nn.Linear with a trainable low-rank update.

    Forward: h = W0 x + (alpha/r) B A x   with W0 frozen, only B and A trained.
    """
    def __init__(self, base: nn.Linear, r: int = 8, alpha: float = 16.0):
        super().__init__()
        self.base = base
        for p in self.base.parameters():
            p.requires_grad = False
        self.r = r
        self.scaling = alpha / r
        in_f, out_f = base.in_features, base.out_features
        # A: Kaiming uniform, B: zero. Update starts at zero so the LoRA-
        # injected model is initially identical to the frozen base.
        self.A = nn.Parameter(torch.empty(r, in_f))
        self.B = nn.Parameter(torch.zeros(out_f, r))
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))

    def forward(self, x):
        return self.base(x) + self.scaling * (x @ self.A.T) @ self.B.T

Open In Colab

That's it. Two new tensors per linear layer, both small. Every existing weight stays frozen. The model produces the same output as the original at initialisation (because B is zero), then learns a low-rank correction during fine-tuning.

To put this on a real workload: pretrain a small 4-block Transformer-style model on Task A, freeze it, then fine-tune on a related Task B using three strategies:

StrategyTrainable paramsTask B test accuracy
Frozen baseline (no fine-tune)026.0%
Full fine-tune67,78093.6%
LoRA r=8 on FFN layers12,28894.2%

LoRA matches and slightly exceeds full fine-tuning while training 6x fewer parameters. That ratio gets dramatically better as the base model grows: on a 7-billion-parameter LLM, a typical LoRA configuration trains around 4 million parameters, a 1750x reduction.

The low-rank intuition

The idea behind LoRA rests on one observation about how fine-tuning works in practice. When you take a pretrained model and adapt it to a downstream task, the matrix of updates ΔW\Delta W you apply to each weight matrix turns out to have very low intrinsic rank. Even though ΔW\Delta W is technically d×kd \times k (potentially millions of entries), the directions in which it moves the weights live in a tiny subspace.

Hu et al. tested this empirically. They froze W0W_0, computed ΔW\Delta W from a normal fine-tuning run, then SVD-decomposed ΔW\Delta W and found that the top 8 to 64 singular values captured almost all of its mass. The other thousands of dimensions of ΔW\Delta W were effectively zero. This means you don't need to learn a full-rank update; you only need to learn its low-rank approximation.

LoRA bakes this directly into the architecture by replacing the unconstrained update ΔW\Delta W with an explicit low-rank decomposition:

ΔW=BA,B∈Rd×r,A∈Rr×k,r≪min⁡(d,k)\Delta W = B A, \qquad B \in \mathbb{R}^{d \times r}, \quad A \in \mathbb{R}^{r \times k}, \quad r \ll \min(d, k)

The forward pass through a LoRA-injected layer becomes:

h=W0x+ΔWx=W0x+BAxh = W_0 x + \Delta W x = W_0 x + B A x

W0W_0 is frozen at the pretrained value. Only BB and AA are trainable. The trainable parameter count drops from dkdk (for the original matrix) to r(d+k)r(d + k) (for the two LoRA factors). When d=k=4096d = k = 4096 and r=8r = 8, that's 16M parameters → 65k, a 246x reduction per layer.

There's one detail worth flagging. LoRA initialises AA randomly (Kaiming uniform) and BB to zero. That means at the start of fine-tuning, BA=0BA = 0 and the LoRA-injected model produces exactly the same output as the frozen base. This is what makes LoRA training so stable: you start from "no change" and the optimiser carves out the update from there, never destabilising the pretrained features. There's also a scalar α/r\alpha / r applied to the LoRA branch in practice; Hu et al. recommend setting α=2r\alpha = 2r so the effective scale stays roughly constant as you sweep rr.

The Minimum Usable Rank

Sweeping the rank rr from 1 up to 32 on the same toy fine-tuning task gives a striking result:

Rank ablation: LoRA matches full FT at r=2 with 22x fewer trainable parameters; the curve plateaus at r=4 and stays flat

At r=1r = 1, accuracy is 89.4%. At r=2r = 2, it's already 93.6%, exactly matching the full fine-tune. Above r=4r = 4 everything plateaus around 94%. The right panel makes the parameter-efficiency point explicit: the entire curve fits in a parameter budget two orders of magnitude smaller than full FT, with no quality loss.

This is not specific to toy data. The LoRA paper reports the same plateau on GPT-3 175B fine-tuning: rank 8 on the attention QQ and VV matrices matches full fine-tuning across 8 different downstream tasks. Rank 1 is sometimes enough. The rank you need depends on how much of the pretrained model's behaviour the new task changes, which turns out to be very little for most fine-tuning scenarios.

The training-curve view of the same experiment makes it visually obvious that rank-1 underfits while r ≥ 4 essentially behaves identically to full fine-tuning:

Training accuracy curves over 80 epochs for LoRA r=1, r=4, r=16, and full FT. r=4 and above track full FT closely while r=1 lags

Where to inject LoRA

LoRA is layer-targeted: you choose which weight matrices in the network get a low-rank adapter. The targets matter more than the rank.

Where to inject LoRA: FFN layers capture nearly all the benefit; head-only barely beats the frozen baseline

Adapting only the classifier head (just 544 trainable parameters) gets 83.2% on Task B; adapting the FFN layers gets 94.2%. The lift is overwhelmingly in the dense feed-forward weights, not the output layer. In real LLMs the picture is similar: the most common LoRA configuration adapts the attention Q,K,V,OQ, K, V, O projections (which you can think of as eight large dense matrices per Transformer block) and leaves the rest of the network untouched. Recent work (Loftq, DoRA) extends this to the FFN layers and claims further gains.

Adding "everything except the head" doesn't beat "all FFN linears" on this task, suggesting the embedding layer doesn't need adapting at all when the input distribution stays similar to pretraining. For tasks with very different inputs (medical imaging fine-tuned from natural images, for example), you'd see the embedding LoRA contribute more.

QLoRA: 4-bit quantisation of the frozen base

LoRA shrinks the trainable parameter count, but the frozen base still has to live in memory during the forward and backward passes. For a 7B model that's 28GB in FP32 or 14GB in BF16. For 65B, the frozen base needs over 130GB. Even with LoRA, you can't fine-tune those models on consumer hardware.

QLoRA's contribution is to attack the frozen-base side of the equation. The base weights are quantised to 4 bits each, so a 7B model goes from 14GB (BF16) to roughly 3.5GB. The LoRA adapters stay in higher precision (BF16 or FP32) because those are the parameters that need to update; gradient flow through 4-bit weights is fine because those weights are constants from the optimiser's perspective.

Here's a uniform-quantile 4-bit quantiser that captures the core trade-off (the real QLoRA paper introduces a non-uniform NormalFloat4 codebook calibrated to standard normal weight distributions, but the accuracy story is the same):

def quantise_4bit(W: torch.Tensor, n_levels: int = 16):
    """Symmetric per-tensor 4-bit quantisation."""
    absmax = W.abs().max() + 1e-9
    scale = absmax / ((n_levels - 1) / 2)
    q = torch.round(W / scale).clamp(-(n_levels // 2), n_levels // 2 - 1)
    return q * scale  # de-quantise back to FP32 for the matmul

Run the same Task A → Task B fine-tune with three configurations and the result speaks for itself:

QLoRA: 4-bit base barely costs any accuracy (94.0% → 92.6%) but cuts base storage by 8x

A 1.4 percentage-point accuracy drop for an 8x reduction in base-model storage. On real LLMs the QLoRA paper reports near-zero accuracy degradation across a wide range of benchmarks; the small gap here is mostly because uniform 4-bit quantisation is cruder than NormalFloat4. The scaling is exactly what you need for consumer hardware: a 65B model in INT4 is around 32.5GB, and the LoRA adapters add only a few hundred MB. That fits on a single A6000 or even a 24GB consumer GPU after careful memory management.

QLoRA also introduces two engineering tricks that matter at scale: double quantisation (quantising the quantisation constants themselves, saving another 0.4 bits per parameter on average) and paged optimisers (using NVIDIA unified memory to handle gradient-checkpoint memory spikes without OOM). These are systems contributions; the algorithmic core is just LoRA + 4-bit base.

The checkpoint-size dividend

Beyond memory at training time, LoRA produces a much smaller checkpoint than full fine-tuning. You only have to save the adapter weights, not the entire model:

LoRA stores only the adapter, 6x smaller than the full-FT checkpoint on this toy model, and far higher ratios at LLM scale

At LLM scale this matters operationally. A team running 100 fine-tuned variants of a 7B base model would need 1.4TB of full-FT checkpoints in BF16. With LoRA at r=8r = 8, the same 100 adapters total around 1-2GB. You can ship them by email. You can also swap adapters at inference time without reloading the base, which is what makes adapter hubs (Hugging Face's PEFT, the OpenAI fine-tuning API's per-customer adapter routing) commercially feasible.

There's a related trick called LoRA merging: at deployment time, you can compute W=W0+BAW = W_0 + BA once and replace the two LoRA matmuls with a single matmul against the merged weight. The merged model has zero LoRA-induced inference overhead and is bit-exact equivalent to the LoRA-adapted model. This is why LoRA is so widely deployed: the training and storage savings are large, and the inference cost is unchanged.

When LoRA is the right tool

A short decision tree:

SituationReach for
Adapting a pretrained LLM to a domain or taskLoRA (or QLoRA if VRAM is tight)
Many fine-tuned variants of one base model (per-customer, per-task)LoRA adapters swapped at inference time
Pretraining from scratchFull training; LoRA is for adapting, not for training the base
Domain shift is so extreme that the pretrained features are uselessFull fine-tune or pretrain from scratch on the new domain
Tiny model where full FT already fits in memoryFull fine-tune; LoRA's overhead isn't worth it

The two scenarios where LoRA dominates are (1) when the base is too large to fully fine-tune on your hardware, and (2) when you need many task-specific variants of a single shared base. Both situations describe almost every production LLM deployment in 2026.

LoRA also composes elegantly with other techniques covered on this site. The adapter checkpoints can be merged into sparse mixture-of-experts layers (one expert per task, gated at inference time). The QLoRA quantisation story sits alongside TurboQuant's random-rotation quantisation and the KV cache memory bottleneck as the three biggest LLM-efficiency levers in active use.

Foundations: Hu, Aghajanyan, Dettmers

The LoRA story rests on three papers in chronological order.

Aghajanyan, Zettlemoyer, Gupta (2020), "Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning", ACL 2021. This is the empirical result that LoRA is built on. The authors showed that for many downstream tasks, you can fine-tune a model by optimising only a low-dimensional reparameterisation of its weights and still match full fine-tuning quality. Their reparameterisation was a random projection; LoRA's contribution was to learn the projection too.

Hu et al. (2021), "LoRA: Low-Rank Adaptation of Large Language Models", ICLR 2022. The paper that crystallised the technique. They showed that on GPT-3 175B, LoRA at r=4r = 4 matches full fine-tuning across 5 different benchmarks while training 10,000x fewer parameters. They also introduced the Kaiming-uniform-AA / zero-BB initialisation that makes LoRA training stable, and the α/r\alpha/r scaling that lets you change rank without re-tuning the learning rate.

Dettmers et al. (2023), "QLoRA: Efficient Finetuning of Quantized LLMs", NeurIPS 2023. The follow-up that made consumer-hardware fine-tuning of 65B models a reality. Their three contributions: (1) NormalFloat4, a non-uniform 4-bit data type calibrated to the standard normal distribution that pretrained weights tend to follow; (2) double quantisation of the quantisation constants; (3) paged optimisers. They demonstrated 99.3% of full-FT performance on the Vicuna benchmark with a 33B model fine-tuned on a single 24GB GPU.

Interactive Tools

  • LLM Cost Calculator: Estimate fine-tuning cost differences between full FT and LoRA at typical adapter sizes; the trainable-parameter ratio drives most of the cost difference.
  • Overfitting Explorer: LoRA's bias toward low-rank updates is itself a form of regularisation; this tool builds intuition for the underlying bias-variance trade-off.
  • Convolution Visualiser: Linear-operator intuition that transfers directly to understanding what BABA does as a learned low-rank correction to W0W_0.

Stay in the loop

Get notified when new tutorials and deep-dives are published. No spam, unsubscribe anytime.

Related Posts

Frequently Asked Questions