AI Experts Are Dead. Long Live the AI Experts.

AI is democratised and monopolised at once. Everyone can prompt, training costs billions. Fine-tuning at a few hundred dollars is the practical moat.

Cover Image for AI Experts Are Dead. Long Live the AI Experts.

Last month, my eight-year-old built a Flappy Bird clone from scratch. He can't really type yet. He certainly can't write Python. What he can do is talk to Claude while I whisper in his ear what to say next. Within an hour, he had a working game: a bird, pipes, a score counter, gravity. He's an "AI expert" now.

And honestly? So is your dentist, your cousin's teenager, and the recruiter who just messaged you on LinkedIn. The barrier to "using AI" has collapsed to the cost of typing a sentence in English. This is genuinely wonderful. Democratisation of powerful technology is how we got the internet, smartphones, and open-source software.

But there's an asymmetry hiding behind this accessibility: while using AI has never been cheaper, building AI has never been more expensive. Training GPT-4 cost over $100 million. Llama 3 required 24,000 GPUs running for months. The companies that can afford to train foundation models from scratch fit comfortably in a single conference room. We've democratised the interface and monopolised the engine.

So where does that leave the engineers, the domain experts, the people who actually know things about medicine, law, finance, or logistics? Somewhere in between. And that somewhere has a name: fine-tuning. For a few hundred to a few thousand dollars, you can take a foundation model and make it yours, trained on your data, speaking your domain's language, following your formatting rules. Not building the engine from scratch, but tuning it to your track. This is where fractional AI advisory for UK firms has moved in: senior domain judgement on build-or-buy, data posture, and where fine-tuning actually earns its keep, part-time, without a full-time CAO hire.

By the end of this post, you'll fine-tune a model on Azure OpenAI, understand the LoRA algorithm that makes it computationally feasible, and know exactly where fine-tuning sits in the hierarchy from prompt engineering to pre-training.

Quick Win: Fine-Tune on Azure in 20 Lines

Let's start with the punchline. Here's everything you need to fine-tune a GPT model on Azure OpenAI. No Colab badge here (you'll need Azure credentials), but the code itself is almost disappointingly simple.

Prepare Your Training Data

Azure expects JSONL format: one JSON object per line, each containing a conversation. Here's what training data looks like for a medical coding assistant:

{"messages": [{"role": "system", "content": "You are a medical coding assistant. Map clinical descriptions to ICD-10 codes."}, {"role": "user", "content": "Patient presents with acute bronchitis"}, {"role": "assistant", "content": "J20.9 — Acute bronchitis, unspecified"}]}
{"messages": [{"role": "system", "content": "You are a medical coding assistant. Map clinical descriptions to ICD-10 codes."}, {"role": "user", "content": "Type 2 diabetes with diabetic chronic kidney disease"}, {"role": "assistant", "content": "E11.22 — Type 2 diabetes mellitus with diabetic chronic kidney disease"}]}
{"messages": [{"role": "system", "content": "You are a medical coding assistant. Map clinical descriptions to ICD-10 codes."}, {"role": "user", "content": "Essential hypertension"}, {"role": "assistant", "content": "I10 — Essential (primary) hypertension"}]}

Each line is a complete conversation with a system prompt, user input, and the desired assistant response.

Upload Data and Launch Fine-Tuning

from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint="https://YOUR_RESOURCE.openai.azure.com/",
    api_key="YOUR_API_KEY",
    api_version="2025-03-01-preview",
)

# Upload training file
training_file = client.files.create(
    file=open("training_data.jsonl", "rb"),
    purpose="fine-tune",
)

# Create fine-tuning job
job = client.fine_tuning.jobs.create(
    training_file=training_file.id,
    model="gpt-4o-mini-2024-07-18",
)

print(f"Job ID: {job.id}")
print(f"Status: {job.status}")

Check Status and Use Your Model

# Check progress
job = client.fine_tuning.jobs.retrieve(job.id)
print(f"Status: {job.status}")  # queued → running → succeeded

# Once succeeded, deploy and use your model
if job.status == "succeeded":
    response = client.chat.completions.create(
        model=job.fine_tuned_model,
        messages=[
            {"role": "system", "content": "You are a medical coding assistant."},
            {"role": "user", "content": "Chronic obstructive pulmonary disease with acute exacerbation"},
        ],
    )
    print(response.choices[0].message.content)
    # J44.1 — COPD with (acute) exacerbation

That's it. Azure handles the infrastructure, the training loop, the checkpointing, and the deployment. You provide the data; it returns a model that speaks your domain.

What Just Happened?

Behind those 20 lines, a lot happened. Let's unpack it.

The JSONL Format

Each training example is a conversation in the chat completions format you already know. The key fields:

  • role: "system": Sets the persona. Keep this consistent across examples.
  • role: "user": The input your model will receive in production.
  • role: "assistant": The exact output you want the model to learn.

You can optionally add a "weight": 0 field to any message to exclude it from the loss computation. This is useful when you want the model to see context but only learn from specific responses.

The Training Pipeline

When you call client.fine_tuning.jobs.create(), Azure kicks off a pipeline:

  1. Validation: Checks your JSONL for formatting errors, token limits, and minimum example counts (at least 10 examples required).
  2. Queuing: Your job waits for GPU capacity.
  3. Training: The model is fine-tuned using LoRA (more on this shortly). Azure automatically creates checkpoints.
  4. Results: A results.csv file is generated with training and validation loss at each step.

Hyperparameters

You can customise the training with the hyperparameters argument:

ParameterDefaultWhat It Controls
n_epochsAuto (based on dataset size)Number of passes through the training data
learning_rate_multiplierAutoScales the base learning rate. Higher means faster but riskier.
batch_sizeAutoExamples per gradient update. Larger is more stable but uses more memory.
seedNoneFor reproducibility
job = client.fine_tuning.jobs.create(
    training_file=training_file.id,
    model="gpt-4o-mini-2024-07-18",
    hyperparameters={
        "n_epochs": 3,
        "learning_rate_multiplier": 1.8,
        "batch_size": 4,
    },
)

For most use cases, the defaults work well. Azure auto-selects values based on your dataset size. If you want to systematically search for optimal values, hyperparameter optimisation methods like Bayesian optimisation can help.

Pricing

Fine-tuning costs vary by model tier:

TierTraining CostHosting CostBest For
StandardHigher per-tokenDedicated deploymentProduction workloads
Global StandardModeratePay-per-useCost-effective production

Training costs are measured in tokens processed. A dataset of 1,000 examples at ~200 tokens each, trained for 3 epochs, processes about 600K tokens, typically a few dollars for smaller models like gpt-4o-mini.

Going Deeper

What Fine-Tuning Actually Does

A language model is a probability distribution over the next token. When you prompt GPT-4o with "The capital of France is", it assigns high probability to "Paris" and low probability to "pizza". These probabilities are determined by the model's weights, billions of numbers learned during pre-training.

Fine-tuning shifts these probability distributions. For our medical coding assistant, the base model might assign:

  • P("J20.9") = 0.001 (it's seen ICD codes, but rarely)
  • P("The patient has") = 0.15 (a more "natural" continuation)

After fine-tuning on hundreds of medical coding examples, the distribution shifts:

  • P("J20.9") = 0.85
  • P("The patient has") = 0.002

Before vs after fine-tuning: the probability distribution over next tokens shifts from favouring generic continuations to favouring domain-specific outputs.

The training objective is cross-entropy loss on the assistant tokens, the same maximum likelihood estimation objective used in pre-training, just applied to a much smaller dataset. The model learns to maximise the probability of producing exactly the outputs in your training data.

The gradients that update the weights flow through the same backpropagation algorithm used in pre-training. The difference is scope: pre-training processes trillions of tokens across the entire internet; fine-tuning processes thousands of tokens from your specific domain.

LoRA: The Algorithm Under the Hood

Here's the problem with naive fine-tuning: GPT-4o has hundreds of billions of parameters. Updating all of them requires enormous GPU memory and risks catastrophically forgetting what the model already knows. This is where LoRA (Low-Rank Adaptation) comes in, and it's what Azure uses under the hood.

The key insight from Hu et al. (2021): when you fine-tune a large language model, the weight updates have low intrinsic rank. In plain English, the changes needed to adapt a model to a new task live in a much smaller subspace than the full parameter space.

Instead of updating a weight matrix W∈Rd×kW \in \mathbb{R}^{d \times k} directly, LoRA decomposes the update into two smaller matrices:

LoRA decomposes the weight update into two small, trainable low-rank matrices B and A, leaving the original weights frozen.

W′=W+ΔW=W+BAW' = W + \Delta W = W + BA

Where:

  • WW is the original frozen weight matrix (d×kd \times k)
  • B∈Rd×rB \in \mathbb{R}^{d \times r} and A∈Rr×kA \in \mathbb{R}^{r \times k} are the trainable low-rank matrices
  • r≪min⁡(d,k)r \ll \min(d, k) is the rank, typically 8, 16, or 64

The parameter reduction is dramatic. Consider a weight matrix in a large transformer:

  • Original: d=4096,k=4096d = 4096, k = 4096 → 16.8 million parameters
  • LoRA with r=16r = 16: (4096×16)+(16×4096)(4096 \times 16) + (16 \times 4096) = 131,072 parameters
  • Reduction: 99.2%

Across the entire model, LoRA typically trains only 0.1–1% of the original parameters. This means:

  1. Memory: You can fine-tune on a single GPU instead of a cluster.
  2. Speed: Fewer parameters to update means faster training.
  3. Storage: Each fine-tuned version is just the small BB and AA matrices, megabytes instead of gigabytes.
  4. No extra inference latency: At deployment, BABA is merged back into WW. The final model has exactly the same architecture and speed as the original.

The initialisation matters too: BB is initialised to zero and AA to random Gaussian, so ΔW=BA=0\Delta W = BA = 0 at the start. Training begins from the exact pre-trained model, with no disruption.

The PEFT Family

LoRA belongs to a broader family called Parameter-Efficient Fine-Tuning (PEFT) methods. Here's how they compare:

MethodTrainable ParamsMemoryQualityInference Overhead
Full fine-tuning100%Very highBestNone
LoRA0.1–1%LowNear-fullNone (merged)
QLoRA0.1–1%Very lowGoodSlight (quantisation)
Prefix tuningUnder 0.1%Very lowModerateSlight (extra tokens)
Adapters1–5%LowGoodSlight (extra layers)

LoRA is the most popular because it hits the sweet spot: near-full fine-tuning quality with no inference overhead. QLoRA adds 4-bit quantisation of the base model, reducing memory further. You can fine-tune a 65B parameter model on a single 48GB GPU. Prefix tuning prepends learnable "virtual tokens" to the input, but quality degrades for complex tasks. Adapters insert small trainable layers between existing transformer blocks.

The Moat Spectrum

Not all AI customisation is equal. Here's the full spectrum, from least to most defensible:

The moat spectrum: from prompt engineering (free, no moat) through RAG and fine-tuning to pre-training (very high cost, strong moat). Fine-tuning is the sweet spot.

ApproachCostSetup TimeMoat StrengthWhen to Use
Prompt engineeringFreeMinutesNonePrototyping, one-off tasks
RAG (Retrieval-Augmented Generation)$10s–100s/moDaysWeak (data can be copied)Need current information, citations
Fine-tuning$100s–1,000sDays–weeksModerate (behaviour is learned)Consistent formatting, domain tone, cost at scale
Pre-training$10Ms–100Ms+MonthsStrong (architecture + data)You're OpenAI, Google, or Meta

Prompt engineering is where most people stop. It works surprisingly well but offers zero competitive moat; anyone can copy your prompt. RAG adds your own data at inference time, which is powerful for knowledge-intensive tasks but the behaviour is still the base model's. Fine-tuning embeds behaviour into the weights. The model doesn't need to be told how to respond; it just does. Pre-training is building the engine from scratch, and unless you have a few hundred million dollars and a research lab, it's not your game. For investors writing cheques into companies at any of these tiers, the moat-spectrum question is exactly what AI due diligence is built to answer: is the target sitting on real weights they own, or on a prompt anyone can clone.

When Fine-Tuning Beats Prompting

Fine-tuning wins over prompting when you need:

  • Consistent output formatting. JSON schemas, code conventions, structured reports. A fine-tuned model follows the format without lengthy system prompts.
  • Domain-specific behaviour. Medical coding, legal analysis, financial compliance. The model internalises domain norms.
  • Tone and style. Brand voice, technical writing style, conversational patterns.
  • Cost at scale. A fine-tuned model with a short prompt is cheaper per request than a base model with a 2,000-token system prompt.
  • Latency. Shorter prompts mean fewer input tokens to process.

When Fine-Tuning Loses to RAG

Fine-tuning embeds knowledge into weights, but weights are frozen after training. If the information changes frequently (stock prices, medical guidelines, product catalogues), RAG is the better choice. RAG retrieves current documents at inference time, so the model always has access to the latest information.

The best systems often combine both: fine-tune for behaviour (how to respond), RAG for knowledge (what to respond with).

Where This Comes From

LoRA: Low-Rank Adaptation of Large Language Models

LoRA was introduced by Hu et al. (2021) at Microsoft Research. The paper's central hypothesis is elegant:

"We hypothesize that the change in weights during model adaptation also has a low 'intrinsic rank,' which leads us to propose Low-Rank Adaptation (LoRA)."

The authors demonstrated that for GPT-3 175B, LoRA with rank 4 matched or exceeded full fine-tuning performance on multiple benchmarks while training only 0.01% of the parameters. They tested on natural language understanding (GLUE), natural language generation (E2E NLG), and instruction following. LoRA matched full fine-tuning across the board.

A key practical insight from the paper: LoRA is most effective when applied to the attention weight matrices (WQW_Q and WVW_V), rather than the feed-forward layers. This is because attention matrices control the model's "routing" of information (which tokens attend to which), and task-specific behaviour is largely about changing these routing patterns.

ULMFiT: The Transfer Learning Paradigm

Before LoRA, there was ULMFiT. Howard & Ruder (2018) established the now-standard paradigm: pre-train on a large corpus, then fine-tune on your task. Their key innovations, discriminative fine-tuning (different learning rates per layer) and gradual unfreezing, are the conceptual ancestors of LoRA's approach.

The Broader Lineage

The idea that pre-trained representations can be adapted to new tasks has a long history:

  1. ImageNet transfer learning (2012–2014). Training on ImageNet, fine-tuning on medical images. Computer vision proved the concept.
  2. ULMFiT (2018). Brought transfer learning to NLP. Demonstrated that language model pre-training produces universal features.
  3. BERT (2018) and GPT (2018). Scaled the paradigm. Pre-train once, fine-tune for everything.
  4. LoRA (2021). Made fine-tuning efficient enough for massive models. You don't need to update every parameter.

Each step reduced the barrier. LoRA's contribution is making fine-tuning feasible for models so large that full fine-tuning would require a cluster.

Further Reading

Try It Yourself

  1. The format experiment. Take a task where you want structured output (e.g., JSON with specific fields). Compare: (a) a detailed system prompt describing the format, vs (b) a fine-tuned model trained on 50 examples of the correct format. Measure how often each produces valid output.
  2. Data quality vs quantity. Create two training sets for the same task: 50 carefully curated, high-quality examples vs 500 noisy, auto-generated examples. Fine-tune on each. Quality almost always wins. This is the moat.
  3. The moat test. Fine-tune a model on a specific domain task. Then try to replicate the same behaviour using only prompt engineering. How close can you get? Where does prompting fall short?
  4. LoRA from scratch. Implement a toy LoRA layer in PyTorch. Freeze a pre-trained GPT-2 model, add BABA matrices to the attention layers, and fine-tune on a small text classification task. Compare the parameter count to full fine-tuning.

Interactive Tools

  • Explore Our Free Tools: Hands-on calculators and visualisers for statistics, machine learning, and quantitative finance
  • AI Readiness Assessment: The benchmarked version of the question this post keeps returning to, scoring an organisation against ONS adoption data rather than against a vendor's checklist
  • Maximum Likelihood Estimation in Python from Scratch. Fine-tuning's loss function (cross-entropy) is maximum likelihood estimation. Understanding MLE gives you intuition for what the training loop is optimising.
  • Backpropagation and Neural Nets from First Principles. The gradient computation that makes both pre-training and fine-tuning work. LoRA reduces the number of parameters, but the gradients still flow through the same algorithm.
  • Hyperparameter Optimisation: Grid, Random, and Bayesian. Fine-tuning has its own hyperparameters (learning rate, epochs, LoRA rank). This post covers systematic approaches to tuning them.
  • LLM Cost Calculator: See live pricing across 300+ LLM APIs and what fine-tuning actually costs today
  • AI Strategy Consultant London: The practical engagement that sits behind this essay: a named principal at the table rather than a pyramid of associates.
  • AI Strategy for Hospitals and NHS Trusts: Board-ready AI strategy for NHS trusts and hospital groups: one named advisor who reads the MHRA, NICE, and DMA surfaces at the same time.
  • LLM Implementation Consulting: When strategy has to become code: model selection, RAG architecture, evaluation harness, and pilot-to-production scaling, by one senior advisor who has shipped production AI.
  • AI Readiness Assessment for CTOs: The scoped, CTO-facing version of this argument: a written six-pillar scorecard, a 12-month remediation plan, and a board briefing pack grounded in Gartner and McKinsey data.
  • AI Pilot Design Consultant: Pilot shortlisting, evaluation harness design, governance charter, and a 90-day execution plan for teams that want production, not a proof-of-concept. With 30–50% of gen AI pilots abandoned after POC, pilot design is where specialist AI judgement earns its keep.
  • AI Consulting for Mid-Market Companies: For UK mid-market firms weighing a Big-4 proposal: same senior judgement, 40–60% of the cost, no hand-off to an associate. The practical economics of this editorial argument, for buyers who have to defend the bill.
  • Boutique AI Consultancy UK: The hub page that organises the whole argument: twelve specialist practices, one named principal, built as the deliberate alternative to Big-4 associate-heavy consulting.
  • The AI Jobpocalypse, Your Kids, and Chicken Impossible: A related reflection on AI, work, and the next generation, picking up the thread this essay opens with an eight-year-old shipping a game.
  • Barriers to AI Adoption by Firm Size: the ONS asks UK businesses what has prevented or delayed them adopting AI, and expertise leads at every size band, which is the survey evidence under this essay's argument.

Stay in the loop

Get notified when new tutorials and deep-dives are published. No spam, unsubscribe anytime.

Related Posts

Frequently Asked Questions