Anomaly Detection from Scratch: 3 Detectors Compared

Build three unsupervised anomaly detectors from scratch: Isolation Forest, One-Class SVM and an autoencoder. See why ROC-AUC lies on imbalanced fraud data.

Cover Image for Anomaly Detection from Scratch: 3 Detectors Compared

A factory sensor has been recording ordinary operation for weeks, and you want an alert when its readings begin to look unusual. You may have plenty of normal data but few examples of a failure, making it hard to train a classifier for every fault. Anomaly detection starts from the patterns in the data you do have and looks for observations that depart from them. The next challenge is deciding which departures deserve attention.

We test alerts on rare credit-card fraud. Isolation Forest scores ROC-AUC 0.951, but average precision is only 0.148. That ROC score can coexist with many false alerts. We compare isolation, boundary fitting and reconstruction error, checking their definitions of unusual and their disagreements.

This post builds all three from first principles: an Isolation Forest in pure NumPy, a One-Class SVM boundary, and a PyTorch autoencoder scored by reconstruction error. We run them on real credit-card fraud, compare them with one ROC and precision-recall panel, and confront the metric trap that makes every one of them look near-perfect when it is not. By the end you will have implemented isolation from scratch and you will know which number to trust.

Let's Build It

First, meet the haystack. The classic ULB credit-card fraud dataset holds 284,807 European card transactions from September 2013, of which 492 are fraud. That is 0.172 percent, one fraud in every 579 transactions. The features are 28 anonymised PCA components plus the transaction amount.

Left: a bar chart on a log scale showing 284,315 legitimate transactions towering over 492 fraud cases. Right: a scatter plot of two features, with legitimate transactions forming a dense grey cloud and fraud cases in red spreading out along the fringe, though a stubborn group of fraud points sits buried inside the legitimate cloud.

Open the notebook and run the whole pipeline in Colab:

Open In Colab

Here is the fastest possible detector. No labels go into training, only the raw features:

import numpy as np
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import IsolationForest
from sklearn.metrics import roc_auc_score, average_precision_score

df = fetch_openml(data_id=1597, as_frame=True).frame          # ULB credit-card fraud
y = df.pop("Class").astype(int).values
X = df.select_dtypes("number").values

Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.5, stratify=y, random_state=42)
Xtr, Xte = (s := StandardScaler().fit(Xtr)).transform(Xtr), s.transform(Xte)

model = IsolationForest(n_estimators=100, max_samples=256, random_state=0).fit(Xtr)
scores = -model.score_samples(Xte)                            # higher = more anomalous

print(f"ROC-AUC = {roc_auc_score(yte, scores):.3f}")          # 0.951
print(f"PR-AUC  = {average_precision_score(yte, scores):.3f}") # 0.148

A ROC-AUC of 0.951 out of the box, with zero labels and zero tuning. That number looks like a solved problem. It is not. The precision-recall score sitting right next to it is 0.148, and the gap between those two numbers is the whole story of this post. Hold that thought. First we need to understand what the forest did, and to do that we build it ourselves.

What Just Happened?

Novelty versus outlier detection

Two setups hide under the word "anomaly", and mixing them up causes most of the confusion in practice.

Outlier detection assumes your training data is already contaminated with a few anomalies, and you want to find them in that same batch. The Isolation Forest above works this way: it saw all 142,403 training transactions, fraud included, and still ranked the strange ones highly.

Novelty detection assumes you can train on a clean, anomaly-free sample of normal data, then flag anything that deviates at test time. The One-Class SVM and the autoencoder we build later work best this way: fit them on legitimate transactions only, and fraud shows up as a stranger.

The knob that connects the two is contamination, your assumed fraction of anomalies. It sets where the score threshold falls, not the ranking itself. Get the ranking right first; choose the threshold second.

Anomalies are easy to isolate

The Isolation Forest rests on one idea you can see in a picture. Pick a random feature, pick a random split point, and cut the data in two. Repeat inside each half. A point sitting far from the crowd gets fenced off into its own region after only a few cuts, because almost any random split separates it from the pack. A point buried in a dense cluster survives many cuts before it is finally alone. The number of splits needed to isolate a point is therefore a measure of how anomalous it is: short is strange, long is normal.

A two-panel animation. Left panel, an anomaly sitting apart from a Gaussian blob is fenced into its own cell by four random horizontal and vertical cuts. Right panel, the same process run on a normal point in the middle of the blob needs eleven cuts, with the splitting lines crowding densely around it before it is isolated.

The whole algorithm is a handful of NumPy. An isolation tree recursively splits a small random subsample until every point is alone or a height limit is hit:

class Node:
    __slots__ = ("feature", "thresh", "left", "right", "size", "depth", "leaf")

def build_tree(X, depth, height_limit, rng):
    node = Node(); n = len(X)
    if depth >= height_limit or n <= 1:
        node.leaf, node.size, node.depth = True, n, depth
        return node
    f = rng.integers(0, X.shape[1])                 # random feature
    lo, hi = X[:, f].min(), X[:, f].max()
    if lo == hi:
        node.leaf, node.size, node.depth = True, n, depth
        return node
    t = rng.uniform(lo, hi)                          # random split point
    mask = X[:, f] < t
    node.leaf, node.feature, node.thresh, node.depth = False, f, t, depth
    node.left  = build_tree(X[mask],  depth + 1, height_limit, rng)
    node.right = build_tree(X[~mask], depth + 1, height_limit, rng)
    return node

Each tree sees only a subsample of ψ=256\psi = 256 points, so trees are cheap and diverse. A forest is just a list of them:

def fit_iforest(X, n_trees=100, psi=256, seed=0):
    rng = np.random.default_rng(seed)
    height_limit = int(np.ceil(np.log2(psi)))       # average tree height
    return [build_tree(X[rng.choice(len(X), min(psi, len(X)), replace=False)],
                       0, height_limit, rng) for _ in range(n_trees)]

To score a point, drop it through every tree, record the depth at which it lands, and average. Anomalies land shallow, normal points land deep.

Two histograms. Left: the average path length across 100 trees, where fraud cases pile up around 5 to 8 splits while legitimate transactions cluster near 14. Right: the same points after the score transform, where legitimate transactions sit near 0.4 and fraud spreads out toward 0.7, with a clear separation and a small overlapping tail.

Run the from-scratch forest against sklearn's and they agree closely: our path-length scores correlate with sklearn's at a Spearman ρ\rho of 0.94, and the ROC-AUC lands at 0.956 versus sklearn's 0.951. The 60 lines above are the real algorithm, not a toy.

Going Deeper

Expected path length and the anomaly score

Averaging raw depths has a bug: a point in a big subsample naturally sits deeper than one in a small subsample, so depth is not comparable across trees. Liu, Ting and Zhou fix this by normalising against the depth you would expect by chance. An isolation tree is a binary search tree, and the average depth of a node in a random BST of nn points is known in closed form:

c(n)=2H(n−1)−2(n−1)n,H(i)≈ln⁡(i)+0.5772c(n) = 2 H(n-1) - \frac{2(n-1)}{n}, \qquad H(i) \approx \ln(i) + 0.5772

c(n)c(n) is the yardstick: the typical number of splits to isolate one of nn points if nothing were unusual. The anomaly score for a point xx compares its observed mean depth E[h(x)]E[h(x)] against that yardstick:

s(x,n)=2− E[h(x)] / c(n)s(x, n) = 2^{-\,E[h(x)] \,/\, c(n)}

Read the exponent. If a point takes far fewer splits than expected, E[h(x)]≪c(n)E[h(x)] \ll c(n), the ratio is small, and the score pushes toward 1 (anomaly). If it takes about as many as expected, the ratio is near 1 and the score sits near 0.5. If it takes many more, the score falls toward 0 (firmly normal). That is the right panel of the histogram above: the score is nothing more than the path length, folded through this exponential so that "short path" becomes "high score".

EULER = 0.5772156649
def c_factor(n):
    return 2.0 * (np.log(n - 1) + EULER) - 2.0 * (n - 1) / n if n > 1 else 0.0

def anomaly_score(paths, psi):
    return 2.0 ** (-paths / c_factor(psi))          # paths = mean depth per point

The forest has two hyperparameters and both come straight from the paper: 100 trees and a subsample size ψ=256\psi = 256. The subsample is the important one. A small ψ\psi is a feature, not a limitation: it keeps each tree's "normal" region tight, so a cluster of near-duplicate anomalies (a fraud ring reusing one pattern) cannot hide by making itself look dense.

A boundary around the normal region

The Isolation Forest asks how easy a point is to cut off. A One-Class SVM asks a geometric question instead: draw the smallest region in feature space that contains almost all the normal data, then flag anything outside it. Schölkopf's formulation separates the data from the origin with a maximum-margin hyperplane in a kernel feature space; the equivalent SVDD of Tax and Duin wraps it in the smallest enclosing hypersphere. With an RBF kernel both bend into a flexible contour that hugs the shape of the normal cloud, even when that cloud has several separate lumps.

A two-dimensional scatter with two grey clusters of normal training points, each wrapped in a smooth amber boundary curve produced by an RBF One-Class SVM. Red X markers scattered across the rest of the plane, outside both boundaries, mark the novel points the detector rejects.

It is the same margin machinery from Support Vector Machines, retargeted from "separate two classes" to "enclose one class". You get it in three lines, fitting on legitimate transactions only so it is a true novelty detector:

from sklearn.svm import OneClassSVM

normals = Xtr[ytr == 0]                              # train on legitimate only
oc = OneClassSVM(kernel="rbf", nu=0.05, gamma="scale").fit(normals[:6000])
oc_scores = -oc.decision_function(Xte)               # outside the boundary = high

The ν\nu parameter (here 0.05) does double duty: it upper-bounds the fraction of training points allowed outside the boundary and lower-bounds the fraction of support vectors. It is your contamination assumption in disguise. The RBF kernel scales poorly with sample count, which is why we fit on a 6,000-point subsample. On the fraud test set this boundary scores a ROC-AUC of 0.950, level with the forest.

Reconstruction error as a score

The third lens is the most flexible and the one that carries into deep learning. Train an autoencoder, a neural network that compresses its input to a small bottleneck and rebuilds it, on normal data alone. It learns to reconstruct legitimate transactions with tiny error because it has seen thousands of them. Feed it fraud and it stumbles: the patterns it learned do not fit, so the reconstruction error spikes. The error is the anomaly score.

import torch, torch.nn as nn

Xn = torch.tensor(Xtr[ytr == 0], dtype=torch.float32)          # legitimate only
ae = nn.Sequential(nn.Linear(29, 16), nn.ReLU(), nn.Linear(16, 8), nn.ReLU(),
                   nn.Linear(8, 16), nn.ReLU(), nn.Linear(16, 29))
opt, loss_fn = torch.optim.Adam(ae.parameters(), lr=1e-3), nn.MSELoss()

for epoch in range(30):
    for i in range(0, len(Xn), 512):
        batch = Xn[torch.randperm(len(Xn))[i:i + 512]]
        opt.zero_grad(); loss = loss_fn(ae(batch), batch); loss.backward(); opt.step()

with torch.no_grad():
    Xt = torch.tensor(Xte, dtype=torch.float32)
    ae_scores = ((ae(Xt) - Xt) ** 2).mean(1).numpy()           # reconstruction MSE

The training loss falls smoothly from 0.87 to 0.46 and flattens, the sign of a fit that has learned the normal manifold without memorising noise.

A training curve of reconstruction MSE against epoch, falling steeply from 0.87 at epoch 1 to about 0.50 by epoch 10, then flattening gradually to 0.46 by epoch 30.

Scored on the test set, legitimate transactions reconstruct with near-zero error while fraud spreads across a long high-error tail. A threshold at the 95th percentile of all errors catches most of the fraud at a cost of some false alarms.

A histogram of reconstruction error. Legitimate transactions form a tall spike near zero error, while fraud cases spread out into a long tail extending far to the right. A dashed vertical line marks the 95th-percentile threshold just past the legitimate spike.

The compress-then-reconstruct idea, and the discovery that reconstruction error carries a signal, is the same instinct behind JEPA and modern self-supervised learning, which push it further by predicting in latent space instead of input space.

The visually similar mimic

Reconstruction error shines on the problem that seeded this post: telling apart two things that look almost identical. Train the autoencoder on one class only, then show it a lookalike. Here the network learns handwritten 3s and never sees an 8. An 8 is close to a 3 with its left side closed, so a human glances and hesitates. The autoencoder does not: it tries to rebuild the 8 using only the vocabulary of 3s, erases the left loops, and returns something that looks like a 3. That failure is exactly the signal.

Top left, four handwritten 3s above their autoencoder reconstructions, which are clean 3s. Top right, four handwritten 8s above their reconstructions, which have been dragged toward 3-like shapes with the left loops erased. Bottom, a histogram showing digit-3 reconstruction error clustered low and digit-8 error clustered high, separated with a ROC-AUC of 0.96.

The reconstruction of the mimic is visibly wrong, and the error separates the two digits at a ROC-AUC of 0.96, without the network ever training on a single 8. This is the shape of the medical use case, a detector that learns healthy tissue and flags the lookalike that is not, and it is why reconstruction methods matter even where a boundary or an isolation tree would struggle.

Accuracy is the wrong metric

Now the reckoning. All three detectors land near a ROC-AUC of 0.95. Plot the ROC curves and they hug the top-left corner, the picture of success.

Two panels. Left, ROC curves for all three detectors climbing steeply to the top-left corner, with AUCs of 0.956 for Isolation Forest, 0.950 for One-Class SVM, and 0.966 for the autoencoder, all far above the random diagonal. Right, precision-recall curves for the same three detectors collapsing toward the bottom of the plot, with average-precision scores of only 0.123, 0.229, and 0.248, far below any reasonable operating point.

The precision-recall panel on the right tells a different story. The same three models score an average precision of 0.12 to 0.25. Why the disagreement? ROC-AUC measures the true-positive rate against the false-positive rate, and with 284,315 legitimate transactions the false-positive rate barely moves even when you wrongly flag thousands of them. It is measured against a denominator so large that mistakes vanish. Precision measures how many of your alerts are real, and with 492 needles in 284,807 pieces of hay, most alerts are false by construction. Accuracy is worse still: a model that predicts "legitimate" for everything scores 99.83 percent accuracy and catches zero fraud.

The metric that survives contact with this imbalance is precision@k: of the top kk transactions your model flags as most anomalous, how many are real fraud? Set kk to the 246 true anomalies in the test set and the three detectors catch 52, 70, and 75 of them respectively. The autoencoder, top of the class, still buries two false alarms for every real fraud it surfaces.

A grouped bar chart with three metric groups. Under ROC-AUC all three detectors sit near 0.95 to 0.97. Under PR-AUC they collapse to 0.12, 0.23, and 0.25. Under Precision@246 they sit at 0.21, 0.28, and 0.30. An annotation points at the tall ROC-AUC bars reading "ROC-AUC flatters: all three look near-perfect".

This is not a failure of the detectors, it is the reality of a 0.17 percent base rate, and it is the exact failure mode we dissected in SMOTE vs class weights. Report ROC-AUC on a problem like this and you will convince yourself a mediocre model is excellent. Report PR-AUC and precision@k and you see the real operating point: a triage tool that surfaces a manageable review queue in which a useful minority are true, not an oracle.

Choosing the threshold

A ranking is not yet a decision. Turning scores into alerts means picking a cut, and the honest way to pick it is by capacity, not by a magic number. If your review team can investigate 200 cases a day, threshold at the 200 highest scores and measure the precision you get there. If a missed fraud costs far more than a wasted review, push the threshold down and accept more false alarms to lift recall. The contamination argument in scikit-learn and the ν\nu in the One-Class SVM are just conveniences that pick this cut for you from an assumed anomaly rate; when you know your real review budget, set it directly. For anomalies that arrive as a shift in a stream over time rather than as points in a fixed batch, a sequential method fits better: see Bayesian Online Changepoint Detection.

Where This Comes From

Anomaly detection is not one algorithm but a family, and the three lenses here trace to distinct lineages two decades deep.

A horizontal timeline of five landmark papers, colour-coded by paradigm: One-Class SVM by Schölkopf and colleagues in 2001 and SVDD by Tax and Duin in 2004 as boundary methods, Isolation Forest by Liu, Ting and Zhou in 2008 as an isolation method, autoencoder anomaly detection by Sakurada and Yairi in 2014 as a reconstruction method, and the Ruff unifying review in 2021 as a synthesis.

Isolation Forest. Liu, Ting and Zhou introduced it in "Isolation Forest" (ICDM 2008). Their move was to break with every method that came before it. Earlier detectors first build a profile of normal points (a density, a distance, a boundary) and call anything far from it an anomaly, which is expensive and struggles in high dimensions. Liu and colleagues instead isolate anomalies directly, and they prove the key claim: because anomalies are "few and different", they are more susceptible to isolation, so their expected path length is shorter. The score s(x,n)=2−E[h(x)]/c(n)s(x,n) = 2^{-E[h(x)]/c(n)} and the normaliser c(n)c(n) we implemented are equations 1 and 2 of the paper, verbatim.

"Anomalies are 'few and different', which make them more susceptible to isolation than normal points." (Liu, Ting & Zhou, 2008)

One-Class SVM and SVDD. Schölkopf, Platt, Shawe-Taylor, Smola and Williamson posed one-class classification as estimating the support of a high-dimensional distribution (Neural Computation, 2001): find the maximum-margin hyperplane separating the data from the origin in a kernel space. Tax and Duin's Support Vector Data Description (Machine Learning, 2004) recast the same idea as the smallest enclosing sphere, which is equivalent for the RBF kernel and often more intuitive.

Autoencoder reconstruction. Sakurada and Yairi's "Anomaly Detection Using Autoencoders with Nonlinear Dimensionality Reduction" (MLSDA 2014) showed that reconstruction error from an autoencoder detects anomalies where linear PCA fails, because the nonlinear bottleneck captures curved manifolds a linear subspace cannot.

The synthesis. Ruff and colleagues' "A Unifying Review of Deep and Shallow Anomaly Detection" (Proceedings of the IEEE, 2021) is the map of the whole territory. It places isolation, boundary, and reconstruction methods in one framework and is the single best next read once the three from-scratch builds here have made the ideas concrete.

Further Reading

Interactive Tools

  • Classification Metrics Calculator: Feed in a confusion matrix and watch precision, recall, and F1 diverge from accuracy exactly the way they do under the fraud imbalance in this post.
  • Medical Statistics Calculator: Sensitivity, specificity, and PPV under a low base rate: the clinical mirror of precision@k, where a "99% accurate" test still yields mostly false positives when the disease is rare.

Stay in the loop

Get notified when new tutorials and deep-dives are published. No spam, unsubscribe anytime.

Related Posts

Frequently Asked Questions