Tabular Foundation Models: How TabPFN Predicts Without Training

TabPFN is a transformer pre-trained on millions of synthetic datasets that predicts in one forward pass: no training. Benchmarked against XGBoost.

Cover Image for Tabular Foundation Models: How TabPFN Predicts Without Training

You have a small labelled table and want predictions for a few new rows. A model such as gradient-boosted trees learns from your training rows before it can make those predictions. TabPFN takes another approach, reading the labelled and unlabelled rows together as context for a model trained in advance. The labelled rows supply examples at prediction time, so there is no optimiser run on your own dataset.

We test that prediction route against models fitted to each table. The 2025 Nature paper reports TabPFN v2 beating tuned baselines; ours are untuned. Prediction takes under a second on small datasets but nearly two minutes on digits using a CPU. We compare accuracy and elapsed time.

This post puts that claim under load. We run TabPFN against logistic regression, XGBoost, and histogram gradient boosting on six small datasets, measure accuracy and wall-clock, and watch its decision boundary form as we feed it more in-context examples. By the end you will understand what "prior-data fitted network" means, why a single forward pass can substitute for training, and exactly where this approach still falls down. Averaged over our six datasets, TabPFN scored 0.955 against XGBoost's 0.921.

Let's Build It

Click the badge to open the interactive notebook:

Open In Colab

Watch the decision boundary form. Each frame fits TabPFN on a few more in-context examples and re-plots its predicted boundary. There are no training epochs between frames; every boundary is a single forward pass over the points shown so far.

Animated decision boundary on a two-moons dataset. Starting from 5 in-context examples the boundary is vague, and as the count grows to 300 it sharpens into a clean curved separator between the red and blue classes. Each frame is one forward pass with no gradient steps.

Using it is anticlimactic, which is the point. There is no training loop to write:

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from tabpfn import TabPFNClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

clf = TabPFNClassifier(device="cpu")
clf.fit(X_train, y_train)        # no optimisation happens here -- it just stores the data
print("accuracy:", clf.score(X_test, y_test))
accuracy: 0.9766

That fit call does not train anything. It stores your training rows so they can be fed into the transformer alongside the test rows at prediction time. The pre-trained weights never change. Compare that with XGBoost, which runs hundreds of boosting rounds on every new dataset, and the shift in mental model is the whole story.

What Just Happened?

In-context learning, moved to tables

If you have used a large language model, you have already seen this trick. Give an LLM a few examples in the prompt ("translate hello to French: bonjour; translate cat to French: chat; translate dog to French:") and it completes the pattern without any weight updates. That is in-context learning: the model learns the task from the examples in its context window, at inference time, using only a forward pass.

TabPFN does exactly this for tables. Your training rows are the examples in the context; your test rows are the queries to complete. The transformer attends across all the rows at once, the labelled ones informing the unlabelled ones, and emits a predicted class distribution for each test row. The "learning" is the forward pass attending from test rows to training rows, not an optimiser grinding on a loss.

This is why the linear regression mental model (fit coefficients, store them, discard the data) does not apply. TabPFN keeps the data and throws away the idea of fitted parameters per dataset entirely.

The decision boundary tells the story

Different model families carve up feature space in characteristically different shapes. Plotting their decision boundaries on the same two-dimensional data makes the difference obvious.

Three side-by-side decision boundaries on the same two-moons data. Logistic regression draws a single straight line that cannot separate the crescents. Gradient boosting draws blocky axis-aligned steps that follow the curve roughly. TabPFN draws a smooth curved boundary that hugs the gap between the two classes.

Logistic regression can only draw a straight line, so the interleaved crescents defeat it. Gradient boosting stacks axis-aligned splits, giving the blocky staircase boundary that trees always produce. TabPFN draws a smooth, almost hand-drawn curve that follows the gap between the classes. That smoothness is not an accident: TabPFN was trained to approximate the Bayesian posterior predictive distribution, which we will unpack shortly. It behaves less like a fitted classifier and more like a calibrated probabilistic model.

Going Deeper

The benchmark: no training, still competitive

We ran four methods on six small classification datasets (four real, two synthetic), three random splits each, all out of the box with no hyperparameter tuning. The baselines (logistic regression, XGBoost, histogram gradient boosting) train normally on each split. TabPFN does its single forward pass.

Grouped bar chart of test accuracy across six datasets for LogReg, XGBoost, HistGBDT, and TabPFN. On the real datasets all methods cluster high with TabPFN slightly ahead; on the two synthetic datasets TabPFN opens a clear gap, scoring around 0.90 and 0.88 where the trees sit near 0.81 and 0.84 and logistic regression collapses to 0.64 and 0.73.

DatasetLogRegXGBoostHistGBDTTabPFN
breast cancer0.9610.9590.9530.969
wine0.9880.9811.0001.000
iris0.9850.9700.9700.993
digits0.9650.9630.9720.990
synth (overlap)0.6380.8080.8060.903
synth (wide)0.7330.8420.8090.878
mean0.8780.9210.9180.955

TabPFN wins or ties on every dataset, and the margin is widest on the harder synthetic problems where the classes overlap. This matches what the TabPFN authors report on larger benchmark suites: on small tabular data, an untuned forward pass competes with carefully tuned gradient boosting. Remember the baselines here are also untuned; a heavily tuned XGBoost would close some of this gap. The remarkable part is that TabPFN needs no tuning at all to land where it does.

The catch: it is a small-data model that wants a GPU

Nothing about this is free. The same forward-pass design that removes training also caps what TabPFN can handle.

Scatter plot of accuracy against fit-plus-predict wall-clock on a log scale. Most points sit under a second. The four TabPFN points on small data are fast and high-accuracy, but the digits point sits far to the right at over 100 seconds, annotated as the >1000-sample CPU-discouraged regime.

On the small datasets TabPFN predicts in well under a second. On digits (1,797 rows) it took nearly two minutes per split on a CPU, because attention cost grows with the number of in-context rows and the library deliberately discourages large CPU runs. The original TabPFN was capped around 1,000 training rows and 100 features; TabPFN v2 pushes that to roughly 10,000 rows, and a GPU makes the forward pass tractable. Past those limits, gradient-boosted trees remain the right tool. TabPFN is a precision instrument for the small-data regime, not a general replacement.

When to reach for it, and when not to

Use TabPFN when you have a small labelled dataset (up to a few thousand rows), a modest number of features, and you want a strong baseline in seconds without tuning. It shines exactly where deep learning usually struggles and where practitioners reach for trees by reflex. Its probabilities are well calibrated out of the box, which matters when you act on confidence rather than the top class, a theme we explored in handling class imbalance.

Avoid it when you have hundreds of thousands of rows, very high-dimensional inputs, or hard latency budgets on CPU. And like any dimensionality-sensitive method, it benefits from sensible feature handling; it is robust, not magic.

Where This Comes From

TabPFN is the tabular instance of a broader idea called a Prior-data Fitted Network (PFN), introduced by Müller et al. (2022) in "Transformers Can Do Bayesian Inference". The insight is worth stating carefully because it reframes what "training a model" means.

A Bayesian predictor answers: given a prior over possible data-generating functions and some observed data, what is the posterior predictive distribution over a new point's label? That integral is usually intractable. PFNs sidestep it by amortisation. You define a prior, sample millions of synthetic datasets from it, and train a transformer once to predict held-out labels given the rest of each synthetic dataset. In doing so the transformer is forced to learn the mapping from "observed rows" to "posterior prediction" directly. After this one-time pre-training, a forward pass on a real dataset approximates Bayesian inference under that prior, with no integration and no per-dataset fitting.

"We train a network to do Bayesian prediction for arbitrary priors... at inference time, the trained network can do Bayesian prediction in a single forward pass." (Müller et al., 2022)

Two-stage pipeline diagram. Stage one (pre-training, done once by the authors): a prior over synthetic tabular datasets and millions of generated train-plus-test sets feed a transformer optimised to predict held-out labels, after which its weights are frozen. Stage two (inference, every use): your real dataset of train plus test rows enters the frozen transformer, which produces predictions for the test rows in a single forward pass with no optimiser.

Hollmann et al. (2023), "TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second", applied this to tables. The prior is the crucial design choice. TabPFN's prior is built to mimic the kinds of structures real tabular data exhibits: causal relationships between columns, mixtures of feature types, varying noise levels. The transformer pre-trained on this prior generalises to genuinely unseen real datasets because they look, statistically, like draws from a similar prior. The connection to Bayesian model comparison and information criteria is direct: TabPFN is implicitly averaging over models the way a fully Bayesian treatment would, rather than selecting one and fitting it.

Hollmann et al. (2025), the Nature paper "Accurate predictions on small data with a tabular foundation model", is the TabPFN v2 release. It scales the row and feature limits, adds regression, and demonstrates the approach beating tuned baselines on standard benchmark suites while running in seconds. Subsequent releases continue to push the size ceiling. The throughline from Müller's PFNs to TabPFN v2 is the same bet: replace per-dataset optimisation with one-time amortised pre-training over a well-chosen prior.

Further Reading

Interactive Tools

  • Classification Metrics Calculator: Turn the accuracy numbers from the benchmark into precision, recall, and F1; TabPFN's calibrated probabilities make threshold choice especially worthwhile
  • Overfitting Explorer: See the bias-variance trade-off that gradient boosting tunes against and TabPFN sidesteps by amortising over a prior

Stay in the loop

Get notified when new tutorials and deep-dives are published. No spam, unsubscribe anytime.

Related Posts

Frequently Asked Questions