Suppose you've fitted a neural network to observations from a physical process. It predicts them accurately, and you'd now like to understand the relationship it found between the inputs and the output. A standard MLP stores that relationship across its weight matrices, which can make a simple underlying formula difficult to recognise. The Universal Approximation Theorem explains its capacity to fit functions, but gives no guarantee that the fitted function will be easy to read. Kolmogorov-Arnold networks offer another way to represent it, using a learned curve on each edge.
We fit to check whether the learned curves reveal its terms. A 39-parameter KAN recovers the sine and square at after affine alignment. We implement Liu and colleagues' 2024 architecture in NumPy with hand-derived gradients. Each edge applies a univariate function and each node sums its inputs. On 8x8 digits, the MLP is more accurate with fewer parameters. Noisy targets also expose spline overfitting.
Let's Build It
Open this in Google Colab and run it top to bottom. It builds a cubic B-spline basis, a KAN layer with hand-derived gradients, Adam, and then fits a two-variable formula with 39 parameters. It takes about twelve seconds on a CPU.
import numpy as np
# --- The whole "library": a cubic B-spline basis by Cox-de Boor recursion ---
class BSpline:
def __init__(self, grid, degree=3):
h = grid[1] - grid[0]
self.grid, self.degree = grid, degree
self.knots = np.concatenate([grid[0] - h * np.arange(degree, 0, -1), grid,
grid[-1] + h * np.arange(1, degree + 1)])
self.n_basis = len(self.knots) - degree - 1
def _b(self, x, p):
t = self.knots
b = ((x[..., None] >= t[:-1]) & (x[..., None] < t[1:])).astype(float)
for d in range(1, p + 1):
la, ra = t[d:-1] - t[:-d - 1], t[d + 1:] - t[1:-d]
b = (np.where(la > 0, (x[..., None] - t[:-d - 1]) / np.where(la > 0, la, 1), 0) * b[..., :-1]
+ np.where(ra > 0, (t[d + 1:] - x[..., None]) / np.where(ra > 0, ra, 1), 0) * b[..., 1:])
return b
def __call__(self, x):
return self._b(np.clip(x, self.knots[0], self.knots[-1] - 1e-9), self.degree)
def deriv(self, x):
p, t = self.degree, self.knots
lo = self._b(np.clip(x, t[0], t[-1] - 1e-9), p - 1)
da, db = t[p:-1] - t[:-p - 1], t[p + 1:] - t[1:-p]
return (np.where(da > 0, p / np.where(da > 0, da, 1), 0) * lo[..., :-1]
- np.where(db > 0, p / np.where(db > 0, db, 1), 0) * lo[..., 1:])
silu = lambda x: x / (1 + np.exp(-np.clip(x, -60, 60)))
# --- A KAN layer: every edge is a curve, every node just adds up ---
class KANLayer:
def __init__(self, n_in, n_out, grid, rng):
self.basis = BSpline(grid)
self.c = rng.normal(0, 0.1, (n_out, n_in, self.basis.n_basis)) / np.sqrt(n_in)
self.ws = np.ones((n_out, n_in)) / np.sqrt(n_in)
self.wb = rng.normal(0, 1, (n_out, n_in)) / np.sqrt(n_in)
def forward(self, x):
self.x, self.B = x, self.basis(x)
self.spl = np.einsum("nib,oib->noi", self.B, self.c)
return (self.ws * self.spl).sum(-1) + silu(x) @ self.wb.T
def backward(self, g): # g = dL/dy, shape (N, n_out)
self.gc = np.einsum("no,oi,nib->oib", g, self.ws, self.B)
self.gws = np.einsum("no,noi->oi", g, self.spl)
self.gwb = g.T @ silu(self.x)
s = 1 / (1 + np.exp(-np.clip(self.x, -60, 60)))
d = np.einsum("nib,oib->noi", self.basis.deriv(self.x), self.c)
return np.einsum("no,oi,noi->ni", g, self.ws, d) + (g @ self.wb) * s * (1 + self.x * (1 - s))
def phi(self, i, j, xs): # the learned function on one edge
return self.wb[j, i] * silu(xs) + self.ws[j, i] * (self.basis(xs) * self.c[j, i]).sum(-1)
class KAN:
def __init__(self, widths, grid_points=8, span=(-1.2, 1.2), seed=0):
rng = np.random.default_rng(seed)
self.layers = [KANLayer(a, b, np.linspace(*span, grid_points + 1), rng)
for a, b in zip(widths[:-1], widths[1:])]
def forward(self, x):
for l in self.layers:
x = l.forward(x)
return x
def fit(self, x, y, steps=2500, lr=0.02):
m_state = [[np.zeros_like(p) for p in (l.c, l.ws, l.wb)] for l in self.layers]
v_state = [[np.zeros_like(p) for p in (l.c, l.ws, l.wb)] for l in self.layers]
for t in range(1, steps + 1):
if t % 200 == 1 and t > 1: # re-centre each grid on what it sees
h = x
for l in self.layers:
lo, hi = h.min(), h.max()
pad = 0.1 * max(hi - lo, 1e-3)
l.basis = BSpline(np.linspace(lo - pad, hi + pad, len(l.basis.grid)))
h = l.forward(h)
g = 2 * (self.forward(x) - y) / len(x)
for l in reversed(self.layers):
g = l.backward(g)
for li, l in enumerate(self.layers):
for k, (p, gr) in enumerate(zip((l.c, l.ws, l.wb), (l.gc, l.gws, l.gwb))):
m, v = m_state[li][k], v_state[li][k]
m *= 0.9; m += 0.1 * gr
v *= 0.999; v += 0.001 * gr ** 2
p -= lr * (m / (1 - 0.9 ** t)) / (np.sqrt(v / (1 - 0.999 ** t)) + 1e-8)
return self
# --- Fit f(x,y) = exp(sin(pi x) + y^2) with a [2,1,1] KAN: 39 parameters ---
rng = np.random.default_rng(0)
X = rng.uniform(-1, 1, (1000, 2))
y = np.exp(np.sin(np.pi * X[:, 0]) + X[:, 1] ** 2)[:, None]
y = (y - y.mean()) / y.std()
kan = KAN([2, 1, 1], grid_points=8, seed=1).fit(X, y)
print("train RMSE:", round(float(np.sqrt(np.mean((kan.forward(X) - y) ** 2))), 5))
# The claim: read the two edges leaving the inputs, compare them to the true terms.
def affine_r2(learned, truth): # the split is unique only up to scale
A = np.column_stack([truth, np.ones_like(truth)])
resid = learned - A @ np.linalg.lstsq(A, learned, rcond=None)[0]
return 1 - (resid ** 2).sum() / ((learned - learned.mean()) ** 2).sum()
xs = np.linspace(-1, 1, 300)
print("edge from x vs sin(pi x):", round(affine_r2(kan.layers[0].phi(0, 0, xs), np.sin(np.pi * xs)), 6))
print("edge from y vs y^2 :", round(affine_r2(kan.layers[0].phi(1, 0, xs), xs ** 2), 6))
train RMSE: 0.00129
edge from x vs sin(pi x): 0.999999
edge from y vs y^2 : 0.999999
Thirty-nine parameters, and the two edges leaving the inputs are the two terms of the formula.
What Just Happened?
The edge is the parameter
An MLP layer computes . The learnable object is , a grid of numbers, and is whatever you picked. A KAN layer computes
where each is its own learnable univariate function. The node contributes a summation and nothing else. Everything the model knows lives on the edges.

That only helps if a learnable function is cheap to represent and cheap to differentiate. Liu and colleagues use a B-spline plus a residual term:
The are the spline's control points, and they are ordinary weights: numbers that gradient descent moves. The term gives every edge a sensible non-zero starting shape, which matters because a spline initialised near zero has no gradient signal to work with. So an edge costs spline coefficients plus two scalars, where is the number of grid intervals and the spline degree. In the run above, and gives 11 coefficients plus 2, so 13 numbers per edge and three edges: 39 parameters.
B-splines in eight lines
A B-spline basis function is a bump with local support: it is non-zero over a few grid intervals and exactly zero everywhere else. Locality is the property that makes the whole thing work, because moving one control point changes the curve near that knot and leaves the rest alone.
The Cox-de Boor recursion builds degree- basis functions from degree- ones, starting from indicator functions on the knot spans:
That is the _b method: start with the half-open interval indicators, then run the recursion three times. SciPy will do this for you, but it is eight lines and the point of building from scratch is to see them.
The backward pass
Because is linear in , the gradient with respect to a control point is the basis function's value:
which is the gc line. Passing gradient back to the input needs the spline's derivative, and B-splines differentiate into lower-degree B-splines:
which is deriv. Nothing here needs autodiff. The layer is backpropagation over a different parameterisation, and Adam trains it unchanged.
One piece of the fit loop is not cosmetic. Every 200 steps the grids are re-centred on the values each layer currently sees. The first layer's inputs are the data and hold still, but the second layer's inputs are the first layer's outputs, which move throughout training. Leave the grid fixed and the signal drifts off the end of the spline's support, where every basis function is zero and no gradient flows.
Reading What the Edges Learned
Here is the reason to care. The network was given pairs and a target. It was not told that the target factorises, nor which variable belongs in which term. Plot the three edge functions against the true terms:

The edge leaving is . The edge leaving is . The single edge in the second layer is . Each at against the true term, and at against the other one, so the match is specific rather than a curve that happens to fit anything smooth.
Watching it happen is more convincing than the endpoint:

Both edges start as the silu residual and neither takes long. Scoring each frame, both edges pass against their true term by Adam step 50 and by step 200, out of 2,500. The parabola briefly leads at step 25 (0.952 against 0.892) and after that they are level. The shape of the answer is settled in the first 2% of training, and the rest of the run is refinement.
Going Deeper
The scale is free, the ratio is not
above is computed after fitting an affine map between the learned edge and the true term, and that is not a cheat, it is forced by the architecture. Layer one computes and layer two applies to the result. Multiply both first-layer edges by 3, add 7, and let undo it: identical output, different edges. The decomposition is unique only up to that shared affine transformation, so measuring shape rather than absolute values is the honest test.
The freedom is visible across seeds. Over five runs the amplitude of the recovered edge ranges from 1.10 to 2.46, and every seed still lands at . What does not vary is the ratio between the two edges' amplitudes: 1.9930, 1.9950, 1.9959, 1.9965, 1.9969. The true terms have ranges 2 and 1 over , so their ratio is exactly 2. The network is free to choose the overall scale and is not free to choose the relative one, and it gets the relative one right to four significant figures every time.
Grid extension
Splines support an operation that has no MLP analogue. Once an edge function is learned on a coarse grid, you can fit a finer spline to the curve the coarse one already describes, and carry on training from there. The curve survives the change of representation; only the resolution available to it improves.

Fitting with a grid of 5 gets to a test RMSE of 0.0710, and the left panel shows what that misses: the coarse fit tracks the slow component and rounds off the fast one. Refit the coefficients onto a grid of 20 and the error is 0.0711, unchanged, which is the point. The refit is a change of basis, not a retrain. Then 1,500 more Adam steps take it to 0.000731.
The control matters. Training a grid-20 model from scratch for the same total number of steps reaches 0.00257, three and a half times worse. Starting coarse and refining is not just convenient, it lands somewhere better than starting fine.
Two things in the right-hand panel need explaining rather than ignoring. The sawtooth through the first half is the grid re-centring every 200 steps: the coefficients briefly mismatch the moved knots, the loss jumps, and Adam recovers within a few dozen steps. The spikes in the teal tail have a different cause, since grid updates are off after refinement. At a loss around Adam's normalised step no longer shrinks with the gradient, so it periodically kicks itself out of a very sharp minimum and walks back in.
The parameter advantage, measured
Recovering a formula is a narrow win. The broader claim made for KANs is parameter efficiency, so here is that claim measured on a target with structure a local basis should suit: , a slow component with a fast one riding on it.

The KAN's error falls steeply and keeps falling: 0.0733 at 60 parameters, 0.00133 at 150, 0.00064 at 222. The MLP's error falls and then flattens. Going from 321 parameters to 16,897, a factor of 53, moves it from 0.0104 to 0.0076, and the largest model is slightly worse than the second largest (0.0076 against 0.0073). The best KAN here is eleven times more accurate than the best MLP while using twenty times fewer parameters.
The residual panel shows where the MLP's remaining error sits. It oscillates at the frequency of the fast component and grows towards the edges of the domain, which is a global basis of tanh units struggling to place fine detail without disturbing the coarse fit it already has. The KAN's residual is close to flat, because moving one spline control point changes the curve near one knot and leaves everything else alone.
One caution about how this number was produced, because it nearly went in wrong. An earlier version of this sweep trained every model for 2,000 Adam steps and showed the MLP stuck near 0.06 at every width, which reads as "an MLP cannot fit this". It can. At 8,000 steps the same 321-parameter MLP reaches 0.0104, a factor of seven better, and the chart above gives every model 8,000 steps with two learning rates tried for each MLP and the better kept. The first chart measured the optimiser and the training budget, not the architecture. If you benchmark two model families against each other, converge both or you are publishing a fact about your patience.
The honest scope of the result: this is a one-dimensional target with multiscale structure, which is close to the best case for a locally-supported basis. Take it as evidence that KANs are parameter-efficient where the data has that shape, not as a general ranking.
The cost nobody puts in the abstract
Run the same architecture on 8x8 handwritten digits from scikit-learn and the picture inverts.

Averaged over three seeds, a 4,810-parameter MLP reaches 97.9% test accuracy in 0.2 seconds. The best KAN here reaches 97.0% with 8,320 parameters in 9.3 seconds. Worse accuracy, more parameters, thirty-eight times the wall-clock. The seed spreads do not overlap, so the accuracy gap is real rather than noise.
The cost is structural, not an artefact of this implementation. An MLP edge is one multiply. A KAN edge evaluates basis functions through a three-pass recursion, then a weighted sum, and the backward pass runs the recursion again for the derivative. Roughly an order of magnitude more arithmetic per edge buys you nothing on a task with no formula underneath it, because there is no symbolic structure for the edges to find. Pixels are not .
Swapping the basis for sinusoids
Nothing about a KAN requires B-splines. The layer needs a set of basis functions and their derivatives, so any parameterised family will do, and Reinhardt, Ramakrishnan and Gleyzer (2024) proposed sine and cosine harmonics instead. Swap the class and change nothing else:

On this target the spline curve sits below the sinusoidal one across the whole range, reaching 0.0045 at 150 parameters against 0.0091 for 132 parameters of sinusoids. The sinusoidal basis is about 2.3 times faster to train, which is the trade its authors emphasise.
The interesting part is the shape of the sinusoidal curve. At 2 and 4 harmonics it is stuck near 0.20, barely better than predicting the mean, then it drops to 0.022 at 6 harmonics. A global basis cannot represent a frequency it does not contain, so below the threshold it fails completely; above it, it works. The locally-supported spline basis has no such cliff and degrades smoothly all the way down. That is the practical difference between local and global bases, and it is worth more than either error number.
Failure modes and limits
The pattern in these results is consistent. KANs pay off where the data has low-dimensional compositional structure worth recovering: symbolic regression, fitting physical laws, any setting where you intend to look at what was learned rather than only use it. They cost you on everything else. On high-dimensional perceptual data an MLP is faster, smaller and more accurate, and the readability that justifies the expense has nothing to read.
Two failure modes are worth naming, and the first one caught me out while writing this.
Too many knots. The grid is the capacity knob, so a KAN should overfit like any flexible model. On the clean target above it does not, at first: pushing from 20 to 32 intervals improves test error, and the sweep gives no warning. Add noise and the textbook picture appears immediately.

With noise on 600 training targets, training error falls monotonically from 0.102 to 0.050 as the grid goes from 5 intervals to 128, while test error against the clean function bottoms out at 12 intervals and then climbs to 0.319, roughly ten times worse. That is the classic U-curve, and the number of knots is the knob that drives it.
On clean data the failure arrives later and looks different. Test error keeps improving to 32 intervals, then the ratio between test and training error jumps from 1.1 to 8.9 at 48 intervals, and by 128 intervals the fit collapses entirely, with training error at 0.020 and test at 0.203. At that point there are 131 basis functions per edge and 600 training points spread across them, so some knot spans contain almost no data and their coefficients are barely constrained. That is not variance in the usual sense; it is a grid finer than the data can support. Either way the practical rule is the same: choose the grid on held-out error, not on training error.
A grid in the wrong place. A KAN needs its grid to match its input range. Unnormalised inputs, or a badly chosen span, put the data where every basis function is exactly zero, no gradient reaches the coefficients, and the layer silently learns nothing at all.
Where This Comes From
The architecture takes its name and its licence from a 1957 theorem. Andrey Kolmogorov, answering Hilbert's thirteenth problem, proved that any continuous function of variables can be written using only univariate functions and addition:
Vladimir Arnold sharpened the result, and the pair became the Kolmogorov-Arnold representation theorem. Read the right-hand side as a diagram and it is a two-layer network with univariate functions on the edges and sums at the nodes, which is the architecture built above.
The theorem sat unused as a network design for decades, and the objection was reasonable. It is an exact representation result rather than an approximation one, the inner functions it guarantees can be pathological, non-smooth and fractal, and the fixed width gives no way to trade capacity for accuracy. Nothing about "these functions exist" implies "gradient descent will find nice ones".
Liu, Wang, Vaidya, Ruehle, Halverson, Soljačić, Hou and Tegmark (2024) got past the objection by declining to honour the theorem exactly. They keep the shape, edges hold univariate functions and nodes add, and drop the fixed depth-2, width- prescription, stacking arbitrary layers of arbitrary width. The result is no longer the theorem's guaranteed exact representation. It is trainable, and it keeps the readability that made the form attractive in the first place.
The contrast with the Universal Approximation Theorem is the useful way to hold both. Cybenko's theorem says a wide enough single hidden layer approximates anything, and it is the licence MLPs run on. Kolmogorov-Arnold says a fixed two-layer form represents anything exactly. One buys accuracy with width, the other claims exactness at fixed size and pays for it with functions you may not be able to learn. Both are existence results, and neither says gradient descent will get there.
That gap is exactly what the 39-parameter fit at the top of this post probes. The theorem never promised that training a KAN on samples of would return , a square and an exponential on the edges. It does, reliably, across seeds, and that is an empirical finding about optimisation rather than a corollary of the 1957 result. It is also the same question mechanistic interpretability asks of ordinary networks, made easy by an architecture that puts its learned functions somewhere you can plot them.
Interactive Tools
- Gradient Descent Visualizer: Adam is what moves the spline control points in this post, and this shows how it behaves against SGD and momentum on the same surface.
- Regression Playground: fit curves interactively and read the used here to score the recovered edge functions.
- Overfitting Explorer: the grid-size failure mode above is polynomial overfitting with the number of knots as the capacity knob.
Related Posts
- The Universal Approximation Theorem in NumPy: the sibling theorem, and the licence MLPs run on rather than KANs.
- Backpropagation Demystified: the chain-rule machinery the hand-derived KAN gradients above are built from.
- SGD, Momentum, and Adam: the optimiser training the spline coefficients, and why its normalised step causes the spikes in the refinement chart.
- Gaussian Process Regression from Scratch: another model built on a basis-function view of fitting curves, with the basis fixed rather than learned.
- Mechanistic Interpretability: One Neuron, Two Features: what reading a network's learned features costs when the architecture does not hand them to you.





