JEPA from Scratch: Predicting in Latent Space

Build a Joint-Embedding Predictive Architecture in PyTorch. Predict in latent space, watch representation collapse, and learn LeCun's world-model bet.

Cover Image for JEPA from Scratch: Predicting in Latent Space

You can recognise an object in a photograph even when you cannot predict every pixel in a hidden part of it. Some details matter for identifying the object, while others may be incidental or hard to guess. LeCun's 2022 proposal argues for predicting representations that retain useful information without specifying every pixel.

We test that idea with a label-free FashionMNIST encoder in PyTorch. Its representations support 82.6% linear-probe accuracy, against 70.9% for a random encoder. Collapse prevention matters because identical codes would make prediction easy. I-JEPA applied the approach to images in 2023; V-JEPA 2 extended it to video and planning in 2025.

Predicting Representations, Not Pixels

A generative model of images predicts in pixel space: given some context, produce the missing pixels. This is what autoencoders, inpainting models, and (in spirit) diffusion models do. The trouble is that pixel space is full of detail the model cannot predict and does not need: the exact phase of a texture, sensor noise, lighting jitter. Forcing the model to reproduce all of it wastes capacity, a problem we ran into directly in perceptual loss and super-resolution, where pixel-wise MSE produced blurry, characterless reconstructions.

A JEPA sidesteps this. It encodes both the context and the target into a learned latent space, then predicts the target's encoding from the context's encoding. Anything the encoder chooses to discard (unpredictable surface detail) simply stops being part of the target. What remains is the structure that is predictable from context: shape, category, layout. That is the signal a downstream task wants.

Quick Win: A JEPA That Trains Itself

Let's see it work before dissecting it. The model below trains on FashionMNIST with no labels, predicting the latent representation of a masked patch region from the surrounding patches. Click to open the full notebook:

Open In Colab

As it trains, here is the encoder's representation of the test set, projected to 2D. The colours are the true classes, used only for the plot. The model never sees them.

FashionMNIST representations learned by JEPA separating into class clusters over training epochs, with no labels used

The structure sharpens on its own as training proceeds. The signal is purely "predict the latent code of the hidden region", yet to do that well the encoder has to learn that a sneaker patch belongs with other sneakers. Here is the core training step:

# context = 10 visible patches, target = a hidden 6-patch region
ctx = encoder(patches[:, ctx_idx], ctx_idx)            # online encoder
pred = predictor(ctx, ctx_idx, tgt_idx)                # predict target codes

with torch.no_grad():                                  # target encoder: EMA copy
    target = target_encoder(patches, all_idx)[:, tgt_idx]   # stop-gradient

loss = F.mse_loss(pred, target)                        # L2 in latent space
loss.backward(); opt.step()
ema_update(target_encoder, encoder, tau)               # slow-moving target

After 35 epochs, a linear probe on the frozen encoder reaches 82.6% accuracy on FashionMNIST. A randomly initialised encoder of the same size manages 70.9%. The encoder learned useful features from a task that used no labels and no pixel reconstruction at all, and we did it with a tiny Transformer on a laptop GPU.

What Just Happened

Four pieces do the work: a patch split, a masking rule, two encoders, and a predictor.

Patchify. Each 28x28 image is cut into a 4x4 grid of 7x7 patches, 16 patches in total, each flattened to a 49-vector. This is the same tokens-from-an-image idea that Vision Transformers use, and it lets us talk about "blocks" of an image as sets of patch indices.

The mask. We pick a contiguous region of 6 patches as the target and use the other 10 as context. The size of the masked region matters: a large hidden region makes pixel reconstruction ambiguous (many plausible fills), which is the regime where predicting an abstraction rather than pixels starts to earn its keep. The model sees the context and predicts what the target looks like in latent space.

An input image split into a 4x4 patch grid, with 10 context patches kept and a 6-patch target region masked in red

Two encoders. The context encoder fθf_\theta is a small Transformer that encodes the 10 visible patches. The target encoder fθˉf_{\bar\theta} is a copy of it that encodes the full image; we read off its outputs at the target positions and treat those as the prediction target. Crucially the target encoder is not trained by gradient descent. Its weights are an exponential moving average of the context encoder's, and we block gradients from flowing into it. We will see in a moment why that asymmetry is the whole game.

The predictor. A second, smaller Transformer gϕg_\phi takes the encoded context tokens plus a learnable "mask token" at each target position (carrying only positional information) and predicts the target encoder's codes. The loss is plain L2 between predicted and actual target codes.

The shapes mirror the architecture: context in, latent prediction out, compared against a latent target.

JEPA architecture: a context encoder and an EMA target encoder feed a predictor that matches target representations under a stop-gradient

Notice what is absent. There is no decoder, no reconstruction of pixels, no contrastive term comparing one image against a batch of negatives. The supervision is entirely self-generated, the same trick that powers transfer learning for text: build a pretext task out of the data itself, then reuse the learned features. The representation the encoder produces is exactly the kind of dense embedding we used in collaborative filtering from scratch, learned here without any labels.

Going Deeper: Collapse and the EMA Trick

A self-predicting model has an obvious cheat. If the encoder maps every image to the same constant vector, then the target codes are trivially predictable: the predictor outputs that constant and the loss hits zero. The model has "solved" the task and learned nothing. This is representation collapse, and it is the central difficulty of every joint-embedding method.

Watch it happen. The two runs below are identical except for one thing: the left uses the EMA target encoder with a stop-gradient; the right computes targets with the online encoder and lets gradients flow through them.

Side-by-side animation: the healthy JEPA keeps its embeddings spread out while the no-stop-gradient variant collapses every image to a single point

On the right the cloud shrinks to a dot. The loss falls beautifully and the representation becomes worthless. On the left the cloud stays spread out and structured. The only difference is the asymmetry between the two encoders.

Why does the asymmetry prevent collapse? When the target is produced by a separate encoder whose weights you do not differentiate through, the model cannot reach the constant solution by simply pushing both sides to zero together. The target is a moving but non-cooperative reference. The online encoder has to match it, and the EMA keeps that reference stable enough to chase yet fresh enough to improve. The update is a simple Polyak average:

θˉ←τ θˉ+(1−τ) θ\bar\theta \leftarrow \tau\, \bar\theta + (1 - \tau)\, \theta

with τ\tau close to 1 (we ramp it from 0.996 towards 1.0). A large τ\tau means the target encoder changes slowly, giving the predictor a near-stationary objective within each short window. This is the same mechanism behind BYOL, where a stop-gradient plus a slow target was shown to be enough to avoid collapse without any negative pairs.

You can read collapse off a single scalar: the standard deviation of the embeddings across the dataset. The healthy run keeps it around 0.6 to 0.8; the collapsed run drives it to nearly zero. Its loss, meanwhile, falls to almost exactly zero, lower than the healthy run's, because a constant prediction is trivially correct. Low loss, dead representation.

Embedding standard deviation per epoch: the healthy run stays around 0.6 to 0.8 while the collapsed run sits at zero

Latent prediction versus pixel prediction. We can put the two objectives head to head. Take the same context encoder and predictor, and change only the target: instead of latent codes, predict the raw pixels of the masked region, a masked-autoencoder style objective. Then linear-probe both frozen encoders.

Linear-probe accuracy bar chart: pixel-MAE at 84% and JEPA at 83%, both far above a random-init encoder at 71%

Both self-supervised objectives crush the random-init baseline of 70.9%: the pixel encoder reaches 84.0% and the latent encoder 82.6%. On FashionMNIST the two run roughly level, with pixel prediction a hair ahead. That is the honest result at this scale, and it is worth understanding why.

FashionMNIST is grayscale, smooth, and centred. The pixels of a masked region are highly predictable, so reconstructing them is a perfectly good pretext task that wastes little capacity. The advantage of predicting in latent space appears when the pixels are not easy to predict: natural images full of texture, lighting, and detail that no model can reproduce exactly. That is the setting of the original I-JEPA results. On ImageNet, latent prediction matches pixel-level masked autoencoders while being substantially more compute-efficient, and it does so without ever rendering a pixel. The mechanism we built here is the same one; its payoff scales with how much unpredictable detail the data contains.

Treat this small-scale demo as a proof of mechanism, not of superiority: a from-scratch JEPA that trains, learns useful features without labels, and shows both the collapse failure mode and its cure. The "latent beats pixels" claim is real, and it lives at a scale a laptop will not reach in one blog post.

When not to reach for JEPA. If your downstream task needs pixels (denoising, super-resolution, generation), a model that throws away surface detail is the wrong tool; use a generative objective. JEPA also has more moving parts than a plain autoencoder (the EMA schedule, the masking design, the stop-gradient), and getting them wrong gives you silent collapse rather than a loud error. For small labelled datasets where supervised training already works, the self-supervised detour buys you little. JEPA earns its keep when labels are scarce, the downstream task is semantic (classification, retrieval, planning), and you have unlabelled data to spare.

Where This Comes From

LeCun laid out the JEPA vision in his 2022 position paper, A Path Towards Autonomous Machine Intelligence. The paper argues that the dominant paradigms (generative pixel prediction and contrastive learning) are each flawed, and proposes energy-based joint-embedding prediction as the route to models that build usable internal "world models". The energy-based framing is what makes collapse the natural enemy: a model can trivially assign low energy everywhere by ignoring its input, and the architecture has to be designed so it cannot.

The capacity of the [target] representation to be invariant to irrelevant details of the input is what allows a JEPA to capture useful abstractions, while the prediction is performed in representation space.

The first concrete, working instantiation for images came a year later. Assran et al. (2023), Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture (I-JEPA, CVPR 2023), is the paper our code is a miniature of. Their recipe is exactly what we built, scaled up: a Vision Transformer context encoder, an EMA target encoder, and a predictor that reconstructs the representations of several masked target blocks from a single context block. Their key empirical finding is about the masking: the target blocks must be reasonably large and semantic, and the context must be informative, otherwise the encoder learns shortcut features. On ImageNet, I-JEPA matched masked-autoencoder quality with no pixel-level loss, validating the "predict in latent space" claim at scale.

The line continues into video and planning. V-JEPA (Bardes et al., 2024) extends the same masked latent-prediction idea to spatiotemporal tubes in video, and V-JEPA 2 (2025) couples the learned world model to action for prediction and planning. The thread running through all of them is the one we just implemented: encode, mask, and predict where it counts, in representation space.

Timeline of the JEPA lineage from LeCun's 2022 position paper through I-JEPA for images, V-JEPA for video, and V-JEPA 2 for planning

Further Reading

  • The position paper: LeCun (2022), A Path Towards Autonomous Machine Intelligence. Read sections on JEPA and energy-based models for the conceptual core.
  • The image instantiation: Assran et al. (2023), I-JEPA. The masking-strategy section is the practical heart of the paper. Official code: facebookresearch/ijepa.
  • Collapse, demystified: Grill et al. (2020) BYOL shows a stop-gradient plus a slow target is enough to avoid collapse without negatives, and Chen & He (2021) SimSiam shows the stop-gradient alone is enough, with no slow target.

Interactive Tools

  • 2D Convolution Visualiser: Builds the patch-and-receptive-field intuition that underlies how the encoder reads a 4x4 grid of image patches.
  • Overfitting Explorer: A frozen self-supervised encoder is judged by how well its features generalise; this tool builds the bias-variance intuition behind that linear-probe test.
  • Classification Metrics Calculator: The linear probe reduces representation quality to a single accuracy number; this calculator unpacks what that metric does and does not capture.

Stay in the loop

Get notified when new tutorials and deep-dives are published. No spam, unsubscribe anytime.

Related Posts

Frequently Asked Questions