Multi-Armed Bandit Simulator

Four strategies pull the same arms at the same time: epsilon-greedy, decaying epsilon, UCB1 and Thompson sampling. Watch cumulative regret separate them, see where each one spends its pulls, and follow the posterior over every arm as the evidence arrives.

Please verify you are human to use the simulator.

Exploration, Exploitation and Regret

The Trade-off

You only learn about the arm you pull. Testing a weak arm costs you earnings; sticking with your favourite costs you the chance of finding a better one.

Regret

What the run cost against always pulling the best arm. A curve that flattens means the strategy has settled; one that keeps climbing is still paying to explore.

Posterior Beliefs

Each arm carries a Beta distribution over its rate. Width is uncertainty, and Thompson sampling turns that width directly into how often the arm gets another try.

Horizon

Exploration is an investment that has to pay back inside the run. A strategy tuned for an infinite horizon can look poor over a thousand pulls.

Reading the Race

The regret curves are the headline, and the pull histograms explain them. A strategy with low regret is one whose green bar, the best arm, towers over the others early. The one-good-arm preset separates the four most clearly, because seven of the eight arms are poor and the whole task is finding the eighth. Thompson sampling gets there first on most seeds. Decaying epsilon is the one to watch: it either finds the good arm before its exploration rate fades and then almost never leaves it, or it fades first and spends the rest of the run on a bad arm. Press New run a few times and you will see both.

The nearly-tied preset is worth loading for the opposite reason. Every strategy scores low regret there and none of them reliably identifies the best arm, which looks like failure and is not. When the arms are four points apart, picking the wrong one costs almost nothing per pull. Low regret and a confident answer are different objectives, and only one of them is what a bandit optimises.

For the reinforcement learning these ideas lead into, our Q-learning visualiser adds state to the problem, and Q-learning on FrozenLake from scratch builds it in NumPy. The same epsilon-greedy rule reappears there. For the policy-search route instead of the value-based one, see policy gradients and REINFORCE and the cross-entropy method. If you want the fixed-split comparison rather than the adaptive one, our A/B test calculator handles the significance and sample-size side.

The choice between a bandit and a fixed split is a business question before it is a statistical one, because it turns on what a losing arm costs while the test runs and on whether anyone needs a defensible effect size afterwards. Framing that question, and setting the success criteria and kill gates before the traffic is spent, is what our AI pilot design work covers.

Related Tools

Stay in the loop

Get notified when new tutorials and deep-dives are published. No spam, unsubscribe anytime.

Multi-Armed Bandit FAQ

Common questions about epsilon-greedy, UCB1, Thompson sampling and regret