Please verify you are human to use the simulator.
You only learn about the arm you pull. Testing a weak arm costs you earnings; sticking with your favourite costs you the chance of finding a better one.
What the run cost against always pulling the best arm. A curve that flattens means the strategy has settled; one that keeps climbing is still paying to explore.
Each arm carries a Beta distribution over its rate. Width is uncertainty, and Thompson sampling turns that width directly into how often the arm gets another try.
Exploration is an investment that has to pay back inside the run. A strategy tuned for an infinite horizon can look poor over a thousand pulls.
The regret curves are the headline, and the pull histograms explain them. A strategy with low regret is one whose green bar, the best arm, towers over the others early. The one-good-arm preset separates the four most clearly, because seven of the eight arms are poor and the whole task is finding the eighth. Thompson sampling gets there first on most seeds. Decaying epsilon is the one to watch: it either finds the good arm before its exploration rate fades and then almost never leaves it, or it fades first and spends the rest of the run on a bad arm. Press New run a few times and you will see both.
The nearly-tied preset is worth loading for the opposite reason. Every strategy scores low regret there and none of them reliably identifies the best arm, which looks like failure and is not. When the arms are four points apart, picking the wrong one costs almost nothing per pull. Low regret and a confident answer are different objectives, and only one of them is what a bandit optimises.
For the reinforcement learning these ideas lead into, our Q-learning visualiser adds state to the problem, and Q-learning on FrozenLake from scratch builds it in NumPy. The same epsilon-greedy rule reappears there. For the policy-search route instead of the value-based one, see policy gradients and REINFORCE and the cross-entropy method. If you want the fixed-split comparison rather than the adaptive one, our A/B test calculator handles the significance and sample-size side.
The choice between a bandit and a fixed split is a business question before it is a statistical one, because it turns on what a losing arm costs while the test runs and on whether anyone needs a defensible effect size afterwards. Framing that question, and setting the success criteria and kill gates before the traffic is spent, is what our AI pilot design work covers.
Watch a Q-learning agent train in a 5×5 grid world. Live Q-value heatmap, policy arrows, and configurable hyperparameters.
Try itFrequentist and Bayesian A/B test analysis. Z-test for proportions, Beta posterior distributions, probability of winning, and expected loss.
Try itCalculate posterior probabilities with interactive visualisation. Includes classic presets (medical tests, spam filtering) and sequential evidence updates.
Try itInteractive probability distribution calculators. Visualise Normal, Binomial, Poisson, Beta, and 12 more distributions with live PDF/CDF charts.
Try itGet notified when new tutorials and deep-dives are published. No spam, unsubscribe anytime.
Common questions about epsilon-greedy, UCB1, Thompson sampling and regret