Skill or luck? Bayesian updating

Watch a fund manager's record come in and update your belief, one result at a time.

Prior: share of managers with real skill10%

What you believe before seeing a single month.

A skilled manager beats the market60% of months

A lucky one wins exactly half the time.

Behind the curtain, this manager is

Track record, one square per month

Months observed0 of 120

Months beating the marketnone yet

Belief it's skill10%

Lucky managers this good–pure chance, same record or better

Blue: your updated belief that the manager is skilled. Dashed: where you started.

Before any results you believe 10% of managers are skilled. Press play and watch how much evidence it takes to move that belief.

The advanced view: what is the win rate, really?

Instead of two types, treat the manager's true win rate as unknown and give every value a probability. Start with a sceptical Beta(6, 6) prior centred on 50%. Each win adds one to the first number, each loss one to the second.

PosteriorBeta(6, 6)

90% credible range27% to 73%

Chance the edge is real51%win rate above 50%

Try this

Keep the manager just lucky and press new record a few times. How often does a lucky streak look like skill for a year or two?
Make the manager truly skilled at 55%. How many years until you're 90% sure?
Drop the prior to 1%. Extraordinary claims need extraordinary evidence: how much longer does it take now?

The math

P(S \mid D) = \frac{P(D \mid S)\,P(S)}{P(D \mid S)\,P(S) + P(D \mid L)\,P(L)}

$P(S)$: the prior: your belief in skill before the evidence
$P(D \mid S)$: chance of this record if skilled, q^k (1 − q)^(n − k)
$P(D \mid L)$: chance of this record from pure luck, 0.5^n
$P(S \mid D)$: the posterior: your updated belief

Ask the Loop about this Next: Gambler's ruin