This is partly a story about lotteries, and partly a story about how math puzzles get discussed in academia and the blogosphere.
Back in 2003, as part of my research for The Decline Effect, I looked for declines in all kinds of random-number generators, or what an earlier generation called ‘chance machines’. I found so many examples, and generated so many more in my own experiments, that I had to split off the material into a separate, future book entitled Experiments in Decline.
One of the ‘experiments’ I did was to study the highly popular ball-and-tumbler lotteries that are found today in nearly every developed country. I obtained the records of a number of lotteries and looked for evidence of systematic over-repetition. And I found some. The analysis is fairly technical, and there are subtle factors that have to be taken into account, such as the practice of rotating without prior notice among several separate machines and/or several separate ball sets. However, I did find that, on a detail level, lotteries do tend to over-repeat. If there are 49 balls in the set, and six balls are drawn from the machine on a given day — 3, 12, 15, 22, 38, and 44, let’s say — then the classical odds of seeing Ball 44 drawn again the very next time are 1 in 49. But if maximum entropy constraints are taken into consideration, the odds can be significantly better, at least for a lottery using brand-new machines and balls. The gain is not high enough to even consider trying to ‘beat’ the lottery, of course, as the house edge is simply too great. But I found enough evidence that a statistician would find the phenomenon odd.
Now, this was an important finding from a technical standpoint, but it was also just one more brick in the enormous edifice I was trying to construct. What would really be handy, I thought, would be for a case to turn up in which all six numbers repeated on the very next draw. But while my study suggested the odds were higher than classical for this to occur, they were still roughly millions to one against. Meaning it would surely happen, but probably not in the next few months. So after confirming my finding, I stuck the lottery data in a file and moved on to other tasks.
I won’t reproduce that analysis in a brief blog post. I bring it up simply to say that I have had a private Bayesian ‘prior’ about lotteries not behaving in classical fashion for some time now.
A few weeks ago my good friend Carmen, who is 13 and fascinated with physics, sent me an e-mail asking if I had heard about the recent case of the Israeli national lottery Mifal HaPayis repeating. I Googled around and found it:
This was not strictly speaking a repeat. Rather it was a ‘near-miss,’ which MaxEnt also predicts should occur more often. The lottery, which has a main draw of six balls in 34, held draws once a week, and so for the number to show up after three weeks meant it had repeated on the third try. Interestingly there were 95 (!) winners claiming part of the prize. Apparently there are a few gamblers out there who favor repetition strategies.
There was a brief but spirited discussion about the event on the Freakonomics blog, and elsewhere. For the most part, the mathematicians among us yawned. or complained about the press being mathematically illiterate, pointing to a widely repeated quote that the odds of a repeat were ‘trillions to one’. Which of course they are not. This was an event that could be expected, classically, to turn up once every 10,000 years for a lottery of the given type. And of course if you have several hundred lotteries around the world, operating for 40-50 years apiece, then the odds of a repeat sometime during that period are pretty good. Hence all the yawning. A lottery repeat or near-miss is a little unusual, but to a classically trained statistician, it is merely that. It’s not grounds for doubting the classical framework.
Well, but then I Googled a little more and found something else. A lottery of this type actually has repeated, on its very next draw. It happened in Bulgaria in 2009, to their national 6/45-type draw. A total of 18 people had bet on the previous draw’s numbers repeating, and so split the prize. Suspicions in that case were considerably more intense, and would grow still more intense after the next drawing, when three of the six numbers turned up a third time. A state commission of inquiry was called, but it found nothing amiss. The draw had been televised, and the machines were exquisitely simple in construction. There was no obvious way the results could have been rigged, and no evidence of a conspiracy.
So then I Googled some more . . . and found that the Wisconsin lottery also has a habit of repeating old numbers at short intervals. The Wisconsin lottery is a 6/39 type, drawn every single day for the past two decades. It has so far repeated once after ten draws, and on another occasion after twenty draws.
The Wisconsin case can be explained away, at least partly, as being just a larger example of the classic ‘same birthday’ puzzle that stats professors like to toss out at the start of each semester. How many people have to be in the same room before there is a 50-50 chance that two of them share the same birthday? Hint: it’s not 182.5. It’s 23 people. The reason is that the number of permutations goes up roughly as the square of the number of people present. Person 1 can share a birthday with 2, 3, 4, 5, 6, etc. Person 2 can share a birthday with 3, 4, 5, 6, etc. After about 7,000 draws, the Wisconsin lottery could expect to have repeated about a dozen numbers, and indeed it has done so.
What is improbable, though, is that these two particular repetitions were so close in time. Mostly the Wisconsin lottery repeats after several years, not several days. Such long-distance repeats excite no one’s imagination; they pass unremarked on many lotteries. To repeat on the tenth try though, as a classically trained statistician would agree, is still unusual.
So now we have four cases to ponder. Plus it seems quite possible, given the very brief and low-key coverage of these events, that there are more such near-misses out there. Only by searching through each lottery’s records can we be sure.
The question we need to ask is a bit demanding, and not one that bloggers or academics seem to have tackled so far: Exactly how likely is it, given the number of lotteries operating around the world, their respective standard odds, and the length of time each has operated, that we would have seen as many as four repeats and/or near-misses so far?
It’s pretty unlikely. Not impossible, of course, but quite unlikely in classical terms. Whereas if I am right and maximum entropy considerations intervene, we will see still more repeats and near-misses in the next few years.
I hope to deal with the classical odds, and the mechanics of MaxEnt as applied to lotteries, in another post. (UPDATE: I published this in January 2011, and still hadn’t gotten around to it in November 2011. I have accordingly amended the title, which originally read “Lottery Repeats (First Part)”. Oh well.)
The thought I want to leave the reader with for the moment is this: In probability, and indeed in science in general, context is everything.
This episode illustrates the point I make throughout The Decline Effect. A skeptic may ask, why haven’t we realized before that probability theory is wrong? The answer is, without a conceptual framework of what we are expecting to see, a suitable ‘prior,’ we can look at a phenomenon every day, and just not see how weird or important or paradigm-breaking it is. We may notice evidence of the failure every day, as we have done with these lottery anomalies. But the bare fact of a repeat or a near-miss proves nothing by itself. Without an alternative theory, all they are is mute anomalies.