![](\"https:\/\/measuringu.com\/images\/ncaa-bracket.jpg\" "\\"Bracket")
It’s that time of year again: March Madness.<\/p>\nIt’s that time of year again: March Madness.<\/p>\n
The Madness in March comes from the NCAA College basketball tournament, with unanticipated winners and losers with dozens of games packed into the final days of March.<\/p>\n
It’s also the time of year where a lot of people start working directly with probability, whether they know it or not. Individuals and groups of colleagues around the US will put together their best guess to who will be the “Final Four” of the 64 contending teams and ultimately who will win the tournament.<\/p>\n
What does this have to do with usability testing?\u00a0 Well, every time researchers conduct a usability test to uncover problems they’re also working with probabilities, even if they tell you they hate math! To understand the role of probabilities in usability testing it helps to see how they are used when picking winning teams for the tournament.<\/p>\n
The sport of picking the most winners in each divisional bracket of the tournament (“bracketology”) has itself become as interesting as the outcome of the games.<\/p>\n
Interest this year was heightened when billionaire investor Warren Buffett pledged $1 billion dollars<\/a> to anyone who successfully predicts the outcome of ALL 63 games in the tournament. You might wonder, what are the odds and why would someone, even one of the richest people in the world, risk losing a billion dollars? After all, like a lottery, won’t someone win?<\/p>\n Finding out your chances of winning will help with finding the right sample size for a usability test<\/a>.<\/p>\n Let’s start with making the correct decision for just a single hypothetical game, say between Stanford and Virginia. There are only two possible outcomes, Stanford wins or Virginia wins. Your chances of guessing correctly with no knowledge of basketball or the teams are 50\/50. It’s like tossing a coin. or .5 or 50% when expressed as a percentage.<\/p>\n Now, what are the chances of correctly guessing TWO games right?\u00a0 It’s just .52<\/sup> or .25. For getting three games correct it’s 12.5% and for four it’s 6%.\u00a0 The chances of randomly guessing just 10 of the 63 games right is slightly less than .1%. In other words, from random guessing alone, there’s a 99.9% chance you’ll get one of 10 wrong.<\/p>\n You can see that the numbers rapidly start working against your favor.\u00a0 In fact, the chances of picking all 63 games correctly (a perfect bracket) is .563<\/sup> or a 1 in 9.2 quintillion chance (1\/9,223,372,036,854,750,000).<\/p>\n Now you might think that putting together a tournament bracket involves more than just flipping a coin.\u00a0 Participants certainly use knowledge of teams’ performance and some historical data. Each team has a ranking from 1 to 16, called a seed, based on how well they did in the regular season.<\/p>\n Teams with higher seeds are more likely to win. In fact, every #1 seeded team (which gets paired with the lowest seeded #16 team) has won its first game in the tournament 116 times over the last 29 years.\u00a0 Some have put the odds of winning with more “expert” knowledge at 1 in 128 billion<\/a> or even as low as 1 in 7 billion<\/a>, as Nate Silver calculated.<\/p>\n So picking the higher seeded teams seems to be a strategy that would improve your odds. The trouble is, there are plenty of examples of highly seeded teams getting beaten by lower seeded teams. For example, last week, the perennial tournament performer, Duke, seeded #3 lost to #14 seeded Mercer in a major upset.<\/p>\n Such unpredictable “upsets” are common in one-game playoffs (hence the madness). Even after taking into account additional variables, Nate’s performance after the first round was 25 out of 32. Not bad, but that’s the same performance as our associate who used the “I like this school’s colors” along with “My dad went to that school” strategy.<\/p>\n It’s likely the case that the actual odds of winning are better than 63 coin flips, but many more sophisticated calculations are based on assumptions, many of which turn out to be wrong. To play things safe, you assume a worse-case scenario and work from there. A similar strategy works when planning small sample sizes in usability tests.<\/p>\n So, how many brackets would need to be submitted to have a decent chance of winning Buffett’s billion?\u00a0 It’s not just 9.2 quintillion, you actually need more than the probability itself to have a good chance of the event occurring. To find out we work backwards from the binomial probability formula<\/a> using logarithms (who doesn’t love logarithms<\/a>?).\u00a0\u00a0 The formula is :<\/p>\n So to have an 85% chance of seeing a coin flipped tails once, you should plan on flipping it three times\u201485% of those 3 flips you’ll see a tail at least one time.<\/p>\n log (1-.85)\/log(1-.50) = 2.74<\/i><\/p>\n <\/p>\n<\/div>\n So using the same formula, to have an 85% chance of getting a perfect bracket, you’d need about 17 quintillion submissions:<\/p>\n That would mean more than 2.4 billion brackets for every person on earth.\u00a0\u00a0 Buffett has limited the number of submissions to 15 million.\u00a0 This is his sample size. To find the odds of an event occurring given a sample size we just rework the formula again<\/a>.<\/p>\n With just 15 million submissions, chance alone would predict a 1 in 615 billion chance of a winner:<\/p>\n A typical power ball lottery has odds of winning in the 1 in 175 million range<\/a>. In other words, your odds of winning the power ball are about 3,514 times better that getting Buffett’s billion.<\/p>\n Even using the most generous 1 in 7 billion odds, we would still need on average about 13 billion submissions to have a decent chance of winning.<\/p>\n log (1-.85)\/log(1- 0.00000000014) <\/i>= 13,279,837,320<\/i><\/p>\n<\/div>\n <\/p>\n The same formula that tells us there’s an infinitesimal chance of getting a perfect bracket also tells us the chances we’ll observe a usability problem given an estimate of how common a problem is.\u00a0 For example, given usability problems that affect 31% of users (a relatively common problem<\/a>), you should plan on testing with 5 users to have an 85% chance of seeing the problem at least one time :<\/p>\n To observe problems that are less frequent (affecting 10% of the entire user population), you should plan on a usability test with 18 users<\/p>\n log (1-.85)\/log(1- .10) = 18.0<\/i><\/p>\n <\/p>\n<\/div>\n It’s often that we have our sample size dictated for us. If you have two days to test before the next round of changes, it’ll be difficult to test more than 10 users total (five per day). Given a sample size of 10, the probability of observing problems that affect 10% of users is 65%:<\/p>\n 1-((1-.1)10<\/sup>) = .65<\/i><\/p>\n <\/p>\n<\/div>\n The table below shows the comparison between the tournament calculations and sample sizes for uncovering problems in usability tests.<\/p>\nComputing The Odds<\/h2>\n
Guessing vs. Experience<\/h2>\n
What are the chances?<\/h2>\n
log (1-chance of seeing)\/log(1- probability)<\/i><\/div>\n\n\u00a0log (1-.85)\/log(1- 0.00000000000000000011) <\/i>= 17,246,500,000,000,000,000<\/div>\n= 1-(1-0.00000000000000000011) 15,000,000<\/sup><\/i><\/div>\n\nSample Sizes for Discovering Problems in Usability Tests<\/h2>\n
log (1-.85)\/log(1- .31) = 5.11<\/i><\/div>\n\n\n\n\n
\n <\/td>\n NCAA Perfect Bracket<\/strong><\/td>\n Usability Test<\/strong><\/td>\n<\/tr>\n \n <\/td>\n <\/td>\n \n Value<\/div>\n<\/td>\n<\/td>\n \n Value<\/div>\n<\/td>\n<\/tr>\n\n Probability of Occurrence<\/td>\n Getting all 63 Games Right<\/td>\n 1 in 9.2 quintillion<\/td>\n User Encountering a less common Interface Problem<\/td>\n .10<\/td>\n<\/tr>\n \n Sample Size Needed<\/td>\n \n To have an 85% chance of having at least 1 winner<\/div>\n<\/td>\n\n