Confidence intervals are your frenemies.
They are one of the most useful statistical techniques you can apply to customer data. At the same time they can be perplexing and cumbersome.
But confidence intervals provide an essential understanding of how much faith we can have in our sample estimates, from any sample size, from 2 to 2 million. They provide the most likely range for the unknown population of all customers (if we could somehow measure them all).
A confidence interval pushes the comfort threshold of both user researchers and managers. People aren’t often used to seeing them in reports, but that’s not because they aren’t useful but because there’s confusion around both how to compute them and how to interpret them. While it will probably take time to appreciate and use confidence intervals, let me assure you it’s worth the pain. Here is a peek behind the statistical curtain to show you that it’s not black magic or quantum mechanics that provide the insights.
To compute a confidence interval, you first need to determine if your data is continuous or discrete binary. Continuous data are metrics like rating scales, task-time, revenue, weight, height or temperature. Discrete binary data takes only two values, pass/fail, yes/no, agree/disagree and is coded with a 1 (pass) or 0 (fail).
To compute a 95% confidence interval, you need three pieces of data:
- The mean (for continuous data) or proportion (for binary data)
- The standard deviation, which describes how dispersed the data is around the average
- The sample size
Continuous data example
Imagine you asked 50 customers how satisfied they were with their recent experience with your product on an 7 point scale, with 1 = not at all satisfied and 7 = extremely satisfied.
- Find the mean by adding up the scores for each of the 50 customers and divide by the total number of responses (which is 50). If you have Excel, you can use the function =AVERAGE() for this step. For the purpose of this example, I have an average response of 6.
- Compute the standard deviation. You can use the Excel formula = STDEV() for all 50 values or the online calculator. I have a sample standard deviation of 1.2.
- Compute the standard error by dividing the standard deviation by the square root of the sample size: 1.2/ √(50) = .17.
- Compute the margin of error by multiplying the standard error by 2. 17 x 2 = .34.
- Compute the confidence interval by adding the margin of error to the mean from Step 1 and then subtracting the margin of error from the mean:5.96+.34=6.3
5.96-.34=5.6
We now have a 95% confidence interval of 5.6 to 6.3. Our best estimate of what the entire customer population’s average satisfaction is between 5.6 to 6.3.
If you have a smaller sample, you need to use a multiple slightly greater than 2. You can find what multiple you need by using the online calculator. Note: There is also a special calculator when dealing with task-times.
Now try two more examples from data we’ve collected.
Example 1
Fourteen users attempted to add a channel on their cable TV to a list of favorites. After the task they rated the difficulty on the 7 point Single Ease Question. Compute the 95% confidence interval. The responses are shown below
2, 6, 4, 1, 7, 3, 6, 1, 7, 1, 6, 5, 1, 1
Example 2
The brand favorability rating of LinkedIN on a five point scale from 62 participants was 4.32 with a standard deviation of .845. What is the 95% confidence interval?
Discrete Binary example
Imagine you asked 50 customers if they are going to repurchase your service in the future. Using a dummy variable you can code yes = 1 and no = 0. If 40 out of 50 reported their intent to repurchase, you can use the Adjusted Wald technique to find your confidence interval:
- Find the average by adding all the 1’s and dividing by the number of responses. 40/50=.8
- Adjust the proportion to make it more accurate by adding 2 to the numerator (the number of 1s) and the adjusted sample size by adding 4 to the denominator (total responses). Then divide the result.
40+2 = 42
50+4 = 54 (this is the adjusted sample size)
42/54 = .78 (this is your adjusted proportion) - Compute the standard error for proportion data.
- Multiply the adjusted proportion by 1 – the adjusted proportion.
.78 * ( 1-.78 )=.17 - Divide the result of step a by the adjusted sample size from step 2.
.17/ 54 = .0032 - Take the square root of the value from step b.
0032= .056
- Multiply the adjusted proportion by 1 – the adjusted proportion.
- Compute the margin of error by multiplying the standard error (result from step 3c) by 2.
.56×2=.11 - Compute the confidence interval by adding the margin of error from the sample proportion from step 2 and then subtracting the margin of error from the sample proportion.
.8+.11=.91
.8-.11=.69
The 95% confidence interval is .69 to .91. Our best estimate of the entire customer population’s intent to repurchase is between 69% and 91%.
Note: I’ve rounded the values to keep the steps simple. If you want more a more precise confidence interval, use the online calculator and feel free to read the mathematical foundation for this interval in Chapter 3 of our book, Quantifying the User Experience.
Now try some examples yourself from actual data we’ve collected.
Example 1:
If 6 out of 8 participants have a problem installing a printer from the printed installation instructions, what’s the best estimate for the minimum number of customers that would also have a problem.
Example 2:
If 5 out of 16 participants in a study mention they don’t pay credit card bills online because they fear their credit card information will be stolen, what’s the best estimate for the percent of all customers who feel this way?
Example 3:
If 3 out of 11 website visitors had a problem downloading and installing AutoCAD because they picked the wrong operating system on the download screen, what is our best estimate for the total percentage of website visitors who will also encounter this problem?
