# How Confident Do You Need to be in Your Research?

Jeff Sauro, PhD

Every estimate we make from a sample of customer data contains error. Confidence intervals tell us how much faith we can have in our estimates.

Confidence intervals quantify the most likely range for the unknown value we’re estimating. For example, if we observe 27 out of 30 users (90%) completing a task, we can be 95% confident that between 74% and 97% of all real-world users can complete that task.

But what exactly does a confidence level of 95% mean?

Well, if you were to take a sample from the same customer population 100 times and then compute a confidence interval around the task-completion rate each time, 95 of those intervals would contain the true task-completion rate. The other 5 times, the interval would not contain the true task-completion rate. You can never know that the interval you compute contains the true percentage. Statistics is about understanding and managing the risk of being wrong.

## Confidence and the p-value

The confidence level is related to the p-value obtained when conducting statistical comparisons. We usually consider something “statistically significant” if its p-value is less than 0.05 (or 5%).

The confidence level and the p-value that determines the threshold for statistical significance are values we set ahead of time, using what we call the alpha level. If we choose an alpha level of 0.05, for example, then a p-value smaller than 0.05 is considered statistically significant, and our confidence level (1–alpha) is 0.95.

Although we most often set alpha to 0.05, it can take any value from just above 0 (e.g., 0.00001)  to just below 1 (e.g., 0.99999). I’m often asked what the best level of confidence to use is. The answer is that it depends on the consequences of being wrong. To help put that into context, here are different thresholds commonly used for confidence (and p-values) that you can apply. Select the level which most closely matches your situation.