{"id":24190,"date":"2021-04-20T19:13:56","date_gmt":"2021-04-20T19:13:56","guid":{"rendered":"https:\/\/measuringu.com\/?p=24190"},"modified":"2022-03-21T16:12:35","modified_gmt":"2022-03-21T22:12:35","slug":"how-does-hypothesis-testing-work","status":"publish","type":"post","link":"https:\/\/measuringu.com\/how-does-hypothesis-testing-work\/","title":{"rendered":"How Does Statistical Hypothesis Testing Work?"},"content":{"rendered":"
<\/a>Statistically significant. p-value. Hypothesis.<\/p>\n These terms are not only commonly used in statistics but also have made their way into the vernacular. Making sense of most scientific publications, which can have practical, significant effects on public policy and your life, means understanding a core framework with which we derive much knowledge. That framework is called hypothesis testing<\/em>, or more formally, Null Hypothesis Significance Testing<\/a> (NHST).<\/p>\n When comparing two measurements, whether they be completion rates, ease scores, or even the effectiveness of different vaccines, a hypothesis test’s primary goal is to make a binary classification decision: is the difference statistically significant or not? Yes or no.<\/p>\n The yes<\/em> or no<\/em> output from a hypothesis test looks simple, indistinguishable from a guess or a coin flip. It\u2019s the process of getting to yes<\/em> or no<\/em> that\u2019s different in NHST.<\/p>\n While we certainly want to make the right decision, we can never be correct 100% of the time. There will always be mistakes; there will always be errors. It\u2019s through the framework of hypothesis testing that we can control the frequency of errors over time.<\/p>\n Hypothesis testing can be a confusing concept, so we\u2019ve broken it into steps with examples. In a follow-up article, we\u2019ll discuss how things can go wrong in hypothesis testing and some criticisms of the framework.<\/p>\n Statistical hypothesis testing starts with something called the Null Hypothesis. Null<\/em><\/a> means none or no difference. A few ways to think of the Null Hypothesis are as the no-difference hypothesis or as the hypothesis that we want to nullify (disprove).<\/p>\n Mathematicians love using symbols, and that applies to hypothesis testing. The Null Hypothesis is represented as H0<\/sub>. H stands for hypothesis, and the 0\u2014that\u2019s a zero in the subscript, not the letter O\u2014reminds us it\u2019s the hypothesis of 0 (no, null) difference.<\/p>\n We start with the Null Hypothesis because it\u2019s easier to disprove something than to prove something. If we disprove the Null Hypothesis (reject the null), then we know there\u2019s at least some relationship between the variables we\u2019re measuring (e.g., different websites and ease scores).<\/p>\n Here are four examples of Null Hypotheses (H0<\/sub>) from studies we\u2019ll refer to throughout this article.<\/p>\n You need data to make decisions. Use methods such as surveys, experiments, and observations to collect data, and then calculate the differences observed. Data can be collected using either a between-subjects (different people in each condition) or within-subjects (same people in both) approach<\/a>.<\/p>\n Building on the examples from Step 1,<\/p>\n Table 1 summarizes the four examples (metrics and observed differences).<\/p>\n\nStep 1: Define the Null Hypothesis<\/h2>\n
\n
Step 2: Collect Data and Compute the Difference<\/h2>\n
\n