Sample size estimation is a critical step in research planning. It can also seem like a mysterious and at times controversial process. But sample size estimation, when done correctly, is based mostly on math, not magic. The challenge is that the math can get complex, so it becomes easier to defer to simple rules or a one-size-fits-all number. Before 2016, there was no well-defined method for computing confidence intervals for the Net Promoter Score (NPS). That was a major drawback in using the NPS for both comparative and longitudinal research.
That changed when Brendan Rocks (2016) published a paper in The American Statistician in which he investigated several approaches for constructing NPS confidence intervals. Rocks’ key finding was that the best-performing method was a variant of the adjusted-Wald procedure.
We investigated this method with our own data and, consistent with Rocks’ results, found it to work well with real-world NPS datasets varying in sample size and magnitude.
Once you have a method for constructing confidence intervals, it’s possible to derive a method for estimating the sample size required to achieve a desired margin of error.
In this article, we show how to develop a sample size estimation process for NPS confidence intervals, demonstrate how it works, and provide a quick lookup table based on the new method.
From Confidence Interval to Sample Size Estimation
The process for computing a sample size involves working backward from the statistical computation you want to perform. In an earlier article, we recommended using the adjusted-Wald method to compute confidence intervals for Net Promoter Scores. We just need to work backward from that adjusted-Wald formula by solving for the sample size algebraically. (If you’re interested in the math, see the appendix.)
At a high level, the sample size estimation process has four steps:
- Start with the level of precision needed.
- Estimate the standard deviation.
- Determine the appropriate level of confidence (usually 90% or 95%).
- Compute the sample size (n) or use lookup tables to avoid the messy math.
To keep computations simple in this article, we’ve used proportions rather than percentages. In practice, NPS are usually reported as percentages.
Simple Approximate Formula Assuming Maximum Variance
If you have no idea what your NPS data will look like, you can use a simple formula that guarantees your sample size will be large enough to achieve your precision goal. That formula is
n = Z2/d2 − 3
In the formula, Z is a Z-score whose value is determined by the desired level of confidence. (Common choices are 1.96 for 95% confidence or 1.645 for 90% confidence.) The variable d is the size of the margin of error that reflects your desired precision of measurement (the margin of error). Examining the formula shows that the more confident you need to be (larger value of Z) and the more precision you need (smaller value of d), the larger the sample size will need to be.
For example, if you’re planning a study and are considering targets of 95% confidence and precision of ± .10, the sample size will be 1.962/.102 − 3 = 382 (always round up). If you relax confidence to 90% and precision to ± .15, the sample size would be 1.6452/.152 − 3 = 118.
This formula is simple, but it has a notable downside. It guarantees an adequate sample size for the study’s goals, but because it assumes maximum variance, the sample size it estimates will often be very conservative. In other words, it will usually recommend a sample size that is much larger than needed.
Simple Approximate Formula Assuming Maximum Realistic Variance
Our next formula is still simple, but it has a slight modification that reduces the previous one’s overestimation.
n = .75(Z2)/d2 − 3
This is the same as the previous formula except for the .75 in the numerator, which reduces its estimates by about 25%. So, where did the .75 come from?
As shown in the appendix, before adjustment and simplification, the formula is
n = (ppro + pdet)( Z2)/d2
In this formula, ppro is the unadjusted proportion of promoters (those who rate their likelihood to recommend as 9 or 10 on a scale from 0–10); pdet is the unadjusted proportion of detractors (those who rate their likelihood to recommend as 0–6).
The variability of the NPS is maximized when half of the respondents are detractors and half are promoters. This means ppro = .5 and pdet = .5, so ppro + pdet = 1. Because 1(Z) = Z, the simplified formula drops the 1.
In our experience, however, we’ve never seen this happen. In previous research, we computed ppro.adj and pdet.adj for 18 sets of NPS data. Across those analyses, the value of (ppro.adj + pdet.adj) ranged from .54 to .80 with a mean (and 99% confidence interval) of .69 ± .05. Given an upper limit of .74 for the 99% confidence interval and a maximum observed value of .80, we settled on using .75 as a reasonable estimate of maximum realistic variance. As often happens in applied statistics, this decision is a mix of math and judgment, and it is subject to change pending the analysis of more data.
Applying this formula to the previous two examples, the sample size for 95% confidence and precision of ± .10 would be .75(1.962)/.102 − 3 = 286 (instead of 382). If relaxing confidence to 90% and precision to ± .15, the sample size would be 1.6452/.152 − 3 = 88 (instead of 118).
Maximum Accuracy Method Given Proportions of Promoters and Detractors
If you know something about the proportions of promoters and detractors, you can use a more complex method to estimate the required sample size more accurately. If you have data from a previous study, you can use that. Or you can run a pilot study and use what you learn to estimate the required sample size for the full study.
This is a two-stage process. In the first stage, estimate the sample size without any adjustments; in the second stage, fine-tune the initial estimate using adjusted-Wald proportions.
- From the previous/pilot study, get the values for the proportions of promoters and detractors (ppro and pdet).
- Compute the NPS as a proportion (NPS = ppro − pdet).
- Decide how precise you need your estimation to be (value for d as a proportion).
- Decide what level of confidence you need (value for Z).
- Compute the initial estimate of n (n.init = Z2(ppro + pdet − NPS2)/d2).
- Compute the adjusted number of promoters (npro.adj = ppro(n.init) + ¾).
- Compute the adjusted number of detractors (ndet.adj = pdet(n.init) + ¾).
- Compute the adjusted sample size (n.adj = n.init + 3).
- Compute the adjusted proportion of promoters (ppro.adj = npro.adj/n.adj).
- Compute the adjusted proportion of detractors (pdet.adj = ndet.adj/n.adj).
- Compute the adjusted NPS (NPS.adj = ppro.adj – pdet.adj).
- The values selected for d and Z in Stage 1 remain the same.
- Compute the estimated sample size requirement (n = (Z2(ppro.adj + pdet.adj − NPS.adj2)/d2) − 3). Always round up.
To check this process, consider the data from the two NPS studies shown in Table 1. In a UX survey of online meeting services conducted in 2019, we collected likelihood-to-recommend ratings. For GoToMeeting (GTM), there were 8 detractors, 13 passives, and 15 promoters, for an NPS of 19% (n = 36). For WebEx, there were 12 detractors, 12 passives, and 7 promoters, for an NPS of −16% (n = 31). Table 1 shows the steps to compute their 90% confidence intervals.
If the process works, we can start with the observed numbers of promoters and detractors, Z-values, and d (the MoE90 from the table), and then get an estimated sample size close to the actual n of the study. We must allow for some uncertainty due to rounding errors (36 ± 1 for GTM, 31 ± 1 for WebEx).
As shown in Table 2, the estimated sample size requirement produced by this process for GTM was 37 and for WebEx was 32, each just one more than the sample size from the studies. If these had been pilot studies where the goal was to have a precision of .10 but keeping everything else the same, the sample size for GTM would be 160 and for WebEx would be 156, about 20% less than the maximum realistic variance method would have estimated (n = .75(1.6452)/.102 − 3 = 200).
Sample Size Lookup Tables
As shown in the previous section, it’s best if you have some idea about the expected proportions of promoters and detractors, but that might not always be possible. Table 3 shows maximum realistic variance estimates for a range of values of d (this time is shown as percentages) and for two confidence levels (90% and 95%). (If you need an estimate that isn’t in the table, you can compute it using the formula.)
|Precision (±)||n (90% confidence)||n (95% confidence)|
If you have no choice but to collect all data in a single administration of the survey, the table can help with decisions about confidence and precision, even though the estimated sample sizes are likely to be overestimates. If you have resources for a sample size of 50 only, you should probably set confidence to 90% and expect precision around ± 20%. If anyone suggests that you need 95% confidence with a precision of ± 1%, showing that those requirements need n = 28,808 should redirect the conversation to more reasonable grounds.
If you can break the survey into multiple administrations, you can use the table for an initial sample size estimate for budgeting purposes. Plan to stop data collection after you have about 30 responses, and then use the observed proportions of promoters and detractors to get a more accurate estimate. Check the precision of your NPS confidence intervals when you get to the new estimate of n. If you’ve achieved your precision goal, you can stop, but if you’re not quite there yet, you still have a budget that lets you continue data collection.
Summary and Discussion
Sample size estimation is an important step in research planning. Based on an adjusted-Wald method for constructing NPS confidence intervals, we’ve developed several associated methods for estimating sample size requirements. The methods differ in their complexity.
The simplest approach, the maximum variance method, assumes no prior knowledge about likely NPS proportions of promoters and detractors. Consequently, it’s likely to overestimate the required sample size by a large amount.
The next simplest approach, the maximum realistic variance method, also assumes no prior knowledge about likely NPS proportions of promoters and detractors but reduces the maximum variance estimates by about 25% using an adjustment based on an analysis of 18 real-world NPS datasets.
The most complex approach, the maximum accuracy method, requires prior estimates of the proportions of promoters and detractors. Despite its complexity, we recommend its use whenever possible because it will be substantially more accurate than the other methods.
When it’s not possible to use the maximum accuracy method, refer to the table of sample size estimates. The table is based on the maximum realistic method, which you can use for initial project budgeting. Ideally, researchers should stop data collection after about 30 complete responses. They can use this initial data to estimate proportions of promoters and detractors and then use the maximum accuracy method to fine-tune the study’s sample size estimate.
To get to the formulas used in this article, we begin with the known formula for the variance of two dependent proportions, where the proportions in the context of the NPS are ppro (proportion of promoters) and pdet (proportion of detractors).
To keep things as simple as possible, we start with unadjusted values and will substitute adjusted values later.
var = ppro + pdet − (ppro − pdet)2
We can simplify this equation a bit because ppro − pdet is the NPS (proportion of promoters minus proportion of detractors).
var = ppro + pdet − NPS2
The margin of error for a confidence interval is Z(se), where se is the standard error (the square root of the variance divided by the sample size). The next task is to use algebra to get from the formula for the margin of error (d) of a confidence interval (still unadjusted) to one that solves for n.
d = Z[(ppro + pdet − NPS2)/n]½
Square both sides.
d2 = Z2(ppro + pdet − NPS2)/n
Multiply both sides by n and divide both sides by d2.
n = Z2(ppro + pdet − NPS2)/d2
Substitute adjusted n and proportions for unadjusted values. (Z and d are not adjusted.)
n.adj = Z2(ppro.adj + pdet.adj − NPS.adj2)/d2
Because for this type of adjusted-Wald procedure, n.adj = n + 3, make the substitution.
n + 3 = Z2(ppro.adj + pdet.adj − NPS.adj2)/d2
Subtract 3 from both sides to get the final formula for the maximum accuracy sample size estimate, where you have values of ppro and pdet from which you can compute estimates of ppro.adj and pdet.adj.
n = Z2(ppro.adj + pdet.adj − NPS.adj2)/d2 − 3
If estimates of ppro.adj and pdet.adj are not available, we can take advantage of the fact that maximum NPS variance occurs when half of respondents are promoters and half are detractors (which means NPS = 0). This is the maximum variance method.
n = Z2(.5 + .5 – 02)/d2 − 3
n = Z2(1)/d2 − 3
n = Z2/d2 − 3
The maximum variance formula is simple and will almost always produce a substantial overestimate for n. It is never the case, in our experience, that for any reasonable sample size you get an even split between promoters and detractors. As described in the body of the article, we can modify this formula to one that uses an estimate of ppro.adj + pdet.adj that will still probably overestimate n to a lesser extent because it’s based on analysis of 18 real-world NPS datasets. This is the maximum realistic variance method.
n = .75(Z2)/d2 − 3