Five-point scales are the best.
No, seven points.
Never use a ten-point scale.
Eleven points “pretend noise is science.”
You never need more than three points.
Few things seem to elicit more opinions (and misinformation) in measurement than the “right” number of scale points to use in a rating scale response option.
For example, here is a discussion on Twitter by Erika Hall making the case that eleven-point scales are difficult for humans to respond to and three points would be better.
This sentiment is echoed in another post, this time about ten-point scales, which apparently “don’t reflect reality.”
And this one from Tomer who also advocates for the simplicity of two- and three-point scales:
Thumbs up, thumbs down is simple. 3-point scales are simple. If you feel strongly about a 4 or 5-point scale, go for it. Not more. If you measure very specific experiences, it is very easy for a user to decide if they are happy or not.
This advice gets repeated by others on the Internet:
Responses collected from a large, 11-point scale are extremely noisy, and meaningful changes in ratings are hard to detect … A better scale would use a 3-option Yes/Maybe/No system or a similar scale with 5 options.
Unfortunately, these articles offer no justification for the position. The references often just point to each other and offer at best a sort of folk wisdom and rationale, sort of like “all content should be reachable in three clicks”—because why make users click more?
While it’s good to start with theories of why one response scale would be better than another (for example, the theory that shorter scales are easier and better), it’s important to verify or falsify those claims with data. And a good place to start is the published literature that contains a wealth of research.
Number of Scale Point Guidance
As a general rule, when measuring a construct that falls on a continuum from low to high (such as satisfaction, ease, and likelihood to recommend), the more points you have in your rating scale, the more reliable (consistent responses) and valid (reflects true attitudes) it generally is. The bluntness of scales with few points reduces correlations and causes the reduction in reliability and validity. We pull much of this from the work of Jum Nunnally, the noted psychometrician who wrote about the reliability of scales in 1967, 1978 and in 1994.
The increase in reliability and validity by adding more points to a scale doesn’t go on forever and it isn’t a linear increase. There is both a diminishing return (just adding more points doesn’t continue to help) and nuances in scale formats, such as the order, labels, and neutral points, that can interfere in improvements.
With more items in a questionnaire, the number of scale points matters less (although it still has an effect). For example, the SUMI questionnaire with its 50 items uses three-point response options. While responses to each of the items is rather coarse (agree, undecided, disagree) the fidelity when averaging across 50 items increases the overall score. In contrast, the SEQ is a single item with seven points, which we found superior to the original five-point version because it has more points of discrimination. The same is the case with the single item Likelihood to Recommend item with 11 points (scaled from 0 to 10).
But what do other published studies show about the number of scale points? Has there been any change since the 1970s?
History of Scale Points
For about as long as we’ve had rating scales, we’ve had questions (and concerns) about the “optimal” number of scale steps (see Boyce, 1915).
For the last 100 years there has been a lot published on the different numbers of scale points in general, but there has also been more specific research on the use of two- and three-point scales versus scales with more points.
Early Research: Two- and Three-Point Scales Are Inadequate
In one of the earliest studies, Ghiselli (1939) found four-point scales performed better than two-point scales. The authors had 200 undergraduate students rate the sincerity of advertising pieces for 41 different brands across 12 commodity types. Half the respondents were given two options (yes or no) while the other half were given a four-point scale: very sincere, fairly sincere, fairly insincere, and very insincere. In both conditions, respondents had the choice of selecting “uncertain.” They found more people responded favorably (top-two box) with the four-point compared to the two-point scale and fewer people selected the “uncertain” response when given four points.
Thus, when a four-step response was permitted, more people were willing to respond than when only a two-step response was permitted. They concluded
to cast grave doubt on the assumption implicit in the use of the 2-step response (as yes-no, favorable-unfavorable, etc.) questionnaire method as a measure of “average” opinion.
In the ensuing decades, the debate continued. Green and Rao summarized the debate on scale points in 1970 and described two factions. One advocates for using fine grained scale points (11 or 21) while another, based on opinions about respondents’ ability to differentiate between different points, advocates for only two or three response options. The loss of fidelity by having only two or three points, they argued, is made up for by asking more questions. Green and Rao conducted a simulation study and showed that adding more questions did NOT make up for the loss in fidelity from using two or three points (at least in their specific study type). So, one point in the column for more than two-point scales.
Three-Point Scales May Be Good Enough When Using Many Items
In response to Green and Rao, in the rather bluntly titled “Three-Point Likert Scales Are Good Enough” article, Matell and Jacoby (1971) argued that three points are good enough in some cases. They had 360 undergraduate psych students answer one of eighteen versions of a 72-item questionnaire of values with the response options varying in the number of scale points between two and nineteen (so there were twenty responses per condition).
Respondents were then given the same questionnaire three weeks later. The authors found little differences in reliabilities and their measure of validity and concluded that as few as two response categories may be adequate in practice. They suggested that both reliability and validity are independent of the number of response categories and their results implied that collapsing data from longer scales into two- or three-point scales would not diminish the reliability or validity of the resulting scores.
Three-Point Scales Frustrate and Stifle
In another paper, Lehmann & Hulbert (1972) argued that the main problem with two- and three-point scales is that they force respondents to choose and introduce rounding error. In the also aptly named “Are Three-Point Scales Always Good Enough?” article, the authors conducted a simulation study with items with three, five, seven, and nine points. They concluded that two or three points are probably fine when averaging across people and across many items. But if the focus of the research is on individual scales, using a minimum of five to six scale points is probably necessary to get an accurate measure of the variable. They found, for example, that even when 30 items are summed, the use of six or seven points instead of three cuts the error approximately in half (see their Figure 3).
Cox (1980) offers one of the more comprehensive reviews of the topic (spanning 80 years). His analysis concluded that “scales with two or three response alternatives are generally inadequate and … tend to frustrate and stifle respondents” and that the marginal return from using more than nine response alternatives is minimal.
Three-Point Scales Contain Less Information
Researchers often use rating scales as dependent variables in regression analysis (e.g., as part of a KDA) to understand what drives brand attitude or satisfaction. Morrison (1972) showed that using discrete points for an underlying continuous variable (like satisfaction) will reduce information. Little is lost at eleven points where 99% of the information is transmitted but 87.5% is transmitted with a three-point scale (illustrating loss of discriminating ability with coarser three-point scales).
Morrison is building off of information theory (Shannon & Weaver, 1949) where the “bits” of information transmitted is the log of the number of response options. A two-point scale communicates only one piece of information (yes/no, agree/disagree). Three-point scales communicate two pieces of information (neutrality and direction). Four-point scales communicate intensity of direction, but no neutral opinion. Five-point scales are then better theoretically because they provide three pieces of information: direction (positive/negative), intensity of opinion, and a neutral point.
More Points Differentiate Better Between Groups
In another study, Loken et al. (1987) examined the criterion validity of various telephone-administered scales through their ability to differentiate between different population groups and found eleven-point scales to be superior to three-point or four-point scales.
Lozano et al. (2008), using a simulation study, analyzed between two and nine response options and concluded “the optimum number of alternatives is between four and seven. With fewer than four alternatives the reliability and validity decrease, and from seven alternatives onwards, psychometric properties of the scale scarcely increase further.” The authors didn’t test ten- or eleven-point scales, however.
Alwin (1997) compared seven- and eleven-point scales from a 1978 life satisfaction survey and found eleven-point scales were superior to seven points. He used a large probability sample of 3,692 U.S. respondents who rated 17 domains of satisfaction (e.g., residence, living, and life satisfaction) and used three scales in the same order: a seven-point satisfied to dissatisfied (endpoints and midpoint labeled); a seven-point delighted-terrible (fully labeled); and an eleven-point feeling thermometer (only end points labeled).
He found that eleven-point scales had higher reliability than seven-point scales and in many cases higher validity coefficients and recommended eleven-point scales when measuring life satisfaction and rejected the idea that the eleven-point scale is more vulnerable to measurement errors.
In a study using a UX questionnaire, Lewis (In Press) found little difference between scales with five, seven, and eleven points but found three-point scales were inadequate. He had 242 participants randomly assigned to a three-, five-, seven-, or eleven-point version of the UMUX-Lite. He found little difference in reliabilities and correlations to other measures except for the three-point version, which had the lowest (and unacceptable reliability alpha = .61).
Three-Point Scales Can’t Identify Extreme Attitudes
In our earlier analysis of top-box scoring, we reviewed studies that showed that extreme responders (the most favorable and least favorable responses) tend to be better predictors of behavior. With three-point scales, as other studies have shown, there is no way to differentiate between extreme and tepid responses. The Net Promoter Score calculation doesn’t split the eleven-point scale into three equal intervals; it uses the difference between the most favorable and least favorable responses. We’ll cover what happens when you turn an eleven-point scale into a three-point scale in an upcoming article.
Three-Point Scales Rated as Quicker but Five, Seven, and Ten Points are Easier.
One of the more relevant and recent compelling studies came from Preston & Colman (2000). In the study, 149 (mostly) students responded to their experiences at a store or restaurant using a 101-point scale and five questions rating their experiences (e.g., promptness of service) repeated 11 times with the only thing changing was the number of points, which ranged from 2 to 11.
All scales were anchored with very poor on the left to very good on the right. Participants also rated the scale formats on ease of use, quickness of use, and allowing them to express feelings adequately. They then had respondents answer the same questionnaire again one to three weeks later to assess test-retest reliability; 129 (86%) completed the second one. The authors found:
- Ease: 5, 7, and 10-point scales were rated as easiest to use and 11- and 101-point scales the least easy to use.
- Quickness: 2, 3, and 4-point scales were rated as the quickest to use and 11 and 101 were rated the least “quick.”
- Express feelings adequately: 2- and 3-point scales were rated “extremely” low on “allowed you to express your feelings adequately,” whereas scales with 9, 10, 11, and 101 points were rated the highest.
- Reliability: 2, 3, and 4-point scales were least reliable (test-retest and internal consistency) while scales with 7–10 points were the most.
The authors concluded that scales with small numbers of response categories yield scores that are generally less valid and less discriminating than those with six or more response categories. They concluded that rating scales with seven, nine, or ten response categories are generally to be preferred.
In a future article, we’ll look to corroborate or clarify the findings in the literature and apply it specifically to the Net Promoter Score using newly collected data.
Summary and Takeaways
A review of 12 studies conducted on the difference between three and more point scales revealed:
More scale points increase reliability. Not much has changed in the 40–50 years since Nunnally’s work. Despite changes in survey formats (especially web surveys), the literature overwhelmingly shows that as you increase the number of scale points you increase reliability. The reliability will increase most when going from three to five points and less is gained as you exceed seven or eleven points.
Three-point scales are not reliable. Across multiple studies, using only three points was shown not only to have lower reliability, but the reliability wasn’t even adequate, and in some studies, even when averaging across items, too much is lost. The low reliability is a consequence of using a coarse scale to represent a continuum. By forcing respondents to choose too few categories, it introduces more error in responding and thus makes responses less consistent in the same study (internal reliability) and over time (test-retest reliability).
There’s a loss of intensity and validity with three-point scales. Using only three points loses all information about intensity, or the strength of people’s opinion. Not everyone feels equally favorably or unfavorably toward a brand, interface, or experience. All but one of the 12 studies we examined recommended using more than three points in a rating scale. In the one study that did suggest three-point scales were enough, it averaged the results across multiple items, not using a single item.
Two- and three-point scales are perceived as quicker and easier. In one study, participants rated two- and three-point scales as being quicker and easier to respond to. This lends credence to the idea that shorter scales require less mental effort to respond to. If speed and the perception of ease is paramount (especially if using multiple items) a researcher may decide that two or three points are good enough. But know that while respondents may save time, it may stifle their ability to actually express their opinions. Or, if you look at the scale as a user interface, you’re preferring a faster and easier but less effective scale.
Two and three points are insufficient to express feelings adequately. While two- and three-point scales may be perceived as faster and easier to respond to, participants overwhelmingly felt they were inadequate in allowing them to express their feelings. We’ll attempt to replicate these findings and measure actual response time (as opposed to perception of speed) in an upcoming article.
The three-point scale superiority is a myth. If you’re concerned about putting a burden on your respondent by including more points, this analysis suggests participants don’t find scales with more than three points necessarily more difficult. In fact, in one study, scales with five, seven, and ten points were rated as EASIER to use than two- and three-point scales.
In other words, using two- and three-point scales may be getting you a slightly quicker but unreliable response, and respondents think they are less easy to use. Along with the idea that all website content should be within 3 clicks, the idea that 3 point scales are always better (or even sometimes better) than scales with more points needs to go into the UX myth dust bin.
Thanks to Jim Lewis for proving comments on this article.
Learn More: UX Measurement Boot Camp
Intensive Training on UX Methods, Metrics and Measurement
|Denver: Aug. 5th-7th, 2020|