How to Report Product Experience Data

Jeff Sauro, PhD • Jim Lewis, PhD

How often do you use your phone? How about a streaming service you subscribe to, like Hulu or Netflix? What about software, like Microsoft Excel?

In an earlier article, we discussed the importance of measuring prior experience. Prior experience is a strong predictor of UX metrics. People who are familiar with a product and use it frequently tend to complete tasks more quickly than unfamiliar users, and they generally have a more favorable attitude toward the product. In short, familiarity breeds content.

Product experience can be measured in at least three ways: how long people have been using an interface (tenure), how often they use it (frequency), and how many features/functions they use (depth).

Product frequency is probably the most commonly used measure of experience. If you had to pick just one to measure, this would be it in most cases. Being able to separate frequent (e.g., daily) from casual (e.g., monthly) users can really help explain differences in UX metrics.

In this article, we describe how to collect and display frequency data.

Collecting Frequency Data

There are two common ways to collect frequency data, one we recommend using and one we don’t.

Vague Quantifiers

The approach that we do not recommend (but which is commonly used in surveys) is the use of vague quantifiers (see Figure 1 for an example).

Figure 1: Frequency scale with vague quantifiers.

If written well, respondents and researchers will generally agree regarding the rank order of vague quantifiers. Using the terms in Figure 1, never will be less frequent than rarely, which will be less frequent than sometimes, which will be less frequent than often. The relative positioning of regularly and often, however, seems less clear.

We don’t recommend using vague quantifiers because of variations in how they are interpreted. What is the difference between regularly and often?

For example, Peterson (2000) describes the words near, very, quite, much, most, few, often, several, and occasionally as ambiguous. Again, what’s the difference between often and occasionally in this list? We’ve seen similar ambiguity with probability words and change verbs.

The caution is more than just quibbling over grammar. It can lead to very different measures of frequency. For example, Peterson reported on a study in which respondents responded “very frequently” to their movie attendance and “very much” to their monthly beer consumption. Later, those respondents assigned frequencies ranging from a low of 52 to a high of 365 as the number of movies per year and beers per month respectively.

Using these response options is fine if you want to gauge the perception of a behavior such as online streaming (do respondents think streaming an hour a day is frequent or moderate) and you have an adjective set with reasonably clear ordinality, but they’re not ideal if you’re researching the frequency of a behavior.

Specific Frequency Intervals

The second way to gauge product frequency is using specific frequency intervals as shown in Figure 2.

Figure 2: Frequency scale with specific frequency intervals.

With this approach, there’s less ambiguity in interpretation, although there will still be errors from how accurately participants can recall and differentiate between, say, something they do once a week (about four times a month) or several times a month (about three times a month using the standard definition of several as “more than two but less than many”).

Presenting Frequency Data

With the frequency data collected, you can present it in four ways.

1.    Provide a frequency distribution for each period.

Our recommended way of displaying product experience data is showing a stacked frequency chart based on the frequency intervals.

This works for both vague quantifiers and specific frequency intervals.

Figure 3 shows a graphical display for the specific frequency intervals and Figure 4 for the vague quantifiers from our 2020 Consumer Software Benchmark Report (modified from the original specific frequency data—for details, see the appendix).

Figure 3: Product usage frequency for Adobe Illustrator and Adobe PDF Reader using specific intervals.
Figure 4: Product usage frequency for Adobe Illustrator and Adobe PDF Reader using vague quantifiers.

Despite the differences, both approaches show the same noticeable difference in the products’ frequencies: Illustrator is used a lot less frequently than PDF Reader.

2.    Consolidate frequency levels.

Too many frequency gradients can make it hard for respondents to visualize and select. To compensate for this, consolidate frequency bands into fewer buckets. (We’ll cover what frequency intervals make sense to combine in an upcoming article.)

High-frequency users (like extreme responders in attitude questions) tend to be strong predictors of breaks in UX metrics. Figure 5 shows that extreme usage of the PDF Reader in our sample was more than three times as high as that for Illustrator.

Figure 5: Consolidating frequency levels to focus on proportions of extremely frequent users.

A benefit of consolidating responses into two categories is that it can be easier to compare the frequencies using a two-proportion test. In Figure 5, for example, there are more than three times more daily PDF Reader users than daily Illustrator users (Z = 1.98, p = .05).

3.    Assign a number to each of your categories and display means.

If you’re working with vague quantifiers, you can take advantage of ordinality in the response options and assign a number from low to high. For example, never could be coded 0, rarely 1, sometimes 2, often 3, and regularly 4. Note that the means of these types of numeric codes can only have an ordinal interpretation, not interval or ratio.

Figure 6. Means of ordinal numbers assigned to vague quantifiers (with 90% confidence intervals).

You can also find the standard deviations and add confidence intervals with the mean values. Figure 6 shows that the relative experience of PDF Reader users is substantially and statistically higher (the confidence interval error bars don’t overlap, t(87) = 5.9, p < .0001). Of course, interpreting exactly what 1.79 or 3.02 means on this frequency scale can be problematic and illustrates the challenge of relying on vague quantifiers (1.79—on average somewhere between rarely and sometimes; 3.02—very close to often, rough ordinal interpretations). Despite this shortcoming, the point is still clear: PDF Reader is used more frequently than Illustrator.

4.    Scale the periods to days per year.

If you have specific intervals but also want to work with means instead of multi-category data, you can convert the categories to number of days per year. Table 1 shows one way to map days per year to the categories.

Frequency CategoryDays/Year
Multiple times a day1,095
Daily365
Several times a week156
Once a week52
Several times a month36
Once a month12
Once a year or less1

Table 1: Assignment of approximate days/year to specific interval categories.

Figure 7 shows what we get when we assign these numbers to the specific interval categories. Although the detailed results are a bit different, the conclusion is the same—users of PDF Reader have substantially more experience (about four times more days per year) with that product than do users of Illustrator (non-overlapping confidence intervals, t(72) = 2.85, p = .006).

Figure 7. Means of approximate days/year assigned to specific interval categories (with 90% confidence intervals).

Summary

Because there is a high correlation between product experience and subjective UX ratings, it’s important to collect usage information when conducting UX research.

Frequency-of-use is a commonly used measure of product experience.

There are two common approaches to measuring frequency: vague quantifiers (e.g., rarely, often) or specific frequency categories (daily, several times a month). Both approaches can provide important information about frequency-of-use as long as they have a clear ordinality, but we generally recommend using specific frequency categories.

Once you have frequency data, there are several ways to present it:

  1. Frequency distributions work well for data collected with vague quantifiers or specific frequency categories, but they can be difficult to characterize statistically.
  2. Consolidating frequency categories into fewer levels can make interpretation easier, and it’s easy to statistically analyze data that has been categorized into two levels.
  3. To conduct statistical analyses such as t-tests or confidence intervals, you can quantify vague labels with an ordinal assignment of numbers (e.g., never = 0, rarely = 1, … often = 3). A shortcoming of this approach is that means can have an ordinal interpretation only.
  4. Estimates of days per year can be assigned to specific frequency categories (e.g., daily = 365, once a week = 52, several times a month = 36, once a month = 12), which can also be analyzed with procedures such as t-tests and confidence intervals.

Appendix

The Illustrator and PDF Reader data analyzed for this article came from our most recent (2020) UX survey of consumer software products. In that research, we used specific frequency categories. Table 2 shows how we assigned those choices to the sample vague, ordinal, and days/year assignments that we used in the analyses in this article. There are, of course, other ways to make these assignments, but it’s unlikely that other assignments would markedly change the outcomes.

IllustratorPDF ReaderSpecificVagueOrdinalDays/Year
17Multiple times a dayRegularly41,095
12DailyRegularly4365
110Several times a weekRegularly4156
65Once a weekOften352
413Several times a monthOften336
58Once a monthSometimes212
255Once a year or lessRarely11

Table 2: Frequency distributions and alternate assignments for Illustrator (n = 43) and PDF Reader (n = 50).

0
    0
    Your Cart
    Your cart is emptyReturn to Shop
    Scroll to Top