Can an experience be useful without meeting your needs?
The UX-Lite™ is a new questionnaire that evolved from the SUS and the UMUX-Lite. It has only two items, one measuring perceived Ease and one measuring perceived Usefulness, as shown in Figure 1.
Because the verbal complexity of the original Usefulness item stands in stark contrast with the simplicity of its Ease item, we conducted numerous studies in 2021 to find simpler ways to express its fundamental meaning without substantially affecting its measurement properties (mean and response distribution). Precise wording is one of the defining characteristics of a standardized questionnaire: the wording (or changes in wording) are properly vetted to understand what, if any, impact it has on scores.
To date, we’ve found that the following alternates produce essentially equivalent UX-Lite scores relative to the original Usefulness item in combination with the Ease item:
- {Product}’s functionality meets my needs.
- {Product}’s features meet my needs.
- {Product}’s functions meet my needs.
- {Product} does what I need it to do.
Except for “does what I need it to do,” the alternates have the same sentence structure as the original, but with simpler words in place of “capabilities” and “requirements.”
We also measured “{Product} is useful” but found that although it correlated almost perfectly with the original, it seems to be easier for people to agree with, so its measurement properties aren’t a close enough match for us to recommend its use as an alternate for the standardized UX-Lite.
If an item is meant to reflect the usefulness of a product or experience, using the word “useful” certainly seems like a good representation of the construct. However, we have three goals with the UX-Lite:
- Describe a construct by replacing multisyllabic words in the original version (e.g., “capabilities”) with shorter, simpler words (e.g., “needs”).
- Keep it to two items to maintain the compactness of the UMUX-Lite.
- Ensure changes in wording don’t impact scores so researchers can compare new UX-Lite scores to earlier UMUX-Lite scores or use the terms interchangeably, depending on the context.
The last variation that we were curious about was a further simplification of the Usefulness wording, from “{Product}’s [functionality|features|functions] meet my needs” to “{Product} meets my needs.”
In this article, we report the results of four additional studies that we conducted to (1) replicate our previous evaluations of “{Product} is useful,” (2) investigate the measurement properties of “{Product} meets my needs,” and (3) see whether respondents prefer the original or a simplified version of the Usefulness item.
Study 1: Replicating “{Product} is useful”
Key Finding
Consistent with our earlier evaluations of this version, respondents clearly found it slightly (but significantly) easier to agree that a website is useful than to agree that its capabilities meet their requirements.
The effect size (defined in this article as the difference in means expressed as a percentage of the possible range of the metric) was a statistically significant 2.5% higher for the “is useful” version of the UX-Lite. The top-box and top-two-box scores for the “is useful” version of the Usefulness item were significantly higher than the original.
Experimental Design
We included the original version of the UX-Lite and a version using the alternate Usefulness item (“{Product} is useful”) as part of a SUPR-Q® survey of grocery websites (Food Lion, HEB, Kroger, Meijer, Publix, Safeway, Walmart, Whole Foods) collected from December 2021 to January 2022 using an online U.S.-based panel (n = 410). Respondents answered the original Usefulness item as part of a grid of about a dozen randomly presented agreement items and then answered the alternate in a second grid of about a dozen randomly presented agreement items.
Results
Figure 2 shows the mean UX-Lite scores from this study (both scores used the same Ease item, but different versions of the Usefulness item). The difference in means was statistically significant (t(409) = 5.98, p < .0001) with an observed difference of 2.5 (95% confidence interval ranging from 1.7 to 3.3).
Figure 3 shows the response distributions for the two versions of the Usefulness item. McNemar tests of top-box and top-two-box scores were highly significant (Top box, z = 4.94, mid-p < .0001, d = 11.0%, 95% confidence interval from 6.7 to 15.2%; Top-two-box, z = 3.5, mid-p < .0001, d = 6.1%, 95% confidence interval from 2.7 to 9.4%).
Study 2: Assessing “{Product} meets my needs”
Key Finding
In this evaluation, the means and response distributions for this alternate version were almost identical to those of the original.
The effect size for means was 0.5% higher for the “meets my needs” version of the UX-Lite. The top-box and top-two-box scores for the “meets my needs” version of the Usefulness item were slightly higher than the original.
Experimental Design
In the survey described above, we also included the original version of the UX-Lite and a version using the alternate Usefulness item (“{Product} meets my needs”) as part of a SUPR-Q survey of eight grocery websites collected from December 2021 to January 2022 using an online U.S.-based panel (n = 410). Respondents answered the original Usefulness item as part of one grid of about a dozen randomly presented agreement items and then answered the alternate in a second grid of about a dozen randomly presented agreement items.
Results
Figure 4 shows the mean UX-Lite scores from this study (both scores used the same Ease item, but different versions of the Usefulness item). The difference in means was very small and not significantly different (t(409) = 1.26, p = .21) with an observed difference of 0.5 (95% confidence interval ranging from −0.3 to 1.7).
Figure 5 shows the response distributions for the two versions of the Usefulness item are virtually identical. McNemar tests of top-box and top-two-box scores were not significant (Top box, z = 0.38, mid-p = .70, d = 0.7%, 95% confidence interval from −3.0 to 4.5%; Top-two-box, z = 0.59, mid-p = .59, d = 1.0%, 95% confidence interval from −2.3 to 4.2%).
Study 3: Replication of “{Product} meets my needs”
Key Finding
In this replication, the means were almost identical, and top-box scores were close, but we found a statistically significant difference in top-two-box scores consistent with respondents finding it harder to agree that a website meets their needs than that a website’s capabilities meet their requirements.
The effect size for means was 0.7% lower for the “meets my needs” version of the UX-Lite. The top-box and top-two-box scores for the “meets my needs” version of the Usefulness item were lower than the original.
Experimental Design
We included the original version of the UX-Lite and a version using the alternate Usefulness item (“{Product} meets my needs”) as part of a SUPR-Q survey of seller marketplace websites (craigslist, eBay, Etsy, Facebook Marketplace, OfferUp, Poshmark) collected from August to October 2021 using an online U.S.-based panel (n = 307). Respondents answered the original Usefulness item as part of one grid of about a dozen randomly presented agreement items and then answered the alternate in a following grid of about a dozen randomly presented agreement items.
Results
Figure 6 shows the mean UX-Lite scores from this study (both scores used the same Ease item, but different versions of the Usefulness item). The difference in means was not significantly different (t(306) = 1.57, p = .12) with an observed difference of 0.7 (95% confidence interval ranging from −1.8 to 1.7).
Figure 7 shows the response distributions for the two versions of the Usefulness item. McNemar tests of top-box and top-two-box scores were not significant (Top box, z = 0.88, mid-p = .38, d = 2.0%, 95% confidence interval from −2.4 to 6.2%; Top-two-box, z=2.19, mid-p = .03, d = 5.7%, 95% confidence interval from 0.5 to 9.2%).
Study 4: Another Replication of “{Product} meets my needs”
Key Finding
The results of this replication indicated that respondents found it harder to agree that a website meets their needs than to agree that its capabilities meet their requirements.
The effect size was a statistically significant 2.2% lower for the “meets my needs” version of the UX-Lite. The top-box and top-two-box scores for the “meets my needs” version of the Usefulness item were significantly lower than the original.
Experimental Design
We included the original version of the UX-Lite and a version using the alternate Usefulness item (“{Product} meets my needs”) as part of a quick screener survey of the eight grocery websites (e.g., Food Lion, Whole Foods) and thirteen news websites (e.g., BuzzFeed, CNN) collected in December 2021 using an online U.S.-based panel (n = 1,443).
Respondents were randomly assigned to rate two websites using both versions of the UX-Lite in one of the following orders:
- Order 1: Grocery with original version then News with alternate version
- Order 2: Grocery with alternate version then News with original version
- Order 3: News with original version then Grocery with alternate version
- Order 4: News with alternate version then Grocery with original version
With this Greco-Latin experimental design, we could focus on within-subjects comparison of the original and alternate versions of the UX-Lite.
Results
Figure 8 shows the mean UX-Lite scores from this study (both scores used the same Ease item, but different versions of the Usefulness item). The difference in means was statistically significant (t(1442) = 3.45, p = .001) with an observed difference of 2.2 (95% confidence interval ranging from 1.0 to 3.5).
Figure 9 shows the response distributions for the two versions of the Usefulness item with some notable differences. McNemar tests of top-box and top-2-box scores were highly significant (Top box, z = 2.78, mid-p = .005, d = 4.5%, 95% confidence interval from 1.3 to 7.6%; Top-two-box, z = 4.97, mid-p < .0001, d = 7.6%, 95% confidence interval from 4.6 to 10.5%).
In this survey, we also asked respondents which of the two versions of the Usefulness they preferred. As shown in Figure 10, among those who expressed a preference, there was roughly a 3:1 advantage for the simpler wording.
Summary and Takeaways
Our prior research and the four studies described in this article has found
“Functionality,” “features,” and “functions” are good substitutes for “capabilities.” Simpler alternates of the UX-Lite Usefulness item that have the same structural form as the original (“{Product}’s [functionality|features|functions] meet my needs”) work well enough to use in place of the original (“{Product}’s capabilities meet my requirements”). Based on this and our previous research, our current recommendation is to use “{Product}’s features meet my needs.” UX researchers can substitute “functionality” or “functions” for “features” when they are more appropriate for a specific research context.
“Useful” and “meets my needs” are not good substitutes. Two other items that deviate slightly from this sentence structure are “{Product} is useful” and “{Product} meets my needs.” The measurement properties of these versions are close to those of the original wording, but they are just different enough that we do not recommend their use as alternate Usefulness items in the UX-Lite standardized questionnaire.
A website can be useful and not meet user needs. Across several replications investigating “{Product} is useful,” we consistently found that it is slightly easier for respondents to agree that a website is useful than to agree that its capabilities meet their requirements. In our three investigations of “{Product} meets my needs,” the results were not perfectly consistent. The results indicate that respondents find it slightly more difficult to agree that a website meets their needs than to agree its capabilities (or features) meet their requirements (or needs).
Simple wording is preferred. We also found that respondents significantly prefer simple wording over the original, more cumbersome wording of the Usefulness item.
