{"id":31727,"date":"2022-03-22T19:57:57","date_gmt":"2022-03-23T01:57:57","guid":{"rendered":"https:\/\/measuringu.com\/?p=31727"},"modified":"2022-03-22T07:02:32","modified_gmt":"2022-03-22T13:02:32","slug":"how-to-report-product-experience-data","status":"publish","type":"post","link":"https:\/\/measuringu.com\/how-to-report-product-experience-data\/","title":{"rendered":"How to Report Product Experience Data"},"content":{"rendered":"

\"\"<\/a>How often do you use your phone? How about a streaming service you subscribe to, like Hulu or Netflix? What about software, like Microsoft Excel?<\/p>\n

In an earlier article, we discussed the importance of measuring prior experience. Prior experience is a strong predictor of UX metrics. People who are familiar with a product and use it frequently tend to complete tasks more quickly than unfamiliar users, and they generally have a more favorable attitude toward the product. In short, familiarity breeds content<\/a>.<\/p>\n

Product experience can be measured in at least three ways<\/a>: how long people have been using an interface (tenure), how often they use it (frequency), and how many features\/functions they use (depth).<\/p>\n

Product frequency is probably the most commonly used measure of experience. If you had to pick just one to measure, this would be it in most cases. Being able to separate frequent (e.g., daily) from casual (e.g., monthly) users can really help explain differences in UX metrics<\/a>.<\/p>\n

In this article, we describe how to collect and display frequency data.<\/p>\n

Collecting Frequency Data<\/h2>\n

There are two common ways to collect frequency data, one we recommend using and one we don\u2019t.<\/p>\n

Vague Quantifiers<\/h3>\n

The approach that we do not recommend (but which is commonly used in surveys) is the use of vague quantifiers (see Figure 1 for an example).<\/p>\n

\"\"<\/a>
Figure 1:<\/strong> Frequency scale with vague quantifiers.<\/figcaption><\/figure>\n

If written well, respondents and researchers will generally agree regarding the rank order of vague quantifiers. Using the terms in Figure 1, never<\/em> will be less frequent than rarely,<\/em> which will be less frequent than sometimes,<\/em> which will be less frequent than often<\/em>. The relative positioning of regularly<\/em> and often<\/em>, however, seems less clear.<\/p>\n

We don\u2019t recommend using vague quantifiers because of variations in how they are interpreted. What is the difference between regularly<\/em> and often<\/em>?<\/p>\n

For example, Peterson (2000)<\/a> describes the words near<\/em>, very<\/em>, quite<\/em>, much<\/em>, most<\/em>, few<\/em>, often<\/em>, several<\/em>, and occasionally<\/em> as ambiguous. Again, what\u2019s the difference between often<\/em> and occasionally<\/em> in this list? We\u2019ve seen similar ambiguity with probability words and change verbs<\/a>.<\/p>\n

The caution is more than just quibbling over grammar. It can lead to very different measures of frequency. For example, Peterson reported on a study in which respondents responded \u201cvery frequently\u201d to their movie attendance and \u201cvery much\u201d to their monthly beer consumption. Later, those respondents assigned frequencies ranging from a low of 52 to a high of 365 as the number of movies per year and beers per month respectively.<\/p>\n

Using these response options is fine if you want to gauge the perception <\/em>of a behavior such as online streaming (do respondents think streaming an hour a day is frequent or moderate) and you have an adjective set with reasonably clear ordinality, but they\u2019re not ideal if you\u2019re researching the frequency of a behavior.<\/p>\n

Specific Frequency Intervals<\/h3>\n

The second way to gauge product frequency is using specific frequency intervals as shown in Figure 2.<\/p>\n

\"\"<\/a>
Figure 2:<\/strong> Frequency scale with specific frequency intervals.<\/figcaption><\/figure>\n

With this approach, there\u2019s less ambiguity in interpretation, although there will still be errors from how accurately participants can recall and differentiate between, say, something they do once a week (about four times a month) or several times a month (about three times a month using the standard definition of several<\/em><\/a> as \u201cmore than two but less than many\u201d).<\/p>\n

Presenting Frequency Data<\/h2>\n

With the frequency data collected, you can present it in four ways.<\/p>\n

1.\u00a0\u00a0\u00a0 Provide a frequency distribution for each period.<\/h3>\n

Our recommended way of displaying product experience data is showing a stacked frequency chart based on the frequency intervals.<\/p>\n

This works for both vague quantifiers and specific frequency intervals.<\/p>\n

Figure 3 shows a graphical display for the specific frequency intervals and Figure 4 for the vague quantifiers from our 2020 Consumer Software Benchmark Report<\/a> (modified from the original specific frequency data\u2014for details, see the appendix).<\/p>\n

\"\"<\/a>
Figure 3:<\/strong> Product usage frequency for Adobe Illustrator and Adobe PDF Reader using specific intervals.<\/strong><\/figcaption><\/figure>\n
\"\"<\/a>
Figure 4:<\/strong> Product usage frequency for Adobe Illustrator and Adobe PDF Reader using vague quantifiers.<\/strong><\/figcaption><\/figure>\n

Despite the differences, both approaches show the same noticeable difference in the products\u2019 frequencies: Illustrator is used a lot less frequently than PDF Reader.<\/p>\n

2.\u00a0\u00a0\u00a0 Consolidate frequency levels.<\/h3>\n

Too many frequency gradients can make it hard for respondents to visualize and select. To compensate for this, consolidate frequency bands into fewer buckets. (We\u2019ll cover what frequency intervals make sense to combine in an upcoming article.)<\/p>\n

High-frequency users (like extreme responders in attitude questions) tend to be strong predictors<\/a> of breaks in UX metrics. Figure 5 shows that extreme usage of the PDF Reader in our sample was more than three times as high as that for Illustrator.<\/p>\n

\"\"<\/a>
Figure 5:<\/strong> Consolidating frequency levels to focus on proportions of extremely frequent users.<\/figcaption><\/figure>\n

A benefit of consolidating responses into two categories is that it can be easier to compare the frequencies using a two-proportion test. In Figure 5, for example, there are more than three times more daily PDF Reader users than daily Illustrator users (Z<\/em> = 1.98, p<\/em> = .05).<\/p>\n

3.\u00a0\u00a0\u00a0 Assign a number to each of your categories and display means.<\/h3>\n

If you\u2019re working with vague quantifiers, you can take advantage of ordinality in the response options and assign a number from low to high. For example, never<\/em> could be coded 0, rarely<\/em> 1, sometimes<\/em> 2, often<\/em> 3, and regularly<\/em> 4. Note that the means of these types of numeric codes can only have an ordinal interpretation, not interval or ratio<\/a>.<\/p>\n

\"\"<\/a>
Figure 6.<\/strong> Means of ordinal numbers assigned to vague quantifiers (with 90% confidence intervals).<\/figcaption><\/figure>\n

You can also find the standard deviations and add confidence intervals with the mean values. Figure 6 shows that the relative experience of PDF Reader users is substantially and statistically higher (the confidence interval error bars don\u2019t overlap<\/a>, t<\/em>(87) = 5.9, p<\/em> < .0001). Of course, interpreting exactly what 1.79 or 3.02 means on this frequency scale can be problematic and illustrates the challenge of relying on vague quantifiers (1.79\u2014on average somewhere between rarely<\/em> and sometimes<\/em>; 3.02\u2014very close to often<\/em>, rough ordinal interpretations). Despite this shortcoming, the point is still clear: PDF Reader is used more frequently than Illustrator.<\/p>\n

4.\u00a0\u00a0\u00a0 Scale the periods to days per year.<\/h3>\n

If you have specific intervals but also want to work with means instead of multi-category data, you can convert the categories to number of days per year. Table 1 shows one way to map days per year to the categories.<\/p>\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\t\n\t\n\t
Frequency Category<\/th>Days\/Year<\/th>\n<\/tr>\n<\/thead>\n
Multiple times a day<\/td>1,095<\/td>\n<\/tr>\n
Daily<\/td>365<\/td>\n<\/tr>\n
Several times a week<\/td>156<\/td>\n<\/tr>\n
Once a week<\/td>52<\/td>\n<\/tr>\n
Several times a month<\/td>36<\/td>\n<\/tr>\n
Once a month<\/td>12<\/td>\n<\/tr>\n
Once a year or less<\/td>1<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n

Table 1<\/strong>: Assignment of approximate days\/year to specific interval categories.<\/p>\n

Figure 7 shows what we get when we assign these numbers to the specific interval categories. Although the detailed results are a bit different, the conclusion is the same\u2014users of PDF Reader have substantially more experience (about four times more days per year) with that product than do users of Illustrator (non-overlapping confidence intervals, t<\/em>(72) = 2.85, p<\/em> = .006).<\/p>\n

\"\"<\/a>
Figure 7.<\/strong> Means of approximate days\/year assigned to specific interval categories (with 90% confidence intervals).<\/figcaption><\/figure>\n

Summary<\/h2>\n

Because there is a high correlation between product experience and subjective UX ratings, it\u2019s important to collect usage information when conducting UX research.<\/p>\n

Frequency-of-use is a commonly used measure of product experience.<\/p>\n

There are two common approaches to measuring frequency: vague quantifiers (e.g., rarely, often) or specific frequency categories (daily, several times a month). Both approaches can provide important information about frequency-of-use as long as they have a clear ordinality, but we generally recommend using specific frequency categories.<\/p>\n

Once you have frequency data, there are several ways to present it:<\/p>\n

    \n
  1. Frequency distributions work well for data collected with vague quantifiers or specific frequency categories, but they can be difficult to characterize statistically.<\/li>\n
  2. Consolidating frequency categories into fewer levels can make interpretation easier, and it\u2019s easy to statistically analyze data that has been categorized into two levels.<\/li>\n
  3. To conduct statistical analyses such as t-tests or confidence intervals, you can quantify vague labels with an ordinal assignment of numbers (e.g., never = 0, rarely = 1, \u2026 often = 3). A shortcoming of this approach is that means can have an ordinal interpretation only.<\/li>\n
  4. Estimates of days per year can be assigned to specific frequency categories (e.g., daily = 365, once a week = 52, several times a month = 36, once a month = 12), which can also be analyzed with procedures such as t-tests and confidence intervals.<\/li>\n<\/ol>\n

    Appendix<\/h2>\n

    The Illustrator and PDF Reader data analyzed for this article came from our most recent (2020) UX survey of consumer software products<\/a>. In that research, we used specific frequency categories. Table 2 shows how we assigned those choices to the sample vague, ordinal, and days\/year assignments that we used in the analyses in this article. There are, of course, other ways to make these assignments, but it\u2019s unlikely that other assignments would markedly change the outcomes.<\/p>\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\t\n\t\n\t
    Illustrator<\/th>PDF Reader<\/th>Specific<\/th>Vague<\/th>Ordinal<\/th>Days\/Year<\/th>\n<\/tr>\n<\/thead>\n
    1<\/td>7<\/td>Multiple times a day<\/td>Regularly<\/td>4<\/td>1,095<\/td>\n<\/tr>\n
    1<\/td>2<\/td>Daily<\/td>Regularly<\/td>4<\/td>365<\/td>\n<\/tr>\n
    1<\/td>10<\/td>Several times a week<\/td>Regularly<\/td>4<\/td>156<\/td>\n<\/tr>\n
    6<\/td>5<\/td>Once a week<\/td>Often<\/td>3<\/td>52<\/td>\n<\/tr>\n
    4<\/td>13<\/td>Several times a month<\/td>Often<\/td>3<\/td>36<\/td>\n<\/tr>\n
    5<\/td>8<\/td>Once a month<\/td>Sometimes<\/td>2<\/td>12<\/td>\n<\/tr>\n
    25<\/td>5<\/td>Once a year or less<\/td>Rarely<\/td>1<\/td>1<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n

    Table 2<\/strong>: Frequency distributions and alternate assignments for Illustrator (n<\/em> = 43) and PDF Reader (n<\/em> = 50).<\/p>\n","protected":false},"excerpt":{"rendered":"

    How often do you use your phone? How about a streaming service you subscribe to, like Hulu or Netflix? What about software, like Microsoft Excel? In an earlier article, we discussed the importance of measuring prior experience. Prior experience is a strong predictor of UX metrics. People who are familiar with a product and use […]<\/p>\n","protected":false},"author":2,"featured_media":31746,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"site-sidebar-layout":"default","site-content-layout":"default","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"default","adv-header-id-meta":"","stick-header-meta":"default","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"_price":"field_56e41332a1ae5","_stock":"","_tribe_ticket_header":"","_tribe_default_ticket_provider":"Tribe__Tickets_Plus__Commerce__WooCommerce__Main","_tribe_ticket_capacity":"0","_ticket_start_date":"","_ticket_end_date":"","_tribe_ticket_show_description":"","_tribe_ticket_show_not_going":false,"_tribe_ticket_use_global_stock":"","_tribe_ticket_global_stock_level":"","_global_stock_mode":"","_global_stock_cap":"","_tribe_rsvp_for_event":"","_tribe_ticket_going_count":"","_tribe_ticket_not_going_count":"","_tribe_tickets_list":"[]","_tribe_ticket_has_attendee_info_fields":false,"footnotes":""},"categories":[183,62,39],"tags":[478,10,45],"acf":[],"ticketed":false,"_links":{"self":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/31727"}],"collection":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/comments?post=31727"}],"version-history":[{"count":13,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/31727\/revisions"}],"predecessor-version":[{"id":32006,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/31727\/revisions\/32006"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/media\/31746"}],"wp:attachment":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/media?parent=31727"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/categories?post=31727"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/tags?post=31727"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}