{"id":502,"date":"2018-10-31T03:19:05","date_gmt":"2018-10-31T03:19:05","guid":{"rendered":"http:\/\/measuringu.com\/seq-prediction\/"},"modified":"2022-03-21T17:36:52","modified_gmt":"2022-03-21T23:36:52","slug":"seq-prediction","status":"publish","type":"post","link":"https:\/\/measuringu.com\/seq-prediction\/","title":{"rendered":"Using Task Ease (SEQ) to Predict Completion Rates and Times"},"content":{"rendered":"

\"seq-predict-feature\"<\/a>Our attitudes both reflect and affect our actions.<\/p>\n

What we think affects what we do and what we do affects what we think.<\/p>\n

It\u2019s not a perfect relationship of course. What people say or think doesn\u2019t always directly correspond to actions in easily predictable ways.<\/p>\n

Understanding and measuring user attitudes early and often can provide a good indication of likely future behaviors. Attitudes can be measured using short (ideally standardized<\/a>) questionnaires\u00a0(e.g. SUS<\/a> and SUPR-Q<\/a>), which can be administered after a study or experience or immediately after a task experience in a usability test (e.g. the SEQ).<\/p>\n

One of the more popular post-task questionnaires is the Single Ease Question (SEQ)<\/a>. It\u2019s a single item that assesses how easy or difficult participants think a task is (Figure 1).<\/p>\n

\"Figure<\/a><\/p>\n

Figure 1: The Single Ease Question (SEQ).<\/p>\n

We found this simple 7-point item performed as well or better than other short questionnaires [PDF]<\/a>. We\u2019ve presented the item labeled as both \u201cOverall, this task was\u201d or \u201cHow easy or difficult was this task?\u201d and found the results indistinguishable.<\/p>\n

Correlating Attitudes with Behavior<\/h2>\n

A fundamental issue in measurement is validation<\/a>. We want our measures to be tied to reality. This is especially important when measuring attitudes, which can be fickle and hard to quantify. Validating attitudinal data like the SEQ involves showing how the numbers we assign to attitudes (things we can\u2019t observe directly) correspond to actions (things we can observe directly).<\/p>\n

In earlier published research we found that UX metrics have a medium to strong correlation<\/a> with each other (between r = .4 <\/i><\/em>and\u00a0.6)<\/i><\/em>. The strength of this correlation changes depending on how the data is correlated and the context in which it\u2019s collected.<\/p>\n

One of the benefits of questionnaires is that they can be easily administered with an interface with any level of fidelity. Encouraging research has shown<\/a>\u00a0that ease of use questionnaires similar to the SEQ are reliable across prototype fidelity levels (from low to high).<\/p>\n

There are times when collecting completion rate and task time data is challenging. This is especially the case when you can\u2019t easily verify the results (in remote unmoderated studies<\/a>) or because the prototype fidelity makes the task time seem too dependent on the fidelity (no load times, for example).<\/p>\n

But how well do questionnaires like the SEQ correlate with (predict) task behavior as measured by completion rates and time?<\/p>\n

SEQ and Completion Rates<\/h2>\n

To understand the relationship between the SEQ and concurrently collected completion rates, we aggregated mean SEQ scores and average completion rates across 286 tasks from data we\u2019ve collected over the past few years in moderated and unmoderated studies. Each task had a sample size between 8 to 601 participants.<\/p>\n

The SEQ and completion rates have a strong correlation (r = .66). This means that perception of ease (SEQ scores) can explain about 44% of task completion rates. This strong relationship shows that what participants think about what happened in a task generally corresponds to what actually happened (but not perfectly). This is similar to what we found with SUS scores<\/a>.<\/p>\n

To visualize this relationship, I converted raw SEQ scores into percentile ranks and then bucketed the scores into deciles (10 groups) along with completion rates as shown in Figure 2.<\/p>\n

\"Figure<\/a><\/p>\n

Figure 2: Relationship between SEQ scores (converted into deciles) and completion rates for 286 tasks.<\/p>\n

For example, a raw SEQ score of 5.1 will correspond to a completion rate of about 68% (with a margin of error of around +\/- 6%). A raw SEQ score of 6.3 will correspond to a completion rate of 90% (with a tighter margin of error of +\/- 2%).<\/p>\n

You can see the linear relationship is strongest in the middle (between the 30% and 70% SEQ deciles). After SEQ scores are above the 70th<\/sup>\u00a0percentile (about a 5.9) the completion rates asymptote and level off at 90%. Conversely, after SEQ scores dip below the 30th<\/sup>\u00a0percentile (a raw score below 4.7), the completion rate levels off at around 50%.<\/p>\n

What\u2019s also interesting is that not only does the completion rate go up as the SEQ goes up, but the margin of error also goes down (from about a width of +\/- 6% at an SEQ of 4.7 to +\/- 2% at an SEQ of 6.5), even with the same or smaller sample sizes at those higher levels (n= 40 and 28 respectively). In other words, the task completion experience becomes more stable and predictable at very high ease levels.<\/p>\n

This finding is also consistent with our earlier research that showed that users were four times as likely to complete a task<\/a>\u00a0when they rate it at the maximum satisfaction versus the minimum satisfaction score.<\/p>\n

SEQ & Task Times<\/h2>\n

I repeated this same services in looking at the relationship between SEQ and concurrently collected task times from 270 tasks (samples sizes between 8 and 601). The task times were for all participants, whether or not the task was completed successfully. We also found a strong correlation (r = .-53<\/i><\/em>) between SEQ and the logarithm of task times (as tasks take longer, people\u2019s perception of ease goes down). This means that perception of ease (SEQ scores) can explain about 29% of task times. Figure 3 shows this relationship.<\/p>\n

\"Figure<\/a><\/p>\n

Figure 3: Relationship between SEQ scores (converted into deciles) and task times rates for 270 tasks.<\/p>\n

While this confirms the relationship we observed almost 10 years ago with a more diverse aggregation of questionnaires<\/a>, I was still surprised to see the strong pattern with task times.<\/p>\n

That is, in theory, how long a task takes should be a function of the context. For example, if it takes two minutes to file your taxes online, that\u2019s fast and should elicit higher SEQ scores. If it takes you 1 minute to log in, that\u2019s excruciatingly slow and should elicit lower SEQ scores (even though it\u2019s half the time of the other task). And maybe that is adding some noise to the data, which is why the correlation is lower than task completion (a good topic for future research). Nevertheless, there\u2019s a solid relationship despite different task contexts.<\/p>\n

I suspect the confined nature of usability testing (where the data came from in this analysis) may partially explain this correlation. In a typical usability test, multiple tasks need to be attempted in one sitting (both collected in moderated and unmoderated studies). With most usability tests lasting between 20 and 60 minutes, any given task can\u2019t take too long<\/i><\/em>\u00a0and usually lasts under 5 minutes. In fact, in this data set, participants took on average 3.5 minutes to complete a task and the longest task average time was 10 minutes. Keep this in mind as you compare your times to ours, especially if the typical task duration differs from your testing setups.<\/p>\n

Estimating with the SEQ<\/h2>\n

We can use the strong relationships to predict completion rates and task times from SEQ scores. Figure 4 shows the corresponding completion rate and task time for each SEQ decile and raw score.<\/p>\n

You can use the data in Figure 4 to estimate completion rates and times, even if you collected the data and have only SEQ data. Or if you have collected the data (even from a small sample size), you can see how well it corresponds with our dataset.<\/p>\n

For example, a raw SEQ score of 4.7 will correspond to a completion rate of 58% and task time of 2.8 minutes. A raw SEQ score of 5.9 will correspond to a completion rate of 86% and task time of about 2 minutes. The yellow bars show average SEQ performance, green above average, and red shows below average scores (which start around scores of 5).<\/p>\n

\"Figure<\/a><\/p>\n

Figure 4: Relationship between SEQ scores, completion rates, and task times.<\/p>\n

Summary<\/h2>\n

The simple one-item Single Ease Question (SEQ) is administered immediately after a task to gauge a participant\u2019s attitude toward the just attempted task experience. Across over 250 task experiences, the SEQ tends to predict concurrently collected task completion rates and task time relatively well. Participants\u2019 attitude about the ease of a task predicts about 44% of completion rates and 28% of task times.<\/p>\n

Asking the SEQ after a task alone can provide some idea about the potential task completion rate and task time, even if these metrics aren\u2019t collected on both prototypes and fully working websites and products.<\/p>\n","protected":false},"excerpt":{"rendered":"

Our attitudes both reflect and affect our actions. What we think affects what we do and what we do affects what we think. It\u2019s not a perfect relationship of course. What people say or think doesn\u2019t always directly correspond to actions in easily predictable ways. Understanding and measuring user attitudes early and often can provide […]<\/p>\n","protected":false},"author":2,"featured_media":2836,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"site-sidebar-layout":"default","site-content-layout":"default","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"default","adv-header-id-meta":"","stick-header-meta":"default","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"_price":"field_56e41332a1ae5","_stock":"","_tribe_ticket_header":"","_tribe_default_ticket_provider":"Tribe__Tickets_Plus__Commerce__WooCommerce__Main","_tribe_ticket_capacity":"0","_ticket_start_date":"","_ticket_end_date":"","_tribe_ticket_show_description":"","_tribe_ticket_show_not_going":false,"_tribe_ticket_use_global_stock":"","_tribe_ticket_global_stock_level":"","_global_stock_mode":"","_global_stock_cap":"","_tribe_rsvp_for_event":"","_tribe_ticket_going_count":"","_tribe_ticket_not_going_count":"","_tribe_tickets_list":"[]","_tribe_ticket_has_attendee_info_fields":false,"footnotes":""},"categories":[66,199,191],"tags":[11,102,130],"acf":[],"ticketed":false,"_links":{"self":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/502"}],"collection":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/comments?post=502"}],"version-history":[{"count":2,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/502\/revisions"}],"predecessor-version":[{"id":34080,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/502\/revisions\/34080"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/media\/2836"}],"wp:attachment":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/media?parent=502"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/categories?post=502"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/tags?post=502"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}