{"id":118,"date":"2012-05-30T23:50:00","date_gmt":"2012-05-30T23:50:00","guid":{"rendered":"http:\/\/measuringu.com\/sum\/"},"modified":"2022-03-21T18:24:58","modified_gmt":"2022-03-22T00:24:58","slug":"sum","status":"publish","type":"post","link":"https:\/\/measuringu.com\/sum\/","title":{"rendered":"10 Things to Know about the Single Usability Metric (SUM)"},"content":{"rendered":"

<\/a>There is no usability thermometer to tell you how easy to use a website or software application is.<\/p>\n

Instead we rely on the outcomes of good and bad experiences which provide evidence for the construct of usability<\/a>.<\/p>\n

Combining multiple usability metrics into a single usability metric (SUM) is something we proposed seven years ago<\/a>[PDF]<\/span> and we wrote about in Chapter 9 of Quantifying the User Experience<\/a>.<\/p>\n

Here are 10 things to know about single measures of usability.<\/p>\n

    \n
  1. Usability is the intersection of effectiveness, efficiency and satisfaction (ISO 9241 pt 11<\/a>). One of the best measures of usability is a combination of metrics that describes each of these aspects.<\/li>\n
  2. The most common usability metrics<\/a> are completion rates and errors (effectiveness), task-times (efficiency) and task-level satisfaction (satisfaction). These metrics tend to have a moderate correlation[PDF]<\/span><\/a> with each other of r<\/span> = .3 to .5. The correlation is strong enough to suggest an overlap (e.g., users that commit more errors tend to take longer) but the correlation isn’t strong enough that one metric can substitute for the other.<\/li>\n
  3. By averaging together a standardized version of completion rates, task-times, task-level satisfaction and errors you generate a Single Usability Metric (SUM) which summarizes the majority of information in all four measures. By averaging you weight each metric equally. Despite many discussions for determining which metric “counts” more, our analysis found that a simple average is least subjective and reflects the data best (from a principal components analysis[PDF]<\/span><\/a>). Keep in mind that if you weight one metric a lot then you must lessen the weight of another, often to a point where an additional metric does little.<\/li>\n
  4. You can have 3 metric or 4 metric versions of SUM: Errors<\/a> are usually the most time consuming and difficult to collect metric (especially in unmoderated<\/a> testing) so completion rates, task-times and task-satisfaction provide the minimum description of effectiveness, efficiency and satisfaction for a single usability metric.<\/li>\n
  5. A single usability metric doesn’t replace the individual metrics; it simply summarizes them in a more condensed way like an abstract to a long paper or like the mean summarizes a large set of numbers. With any summarization comes data loss, but the gain in interpretability usually far outweighs the loss\u2014especially considering you don’t “lose” anything as you can always dive into the individual metrics (like you can read the details of a paper).<\/li>\n
  6. There are a number of reasonable ways to combine usability metrics. One of the best ways we’ve found is to convert everything into a percentage. For discrete metrics (completion rates and errors) this is done by generating a proportion and for continuous metrics (time and satisfaction) we generate a normalized “z-score<\/a>” and convert it to percentage then average the metrics together.<\/li>\n
  7. To convert discrete data so they are amenable to combining:\n