Topics
Topics
Is There A Difference In Usability Data From Remote Unmoderated Tests & Lab-based Tests?
Why isn’t usability testing done more? And when it is done why is the sample size small? One major reason is the cost. It takes a lot of money and time to bring users into a lab and conduct a usability test. Even if users don’t get compensated for their time, it still takes a
How much is a PhD Worth?
Does a PhD pay off financially? I recently helped conduct the statistical analysis of the UPA 2009 salary survey[pdf], and used this opportunity to look into the data to see if I could calculate how much a PhD affects salaries in this profession. The dataset contains salary information for a wide range of jobs in
Do Users Fail A Task And Still Rate It As Easy?
Have you ever watched a user perform horribly during a usability test only to watch in amazement as they rate a task as very easy to use? I have, and as long as I’ve been conducting usability tests, I’ve heard of this contradictory behavior from other researchers. Such occurrences have led many to discount the
Margins of Error in Usability Tests
How many users will complete the task and how long will it take them? If you need to benchmark an interface, then a summative usability test is one way to answer these questions. Summative tests are the gold-standard for usability measurement. But just how precise are the metrics? Just as a presidential poll uses a
Deriving a Problem Discovery Sample Size
Nielsen derives his “five users is enough” formula from a paper he and Tom Landauer published in 1993. Before Nielsen and Landauer James Lewis of IBM proposed a very similar problem detection formula in 1982 based on the binomial probability formula.[4] Lewis stated that: The binomial probability theorem can be used to determine the probability
Task Times in Formative Usability Tests
It is common to think of time-on-task data gathered only during summative evaluations because, during a formative evaluation, the focus is on finding and fixing problems, or at least finding the problems and delivering a report. For a variety of reasons, time-on-task measures often get left out of the mix. In this article, I show
SUM: Single Usability Metric
SUM is a standardized, summated and single usability metric. It was developed to represent the majority of variation in four common usability metrics used in summative usability tests: task completion rates, task time, satisfaction and error counts. The theoretical foundations of SUM are based on a paper presented at CHI 2005 entitled “A Method to
Restoring Confidence in Usability Results
Adding confidence intervals to completion rates in usability tests will temper both excessive skepticism and overstated usability findings. Confidence intervals make testing more efficient by quickly revealing unusable tasks with very small samples. Examples are detailed and downloadable calculators are available. Are you Lacking Confidence? You just finished a usability test. You had
Calculating Sample Size for Task Completion (Discrete-Binary Method)
One of the biggest and usually first concerns levied against any statistical measures of usability is that the number of users required to obtain “statistically significant” data is prohibitive. People reason that one cannot with any reasonable level of confidence employ quantitative methods to determining product usability. The reasoning continues something like this: “I have
Calculating Sample Size for Task Times (Continuous Method)
We already saw how a manageable sample of users can provide meaningful data for discrete-binary data like task completion. With continuous data like task times, the sample size can be even smaller. The continuous calculation is a bit more complicated and involves somewhat of a Catch-22. Most want to determine the sample size ahead of
Calculating A Sigma Level From Task Success
Often the most reported measures of usability is task success. Success rates can be converted into a sigma value by using the discrete-binary defect calculation: Proportion Unsuccessful= Defects/Opportunities Where opportunities are the total number of tasks and defects are the total number of unsuccessful tasks. This calculation provides a proportion that is equivalent to a
Current Usability Solutions Are Unpredictable
It’s a big step when User Centered Design methods are employed in a company to improve the usability of a product. It shouldn’t be the last step and often times it is. Many popular usability testing techniques are the right method to gather user data, however, their results alone will only scratch the surface of