Few things are more revealing than watching just a few users attempt tasks on a website or software.
But while you’re watching users you should ask them some key questions to help put the observations into perspective.
Here are eight recommendations for helping quantify both attitudes and put the insightful observations into context.
Software Usability– Use the System Usability Scale (SUS): This venerable 10 item instrument is the most used questionnaire for measuring perceptions of usability. It’s short, reliable and free (don’t we wish everything was like this?). It’s technology agnostic, which means you can use it on software, mobile apps, hardware, Voice Response Systems and websites.
Websites Quality– Use the SUPR-Q. The 13 item Standardized Universal Percentile Rank Questionnaire (SUPR-Q) provides four essential measures of website quality: usability, loyalty, credibility and appearance. In fact, just four items can accurately predict the System Usability Scale. Raw scores can be associated with a database of 200 other websites.
Task Difficulty– Use the SEQ: After users attempt a task in a usability test, ask them how difficulty or easy they thought it was on this single 7 point scale. We’ve found that this single question performed about as well as more complicated and lengthy questionnaires. User time is limited so the shorter the better. In this case the SEQ is both short and good.
Task Confidence: Users might find ways of getting through task scenarios but just aren’t quite sure they did things correctly. In such cases, getting a measure of confidence can help tell if the task completion rate is masking some underlying issues. We also use confidence along with completion rates to generate Disaster Rates: when users are extremely confident and yet fail the task.
Customer Loyalty–Use the Net Promoter Score: Word of mouth is the engine of viral growth and the 11 point Net Promoter Score has become the defacto measure of customer and user loyalty. If users are saying positive things about your software, website or other product to a friend then growth prospects are good. Conversely, if others have had bad experiences and are telling others then bad news is spreading fast. In both cases you’ll want to have a good measure of customer loyalty to act upon.
Product Usefulness– Use the Technology Acceptance Model: You can have the most usable and elegant product but if no one feels a need to use it, not much else matters. The questionnaire based on the TAM is 20 items, 10 provide a measure of usefulness which can be used as a measure over time or when comparing products with different features. The other 10 items are similar to the System Usability Scale, so when measuring usability use the SUS.
Why?: The items above use closed-ended rating scales. This is helpful for putting problems and actions into context but the numbers alone don’t provide much insight into what the problem is. When possible, ask users to provide in their own words, their thoughts about the product. We find this especially helpful when we get lower ratings on task-difficulty and Net Promoter Scores
What one thing should be fixed?: Once you’re asking what the problem is, use the opportunity to ask users what they would fix in the software or website. While we can’t expect users to be designers or do the job for us, it’s good to capture those problems that are at the top-of-mind of users. You can’t fix everything, but usually there are 1 or 2 things that are not too difficult to fix and the improvement would benefit a lot of users.
