A user-interface problem is anything in the code or design that inhibits a user from completing an intended action. Usability testing is best done in the formative stages of development to find these UI problems and generate quantifiably better designs. Usability testing also has a summative role at the end of development as validation when bugs and functional gaps have been addressed. Most UI problems found during summative usability tests are issues with the interaction of properly functioning code, but a few are bugs.
Even on well-used websites and released applications bugs can find their way into usability tests. For example, the Hotel Pennsylvania website was tested by 17 independent usability teams using a combination of expert reviews and user-testing as part of the CUE-4 evaluation. The teams found a total of 340 unique usability problems of which 13 (3.8%) were classified as bugs.
The nice thing about software bugs is that there’s rarely disagreement about whether they are in fact a defect or a false alarm. Contrast that with some usability issues, which can often lead to heated debates about whether they are legitimate problems or “features.” The problem Hertz had with their calendar was a bug in the software that wasn’t picked up in their coding or QA (Quality Assurance) process. I used it as an example of an indisputable UI problem that affected some percent of users. This example led to some interesting discussions about the roles of software testing and usability testing.
Users are the last line of defense for finding bugs. But testing a few users or even many users is not a substitute for good coding and good testing. You’ll never be able to test enough users to find the problems a tester will find in a QA process. However, even the most skilled tester or the best written script cannot predict the unintended actions only some users will have with well written code. QA testers are not adequate substitutes for real users and usability tests are not adequate substitutes for good QA.
