It’s a big step when User Centered Design methods are employed in a company to improve the usability of a product. It shouldn’t be the last step and often times it is. Many popular usability testing techniques are the right method to gather user data, however, their results alone will only scratch the surface of the true state of usability. Often their results can be misleading.

Is the problem you found with that one user when testing the 10 users using Think-Aloud Protocol a problem 1 out of 10 times or 1 out of 100 times? More important, what is to be done with that information? Is a redesign of the screen warranted? Does the scenario match the user’s real goals? With a software product that has thousands of users executing tasks thousands of times over the life of a product, decisions can have drastic impacts for better or worse.

The data may be partially objective, but the solution typically is not. All too often we well intentioned usability practitioners use our intuition to decide how to make the product more usable. Even when that intuition and experience is well informed, without an objective measure it’s hard to tell either way. Currently we’re throwing darts in the dark. Sometimes we hit our target but it’s usually by chance and even if we do we can’t say with much confidence because we can’t even see the dart board.