Benchmarking the User Experience

Purchase Hard Copy on Amazon  | Download Electronic Copy

This is a practical book about how to measure the user experience of websites, software, mobile apps, products, or just about anything people use. This book is for UX researchers, designers, product owners, or anyone who has a vested interest in improving the experience of websites and products.

In this book, I use practical examples gleaned through my years of experience to illustrate what benchmarking is and how you can use this technique to measure users’ experiences. I use examples of studies that the team at MeasuringU, a quantitative research firm that I founded, have worked on to give you constructive information that will help you formulate and/or understand your benchmarking studies, which ultimately helps you understand your audience and customers.

If you are new to benchmarking, you’ll find the introductory chapters will provide a thorough background and overview of the primary impetus underlying common benchmarking concepts, methods, and metrics. If you are familiar with in-person usability studies, many of the chapters compare and contrast moderated versus unmoderated benchmarks and should have many familiar concepts and examples.

Chapters 2, 3, and 4 provide guidance on the common “how much,” “how long,” and “how do we” questions and practical advice on making the most of your benchmark. If you are familiar with benchmarking and looking for guidance, you’ll find Chapter 5 useful for providing a thorough overview of task- and study-level metrics. Chapter 6 explains how to determine and justify the sample sizes you choose.

The latter chapters on analysis provide several step-by-step guides on gathering raw benchmark data, graphing benchmark data, and using some best practices for statistical analysis and reporting. The free downloadable MeasuringU Benchmark Calculator (see Chapter 10) can help even the most seasoned researcher more efficiently analyze their benchmark data. Appendix A contains a checklist for conducting a benchmark study, and Appendix B contains best practices for competitive benchmarks, referencing relevant chapters for more details.

Table of Contents

Chapter 1: An Introduction to UX Benchmarking
Chapter 2: Planning & Defining Your Study
Chapter 3: Working Through Study Details
Chapter 4: Planning & Logistics
Chapter 5: Benchmark Metrics
Chapter 6: Sample Size Planning
Chapter 7: Writing the Study Script
Chapter 8: Preparing For Data Collection
Chapter 9: Participant Recruitment & Data Collection
Chapter 10: Analyzing & Displaying Your Data
Chapter 11: Advanced Analysis
Chapter 12: Reporting Your Results

Appendix A: A Checklist For Planning a UX Benchmark Study
Appendix B: 10 Best Practices for Competitive UX Benchmarking
Appendix C: 5 Common Mistakes in UX Benchmark Studies
Appendix D: Example Project Booking Form for Hotel Comparison Study
Appendix E: Example Study Script for an Unmoderated Study