UX Measurement Bootcamp 2022

Day 1

Monday

Prerequisites
Before starting the bootcamp on Monday, attendees will receive access to bootcamp-related courses on MeasuringUniversity™, our online training center. The prerequisite videos/quizzes to complete before the bootcamp begins on Monday morning are:

1. UX Methods (60 min): A review of common methods for measuring usability and user experiences.
2. UX Metrics (60 min): A taxonomy of UX metrics including the difference between attitude and action metrics and how to determine the metric you need.
3. Intro to Data Types (25 min): A tour of different types of data (qualitative vs. quantitative, discrete vs. continuous) and an introduction to levels of measurement.

Introductions
The day starts with a brief introduction to UX and introductions of instructors and attendees.

Study Development
Introducing our framework, based on the scientific method, for developing UX studies.

Picking the Right Methods and Metrics
Building on the prerequisite online training, we review key UX methods and metrics with in-class selection exercises.

Top Tasks Analysis
It’s difficult to do everything well all the time. One of the biggest problems in interactions we see in software and websites is that developers and designers think users are doing one thing when in fact users are doing something else. The top task analysis is a simple, yet powerful way of understanding what critical few tasks tend to affect the bulk of the user experience. We reinforce the online training with an in-class top-task exercise.

Day 2

Tuesday

Prerequisites

1. Benchmarking Part 1 (38 min): Introduction to benchmarking essentials including what is it and why use it, when to use it, what type to use, and defining and getting access to users.
2. Benchmarking Part 2 (41 min): Additional benchmarking essentials including task definition, metrics selection, measurement precision, and study length.
3. Confidence Interval Demonstration (12 min): Physical demonstration of the accuracy of confidence intervals.
4. Confidence Intervals: Intro and Rating Scales (48 min): Introduction to confidence intervals including their three ingredients, a quick stats review, and application to rating scale data.
5. Confidence Intervals: Binary and Time Data (28 min): Confidence intervals for binary data (e.g., completion rates) and task time data.

Team Assignment
The day starts with the formation of research teams and some time for team members to start making decisions about the teams’ research projects.

Benchmark Review & Quiz Results
Next is a review of the online benchmarking content and quizzes.

Confidence Intervals Review and Exercises
Sampling users of any size involves sampling error. Confidence intervals provide the level of precision around estimates of findability, completion rates, likelihood to recommend, and any metric you collect from a sample of users. We review the prerequisite online content and go through additional confidence interval exercises.

Day 3

Wednesday

Prerequisites

1. Hypothesis Testing Demonstration (25 min): Physical demonstration of hypothesis testing.
2. Hypothesis Testing Intro (36 min): An introduction to the Null Hypothesis Significance Testing (NHST) framework, including discussion of significance levels and difference between statistical and practical significance.
3. Hypothesis Testing – Conducting Tests (12 min): Considerations when conducting statistical tests (data types and experimental designs).

Hypothesis Testing Review & Exercises
With everyone wide awake on caffeine, we build on the online prerequisite courses, reviewing the content and going through additional in-class exercises.

Day 4

Thursday

Prerequisites

1. Power and Sample Size Estimation – Intro and Problem Discovery (39 min): Introduction to three ways to determine the right sample size and discussion of sample sizes for problem discovery, including whether five users are enough and if more severe problems get discovered first.
2. Power and Sample Estimation – Precision/Estimation (33 min): With a focus on sample sizes for confidence intervals, discusses precision requirements and different methods for different data types (binary vs. continuous).
3. Power and Sample Size Estimation – Comparison (37 min): Discussion of how effect sizes and power affect sample size estimation for statistical hypothesis tests.

Observing & Coding Usability Problems
At the heart of usability testing, whether it be the do-it-yourself kind or the more industrial strength kind, is documenting, diagnosing and reducing usability problems. We will watch videos of users attempting to find the cost of dog grooming, documenting and discussing the usability problems.

Power and Sample Size Review & Exercises
How many users do you test? It’s one of the most common and confusing questions in applied user research. We review the prerequisite online content and go through additional sample size estimation exercises.

Project: Analyze Data & Prepare Presentations
Breaking again into project teams, you’ll get help in preparing your analyses, conducting statistical calculations, and presenting your findings.

Day 5

Friday

Project: Finish Analysis and Report Preparation
Breaking again into project teams, you’ll have 2.5 hours to complete your analyses, statistical calculations, and presentations.

Final Exam
Take and pass a final exam, 50 items based on quizzes taken online during the bootcamp (80% required to pass, no limit on number of times to attempt the test).

Presentations & Ending
We finish with team representatives giving their presentations, then wrapping up with a final Q&A.