{"id":290,"date":"2015-12-01T21:30:00","date_gmt":"2015-12-01T21:30:00","guid":{"rendered":"http:\/\/measuringu.com\/advanced-stats\/"},"modified":"2021-08-12T09:04:56","modified_gmt":"2021-08-12T15:04:56","slug":"advanced-stats","status":"publish","type":"post","link":"https:\/\/measuringu.com\/advanced-stats\/","title":{"rendered":"5 Advanced Stats Techniques & When to Use Them"},"content":{"rendered":"

To answer most user-research questions fundamental statistical techniques like confidence intervals, t-tests, and 2 proportion tests<\/a> will do the trick.<\/p>\n

But to answer some questions most effectively you need to use more advanced techniques.<\/p>\n

Each of these techniques requires specialized software (e.g., SPSS, Minitab, R) and training on how to set up and interpret the results. But even if you aren’t ready to execute these techniques yourself, you can still learn what they are, when to use them, and some gotchas to look out for.<\/p>\n

1. Regression Analysis<\/h2>\n

<\/h2>\n

<\/h2>\n

When you want to understand what combination of variables best predicts a continuous outcome variable like customer satisfaction, likelihood to recommend, time on task, or attitudes toward usability, use regression analysis. This technique also goes by key-drivers analysis<\/a> because you’re able to determine which independent variables have the biggest impact on your dependent (outcome) variable.<\/p>\n

You can use both continuous and discrete variables (dummy coded<\/a>) as independent (predictor) variables. For example, we found that usability for education software had the biggest relative impact on likelihood to recommend<\/a> (NPS) using a multiple regression analysis. Regression analysis is a workhorse of statistical techniques and forms the basis of other methods for identifying optimal combinations of variables such as conjoint analysis<\/a>.<\/p>\n

Gotchas<\/i>: Be sure a linear relationship exists between your variables by graphing them. You also don’t want the independent variables to correlate highly with each other (usually r > .8)\u2014a condition called multicollinearity, which renders the regression equation unreliable.<\/p>\n

2. ANOVA<\/h2>\n

<\/h2>\n

An Analysis of Variance (ANOVA) tells you whether the means from more than two groups have a significant difference, such as the SUPR-Q scores<\/a> across five websites. The more familiar t-test<\/a> is a special case of the ANOVA when there are only two groups to compare.<\/p>\n

What makes an ANOVA powerful though is it allows you to look at multiple variables at a time AND understand what combination results in the largest difference. This is called the interaction effect.<\/p>\n

For example, you may be interested to know which form design users can complete more quickly: a form on one page, or the form split across two pages. That’s one variable (the form design) with two levels (form A and form B). This research question can be answered using a t-test. But if you want to understand how another variable, say the device type (mobile versus desktop), affects form completion time, you would use an ANOVA. It allows you to ensure that the form design that’s fastest for desktop completion isn’t a detriment to form completion on the mobile screen.<\/p>\n

If you think this sounds similar to regression analysis, you’d be right. The ANOVA is just a regression analysis with discrete independent variables. Both techniques are part of a larger technique called General Linear Modeling (GLM), which we explain in detail in Chapter 10 of our forthcoming 2nd Edition of Quantifying the User Experience<\/a>.<\/p>\n

Gotchas<\/i>: When you compare many groups, you increase the chance of finding a difference from chance alone (called alpha inflation). You should plan to use a multiple comparison technique, such as a Bonferroni adjustment<\/a> or my preference, the Benjamini\u2013Hochberg procedure<\/a>, to differentiate the signal from the noise<\/a>.<\/p>\n

3. Factor Analysis<\/h2>\n

<\/h2>\n

Sometimes the variables you’re most interested in are those you can’t directly observe or measure easily. For example, there isn’t a direct measure of usability (no usability thermometer); instead, you have to rely on the outcomes of good and bad experiences to quantify usability. Factor analysis is a technique that takes many observed correlated variables and reduces them to a few latent (hidden) variables called factors.<\/p>\n

For example, we used factor analysis[pdf]<\/a> to identify usability as a single factor from the multiple correlated variables of task-time, completion rates, errors, and perceived task difficulty.<\/p>\n

Factor analysis is also a staple of questionnaire development where you determine which items group together to best embody a construct. I used factor analysis extensively to build the SUPR-Q where I uncovered<\/a> the latent variables of usability, loyalty, trust, and appearance from dozens of potential items.<\/p>\n

Gotchas<\/i>: There are a number of ways to determine how many factors are in your data and different researchers examining the same data may uncover a different number of factors. You generally need a large sample size to conduct a factor analysis and like many of the techniques, the relationship between variables should be linear.<\/p>\n

4. Cluster Analysis<\/h2>\n

<\/h2>\n

When you want to know what items to group together you use cluster analysis. Like factor analysis, these groupings aren’t directly measurable but instead inferred from the data (another type of latent variable). Cluster analysis is the approach used in card sorting<\/a> when you want to know how closely products, content, or functions relate from the users’ perspective.<\/p>\n

Cluster analysis is also the technique we use to segment customers<\/a> and build personas<\/a>. When we conduct segmentation studies for clients we’ll look at a number of variables demographic (income, age) and psychographic (e.g. likelihood to use a new service) to see what defines a segment.<\/p>\n

Gotchas<\/i>: Like factor analysis, there’s some subjectivity involved in determining the number of clusters. You can easily justify more granular or more general clusters based on how closely you want the items to relate.<\/p>\n

5. Logistic Regression<\/h2>\n

<\/h2>\n

It’s often the case that dependent variables are discrete and not continuous. For example, you may be primarily interested in purchase rates (purchase versus not purchase) or conversion rates (recommend, didn’t recommend); both are discrete binary<\/a>.<\/p>\n

In such cases, regular regression analysis won’t work. Instead, you’d use a different, but related technique, called logistic regression analysis, which converts the data using a logit transformation<\/a>. In logistic regression, you still want to know what combination of independent variables best predicts the outcome, as in regular regression–the only difference is the outcome variable is discrete (usually binary).<\/p>\n

For example, we’ve used logistic regression to understand how attitudes toward a service experience (favorable or unfavorable) and customer tenure (new versus existing) affects likelihood to repurchase.<\/p>\n

Gotchas<\/i>: While logistic regression doesn’t have the same linearity assumptions as regular regression, you still need to look for highly correlated independent variables (multicollinearity), need a large sample size, and interpret log-odd ratios<\/a>, which is more challenging.<\/p>\n

Summary<\/h2>\n

Simple statistical techniques can answer many questions posed to UX researchers. There are times however when you need more advanced techniques to best answer the question.<\/p>\n

Here’s how to differentiate between the five techniques based on their focuses, example research questions, and gotchas to watch out for.<\/p>\n\n\n\n\n\n\n\n\n
\u00a0Technique
\n<\/b><\/td>\n		Focus
\n<\/b><\/td>\n		Research Example
\n<\/b><\/td>\n		Gotcha
\n<\/b><\/td>\n<\/tr>\n
Regression Analysis<\/td>\nBest combination of variables to predict continuous outcome variables<\/td>\nWhat features best predict likelihood to recommend?<\/td>\n\u00a0Correlated independent variables and linearity<\/td>\n<\/tr>\n
ANOVA<\/td>\nComparison of multiple variables plus interactions<\/td>\nHow does device type and form type affect task completion time?<\/td>\nAlpha inflation<\/td>\n<\/tr>\n
Factor Analysis<\/td>\nIdentify latent variables that form groups (factors)<\/td>\nWhat combination of items describes the constructs of appearance and trust?<\/td>\nSubjective factors & linearity<\/td>\n<\/tr>\n
Cluster Analysis<\/td>\n\u00a0Uncover how items form latent groups<\/td>\nDo users group two products together?<\/td>\nSubjective clusters<\/td>\n<\/tr>\n
Logistic Regression<\/td>\nBest combination of variables to predict discrete outcome variables<\/td>\nHow much does a service experience and tenure affect purchase rates?<\/td>\nCorrelated independent variables and linearity<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

To answer most user-research questions fundamental statistical techniques like confidence intervals, t-tests, and 2 proportion tests will do the trick. But to answer some questions most effectively you need to use more advanced techniques. Each of these techniques requires specialized software (e.g., SPSS, Minitab, R) and training on how to set up and interpret the […]<\/p>\n","protected":false},"author":2,"featured_media":1853,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"site-sidebar-layout":"default","site-content-layout":"default","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"default","adv-header-id-meta":"","stick-header-meta":"default","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"_price":"field_56e41332a1ae5","_stock":"","_tribe_ticket_header":"","_tribe_default_ticket_provider":"Tribe__Tickets_Plus__Commerce__WooCommerce__Main","_tribe_ticket_capacity":"0","_ticket_start_date":"","_ticket_end_date":"","_tribe_ticket_show_description":"","_tribe_ticket_show_not_going":false,"_tribe_ticket_use_global_stock":"","_tribe_ticket_global_stock_level":"","_global_stock_mode":"","_global_stock_cap":"","_tribe_rsvp_for_event":"","_tribe_ticket_going_count":"","_tribe_ticket_not_going_count":"","_tribe_tickets_list":"[]","_tribe_ticket_has_attendee_info_fields":false,"footnotes":""},"categories":[21],"tags":[22],"acf":[],"ticketed":false,"_links":{"self":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/290"}],"collection":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/comments?post=290"}],"version-history":[{"count":0,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/290\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/media\/1853"}],"wp:attachment":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/media?parent=290"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/categories?post=290"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/tags?post=290"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}