{"id":122,"date":"2012-06-26T20:30:00","date_gmt":"2012-06-26T20:30:00","guid":{"rendered":"http:\/\/measuringu.com\/what-test\/"},"modified":"2022-03-21T18:23:28","modified_gmt":"2022-03-22T00:23:28","slug":"what-test","status":"publish","type":"post","link":"https:\/\/measuringu.com\/what-test\/","title":{"rendered":"What Statistical Test do I Use?"},"content":{"rendered":"<p><a href=\"http:\/\/www.flickr.com\/photos\/ontherocks76\/2530994056\/sizes\/l\/in\/photostream\/\"><img decoding=\"async\" style=\"float: left; padding: 10px; margin-right: 20px; width: 347px; height: 288px;\" title=\"What statistical test do you use? Follow these sign posts\" src=\"https:\/\/measuringu.com\/images\/directionpost.jpg\" border=\"0\" \/><\/a>What do you think the most common question in statistics is?<\/p>\n<p>Several times a year I teach a statistics course for UX professionals and get asked this question a lot.<\/p>\n<p>We&#8217;re offering the class this fall at the <a href=\"http:\/\/www.leanuxdenver.com\/\">LeanUX Denver conference<\/a> and a portion of it is <a href=\"https:\/\/measuringu.com\/products\/uxStats1\">available for download<\/a>.<\/p>\n<p>Some attendees have had statistics classes and for others it&#8217;s their first one. Regardless of the background, almost everyone who uses statistics wants to know: <span style=\"font-weight: bold;\">What statistical procedure do I use? <\/span><\/p>\n<p>It&#8217;s hard enough to grasp many of the concepts in statistics. Most people in UX aren&#8217;t math majors and never intended to use statistics as part of their job.<\/p>\n<p>For this reason we have a decision tree to help you know when to use which statistical procedure in both the <a href=\"https:\/\/measuringu.com\/products\/expandedStats\">Excel calculator<\/a> and in Chapter 2 of our book <a href=\"http:\/\/www.amazon.com\/gp\/product\/0123849683\/ref=as_li_ss_tl?ie=UTF8&amp;tag=meausallc-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=0123849683\">Quantifying the User Experience<\/a>.<\/p>\n<p>Getting to know the decision map is one of the most popular parts of the course because you can click right to the appropriate calculator after answering a couple questions, paste your data and get your answer.<\/p>\n<p>I&#8217;ve included a similar clickable version below with links to several of our free online calculators. To use the decision map you just need to know a few things.<\/p>\n<h2>What type of Data do you have?<\/h2>\n<p>For this decision map, the primary thing you need to know is if your data is binary or not.<\/p>\n<ul>\n<li><span style=\"font-weight: bold;\">Binary <\/span>(pass\/fail, yes\/no, purchase\/didn&#8217;t purchase) and coded as 1 or 0. These include things like completion rates and conversion rates.<\/li>\n<li><span style=\"font-weight: bold;\">Continuous<\/span>: If your data isn&#8217;t coded as 1&#8217;s or 0&#8217;s then we can usually treat it as what&#8217;s called <a href=\"http:\/\/www.usablestats.com\/lessons\/datatypes\">continuous or metric data<\/a>.<\/li>\n<\/ul>\n<p>Once you&#8217;ve established whether your data is binary or continuous, you start with the appropriate decision map below. The rest is just answering a couple questions about what you&#8217;re trying to do. Four examples follow the decision maps to give you some practice.<\/p>\n<p>The blue boxes at the termination points on the two decision maps below link to our free online calculator and if I don&#8217;t have one up yet (more coming soon!) it will link to the <a href=\"https:\/\/measuringu.com\/products\/expandedStats\">Excel calculator<\/a>.<\/p>\n<p>You can also download a <a href=\"https:\/\/measuringu.com\/wp-content\/uploads\/2012\/06\/stats-decisiontree.pdf\">free high-resolution printable version<\/a><span style=\"color: #cc0000;\"> [3.5MB pdf]<\/span> of both maps. Pin it to your wall for help or throw darts at it if statistics just frustrates you too much.<\/p>\n<h2>Continuous Data Decision Map<\/h2>\n<p><img decoding=\"async\" src=\"https:\/\/measuringu.com\/images\/metric-decision-tree2.jpg\" usemap=\"#Map\" border=\"0\" \/><\/p>\n<map id=\"Map\" name=\"Map\">\n<area title=\"Available in the Excel Calculator\" coords=\"251,283,320,352\" shape=\"rect\" href=\"https:\/\/measuringu.com\/products\/expandedStats\" \/>\n<area title=\"Links to the Online Confidence Interval Calculator\" coords=\"444,436,503,505\" shape=\"rect\" href=\"https:\/\/measuringu.com\/ci-calc.php\" \/>\n<area title=\"Links to the Graph and Confidence Interval Calculator for Task Times\" coords=\"350,437,419,505\" shape=\"rect\" href=\"https:\/\/measuringu.com\/time_intervals.php\" \/>\n<area title=\"Links to the Online 1 Sample t Calculator\" coords=\"348,278,415,352\" shape=\"rect\" href=\"http:\/\/www.usablestats.com\/calcs\/1samplet\" \/>\n<area title=\"Links to the Online 2 Sample t Calculator\" coords=\"78,420,138,482\" shape=\"rect\" href=\"http:\/\/www.usablestats.com\/calcs\/2samplet\" \/>\n<area title=\"Available in the Excel Calculator\" coords=\"-1,418,55,478\" shape=\"rect\" href=\"https:\/\/measuringu.com\/products\/expandedStats\" \/>\n<area title=\"Available in the Excel Calculator\" coords=\"67,269,149,340\" shape=\"rect\" href=\"https:\/\/measuringu.com\/products\/expandedStats\" \/>\n<area title=\"Available in the Excel Calculator\" coords=\"183,266,247,330\" shape=\"rect\" href=\"https:\/\/measuringu.com\/products\/expandedStats\" \/> <\/map>\n<p>&nbsp;<\/p>\n<h2>Binary Data Decision Map<\/h2>\n<p><img decoding=\"async\" src=\"https:\/\/measuringu.com\/images\/binary-decision-tree.jpg\" usemap=\"#Map2\" border=\"0\" \/><\/p>\n<map id=\"Map2\" name=\"Map2\">\n<area title=\"Large Sample Usually means having expected cell-counts above 5, although this convention is arbitray.\" coords=\"204,405,348,454\" shape=\"rect\" \/>\n<area title=\"Adjusted Wald Confidence Interval Calculator Link\" coords=\"290,200,367,289\" shape=\"rect\" href=\"https:\/\/measuringu.com\/onep.php\" \/>\n<area title=\"Adjusted Wald Confidence Interval Calculator Link\" coords=\"421,192,498,281\" shape=\"rect\" href=\"https:\/\/measuringu.com\/wald.htm\" \/>\n<area title=\"N-1 Two Proportion Test online Calculator\" coords=\"4,499,81,588\" shape=\"rect\" href=\"https:\/\/measuringu.com\/ab-calc.php\" \/>\n<area title=\"We recommend comparing each pair using the N-1 Two Proportion Test online Calculator\" coords=\"304,488,381,577\" shape=\"rect\" href=\"https:\/\/measuringu.com\/ab-calc.php\" \/>\n<area title=\"Available in the Excel Calculator\" coords=\"171,487,241,565\" shape=\"rect\" href=\"https:\/\/measuringu.com\/products\/expandedStats\" \/>\n<area title=\"We recommend comparing each pair using the McNemher Exact Test. It is available in the Excel Calculator\" coords=\"82,303,159,399\" shape=\"rect\" href=\"https:\/\/measuringu.com\/products\/expandedStats\" \/>\n<area title=\"Available in the Excel Calculator\" coords=\"220,303,290,381\" shape=\"rect\" href=\"https:\/\/measuringu.com\/products\/expandedStats\" \/><\/map>\n<p>&nbsp;<\/p>\n<h3>Example 1<\/h3>\n<p>Let&#8217;s say you had a rating scale question in a survey that went from strongly disagree to strongly agree and was coded from 1 to 5 for each level of agreement. The average score was a 3.9 (sd = 1.2) from 36 people. What test would you use to find out how much that sample mean would fluctuate?<\/p>\n<ol>\n<li>It&#8217;s not a binary measure so we&#8217;d use the continuous decision map.<\/li>\n<li>We aren&#8217;t comparing data, we only want to know how precise our estimate is so we pick &#8220;N&#8221; in the first branch.<\/li>\n<li>We don&#8217;t have a benchmark we&#8217;re testing against (such as testing whether the average exceeds 3.5) so we again pick &#8220;N&#8221;.<\/li>\n<li>The data isn&#8217;t task time so we pick &#8220;N&#8221; and end up at the t confidence interval link. Entering in the mean, standard deviation and sample size as summary data gets us a 95% confidence interval from 3.494 to 4.306.<\/li>\n<\/ol>\n<p>&nbsp;<\/p>\n<h3>Example 2<\/h3>\n<h3><\/h3>\n<p>Let&#8217;s say you had two different groups of users who attempted to locate the nearest rental car location. In one group 12 out of 12 users found the correct location on Budget.com. In another group, 17 out of 24 found the correct location on Enterprise.com. Is there a statistically significant difference between rental car websites?<\/p>\n<ol>\n<li>Completion rates are a binary measure (pass\/fail) so we&#8217;d use the binary decision map.<\/li>\n<li>We ARE comparing groups\u2014we want to know if users on different websites will have different completion rates when finding a location so we pick the &#8220;Y&#8221; in the first branch.<\/li>\n<li>We have different users in each group (called Between Subjects) so we pick &#8220;Y&#8221; again.<\/li>\n<li>We have only 2 groups so we pick &#8220;N&#8221; at the 3 or more groups box.<\/li>\n<li>We end at the N-1 Two Proportion Test with a link to the A\/B Test calculator link. Entering 12 out of 12 and 17 out of 24 gets us a p-value of .03985, which means the difference is statistically significant.<\/li>\n<\/ol>\n<p>&nbsp;<\/p>\n<h3>Example 3<\/h3>\n<h3><\/h3>\n<p>Let&#8217;s say you watched as 20 out of 25 people found a Brother Sewing Machine in a <a href=\"https:\/\/measuringu.com\/blog\/navigation-metrics.php\">tree test<\/a> from <a href=\"http:\/\/www.loop11.com\/?ref=mu\">Optimal Workshop<\/a>. Is there evidence that at least 75% of all users would find the Sewing Machine?<\/p>\n<ol>\n<li>Completion rates are a binary measure (pass\/fail) so we&#8217;d use the binary decision map.<\/li>\n<li>We are not comparing groups so we pick the &#8220;N&#8221; in the first branch.<\/li>\n<li>We ARE testing against a benchmark of 75% so we pick &#8220;Y&#8221; again.<\/li>\n<li>We end at the 1 Sample Binomial Test with a link to the One Proportion Calculator. Entering 20 out of 25, &#8220;Is Greater Than&#8221; and a Test Proportion of .75 tells us there&#8217;s about a 70% chance at least 75% of all users would be able to find the Sewing Maching&#8211;not terribly compelling evidence.<\/li>\n<\/ol>\n<p>&nbsp;<\/p>\n<h3>Example 4<\/h3>\n<p>Let&#8217;s say you administered the <a href=\"https:\/\/measuringu.com\/sus.php\">System Usability Scale (SUS)<\/a> after users spent time using an accounting application. You tested 15 users on an old version and 17 on a new version and wanted to know if there was evidence that users thought the new version was easier to user. The SUS means and standard deviations were 68 (sd=14) and 76 (sd =12) on the respective versions.<\/p>\n<ol>\n<li>SUS scores are not binary so we&#8217;d use the continuous decision map.<\/li>\n<li>We ARE comparing groups so we pick the &#8220;Y&#8221; in the first branch.<\/li>\n<li>We have different users in each group (called Between Subjects) so we pick &#8220;Y&#8221; again.<\/li>\n<li>We have only 2 groups so we pick &#8220;N&#8221; at the 3 or more groups box.<\/li>\n<li>We end at the 2 Sample t test with a link to the online calculator. We enter the summarized values and get a p-value of .0962 which means there&#8217;s good, although not overwhelming evidence, that the difference is statistically significant.<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"<p>What do you think the most common question in statistics is? Several times a year I teach a statistics course for UX professionals and get asked this question a lot. We&#8217;re offering the class this fall at the LeanUX Denver conference and a portion of it is available for download. Some attendees have had statistics [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":11259,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"site-sidebar-layout":"default","site-content-layout":"default","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"default","adv-header-id-meta":"","stick-header-meta":"default","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"_price":"field_56e41332a1ae5","_stock":"","_tribe_ticket_header":"","_tribe_default_ticket_provider":"Tribe__Tickets_Plus__Commerce__WooCommerce__Main","_tribe_ticket_capacity":"0","_ticket_start_date":"","_ticket_end_date":"","_tribe_ticket_show_description":"","_tribe_ticket_show_not_going":false,"_tribe_ticket_use_global_stock":"","_tribe_ticket_global_stock_level":"","_global_stock_mode":"","_global_stock_cap":"","_tribe_rsvp_for_event":"","_tribe_ticket_going_count":"","_tribe_ticket_not_going_count":"","_tribe_tickets_list":"[]","_tribe_ticket_has_attendee_info_fields":false,"footnotes":""},"categories":[21],"tags":[22],"acf":[],"ticketed":false,"_links":{"self":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/122"}],"collection":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/comments?post=122"}],"version-history":[{"count":1,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/122\/revisions"}],"predecessor-version":[{"id":31994,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/122\/revisions\/31994"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/media\/11259"}],"wp:attachment":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/media?parent=122"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/categories?post=122"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/tags?post=122"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}