![](\"https:\/\/measuringu.com\/images\/opposites.jpg\" "\\"Complementary")
As you approach a research project, you are often faced with choices about different methods and concepts.<\/p>\nAs you approach a research project, you are often faced with choices about different methods and concepts.<\/p>\n
While you consider your options among methods<\/a>, it may be that opposing methods actually offer a complementary rather than contradictory view of the user experience.<\/p>\n The following pairs of essential concepts are often portrayed as mutually exclusive while, in fact, the elements of each pair complement each other.<\/p>\n Understanding the different but often complementary nature of these paired concepts can help improve your research.<\/p>\n Inspection methods involve having an interface expert (who, ideally, is also an expert in the domain) evaluate an interface, often against a set of guidelines (called heuristics).<\/p>\n Usability testing<\/a> involves actually observing users as they engage with an interface to accomplish tasks. Each method generates lists of problems and insights. While only usability testing generates performance metrics, such as task-completion rates or task time, we’ve found that inspection methods uncover around 30% of the issues uncovered in usability tests<\/a>.<\/p>\n There was a big debate in the 1990’s around how effective inspection methods are compared to usability testing in generating the most usability issues. The seminal paper<\/a>, Damaged Merchandise, provides a good overview of that debate. In practice, it should not be inspection methods OR usability testing, but rather inspection methods AND usability testing.<\/p>\n Analytic methods are those that don’t involve observing and collecting data directly from participants. These include Heuristic Evaluations<\/a>, Cognitive Walkthroughs<\/a> and Keystroke Level Modeling (KLM)<\/a>.<\/p>\n Analytic methods are particularly helpful when:<\/p>\n Empirical methods, such as usability testing, surveying, and customer interviews, do involve observing and collecting data directly from participants. They’re essential if you need to make definitive statements about actual user performance.<\/p>\n Good research involves a mix of analytic and empirical methods.<\/p>\n Qualitative research<\/a> involves collecting and analyzing primarily nonnumeric data from word-, picture-, and action-oriented activities. Quantitative analysis, of course, involves numbers or things that can be converted to numbers<\/a>. These two analysis types aren’t opposites: every quantitative measure comes first from a qualitative observation, underscoring their complementary nature.<\/p>\n A common misconception<\/a> holds that small sample sizes mean qualitative methods and large sample sizes mean quantitative methods; but this isn’t necessarily true. We may need to quantify behaviors with small sample sizes, and we may need to note and code behaviors with large sample sizes.<\/p>\n To describe and improve the user experience, you need quantitative methods to describe frequency and qualitative methods to understand the why.<\/p>\n Summative evaluation <\/a>establishes benchmarks: How usable? How satisfied? How loyal? Formative evaluation is primarily about finding and fixing interaction problems. In practice, most studies mix summative and formative approaches. Even when we conduct large-scale annual benchmarks for companies, we still diagnose the usability problems and make recommendations on what to fix. When we’re detecting problems with an interface, we include task- and test-level metrics that enable us to gauge whether the usability is improving in each phase<\/a>.<\/p>\n So while it may be convenient to think of studies as either formative or summative, don’t let the lexicon prevent you from mixing these in one study.<\/p>\n Research is often commissioned on a sample of customers to establish causation<\/a> for the entire customer population. For example, you may want to know whether a new design yields higher conversion rates, or whether a new pricing model increases customer loyalty. When doing so, you have to know with certainty that the research conditions aren’t overtly impacted by nuisance variables that may cloud the causation results.<\/p>\n Examples of nuisance variables that may cloud the claims of causation include:<\/p>\n When we account for and control nuisance variables in a study, we can increase our internal validity. When we can show that conditions in the study will mimic the real world situation of the entire customer population, with all its confounding variables, then we can increase our external validity.<\/p>\n Good research balances internal and external validity to be sure that our findings hold up in the lab and real world. It can be tricky to establish high external and internal validity in one study, so consider forming a hypothesis in one study that has, say, higher external validity, and then look to replicate the findings in a study with more internal validity.<\/p>\n While your sample size can’t be at once small and large, the distinction between the two is fuzzy. The demarcation line for some falls at 10 or 30 and for others at 100. There is no magic or universal number at which a sample size goes from being small to being large. The mathematics behind most statistical computations requires at least two participants in a study. All things considered, a larger sample size implies more precise estimates and a greater ability to distinguish causal difference from random noise.<\/p>\n But quantitative methods aren’t limited to large sample sizes<\/a> (whatever number that is), and formative methods aren’t limited to small sample sizes. Find more information on the optimal statistical techniques for large and small sample sizes in our book, Quantifying the User Experience<\/a>.<\/p>\n While these six paired concepts often seem mutually exclusive, they often complement each other. Understand the distinctions between the two, consider how tenuous the distinctions can be, and then look for opportunities to use both.<\/p>\n","protected":false},"excerpt":{"rendered":" As you approach a research project, you are often faced with choices about different methods and concepts. While you consider your options among methods, it may be that opposing methods actually offer a complementary rather than contradictory view of the user experience. The following pairs of essential concepts are often portrayed as mutually exclusive while, […]<\/p>\n","protected":false},"author":2,"featured_media":1643,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"site-sidebar-layout":"default","site-content-layout":"default","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"default","adv-header-id-meta":"","stick-header-meta":"default","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"_price":"field_56e41332a1ae5","_stock":"","_tribe_ticket_header":"","_tribe_default_ticket_provider":"Tribe__Tickets_Plus__Commerce__WooCommerce__Main","_tribe_ticket_capacity":"0","_ticket_start_date":"","_ticket_end_date":"","_tribe_ticket_show_description":"","_tribe_ticket_show_not_going":false,"_tribe_ticket_use_global_stock":"","_tribe_ticket_global_stock_level":"","_global_stock_mode":"","_global_stock_cap":"","_tribe_rsvp_for_event":"","_tribe_ticket_going_count":"","_tribe_ticket_not_going_count":"","_tribe_tickets_list":"[]","_tribe_ticket_has_attendee_info_fields":false,"footnotes":""},"categories":[39],"tags":[40],"acf":[],"ticketed":false,"_links":{"self":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/246"}],"collection":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/comments?post=246"}],"version-history":[{"count":0,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/246\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/media\/1643"}],"wp:attachment":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/media?parent=246"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/categories?post=246"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/tags?post=246"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}1.\u00a0\u00a0 \u00a0Inspection Methods and Usability Testing<\/h2>\n
2.\u00a0\u00a0\u00a0 Analytic and Empirical Methods<\/h2>\n
\n
3.\u00a0\u00a0\u00a0 Qualitative and Quantitative Analysis<\/h2>\n
4.\u00a0\u00a0\u00a0 Summative and Formative Evaluation<\/h2>\n
5.\u00a0\u00a0\u00a0 Internal and External Validity<\/h2>\n
\n
6.\u00a0\u00a0\u00a0 Small and Large Samples<\/h2>\n