{"id":324,"date":"2016-07-26T21:30:14","date_gmt":"2016-07-26T21:30:14","guid":{"rendered":"http:\/\/measuringu.com\/control-brand\/"},"modified":"2021-08-10T09:34:12","modified_gmt":"2021-08-10T15:34:12","slug":"control-brand","status":"publish","type":"post","link":"https:\/\/measuringu.com\/control-brand\/","title":{"rendered":"Controlling for Brand Attitudes in UX Studies"},"content":{"rendered":"<p><a href=\"http:\/\/www.business2community.com\/brandviews\/smsssummit\/5-brands-winning-social-media-can-learn-01433688#0Ple7sc5G7MqjLQU.97\"><img decoding=\"async\" style=\"float: left; padding: 10px; margin-right: 20px;\" title=\"Brands\" src=\"https:\/\/measuringu.com\/images\/brands.jpg\" width=\"245\" height=\"167\" \/><\/a>There are a number of variables that <a href=\"https:\/\/measuringu.com\/blog\/comparative-variables.php\">affect UX metrics<\/a>.<\/p>\n<p>In most cases though, you&#8217;ll simply want to measure the user experience and not these other &#8220;nuisance variables&#8221; that may mask the experience users have with an interface.<\/p>\n<p>This is especially the case when making comparisons. In a comparative analysis you use multiple measures to determine which website or product is superior.<\/p>\n<p>Three of the <a href=\"https:\/\/measuringu.com\/blog\/comparative-variables.php\">most common variables<\/a> that can affect UX metrics and mask differences in designs are:<\/p>\n<ul>\n<li>Prior experience<\/li>\n<li>Domain knowledge and skills<\/li>\n<li>Brand attitudes<\/li>\n<\/ul>\n<p>In an earlier post I discussed the <a href=\"https:\/\/measuringu.com\/blog\/prior-exposure.php\">effects of prior experience<\/a> and how it can be measured and <a href=\"https:\/\/measuringu.com\/blog\/weighted-experience.php\">controlled for<\/a>. Here I&#8217;ll describe the more nebulous nature of <a href=\"https:\/\/measuringu.com\/blog\/branding-survey.php\">brand attitudes<\/a> and how best to measure and control for them in UX studies.<\/p>\n<h2>Attitudes toward People and Brands<\/h2>\n<p>A user&#8217;s attitude toward a company or product can heavily influence (for better or worse) their attitude toward an experience. People treat companies and brands like they treat people. Think of someone in your life who you generally don&#8217;t like (a colleague, manager, or friend of a friend). This person may annoy you, or you disagree with on many issues.<\/p>\n<p>If I asked you to rate your favorability toward them on a seven-point scale, chances are, it&#8217;d be on the low side. This unfavorable attitude would then likely carry over to how you judge the quality of his or her work and ideas. In contrast, think of someone you like, admire, and respect. When you judge this person&#8217;s work and ideas you&#8217;re likely to cast a more <a href=\"https:\/\/en.wikipedia.org\/wiki\/Halo_effect\">positive &#8220;halo&#8221;<\/a> on their efforts.<\/p>\n<p>The same concept applies for companies and their brands. If you don&#8217;t like a company (for example Coca-Cola or Walmart), or one of their brands (Odwalla or Sam&#8217;s Club), you&#8217;re probably not going to like their website and in turn rate the experience less favorably.<\/p>\n<h2>Controlling for Brand<\/h2>\n<p>While it&#8217;s realistic to have a mix of brand lovers and haters in a UX study, you&#8217;ll often want to understand how much the experience differs despite these variations in participants&#8217; brand attitudes.<\/p>\n<p>For example, I have data from a <a href=\"https:\/\/measuringu.com\/blog\/competitive-benchmarking.php\">comparative website benchmark<\/a> study we conducted with well-known consumer retail brands. The study was <a href=\"https:\/\/measuringu.com\/blog\/between-within.php\">between-subjects<\/a> with 592 participants and four websites. At the beginning of the study we asked participants to rate their favorability toward the brands used in the study. At the end of the study, participants answered the 8-item <a href=\"https:\/\/measuringu.com\/products\/suprq\">SUPR-Q<\/a>\u00ae\u2014a reliable measure of the quality of the website user experience. You can see the scores by website in Figure 1 below.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/measuringu.com\/images\/brand-g1.jpg\" \/><br \/>\n<span style=\"font-size: small;\">Figure 1: SUPR-Q scores for four websites (un-corrected). Error bars are 95% confidence intervals.<\/span><\/p>\n<p>In Figure 1, website A had the highest SUPR-Q score and separated itself from the other websites. Running an Analysis of Variance (<a href=\"https:\/\/measuringu.com\/blog\/advanced-stats.php\">ANOVA<\/a>) we find at least one website is statistically significant from the others F(3,589)=6.31; p &lt;.001. You can also see this difference visually from the lack of overlap in the confidence intervals between website A and the other sites.<\/p>\n<p>But Figure 2 below shows the same participants&#8217; brand favorability ratings on a 7-point scale (1 = very unfavorable and 7 = favorable).<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/measuringu.com\/images\/brand-g2.jpg\" \/><br \/>\n<span style=\"font-size: small;\">Figure 2: Brand favorability score for the four websites.<\/span><\/p>\n<p>We see the same pattern as the SUPR-Q scores in Figure 1. Participants in particular had a strong affinity toward Brand A. In fact, the correlation between brand favorability and SUPR-Q scores in this study is reasonably large ( r = .52). But when measuring the website user experience, we usually want to measure how the website affected attitudes about usability and not just a preexisting measure of brand attitude.<\/p>\n<p>In other words, what we want to know is, <b>how much of the differences in SUPR-Q scores are due to the website and not brand favorability?<\/b> To find out we can use two approaches: the ANCOVA or create subgroups by splitting the data.<\/p>\n<h3>The ANCOVA<\/h3>\n<p>The Analysis of Covariance (called ANCOVA) will &#8220;partial&#8221; out the correlated effects of brand attitudes to tell us if the website experiences continue to differ. Both <a href=\"https:\/\/www.youtube.com\/watch?v=_uYASFVUNpQ\">SPSS<\/a> and <a href=\"http:\/\/www.tutorialspoint.com\/r\/r_analysis_of_covariance.htm\">R<\/a> have ANCOVA procedures and they both allow you to save the corrected scores, which you can then use as a new variable and to visualize. Figure 3 below shows the new &#8220;corrected&#8221; means from the ANCOVA along with the original SUPR-Q mean scores by website.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/measuringu.com\/images\/brand-g3.jpg\" \/><br \/>\n<span style=\"font-size: small;\">Figure 3: Raw and corrected SUPR-Q scores for brand favorability for four websites.<\/span><\/p>\n<p>After taking into account brand favorability, the differences between websites are no longer statistically significant F(3,593) = 1.790; p = .148. You can also see that website A&#8217;s error bars are now overlapping with the other websites, also showing the loss of statistical significance. In other words, after accounting for brand favorability, attitudes toward the quality of the website user experience are no longer substantially different. Accounting for brand favorability in this study makes a difference in our conclusions\u2014the sites aren&#8217;t substantially different (not statistically) given the tasks and types of users.<\/p>\n<h3>Compare Brand Subgroups<\/h3>\n<p>If you aren&#8217;t able to run an ANCOVA on your data, you can also take a simpler approach by segmenting your data into High versus Low Brand Favorability groups and then running the comparisons again. For example, I created a new segment of high brand favorability (responses of 7 on the brand favorability scale) and a lower brand favorability segment (1 to 6 on the same scale) and then graphed the SUPR-Q scores.<\/p>\n<p>You can see in Figure 4 below that while there&#8217;s a noticeable difference in scores between the high and low brand groups, there&#8217;s much less differentiation within each group. The <a href=\"https:\/\/measuringu.com\/blog\/ci-10things.php\">confidence intervals overlap<\/a> and the results of the ANOVAs are not significant for the low brand group F(3,160) =1.89; p = .13 or the high groups F(3,195) =1.57; p = .20.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/measuringu.com\/images\/brand-g4.jpg\" \/><br \/>\n<span style=\"font-size: small;\">Figure 4: SUPR-Q scores across four websites for low brand favorability (green bars) and high brand favorability (peach bars).<\/span><\/p>\n<p>As we&#8217;ve seen with this example, brand attitudes matter and they can explain much of the differences in UX metrics. But they don&#8217;t always render the differences as not significant.<\/p>\n<p>In another study with five websites and 492 participants the differences between sites remained even after factoring in brand favorability. Brand favorability again was correlated with SUPR-Q scores (r = .46). The ANCOVA results still showed statistically significant differences between websites after removing existing brand attitudes F(4,467) =3.10; p = .02.<\/p>\n<p>Figure 5 shows how the differences remain after factoring out brand favorability. Website Q had higher SUPR-Q scores than the other websites with the raw scores (orange bars) and, while this difference was reduced, still remained with the corrected scores from the ANCOVA (blue bars).<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/measuringu.com\/images\/brand-g5.jpg\" \/><br \/>\n<span style=\"font-size: small;\">Figure 5: Raw and corrected SUPR-Q scores for brand favorability on five websites.<\/span><\/p>\n<h2>Conclusion<\/h2>\n<p>A few things to keep in mind when measuring the user experience and accounting for brand favorability:<\/p>\n<ol>\n<li>Be sure to track brand favorability. Even a simple single item asking how favorable your participants feel towards a brand is usually sufficient. Collect this prior to any UX metrics or tasks.<\/li>\n<li>Brand favorability has a strong and statistically significant impact on UX metrics. The correlation is often above r = .4, meaning brand accounts for at least 16% of the variation in UX metrics (which often exceeds the differences between interfaces being compared).<\/li>\n<li>To control for the effects of brand favorability you can use an ANCOVA (preferred) or create a segment of low and high brand favorability participants to see whether the patterns in metrics hold.<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"<p>There are a number of variables that affect UX metrics. In most cases though, you&#8217;ll simply want to measure the user experience and not these other &#8220;nuisance variables&#8221; that may mask the experience users have with an interface. This is especially the case when making comparisons. In a comparative analysis you use multiple measures to [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":2016,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"site-sidebar-layout":"default","site-content-layout":"default","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"default","adv-header-id-meta":"","stick-header-meta":"default","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"_price":"field_56e41332a1ae5","_stock":"","_tribe_ticket_header":"","_tribe_default_ticket_provider":"Tribe__Tickets_Plus__Commerce__WooCommerce__Main","_tribe_ticket_capacity":"0","_ticket_start_date":"","_ticket_end_date":"","_tribe_ticket_show_description":"","_tribe_ticket_show_not_going":false,"_tribe_ticket_use_global_stock":"","_tribe_ticket_global_stock_level":"","_global_stock_mode":"","_global_stock_cap":"","_tribe_rsvp_for_event":"","_tribe_ticket_going_count":"","_tribe_ticket_not_going_count":"","_tribe_tickets_list":"[]","_tribe_ticket_has_attendee_info_fields":false,"footnotes":""},"categories":[21],"tags":[22],"acf":[],"ticketed":false,"_links":{"self":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/324"}],"collection":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/comments?post=324"}],"version-history":[{"count":0,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/324\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/media\/2016"}],"wp:attachment":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/media?parent=324"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/categories?post=324"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/tags?post=324"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}