{"id":59,"date":"2011-02-21T22:00:00","date_gmt":"2011-02-21T22:00:00","guid":{"rendered":"http:\/\/measuringu.com\/quantify-comments\/"},"modified":"2023-05-08T20:37:49","modified_gmt":"2023-05-09T02:37:49","slug":"quantify-comments","status":"publish","type":"post","link":"https:\/\/measuringu.com\/quantify-comments\/","title":{"rendered":"How To Quantify Comments"},"content":{"rendered":"<p><a href=\"http:\/\/www.flickr.com\/photos\/declanjewell\/2733974848\/sizes\/o\/in\/photostream\/\"><img decoding=\"async\" style=\"float: left; padding: 10px; margin-right: 20px;\" src=\"http:\/\/www.measuringusability.com\/images\/iPhoneBug.jpg\" border=\"0\" \/><\/a>Just because customer information is qualitative doesn&#8217;t mean you can&#8217;t use some quantitative methods to help interpret and prioritize your findings.<\/p>\n<p>Quantifying the frequency of comments with a confidence interval helps you estimate a sentiment in the total user population.<\/p>\n<p>Analyzing and prioritizing comments is a common task for the user researcher. Open ended comments take all sorts of forms.<\/p>\n<ul>\n<li>Reasons why customers are <a href=\"http:\/\/www.measuringusability.com\/blog\/top-box.php\">promoters or detractors<\/a> for a product<\/li>\n<li>Customer insights <a href=\"http:\/\/www.measuringusability.com\/blog\/customers-observe.php\">from field studies<\/a><\/li>\n<li>Product complaints to <a href=\"http:\/\/www.measuringusability.com\/analytic-testing.php\">calls to customer service <\/a><\/li>\n<\/ul>\n<h3>Step 1: Group<\/h3>\n<p>Consolidating and grouping comments into categories has its own methodology (see for example <a href=\"http:\/\/www.amazon.com\/gp\/product\/1558604111?ie=UTF8&amp;tag=meausallc-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=1558604111\">Contextual Design Chapter 9<\/a>). Some comments will be virtually identical and grouped easily. Others will differ and require additional layers of grouping.<\/p>\n<p>There can be a <a href=\"http:\/\/www.measuringusability.com\/usability-radiology.php\">high amount of variability between evaluators<\/a> grouping items. When possible it is a good idea to independently categorize comments and assess the degree of agreement using a statistic like <a href=\"http:\/\/en.wikipedia.org\/wiki\/Cohen%27s_kappa\">kappa<\/a>.<\/p>\n<h3>Step 2: Count<\/h3>\n<p>Once comments, insights or complaints have been categorized, count the number of comments that appear in a category and the total number of comments. You can then prioritize these comments by generating a confidence interval to understand what percent of all users likely feel this way.<\/p>\n<h3>How Likely are you to Recommend iTunes to a Friend?<\/h3>\n<p>For example, I recently asked 56 users to rate their experience with the iTunes software program. Users answered several questions about the ease of use and quality of iTunes.<\/p>\n<p>I also asked them how likely they&#8217;d recommend the product to a friend on a 0 to 10 scale where 10 means extremely likely. By subtracting the number of responses from 0 to 6 (Detractors) from the number of 9 and 10 responses (Promoters) you can generate the <a href=\"http:\/\/www.measuringusability.com\/blog\/normal-nps.php\">Net Promoter Score<\/a>.\u00a0 Of the 56 respondents, 31 gave a 9 or 10 and 12 responded with a 0 to 6 which generated a NPS of 34% for iTunes (19\/56).<\/p>\n<p>To understand the key reason WHY users were Detractors, I categorized the open ended comments for Detractors.\u00a0\u00a0 In total there were 16 key points I pulled from the comments.<\/p>\n<table class=\"articletable\" border=\"0\" cellspacing=\"2\" cellpadding=\"2\">\n<tbody>\n<tr>\n<th>Category 1<\/th>\n<th>Category 2<\/th>\n<th>Rating<\/th>\n<th width=\"350\">Comment<\/th>\n<\/tr>\n<tr>\n<td>\u00a0Conversion<\/td>\n<td>\u00a0Price<\/td>\n<td style=\"text-align: center;\">2<\/td>\n<td>\u00a0I used to use iTunes much more often before they raised their prices from 99 cents to $1.29 per song.\u00a0 I still use the iTunes player to transfer songs to my iPod but I usually buy songs on Amazon.com now because it&#8217;s a much better deal.<\/td>\n<\/tr>\n<tr>\n<td>Conversion<\/td>\n<td><\/td>\n<td style=\"text-align: center;\">2<\/td>\n<td>\u00a0It&#8217;s okay at best, but I find a lot of the more obscure music I listen to is not available so I rarely use it to find music as a first source. Also, I have to convert anything from iTunes in order to put it on my Zune, which is a pain.<\/td>\n<\/tr>\n<tr>\n<td>\u00a0Difficult to Use<\/td>\n<td><\/td>\n<td style=\"text-align: center;\">3<\/td>\n<td>\u00a0I dislike iTunes. It&#8217;s unnecessary and can be annoying and difficult to use.<\/td>\n<\/tr>\n<tr>\n<td>\u00a0Necessary to Use<\/td>\n<td><\/td>\n<td style=\"text-align: center;\">3<\/td>\n<td>\u00a0People generally say that iTunes kind of sucks. With ipods as ubiquitous as they are, though, iTunes is pretty necessary.<\/td>\n<\/tr>\n<tr>\n<td>\u00a0Necessary to Use<\/td>\n<td><\/td>\n<td style=\"text-align: center;\">3<\/td>\n<td>\u00a0I don&#8217;t really like it but since i have an ipod I&#8217;m obligated to use it<\/td>\n<\/tr>\n<tr>\n<td>\u00a0Proprietary<\/td>\n<td><\/td>\n<td style=\"text-align: center;\">\u00a04<\/td>\n<td>\u00a0There are better software options out there, and I don&#8217;t like the DRM of itunes.<\/td>\n<\/tr>\n<tr>\n<td>\u00a0Difficult to Use<\/td>\n<td>\u00a0Proprietary<\/td>\n<td style=\"text-align: center;\">4<\/td>\n<td>\u00a0It&#8217;s cumbersome and too proprietary.<\/td>\n<\/tr>\n<tr>\n<td style=\"vertical-align: top;\"><\/td>\n<td style=\"vertical-align: top;\"><\/td>\n<td style=\"vertical-align: top; text-align: center;\">4<\/td>\n<td style=\"vertical-align: top;\">&lt;no comment&gt;<\/td>\n<\/tr>\n<tr>\n<td>\u00a0Proprietary<\/td>\n<td><\/td>\n<td style=\"text-align: center;\">5<\/td>\n<td>\u00a0I really don&#8217;t like the way apple handles there control over your own item in which you purchased.<\/td>\n<\/tr>\n<tr>\n<td>\u00a0Difficult to Use<\/td>\n<td>\u00a0Installation<\/td>\n<td style=\"text-align: center;\">5<\/td>\n<td>\u00a0The software installer is very bloated, and there are better music library managers for some people.<\/td>\n<\/tr>\n<tr>\n<td>\u00a0Difficult to Use<\/td>\n<td>\u00a0Necessary to Use<\/td>\n<td style=\"text-align: center;\">5<\/td>\n<td>\u00a0iTunes is good, but it is unnecessarily complex. Only thing which makes someone to have it on their computer is their Apple devices.<\/td>\n<\/tr>\n<tr>\n<td style=\"vertical-align: top;\">Proprietary<\/td>\n<td style=\"vertical-align: top;\">Slow<\/td>\n<td style=\"vertical-align: top; text-align: center;\">6<\/td>\n<td style=\"vertical-align: top;\">It&#8217;s a good program, but it&#8217;s a little slow and I don&#8217;t like the fact that itunes are incompatible with MP3 players.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>I generated a 95% confidence interval around each group of comments <a href=\"http:\/\/www.measuringu.com\/wald.htm\">using the online calculator<\/a> to get an idea about how prevalent these reasons would be in the whole iTunes user population. For example, we can see that one of the primary reasons users are unlikely to recommend iTunes is because it is difficult to use.<\/p>\n<table class=\"articletable\" border=\"0\" cellspacing=\"2\" cellpadding=\"2\">\n<tbody>\n<tr>\n<th>Category<\/th>\n<th># of Comments<\/th>\n<th>Total Comments<\/th>\n<th>%<\/th>\n<th>95% CI Low<\/th>\n<th>95% CI High<\/th>\n<\/tr>\n<tr>\n<td>\u00a0Difficult to Use<\/td>\n<td style=\"text-align: center;\">\u00a04<\/td>\n<td style=\"text-align: center;\">\u00a016<\/td>\n<td style=\"text-align: center;\">\u00a025<\/td>\n<td style=\"text-align: center;\">\u00a010<\/td>\n<td style=\"text-align: center;\">\u00a050<\/td>\n<\/tr>\n<tr>\n<td>\u00a0Proprietary<\/td>\n<td style=\"text-align: center;\">\u00a04<\/td>\n<td style=\"text-align: center;\">\u00a016<\/td>\n<td style=\"text-align: center;\">\u00a025<\/td>\n<td style=\"text-align: center;\">\u00a010<\/td>\n<td style=\"text-align: center;\">\u00a050<\/td>\n<\/tr>\n<tr>\n<td>\u00a0Necessary to Use<\/td>\n<td style=\"text-align: center;\">\u00a03<\/td>\n<td style=\"text-align: center;\">\u00a016<\/td>\n<td style=\"text-align: center;\">\u00a019<\/td>\n<td style=\"text-align: center;\">\u00a06<\/td>\n<td style=\"text-align: center;\">\u00a044<\/td>\n<\/tr>\n<tr>\n<td>\u00a0Conversion Process<\/td>\n<td style=\"text-align: center;\">\u00a02<\/td>\n<td style=\"text-align: center;\">\u00a016<\/td>\n<td style=\"text-align: center;\">\u00a013<\/td>\n<td style=\"text-align: center;\">\u00a02<\/td>\n<td style=\"text-align: center;\">\u00a037<\/td>\n<\/tr>\n<tr>\n<td>\u00a0Installation<\/td>\n<td style=\"text-align: center;\">\u00a01<\/td>\n<td style=\"text-align: center;\">\u00a016<\/td>\n<td style=\"text-align: center;\">\u00a06<\/td>\n<td style=\"text-align: center;\">\u00a01<\/td>\n<td style=\"text-align: center;\">\u00a030<\/td>\n<\/tr>\n<tr>\n<td>\u00a0Price of Songs<\/td>\n<td style=\"text-align: center;\">\u00a01<\/td>\n<td style=\"text-align: center;\">\u00a016<\/td>\n<td style=\"text-align: center;\">\u00a06<\/td>\n<td style=\"text-align: center;\">\u00a01<\/td>\n<td style=\"text-align: center;\">\u00a030<\/td>\n<\/tr>\n<tr>\n<td>\u00a0Slow<\/td>\n<td style=\"text-align: center;\">\u00a01<\/td>\n<td style=\"text-align: center;\">\u00a016<\/td>\n<td style=\"text-align: center;\">\u00a06<\/td>\n<td style=\"text-align: center;\">\u00a01<\/td>\n<td style=\"text-align: center;\">\u00a030<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>It turns out that <a href=\"http:\/\/www.measuringusability.com\/usability-loyalty.php\">ease of use is often a key driver of product loyalty<\/a> and iTunes is no exception. We can be 95% confident between 9% and 50% of all iTunes Detractors feel ease of use is one of the main reasons for not recommending the product to a friend.<\/p>\n<p>Ease of use isn&#8217;t the only major reason for the detracting. By combining the &#8220;Proprietary&#8221; and &#8220;Necessary to Use&#8221; categories, we have 7 out of 16 Detractors (44%) who feel they have no choice but to use a proprietary product. We can be 95% confident between 23% and 67% of all iTunes Detractors feel this way.<\/p>\n<p>While it&#8217;s unlikely Apple&#8217;s business model is going to embrace open source any time soon, there are probably some easy ways to fix the ease of use problems with iTunes.\u00a0 Quantifying open ended comments and using confidence intervals can help you prioritize fixes and allow you to communicate your qualitative findings more confidently.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Just because customer information is qualitative doesn&#8217;t mean you can&#8217;t use some quantitative methods to help interpret and prioritize your findings. Quantifying the frequency of comments with a confidence interval helps you estimate a sentiment in the total user population. Analyzing and prioritizing comments is a common task for the user researcher. Open ended comments [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":11017,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"site-sidebar-layout":"default","site-content-layout":"default","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"default","adv-header-id-meta":"","stick-header-meta":"default","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"_price":"field_56e41332a1ae5","_stock":"","_tribe_ticket_header":"","_tribe_default_ticket_provider":"Tribe__Tickets_Plus__Commerce__WooCommerce__Main","_tribe_ticket_capacity":"","_ticket_start_date":"","_ticket_end_date":"","_tribe_ticket_show_description":"","_tribe_ticket_show_not_going":false,"_tribe_ticket_use_global_stock":"","_tribe_ticket_global_stock_level":"","_global_stock_mode":"","_global_stock_cap":"","_tribe_rsvp_for_event":"","_tribe_ticket_going_count":"","_tribe_ticket_not_going_count":"","_tribe_tickets_list":"[]","_tribe_ticket_has_attendee_info_fields":false,"footnotes":""},"categories":[48],"tags":[50],"acf":[],"ticketed":false,"_links":{"self":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/59"}],"collection":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/comments?post=59"}],"version-history":[{"count":3,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/59\/revisions"}],"predecessor-version":[{"id":37910,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/posts\/59\/revisions\/37910"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/media\/11017"}],"wp:attachment":[{"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/media?parent=59"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/categories?post=59"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/measuringu.com\/wp-json\/wp\/v2\/tags?post=59"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}