It’s hard to collect data for UX research, and once you have it, you have to clean it.

In a simpler world, all respondents would be honest and focused on providing high-quality information rather than maximizing income, but that’s not the world we live in. From past research, we estimate the prevalence of cheating on paid panels to be about 10% of respondents (ranging from 3–20%).

UX researchers can use numerous strategies for screening (stopping bad actors before they get to the actual study) and cleaning (finding and removing poor quality respondents after study completion). These include:

• Identification of speeders
• Disqualifying questions
• Attention checks
• Review of open-ended responses
• Internal consistency
• Straightlining
• Review of session recordings (when available)
• Duplicate and bot detection

AI complicates all these approaches. Modern AI can mimic attentive respondent behavior well enough to slip past most of these detection methods. We are encouraged, however, that many panel operators have taken active steps to restrict AI fraud at the source.

When those safeguards are in place, or when participants come from a verified human population such as a customer list, we propose another dual-purpose and quick approach.

In this article, we demonstrate how to use TAC-10™ response patterns not only for its primary purpose as a measure of tech savviness but also as a type of internal consistency check for screening and detecting inattentive or misrepresenting human respondents.

TAC-10 Basics

In a series of articles, we reviewed the findings of eight years of research into measuring tech savviness. In that research program, we explored several methods for measuring tech savviness, including quizzes (what people know), self-assessment questionnaires (what people feel), and technical activity checklists (what people are confident doing).

After analyzing thousands of participants’ data to see how measures of tech savviness predict task performance, we determined that technical activity checklists had better measurement properties than quizzes or questionnaires. Of the various versions of checklists that we studied, we determined that the one with ten activities and a none-of-the-above option (the TAC-10 shown in Figure 1) has the best balance between conciseness and completeness.

Figure 1: The current version of the TAC-10 (image from the MUiQ^® platform).

The TAC-10 score for a person is the number of selected items. It’s a reliable (consistent) and valid (predictive) measure of tech savviness that, for its primary purpose, can be used (1) to classify participants into groups with low, medium, or high levels of tech savviness and (2) as a tech savviness predictor or covariate in advanced statistical analysis.

Some TAC-10 Response Patterns Are More Plausible than Others

In addition to being a tool to measure and classify tech savviness, TAC-10 response patterns can also be used to identify potentially problematic respondents based on the plausibility of the pattern.

In May 2023, we collected a large sample of completed TAC-16 checklists as part of a screening survey (n = 4,731) to acquire enough data for Rasch analysis of three versions of the TAC (TAC-9, TAC-10, and TAC-16). In this new analysis, we applied various methods to classify response patterns as plausible, implausible, or indeterminate. Examples of plausible patterns are those consistent with perfect or near-perfect Guttman scaling. Response patterns that are logically inconsistent are implausible. Patterns that are not clearly plausible or implausible are indeterminate. For these analyses, we coded each TAC-10 response as a binary string of 0s (not selected) and 1s (selected) for activities in the order shown in Figure 1. For example, 1100000000 indicates a user who selected “installing a new app on your phone” and “setting up a new phone,” but no other activities.

Responses Consistent with Guttman Scaling Are Plausible

Guttman scaling, which dates back to the 1940s, is a deterministic predecessor of probabilistic Rasch scaling. The goal of a Guttman scale is to develop a set of distinctive items, from easy to difficult, that form a unidimensional scale. The range from easy to difficult can refer to characteristics like easy to solve to difficult to solve for math problems or easy to agree with to difficult to agree with for attitudinal scales.

For 10 binary (yes/no) items like the TAC-10, there are 2¹⁰ (1,024) possible arrangements of selected (1) or unselected (0) items. Only 11, however, are consistent with a perfect Guttman scale (all 1s toward the left side of the pattern, all 0s to the right): 0000000000, 1000000000, 1100000000, 1110000000, 1111000000, 1111100000, 1111110000, 1111111000, 1111111100, 1111111110, and 1111111111. We categorized these patterns as plausible.

Other Plausible Response Patterns

In practice, other patterns that are close to Guttman patterns are also likely to be plausible. For example, if someone is comfortable with all activities except HTML, the pattern would be 1111111101. Although it’s unlikely that someone who programs efficiently in C knows nothing about HTML, it’s possible that they would lack sufficient practical or deep familiarity with it to be comfortable selecting it. In most cases, Guttman-like patterns with one or two discontinuities are plausible.

Implausible Response Patterns

Patterns that are the inverse of Guttman patterns (1 and 0 swapped) are categorized as implausible (except for 0000000000 and 1111111111). For example, the pattern 0000000001 indicates someone who programs efficiently but isn’t comfortable with anything else in the list—possible but highly unlikely.

Other problematic patterns are those that start with 01 because, for this to be plausible, the respondent would have to be comfortable setting up a new phone but uncomfortable adding an app to that phone.

Patterns that contain just one 1 (other than 100000000) are implausible and may indicate a respondent who misunderstood the instruction to select all that apply.

Indeterminate Response Patterns

Patterns not categorized as plausible or implausible are provisionally defined as indeterminate.

Plausible TAC-10 Patterns Are Much More Likely in Practice than Implausible Patterns

We investigated the frequency of occurrence of plausible, implausible, and indeterminate patterns in our large sample of TAC-10 scores. Of the 1,024 possible patterns, only 199 appeared at least once in our dataset of 4,731 cases.

Guttman Patterns

Table 1 shows the frequency of Guttman patterns in the large TAC-10 database, accounting for 56.4% of cases.

Table 1: Distribution of Guttman patterns in the large database.

Other Plausible Patterns

Table 2 shows other frequently occurring plausible patterns (each accounting for at least 0.5% of cases in the database). The 21 patterns in the table accounted for 30.7% of cases. In combination, the percentage of the 32 Guttman and other high-frequency plausible patterns in the database is 87.1%.

Table 2: Distribution of other patterns in the large database that are plausible and had frequencies of at least 0.5%.

Implausible Patterns

The database did not contain any cases matching an inverse Guttman pattern.

There were 17 implausible patterns that started with 01, each having a frequency of 1 or 2 for a total of 21, accounting for just 0.4% of the data.

There were only four cases (0.1% of the data) in which a single activity was chosen past the phone activities (three cases with 0010000000 and one with 0001000000, two additional implausible patterns).

Indeterminate Patterns

Because there were 32 plausible and 19 implausible patterns (51) out of a total of 199 patterns, the remaining 148 patterns are indeterminate.

Combined, the indeterminate patterns account for 12.4% of the data, with no individual indeterminate case having a frequency greater than 0.4%.

Summary and Discussion

In addition to its use as a measure of tech savviness, we investigated how well the TAC-10 might be used to identify plausible and implausible response patterns for the purpose of identifying potentially problematic respondents in screening and data cleaning.

Based on our large database of TAC-10 cases (n = 4,731), using two criteria for identifying plausible response patterns (matching Guttman patterns and/or frequently occurring patterns), we found that 56.4% of cases matched Guttman patterns; an additional 21 frequently occurring patterns that slightly deviated from Guttman patterns accounted for 30.7%, for a total of 87.1%. Clearly implausible patterns accounted for only 0.5% of cases, leaving the others indeterminate.

Our key conclusions from these analyses were:

Plausible patterns made up the vast majority (87%) of TAC-10 cases. This suggests that most respondents were attending to the items rather than carelessly checking boxes, especially because we randomized the order of presentation of the items.

Implausible patterns were rare. There were no occurrences of inverse Guttman patterns, and less than 0.5% of the of the responses had a problematic pattern starting with 01 or containing a single 1 (aside from a single 1 for the easiest activity).

TAC-10 responses can be used for screening and data cleaning. These results (a large percentage of plausible and a low percentage of implausible response patterns) are encouraging regarding the application of TAC-10 to identify potentially problematic (implausible or indeterminate) response patterns as part of a battery of approaches used to identify potential cheaters (along with other strategies such as examination of open-ended responses, implausible completion times, distractors in multiple choice items, attention checks, and straightlining).

Not going to solve AI fraud: We don’t think the TAC-10 is necessarily a solution to AI fraud. More sophisticated AI methods can convincingly mimic either a low- or high-skilled human respondent, possibly by training on the articles we’ve published on the TAC-10. However, the TAC-10 remains a valuable screening tool in contexts where respondents come from a known population, such as a customer list, or where other panel-level methods have already confirmed that participants are human.