Post-mortems and retrospective accounts are valuable for understanding what went well and what went poorly. This applies not only to investigations of critical events, such as crimes and plane accidents, but also to experiences with products and services.
But the usefulness of people’s recollections of events and experiences rests on the accuracy of their memories. (Measuring the perception of experiences might be just as valuable as what actually happened, but that’s a topic for another article.)
Memory is, of course, fallible (where did I put my phone again?).
While direct observation and usability testing are great for understanding and preventing problems, they aren’t always feasible. You can’t always observe or collect data while important events are happening. Even contemporaneous records might provide only a partial picture, failing to fully represent the user experience.
You will then need to leverage customers’ recollections of events using methods such as in-depth interviews and surveys. Understanding what causes participants to forget the details needed to answer interview and survey questions correctly is the first step in preventing memory errors and getting more accurate information.
Groves et al. (2009) describe four types of memory problems that can affect the accuracy of answers to questions.
1. Retrieval Failure
You can’t forget what you didn’t notice. Forgetting something means details were encoded into memory in the first place. When trying to recall these details, you either remember (memory retrieval) or forget (retrieval failure).
It’s hard enough to remember details about significant life events, but details about routine events are even harder to remember. Did the problem with finding your premium amount happen on your auto insurance website, or was it your health insurance website? An in-store shopping experience from six weeks ago can be hard to recall, as was the call to customer support last week.
Time may heal all wounds, but time also wounds all recall. There’s a rich history of published work on memory, and one clear finding is that the more time has passed, the less likely you’ll be able to remember things.
Forgetting isn’t an even, linear decay. Instead, we tend to forget much soon after, and then the rate tapers off in a reverse exponential function. Interestingly, a significant number of people remember personally salient facts and events (teacher names, grades, classmates), even after decades. Figure 1 shows the inverse exponential decay of memory for four different types of personal facts.
Figure 1: Memory over time for grades, teacher names, classmate names, and street names (source is Tourangeau et al., 2000).
2. Reconstruction Errors
Similar events may blend to create a sort of generic memory. Were you transferring money, checking your balance, or paying your mortgage when you last used your banking app? Gaps in our memory are often filled in with guesses of what we probably did. There is also evidence that we project our current thinking onto the past, even the recent past (see Pearson et al., 1992). For example, answer the following question that appeared in a recent survey we took:
How many prescriptions have you ordered online in the last two years?
What are the steps you go through to come up with your best answer? How confident are you that your answer is accurate?
Other than the dates of significant events such as birthdays, anniversaries, or holidays, it can be hard to remember the date on which an activity occurred or, as is often the case with surveys, whether something happened in a time period (for example, within the last two years).
People tend to misplace events that really happened earlier or later into the reference period (technically referred to as forward and backward telescoping errors). This leads to respondents overreporting activities such as the frequency of home repairs and underreporting how much money they’ve spent. This likely also affects the accuracy of responses to consumer survey questions about general product purchasing and usage.
3. Memory Distortion
Our memory of significant events such as weddings, graduations, and hospital visits can be affected by contemporary details gathered from photos, videos, or stories from others. This distortion almost certainly happens to some extent for the routine tasks asked about in customer and UX surveys. Was the problem you remember having with your banking app something you encountered? Or might comments on Twitter about the bad experience be distorting your recall?
The psychologist Elizabeth Loftus is famous for showing how easily memories can be created, changed, or manipulated even for significant events. This has implications for the reliability of memories recovered during talk therapy and for the use of eyewitness testimony in courtrooms.
Loftus herself has experienced the formation of a false memory, which she recounted in an NPR interview from 2017:
Well, personally, I had a kind of an amazing experience. I have to preface this with the fact that when I was 14 years old, my mother drowned in a swimming pool. And, you know, jump ahead decades later. I went to a 90th birthday party of one of my uncles. And one of my relatives told me that I was the one who found my mother’s body. And I said, no. No, it didn’t happen. And this relative was so positive that I went back from that family reunion and I started thinking about it. And I started maybe visualizing.
And I started to think maybe it really did happen. I started to make sense of other facts that I did remember in light of this news. And then my relative called me up a week later and said, I made a mistake, it wasn’t you. And so I thought, oh, my gosh, I just had the experience of my subjects, where someone convincingly tells you and you start to visualize and you start to feel it. And then it wasn’t true.
4. Mismatches between Researcher and Respondent Interpretation of Terms
The fourth type of memory problem is that the terms we use to ask respondents about prior events might not trigger recall. Does ordering food from Grubhub count as online shopping? How about ordering a prescription online for pickup at the drugstore? Maybe the researcher is interested in capturing these activities, but the term online shopping might not cause the participant to recall them.
Summary
Using surveys and interviews to ask participants about their past experiences can help you understand what does and doesn’t work for developing better products or services. People, of course, forget details. We forget things for many reasons. Four primary culprits are
- Retrieval failure (the more time has passed and the less salient the detail, the more imperfect the recollection)
- Reconstruction (we fill in gaps with generic memories)
- Distortion (our memories are influenced by contemporary details such as photos and videos of an event)
- Mismatching terms (we might not associate memories with the terms used by interviewers or surveys)
In a future article, we’ll discuss ways of minimizing forgetting; we hope you’ll remember to check back!
