The best-laid plans of researchers often go awry.
No matter how carefully a research project is planned, things may still go wrong.
There are many threats to the success of a customer research project.
At both big and small companies, at agencies and on the client side, research projects can fail to live up to expectations.
Common problems include:
- Time delays
- Cost overruns
- Poor management
- Technology problems
- Insufficient sample sizes
- Inadequate control on threats to reliability
- Failure to get stakeholder buy-in
- Results that go ignored
Each of these outcomes can be challenging to prevent. Some seem more inevitable, like time delays and technology problems; others seem more preventable, like insufficient sample sizes or stakeholder buy-in.
Here are seven techniques we use to help identify and mitigate problems to conduct a successful project.
1. Identify your stakeholders’ stakeholders
Knowing who the primary stakeholders are is something a lot of researchers know at the beginning. Stakeholders often drive the budget and the direction of the project; they help establish and clarify the research objectives. But it’s often when these stakeholders report out the research that new questions or concerns arise.
Try to identify your stakeholders’ stakeholders and their potential concerns and points of view. Even if you don’t interact with them, identifying this secondary audience will often help the primary stakeholders ensure there’s buy-in and you can be sure you’re measuring the right things.
2. Fill in as much of the final report before you collect data
This is an idea we try to adopt wherever we can. It comes from Lean UX. Even if you don’t know what the project results will show, you can likely set up many sections of a report (including the research goals, participants’ characteristics, and metrics).
While most projects are unique, many share some common components; the more sections you can fill in ahead of time, the faster you can get your results to stakeholders (especially when time delays are a common problem).
3. Write the executive summary before the project
Even if you don’t fill out a report, consider what you would write as the executive summary. While you don’t want to fabricate data or bias yourself into a predetermined result, try writing an executive summary ahead of time. Good executive summaries include information on:
- Key metrics
- The “so-what” findings
- Research goals
Knowing what you need to cover in an executive summary ahead of time gives you the opportunity to identify gaps, assumptions, and potential problems during your project.
4. Anticipate major objections
Like an attorney preparing for trial, knowing the objections and preparing for them makes a stronger and more convincing case. You can consider major objections before you collect data to help tailor your research and write the report. In the studies we’ve conducted over the years, some of the more common objections include
- The wrong mix of participants (demographics, prior experience, and occupations)
- Incorrect tasks in usability tests (too easy, or address the wrong functionality)
- Insufficient sample sizes and power
- Wrong interpretation of statistical significance
We’ll often have backup slides ready or included in an appendix that address some of these objections.
5. Conduct a postmortem
Postmortems are conducted on a dead body to fully understand the cause of death. In business, the body is the project and death is the end of the project (successful or unsuccessful). But the intention of both kinds of postmortems is to understand what happened. Postmortems don’t benefit the dead patient or finished project. But they do help researchers and stakeholders on the next project understand the lessons learned.
This can be helpful when components get shared from study to study (recruiting, metrics, analysis, presentations etc.). Use techniques like an affinity diagram to help organize and uncover root causes of problems.
6. Anticipate the reaction to potential findings
Similar to anticipating objections, you also want to anticipate reactions to the findings, even from supportive stakeholders. What will be the reaction if you find no statistical differences, or find a major increase or decrease in a Key Performance Indicator (KPI)? What if the research uncovers no usability problems or several critical ones? What if you discover there’s tepid interest in your new product idea?
Anticipating the reaction to potential findings will help determine if the right people and teams will be able to address the results and make changes if needed.
7. Conduct a premortem
Instead of waiting till the project ends to review what went wrong, do it before the project begins. Research has found that the act of imagining a future event increases the ability to correctly identify what caused the event by 30%. In a premortem, team members get briefed on the project plan and then are asked to imagine it failed and to brainstorm the reasons the project failed. The eminent psychologist Daniel Kahneman has called premortems one of the most effective techniques for identifying and preventing problems.