Building a Process for Design Research

At Automattic, our design teams run their own processes and collect their own data. The Jetpack team recently completed research to identify useful new features for agencies that use the product. We’ll talk through their project as we review the research process.

To find ideas, find problems. To find problems, talk to people.
Julie Zhuo

Step 1: Defining the problem, audience, and goals

Identifying the problems to solve and the research parameters is incredibly important, but it doesn’t have to be complicated.

Assumption mapping is an option you can use to gather ideas from the team on what they think users need. From there, an assumption map breaks down research priority based on evidence.

An assumption map arranges your team’s ideas into four categories to help plan the research:

Important + have evidence

If we have features that we deem important and we have supporting evidence, we should be planning and prioritizing that work.
Important + no evidence

If there are features we assume are important, but don’t have evidence for, we should evaluate further.
Unimportant + have evidence

If there are features we assume are unimportant, and have evidence for, we should defer those ideas.
Unimportant + no evidence

If there are features we assume are unimportant, but don’t have evidence of that, we should run further research to validate those assumptions.

What is considered “evidence?”

Have we spoken to customers directly and been told they need or want these features?
Have we seen something similar in the market working or not working?
Have case studies been done?

Anything that helps show these ideas have been requested or are successful can count as evidence.

Example: Jetpack Agency Research

The Jetpack design team wanted to better understand their agency customers and create a product roadmap to address their unique needs. Their research process began with defining an ‘agency customer’ (defining it as “customers that create and manage sites for others”).

Before completing the assumption map, the team reviewed data that they collected earlier in 2021. Then, they completed a brainstorming activity to gather more internal ideas for what tools they believe they should be building for agencies. They sorted these ideas into the four assumption map categories to help determine what else they already knew about agency desires and what needed further research. From there, the team generated a list of features to investigate.

Step 2: Choosing an approach

After you’ve determined which ideas need further investigation, you’ll need to figure out what research methods you’ll be using. Surveys, general interviews, and usability tests are popular methods to get quantitative and qualitative research data.

The Jetpack team chose to send surveys and schedule interviews with users to collect additional feedback and explore their ideas. They created an exhaustive list of things they wanted to learn, such as:

Which existing features do agency customers value the most and their satisfaction rating on each feature.
Which features agency customers would like to see the most.
A detailed profile of this audience – for example, what makes them remain a Jetpack user, and what would make them leave? How fast do they grow their portfolio? How many sites do they manage?

From this exhaustive list, they were able to determine a focus for their surveys and interviews and move forward.

Step 3: Preparing the research

Next, you’ll need to find people to talk to and determine what prompts you’ll need to provide to gather the right information.

Gather participants — Put a call out on social media for customer feedback, use marketing mailing lists, or find clients directly.
Write your interview script (general interview) or prep your prototypes (usability testing).

The Jetpack team did this, first by compiling a contact list of current and potential agency partners and creating a quick 5-minute product survey. They emailed the list, asking them to fill out the survey along with a prompt to sign up for an in-person interview. They wrote an interview script and revised it where needed based on the survey results.

Step 4: Conduct the research

This is where the action happens. Whatever methods you’re using, this is the step when you’re finally working with customers to gather new data on your product.

Tip: If you’re interviewing, review some best practices for communicating with your interviewees prior to conducting the sessions. Good habits include facilitating a warm introduction, remaining neutral, and being prepared to ask lots of follow-ups to your original questions. Although the format of usability testing is different, you’ll find many common tips about demeanor and questioning are similar.

Step 5: Synthesize

Recap on a video call with your team after each interview. What were the main takeaways that stood out in the interview session?
Re-watch recordings and notes for further tidbits. You might discover more nuanced details when reviewing the recordings.
Create an actionable list with every interview. Be thinking about solutions immediately.

Tip: The Jetpack designers used Grain to transcribe the interviews and quickly create smaller shareable clips.

Step 6: Reporting results

Try and create a consolidated report for easier digestion amongst your team. You should work on packaging your findings in a way that’s convincing to stakeholders and people who make decisions about product direction. You also want to get folks excited. Consider using video clips and quotes to bring the internal team closer to the customer.

Start with the effect and follow with the cause: High-impact actionable items should be high up in your report.
Quantify the qualitative bits to help your team prioritize. (Example: 12 people said X, while only 1 said Y)
If possible, report results with lo-fi design directions. A lot of people don’t know what you have in your head — these visuals help your team visualize your discoveries.

The Jetpack team is currently (as of April 2022) working on shipping enhancements for agencies that came directly from this research process. Follow the blog for future updates on the exciting work our teams are doing at Automattic.