What Happens Before and After User Testing Matters
How might Toss improve its user research participants' journey?
Overview
🖇 Project Details
-
Timeline: April - August 2021
-
Team: UX Research Coordinator, supporting 4 UX Researchers
📌 Problem
Team-facing | the user research process was very inefficient, which prevented the UXR team from taking on and conducting more research projects
Participant-facing | the end-to-end participant journey was not well-managed, which led to a low recruitment rate
🙋🏻♀️ What company is Toss?
Viva Republic (Toss) is a Seoul-based fintech company that created Toss, a super app that provides 40+ financial services to more than a third of South Korea's population.
📌 Solution
Team-facing | Enhanced and expedited the team’s research by implementing new tools and processes to reduce unwanted participant behavior and automate time-consuming tasks
Participant-facing | Designed the participants’ end-to-end journey, driving all aspects of participant recruitment and management for user research
CONTEXT
The company's core mission is to revolutionize the user experience in financial services in Korea, which have traditionally been very inefficient and difficult to navigate.
The UX Research team received a number of research requests from various product teams at Toss but wasn’t able to meet the demand.
One of the reasons was that the UX Research team was relatively small, but another big reason was that there was essentially no research operations in place.
GOAL
How might I support the participants and researchers and create a seamless experience for both?
To figure this out, I had to first identify gaps and problems that made the experience difficult
PROBLEM # 1
Participants were concerned that the recruiting process was a scam.
Many people didn’t respond to the text messages or calls after completing the screener survey. Some even reported the interview as a fraud to the company’s customer service after the interview.
Oftentimes, the recruiting criteria were very narrow and specific, and recruiting each and every user who passed the screener was critical to securing a sufficient sample size for research.
How can I better establish credibility during the recruiting process and increase the response rate?
SOLUTION # 1
Published and shared an official Toss blog post on the research team
How
-
Worked with a Content Designer to create an official Toss blog post that introduced the research team at Toss and what it did
-
Looked through conversation history with past participants and identified their most frequently asked questions to shape the content around what prospective participants would be most curious about
-
Added this link whenever I made the first contact with the participants
Why
Voice and SMS phishing scams, along with spam calls, happen quite often in Korea. To dispel any suspicion and establish credibility, it was important that people first recognize that Toss had an official research team that did recruit and interview its users.
It was also important to show how the recruiting and interview process officially worked so that they could easily make the connection and feel more at ease with proceeding.
SOLUTION # 2
Made the communication flow with the participants smoother by containing it in one single "official" space
Before, communication with participants started with texts or calls and had to later move to a messaging platform called Kakaotalk
How
-
Worked with the internal tool engineering team to re-configure its Kakaotalk messaging system so that it could be used for recruiting purposes
-
Started and ended all conversations with participants in Kakaotalk
Why
Conversations had to happen in one place without the user having to jump from one to another to create a more seamless experience and minimize drop-offs.
Also, since Kakaotalk's business accounts had to be verified, all participant communication through Kakaotalk could more easily be affiliated with Toss and be perceived as more "official."
SOLUTION # 3
Leveraged the Kakaotalk Business Chatbot features to inform and assure participants and encourage them to initiate a Kakaotalk conversation
How
-
Using the chatbot feature, created a series of Q&As so that participants were better informed and felt more comfortable about participating before initiating a conversation
-
Before they said anything on messenger, they could click on different options on the chatbot and get more information
Why
Since our research Kakaotalk account was a business account, it couldn’t initiate conversations with the users. The user had to initiate. Giving users enough information for them to take action was key.
It also reduced my time and effort of having to respond to similar questions from participants.
PROBLEM # 2
It took lots of time to help the participants get set up at the beginning of the interview, taking time away from interviews. There were also quite a bit of no-shows and late participants.
SOLUTION # 1
Provided clear and easy-to-understand prep instructions before the interview
How
Created a visual to share with the participants after scheduling to make sure participants know what to do and do what they need to do before the interview
Why
There were certain participant behaviors that made conducting remote interviews very difficult and disrupted the interview flow, including joining Zoom in places with loud background noise or unstable internet connections.
SOLUTION # 2
Provided multiple "gentle" reminders
How
-
Provided to participants a brief recap of the interview date & time and other things to remember right after scheduling an interview
-
Reminded them a day or a few hours before the interview and shared a Zoom link in advance
Why
No-show or late participants meant that the time and effort of the researchers and other team members who joined to audit the interview were wasted. It was critical that participants showed up on time ready to conduct the interview.
PROBLEM # 3
Many parts of the research process were done manually and often involved repetitive, error-prone, tedious work.
I was supporting four UX Researchers, which meant that I was supporting and working on recruiting participants for multiple research projects, with very different recruiting criteria, at any given time.
It was a lot of work, and I had to carefully choose where my efforts could be the most critical and impactful.
How can I make the research process as efficient as possible and minimize repetitive tasks by automating them as much as possible?
SOLUTION
Create a system and automate, automate, automate
How
Automated the bulk of the communication and operational work, including the following:
-
Once the interview was scheduled, automatically created a Google Calendar event with a Zoom link with the researcher leading the project set as the meeting host
-
After the interview was scheduled, the researcher who was leading that particular research was automatically notified through Slack and Google Calendar
-
After the interview, the Zoom recording was automatically uploaded to a private YouTube channel where it was shared internally
-
The participants’ information that was required to provide the monetary incentive and the incentive amount were automatically notified to the financial managers
_svg.png){kind=icon}
IMPACT
What impact did I make? What changed?
-
More than doubled the participant response rate and reduced the drop-off rate in the recruiting funnel
-
Expedited the recruitment and research process from a few weeks to as short as 3 days, significantly reducing the amount of time taken to recruit for a project
-
Enabled UX Researchers to conduct research projects 1.5 times more quickly and take on more projects
-
Decreased the number of user complaint calls regarding user research