Challenges Posed by Covid-19

For consumer brands, one of the many challenges posed by Covid-19 is how to carry out qualitative testing when the “in person” aspect is currently not possible.

In recent weeks, we’ve been hearing from many marketing leaders that the pandemic’s shelter-in-place requirements are blocking their R&D and innovation initiatives. Specifically, they're not able to carry out central location tests (CLT), sensory research, focus groups, in-person interviews, and other forms of qualitative testing. Insights gleaned from in-person qualitative testing remain the gold standard for many a marketer for understanding customer experiences because participant credibility is guaranteed and it provides one of the best means for answering the hard questions that run-of-the-mill surveys cannot. For instance, measuring and comparing sensory experiences (taste, smell, touch, etc.), unraveling cognitive reasoning & decision making and surfacing those implicit, non-obvious reasons underlying “Why?”

But herein lies a silver lining to the current Covid-19 challenges mentioned above: it’s raising awareness that while they can yield high value insights, the mechanics of these in-person tests remain woefully outdated, slow, expensive and cumbersome. Not to mention, these tests are fraught with multiple biases such as regional bias, small sample size bias, group bias and facilitator bias as detailed in a previous post.

Now you might assume that it should be relatively straightforward to leverage the ubiquitous mobile phone to replicate qualitative testing. Alas, when you start digging into the details you discover a large spectrum exists between the promise vs. the reality of online / mobile methodologies and, consequently, the resulting output can vary extensively which in turn can pose significant risk to your business decision making. As such, having spent years measuring consumer experiences with hundreds of brands, the focus of this post is to provide you with crisp tips and suggestions on how to not only pandemic-proof your qualitative testing for the short term but also how to increase the speed, quality, credibility, and cost effectiveness of your entire consumer insights strategy (including both qualitative and quantitative) for the long term.

I. Ensuring Respondent Quality

What to Avoid: The core promise of qualitative testing is the assurance that you’re interacting with credible, engaged consumers in meaningful ways. Of course, when this testing is done in-person, you have 100% certainty that you’re engaging with real people. However, when those same questions are posed online, that certainty can quickly disappear. The brutal reality is that there’s a huge number of “survey speeders” out there who mindlessly race through your questions — choosing random answers for closed-ended questions and providing effectively useless one liners or totally incoherent responses for open ended questions — so they can maximize point collection as quickly as possible. You know you’ve been duped when you get back results from a large “qualified pool” of respondents but the actual percentage of open ended responses you can utilize is just 20% and of those maybe half, if you’re lucky, truly include cogent responses that surface valuable insights. Applying some simple math, that means that for every 100 respondents, you’re getting just 10 quality responses to your open ended qualitative questions. The risk this poses to your business decision making is totally unacceptable.

What's Recommended: Choose only platforms that incorporate technology smarts for weeding out survey speeders and others who try to game the system. These platforms will employ multiple means to not only verify that your participants are real people who actually reside in your target country (e.g. through mobile phone and geolocation verification) but they will also employ sophisticated means to continuously measure the quality of respondent responses, returning to you only responses that have extremely high statistical likelihood of being from real people who were actually engaged with the subject matter at hand. When done right, instead of getting back 10% quality for your open ended responses, you will get back closer to 100% quality along with the confidence of knowing you can easily audit respondent credibility for yourself. Additionally, these technologies will safeguard your data against bots, a rapidly emerging data pollution challenge that's been recently covered by Wired, New York Magazine, and CBS news, to name but a few.

II. Ensuring Brain Engagement

What to Avoid: It never ceases to amaze how some still think it’s ok to ask scores of questions in a single survey. Eat your own dog food by first taking it yourself and checking not only whether you can realistically finish it in your estimated time but would you want to? And while fewer questions are always a best practice for maximizing completion rates, unless you also effectively and actively engage the minds of your participants, you’re destined to get back mindless drivel. Said another way, cognitively disengaged people will just parrot back whatever you show them revealing little or nothing at all in the way of the deeper insights you seek.

What's Recommended: Choose only platforms that are media rich both in terms of (1) the content they're able to present to participants and (2) how they empower participants to easily speak their minds. Both of these attributes are essential for maximizing cognitive engagement. Looking first at content presentation: Make sure that the platform you choose makes it easy and intuitive for participants to watch videos and view images in addition to presenting various visual questions. More cutting edge platforms will extend content engagement to include physical product interactions via shopping (in-store or online), sensory measurements via shipped products, virtual environments, websites / web pages and 3rd party apps. Turning now to empowering opinion engagement: Ensure that the platform can enable participants to easily and authentically verbalize their thinking — either through audio or video capture. In our experience, audio is especially powerful in that it’s the most comfortable and natural for the majority of us in that we’re free to speak our minds as we would to a friend but we’re not “on camera”. That being said, "front of camera” video can often be quite effective in online panel engagement and “back of camera” video can be ideal for capturing out of box experiences, mapping step-by-step experiences and unraveling physical product experiences. Finally, there's still value in enabling open ended text entry as this proves most effective in loud environments such as at live events, conferences and festivals.

III. Ensuring Actionable Insights Insights

What to Avoid: Beware of online solutions that only speak to the capture side of the equation. Capturing participant responses via video or audio to your open-ended questions can effectively surface in-the-moment thinking and decision making but unless that solution also comes with the intelligence to automatically give structure and meaning to every word that’s being spoken, you can quickly end up with a mountain of raw data that is impossible to make sense of in a short period of time. Some simple math will serve well to illustrate the point: Say you’re using video interviews to understand how consumers choose between your brand and a competitor. Let's say each participant first views one of two products (presented in random order) in isolation, voicing likes and dislikes. Then that participant views the other product and does the same for it. Finally, you present both products side by side and ask which they would buy and why. In total, that's 3 open ended responses per participant. Now assuming that on average each participant provides a 60 second response for each of the 3 questions, that’s 3 minutes of video per participant. If you engage 200 participants that’s 600 minutes or 10 hours of video! Going through all that video manually is not only time inefficient, it can fall prey to reviewer bias.

What's Recommended: Be sure to select a solution that leverages natural language processing and other machine learning capabilities that have the heavy lifting power to quickly make sense of what’s being said so that all the raw data can be transformed into actionable insights such as recurring themes, thematic patterns and cognitive hot spots. The ideal solution will also give you a multi-dimensional view of the data, letting you slice and dice it not only by context and sentiment but also by quantitative dimensions such as demographics (gender, age, etc.), behavioral, psychographic and other closed-ended questions you’ve asked. Importantly, you want to ensure that the raw data is not rolled up to such a high level that you lose the ability to zero in on interesting lower level data points. Specifically, you want to retain access to the response content so that you can easily replay especially insightful videos or soundbites that best exemplify particular cohorts, letting you (and other stakeholders) feel first hand their words, their reasoning and their raw emotion.

About Qbit

I founded Qbit five year ago out of Silicon Valley along with my business partner Ravi Dwivedula. We previously held leadership roles at the Neat Company where we were deeply involved with understanding users' experiences with the company’s consumer and business products. Qbit was born out of our repeated frustrations with slow, cumbersome and costly in-person research. Since then Qbit has been granted multiple patents for its technologies with additional patents pending.

Our mission at Qbit is to help brands grow market share by providing the fastest and most credible way to understand consumer decision-making. Trusted by national brands and public companies, Qbit has crunched millions of spoken words by consumers about their experiences with hundreds of brands. By capturing real-time experiences via audio or video, Qbit ensures 100% authenticity and verifiability. Qbit’s Android and iOS apps empower brands to engage consumers anywhere (virtually, in-store, at home, etc.) and at any point along the customer journey. Qbit's AI rapidly crunches open-ended data into crisp actionable insights based on both context and sentiment, empowering brands to get fast feedback to any business question.

PANDEMIC PROOFing YOUR CUSTOMER INSIGHTS