The “New Normal” of Qualitative Research

What we found in our competitive review of the market research industry

Especially over the last couple of years, when everything was about, well, coping — just getting on with it — we had sort of lost track of what was happening for the broader market research industry. We saw companies and products come and go, heard about others being bought or merging, and then just got a lot of newsletters and reports about in-person versus remote research.

So we decided to do our own research and conduct a competitive review of the larger market research industry. We’ve been particularly interested in the ways the industry is changing — or not — in the face of both rapid technological and cultural change, and also in the face of public health, economic, and geopolitical crises.

We’ve noticed that qualitative research in particular has been both surprisingly resilient, but also surprisingly resistant to change. The shift from in person research to remote research was the biggest adjustment for many in our industry given the general life switch to all things remote in the pandemic. While the shift to remote didn’t surprise us, the number of researchers and firms who weren’t already well versed in online methods surprised us. The Difference Engine has been primarily remote for the last several years, and while we really like in person research, we’re big fans of remote research. For many reasons.

But we’re still getting asked by clients and partners if it’s ‘better’ to be in person now, if it’s even possible, and this suggests that part of the deep need to return to ‘normal’, a return back into focus group facilities feels like the kind of normal a lot of researchers and research clients alike are hoping for. We also think this belies a sense that they were ‘settling’ for remote over the last couple of years. We don’t think of it as settling at all.

So — we’re here to debunk a few of the biggest assumptions or misconceptions around virtual ethnography, focus groups, interviews and other remote techniques.

Let’s start with the assumptions.

In talking with others and reading through many reports over the last few years, many say “you don’t get answers right away, you still have to wait”, or “it’s not actually cheaper to go digital, you just don’t have to pay for travel expenses”, or “remote requires a level of tech savvy and therefore there is a bias in recruiting only those who know how to use the internet”.

There is some merit to these assumptions, but we feel they’re deeply rooted in a time where remote interaction was the exception and not the rule. The pandemic has changed that forever. The new normal is remote, and in the future we’ll want to be more deliberate about when we want to be in person.

Time

One of the oldest complaints about remote research is the idea that ‘you still have to wait’. As so many things moved into digital platforms, it seemed many research buyers — and some researchers, too — believed that research could happen automatically, and fast. There are a lot of tools now that promise that kind of immediacy, but they focus on different parts of the research project lifecycle. So, you can recruit in an hour or two using tools like Respondent.io and UserInterviews.com; you can push surveys to your customer panels and get results in minutes or hours using tools like SurveyMonkey, TypeForm and Suzy; you can complete dozens of ethnographic diaries in a matter of days instead of months of in-person observation. Increasingly, these platforms offer first run analysis tools where they automatically generate charts, graphs, word clouds, heat maps and other visualizations of the data you collect. All of this is welcome, but let’s talk about what still takes time:

• Research objectives — what are we researching, with whom, and why
• Research design — choosing a method, setting up tools and platforms, designing activity guides or questionnaires, creating stimulus for responses
• Sample design — identifying the qualifying characteristics of the people you want to conduct your research with, and selecting the best source for that sample
• Data coding and analysis — while the tools provide some level of automation, including in filtering, segmenting, and in some instances AI enabled transcripts, humans still have to create codes for open-ended or unstructured data, and a lot of that is still done by hand
• Sense-making and report writing — even if the platform automagically creates some charts and graphs for you, especially with qualitative research, we need to bring to life the research through a combination of straight reporting, storytelling, and illustration of ideas through imagery, clips, verbatims, and data visualizations

Cost

The second objection since the late aughts has been that just because research is digital it doesn’t always mean it’s cheaper. We think this has become somewhat less of a concern over time but we’d like to take a moment to talk about the main drivers of cost:

• First the savings: remote research means you’re not paying for researcher travel and travel time. Remote research platforms are significantly less expensive than focus group facility rentals.
• The things that didn’t change: Incentives and recruiting fees. Sometimes we can recruit for less money if the sample we’re looking for is likely to be found on a recruitment platform like UserInterviews or Respondent. But if you’re looking for specialized, low incidence populations, or want to cast very specific profiles, you’ll likely need a blended approach of automatic tools and traditional recruiting — or you may still need the expertise and network of an experienced, specialized recruiter.
• Some things feel like they got more expensive: Increasingly, our clients are choosing to use digital diaries (digital ethnography) instead of traditional interviews or group discussions. The digital platforms enable us to engage dozens of people in ethnographic research. In-person, we make choices about how many days of fieldwork we can afford versus the number of interviews we can complete each day. So, while you might be in digital ethnography fieldwork for a third of the time of traditional ethnography, you may have 3–10 times as much data to collect, parse, code and analyze as you would through traditional ethnographic methods.

Going Deep

There are other concerns as well. Perhaps the most common is the feeling that it may be harder to ‘go deeper’ with digital methods because they are asynchronous, self-guided, or suffer from the usual glitches of cross-talk and lag, or because it’s harder to read body language. We have found that active moderation, use of tools like gestures, polls and chat, and multi-modal studies (diaries + interviews, for example) help us around this concern, as well as traditional tools like homework assignments, pre-interviews, and follow-up questionnaires.

Sample Bias

There is also the concern about biased sample — always a concern, no matter the method — but one that traditionally came up as prioritizing the digitally savvy. But after most of us were basically forced to learn how to use remote communication technology over the course of the pandemic, we think it’s fair to say this assumption around bias in recruiting those who know how to use the internet has diminished quite a bit.

In fact, we actually believe you can often get a more representative sample when going digital. Issues of transportation and accessibility are mostly thrown out the window. Those who have social anxiety or prefer to work from the safety of their home can now participate in research more comfortably. And we now have a plethora of tools at our disposal that allow access to multiple research methods — one on one, groups, activities, etc., offering us more flexibility in study design and consumer engagement.

Tech Friction

One problem we do share: Some research platforms require participants to download an app. We agree, we don’t like that route, which is why we’ve been using tools that are browser-based and mobile friendly. Our personal favorite at the moment for conducting remote ethnographic research or digital diaries is Recollective. The range of activities you can put together with it may actually be more than you could in person, including easy to generate visualizations of responses to certain activities, which you can sort by segments or other criteria.

What About the Culture?

As we look back on the industry reports asking practitioners and clients about what changed for them during the pandemic, we do find ourselves curious about the culture of the industry:

• We were surprised, for instance, that European agencies had not adapted as much to remote methods as US-based agencies prior to the pandemic, so this created a challenge. Is this cultural? Are there regional differences in methodological preferences? Do European clients prefer in-person methods? We’d love to understand this more.
• Is this reluctance to adopt multi-modal, remote research methods generational? Will we see more shifts as senior managers are made up more of so-called digital natives? Does industry tenure affect how leaders set up their practices?
• Some methods are easier to scale than others — what makes some companies ‘precious’ about their preferred methods, and some more flexible? Is it all market demand? Personal preference and expertise? Or some other set of considerations?
• How will the shift to remote translate into shifts in our methods? We’ve been more adventurous with methods over time, and are starting to mix modalities more, even in remote interviews or groups, really stretching the limits of what platforms like Recollective, Zoom, TypeForm, dScout, and others can do. Will we see more of this methodological diversity and creativity, or will people stick to traditional study designs?

As we look at the trends shaping the industry — especially for qualitative research practices — we’re going to be keeping an eye on how we lean in to the affordances and the limitations of digital and in-person methods and tools, how we become more deliberate in our choices and more mindful of the participant experience in our study design in order to drive better research outcomes, or alternatively, how some of us revert to our old habits and preferences. How we design our research is just as important as what people say or do in response. This shift to digital should create the conditions for a lot of professional introspection — about how we do what we do, and why we choose those tools and methods, and how they impact the insight and ideas our clients develop and the decisions they make.

We have a lot to share about the broader market research landscape — both in additional blog posts and newsletter installments (subscribe here) and in bigger reports and maybe even a live event! Stay tuned.

Have questions or suggestions about commissioning research studies, setting objectives, selecting methods, designing sample, conducting analysis, or developing useful reports? Please do get in touch. You can reply in the comments here, or send us an email!