RE-STOKED: When Not to Love Bias (11/14/14)

We frequently find clients and colleagues protesting the use of improvisational, collaborative, or small-sample approaches to design research and hypothesis testing. Aside from the usual complaints about qualitative approaches (takes too long, costs too much, people lie), we find that people are really, really worried about researcher and respondent bias. And too often we joke, as we did last week, about learning to love bias.

But sometimes bias, especially the kind that the researchers deny or are willfully unaware of, is downright destructive. In this piece, Zeynep Tufecki looks at the role of bias in the Hollaback video about street harassment, and reminds us of a very important truth about research design: “every element of research design requires hard thinking”.

With access to cameras, social media, Google, and the occasional syndicated research report, we see brand and product strategists losing some of the social science skills needed to design research (and do design research) well.

But even if this "research" was designed unimpeachably, there could still be multiple explanations for what we see in this video (about street harassment, location, and race). In fact, there almost always are multiple explanations for any set of data - but it's our job as strategists and researchers to make good sense of it. To do this, we need and underlying theory.

To find the right theory is, like everything, an iterative process - one that starts with how we collect data, and never stops, through the selection of data and the interpretation of it. We have to look at many possible designs, many possible selection sets, and many possible explanations to find the best ones.

Designing good and useful research, in other words, isn't all that different from designing good and useful products.

Market Research, Not Marketing Research

I started The Difference Engine for one reason. After 10 years in brand consultancies that based their recommendations on qualitative and quantitative research, I came to believe that the typical mode of market research is broken. I'd like to help fix it.

When I worked in agencies, this image (insight mining, leading to 'nuggets' that get synthesized into 'insights' that somehow magically become great ideas) seemed to sum up the way we'd talk to clients about figuring out what to do with their marketing budgets.

But much of the time, budgets are set based on what was spent last year; quarterly spending is organized around known product announcements and/or seasonality in the category; and briefings to agencies happen sometime after a lot of debate and politicking, research and consulting. Oftentimes, project briefs are widely abstracted from business objectives. KPIs tend to measure the marketing results, not the business results. They're trying to make their ads have better recall, their brands gain greater stated and derived preference, their brand metrics lift.

In other words, the marketing team and their agencies are solving marketing problems, not business problems.

In the past few years, CMO tenure has doubled to nearly 4 years, driven by more accountability to, and involvement by, CEOs and COOs. This is a positive trend. It suggests that CEOs/COOs and CMOs are beginning to look beyond individual campaigns, taking a longer view. I certainly hope so. But it also suggests that perhaps we're getting back to basics about what marketing is.

The nucleus of marketing, in my opinion, is the interaction between the product/service and the end user. Understanding these interactions is critical to both disruptive innovation (spotting market need gaps and opportunities, anticipating threats from challenger brands and technologies, understanding small shifts in user behavior that could kill your category in a year or two), and to sustaining innovation (adapting to user needs, improving performance/service/experience, scaling, reducing cost/price).

Marketing could be a team that provides a critical conduit to innovation within an organization.

But market research today seems largely limited to two branches of business: the big strategy consultancies who may be able to assist with sustaining innovation; and the marketing research companies that provide research to develop and test messaging and imagery.

Both are useful and necessary, but they are inward-oriented. They look at the business as it is today, or the marketing as it is today, and ask how to optimize efficiency, effectiveness and profitability.

But new opportunities are hard to spot when you're looking only at yourself, your most profitable customers, and your closest competitors. You have to branch out.

Watch & Learn

Most of my clients don't sell directly to their end users. They rely on others to do that. Right away we have two problems.

The first is that it changes my clients' definition of a 'customer' - in fact, this is probably why we use the term 'consumer' to describe a probable end user. Their first order customer is a retailer or wholesaler. Their customer's customer is a 'consumer'. Fundamentally, most businesses are at arms' length from their consumers. This limits the ability to see what consumers are really doing with your products, how they're really using your services... You miss out on all those moments when a business' beliefs about the role its products play in the world clash with or combine with reality.

The second is what happens when you're at arms' length from your consumers - you rely on your most profitable customers to tell you what they need. They need more of whatever is selling well today, whatever fits their current value chain.

How can the brand get closer with their consumers/end users? They can certainly do large surveys (and they do), or use large private panels for quant-sized qualitative exercises (and they do), or do focus groups with their core consumer segments (and they do). But many times, these surveys are merely proxies for asking their biggest customers what they want. It tends to reinforce doing more of whatever you're doing today.

[By the way, this is not bad! You need your biggest, most profitable, most frequent consumers and customers. They pay for your products and services and deserve your respect and attention.]

But if you want to learn what to do next, who your next market will be and what they'll need, where the strategic threats are coming from, how technologies can transform your category... You might need to get off the beaten path, get personal, and watch as well as listen.

Behavior is a terrific source of inspiration. How people are hacking products to their own purposes, why people move on from a preferred brand to something else, when people would rather a small, half-baked solution to a full-featured one... These are the places where opportunities so often lie. And to meet these people we have to talk to your smaller consumer segments, or the segments that are large but neglected - even people who might otherwise be called 'rejectors'.

Less Talk, More Frequently

There can be immense value in convening a group of consumers in a room for a few hours to talk about ideas and get feedback on a brand. This is the purpose of a focus group. Brands should do them.

There is a lot of utility in getting a sense check of a creative campaign in 45 minute in-depth interviews with consumers. You can avoid (those over-hyped bogeymen) Group Think and The Dominant Respondent. You can gain intimacy and trust, and detail. By all means, do them.

But when you're looking for new product or service or segment opportunities, you should consider a lighter-weight approach to consumer research. Shorter, more focused conversations and contextual observation of behavior, repeated over time, can provide both the spark and the fuel to the innovation fire.

To do this well, we obviously can't also incur the costs of traditional research facilities. We can't afford two weeks' recruitment time for each conversation. And we can't be too precious about the mode of the interaction. Every conversation doesn't need to be the same. They can be phone calls, or Skype video calls, or visits at home, or meetings at the office. We can even invite them around to ours. Stimulus can be paper prototypes or working betas or finished products. Discussion guides should be much looser, allowing people to do what they would naturally rather than being led by the moderator.

This is a recipe for continuous feedback throughout the life of an innovation effort. Certainly I believe that innovation and R&D should be ongoing efforts; but they should also be focused on specific hypotheses to be tested.

Rather than separating UX testing from any other kind of research, I place a premium on understanding the user's experience from all angles. Certainly, which button they'll press and whether they understand the product, and how they navigate a retail environment or a website or a mobile app will matter a lot. But there's more to this than click maps and eye-tracking (which, like all other quantitative types of research can tell you how much, how many, when, how often - but not why).

I want my clients to feel more than familiarity with their consumers - I want them to feel empathy for them, and to gain this through real intimacy. Taking away the delays, the one-way mirror, the rigid 'protocols', allows us to keep learning and improving. The goal is to get to know your consumers so well you can anticipate their needs, rather than chasing after them.

Market v. Marketing

I'll keep writing more here about what The Difference Engine does and how it does it \- and why - but for now, I'll close where I began.

I like to say we do market research, not marketing research. We don't test ads.

So what do we do, then? We help you identify market opportunities - gaps unfilled by your competitors, opportunities to do things better, underserved consumer segments, changing consumer behavior, burgeoning competitive threats, and more.

I want to put people back at the center of the equation, right next to the product. It's this dynamic that should inform all the rest - design, price, distribution, messaging, offers, and so on. I want my clients to know what people are doing and what this means for their business and product road maps - not only what they're saying and what that'll mean for their ad campaigns.

I won't just diagnose and describe these things, I work with clients to decide what to do next.

And along the way, The Difference Engine is going to introduce methods and technologies that I think will fix some of what is broken about market research. Watch this space.