Beyond the Checkbox: Redesigning Surveys for Reliable Behavioral Insights
Surveys are often considered a goldmine for consumer research. They promise insights and predictions about customer behaviors, preferences, and mindsets. But not all surveys are created equal. In fact, some surveys may be steering you in the wrong direction entirely. Why? Because consumers are unique, wacky, creative, illogical, and forgetful. That is, they’re human. Basic surveys often struggle to account for this human-ness, resulting in simplified insights that don’t represent the end user.
So how can you ensure that your research is providing you with valuable insights and not fool’s gold? And what “bias-reduced” methods can you leverage to reliably understand and predict consumer behaviors? Today we’ll unravel some of the biases present in traditional surveys and introduce three methods that can offer deeper insights into the baffling nature of human behavior.
The “human-ness” of answering simple questions
Think back to your last doctor’s appointment and the ever-troubling questions of: “How much do you exercise? Do you drink enough water? How many hours of sleep do you get a night?” If you’re a fitness buff, water enthusiast, and night-time guru, then you may breeze past these questions without a second thought. But if you’re anything like me (and countless other regular folks), you may be inclined to lie, just a little. Maybe you’d say you sleep eight hours instead of your actual six. Or, you’d say you exercise every day for 30 minutes instead of your actual 10-minute walk to and from the subway.
The comforting (yet perhaps troubling) reality is that you’re not alone in wanting to stretch the truth. The response bias is a commonly known phenomenon not just in the doctor’s office, but also when conducting surveys. Our tendency to respond to questions in a way deemed “socially acceptable” leads us to exaggerate positive behaviors and minimize negative ones.
Wanting to “look good” in surveys is not the only human thing we do that impacts potential responses. There are countless other cognitive biases that sway consumers to report behaviors different from those they actually practice. They may experience choice overload when deciding between multiple options, encounter the optimism bias when predicting their future behavior, or face decision fatigue when answering an onslaught of survey questions.
What this means for survey-based research
So what does this mean for our beloved surveys? Should we unsubscribe from our favorite survey platforms or toss our research out the window? Of course not! We just need to be conscious about how we design our research so that we can minimize biases and maximize our understanding of consumers’ true behavior. Specifically, we need to consider ways that we can assess behavior beyond classic multiple-choice or select-all driven surveys.
Behavioral Science, Democratized
We make 35,000 decisions each day, often in environments that aren’t conducive to making sound choices.
At TDL, we work with organizations in the public and private sectors—from new startups, to governments, to established players like the Gates Foundation—to debias decision-making and create better outcomes for everyone.
Our favorite bias-reduced survey methods
We’re going to explore three research methods that maximize our ability to understand actual behavior. Most importantly, these methods can all be easily deployed through an online survey platform! Although real observation is ideal, we know it’s often impractical – whether you’re constrained by time, geographic location, budget, or language.
There are, of course, many different ways to enhance survey design. These are just some of our favorite “bias-reduced” methods here at TDL! By using best-worst scaling, tasks and scenarios, and game-based design, we aim to reduce common biases to tap into insights that are reflective of real-world behaviors.
Best-worst scaling
In best-worst scaling, respondents compare different options and pick the top and bottom of the pile. (Think of this as sorting favorites from the not-so-favorites.) Options are then repeatedly randomized and shown in different combinations. It’s a pretty straightforward way to quickly reveal consumer priorities.
Superpower: Provides a prioritized list of options based on consumer preferences and aversions.
Overcoming Bias: Reduces choice overload by simplifying complex preferences to simple options.
Survey Upgrade: Transform ranking questions into a best-worst analysis. After all, who among us can really rank 6+ different options?
TDL in Action: Best-worst scaling is one of our go-tos when weighing options for a new behavioral intervention, or ranking features for a website or mobile application.
Tasks and scenarios
Why would you ask a participant what they would do when you could actually get them to do it? For this approach, participants complete pre-set tasks while you capture information like click distributions or selected buttons. For example, participants can navigate to specific pages on your website, find the answer to a question, or interact with an application, all while you measure their behavior!
While traditionally done for interactions with products or digital services, this method can also be used more creatively by having your participants imagine themselves in a scenario before answering a series of traditional survey questions.
Superpower: Assesses firsthand how people would behave in a given scenario – no guesswork involved.
Overcoming Bias: Sidesteps the optimism bias. It's about actions, not just well-meaning intentions.
Survey Upgrade: Replace questions about product interaction (e.g., “How easy is it to find the red button”) with specific tasks (e.g., Find the red button) and metrics (e.g., Time to find the red button).
TDL in Action: Task-based questions are heavy hitters when analyzing navigation behavior on digital interfaces.
Game-Based Designs
Game-based designs transform traditional survey questions into interactive experiences. Participants might progress through levels, earn points, or face challenges that are directly linked to overall research questions. Want to know how someone would spend their money on the stock market? Or how they would design their ideal workout routine? Gamify it!
Superpower: Captures in-the-moment experiences and behaviors.
Overcoming Bias: Eliminates decision fatigue by creating an engaging experience. Let’s be real – we’d all rather do a “which animal are you?” quiz instead of answering redundant questions on a basic survey design.
Survey Upgrade: Create simple games or quizzes as substitutes for hypothetical questions about preferences or behaviors.
TDL in Action: Game-based designs are a known and loved tool for some of our finance-related experiments!
Revamping your research
While surveys have their limitations, they’re not inherently flawed. By leveraging principles of behavioral science to select research methods, we can develop surveys that more accurately reflect real, human behavior.
As you delve into your next round of research, consider: how easy is it for someone to stretch (or forget!) the truth while answering your survey? Are you asking your consumer to predict their future behavior? If so, what bias-reduced methods could work for you?
And of course, drop us a line! Do any of these methods speak to you? Have you tried any of them and had success? Are you looking for support to implement these (or other) bias-reduced research methods? We’d love to hear from you!
About the Author
Alexi Michael
Alexi is a Consultant at The Decision Lab. Her expertise is multidisciplinary, spanning the fields of social innovation, artificial intelligence, and human-centered design.
