The Pitfalls of Behavioral Science: How to avoid untrustworthy data
There is a crisis in the field of behavioral science
The field of behavioral science has grown from obscure to mainstream. Like anything, it’s had growing pains. While the discipline has taken huge strides from its origins, the current crisis is one of evidence.
What does an evidence crisis look like? Well, some behavioral studies only use a narrow range of stimuli to deduce their findings. Others neglect whether their stimuli are generalizable to the real world, outside of clinical trials.1 And others still may have used methods of measurement that are later proven invalid.
Publication issues in behavioral science
Research paper retractions
The media star of the newest behavioral science crisis is the retraction of research papers. Retractions can occur for a number of reasons, including:2
- Mistakes (whether honest or careless) in the data collection process or research methods
- Results that cannot be replicated
- Misconduct, e.g. fabricating results
Retractions in behavioral science have been on the rise - with a faster growth rate than other scientific publications.5 Before their retractions, these studies may have been used as evidence to implement interventions or guide further studies.
The optimistic interpretation: it’s plausible that the reason more articles are being retracted is due to a greater emphasis on flagging errors, misconduct, and faulty research practices in recent years.
Behavioral Science, Democratized
We make 35,000 decisions each day, often in environments that aren’t conducive to making sound choices.
At TDL, we work with organizations in the public and private sectors—from new startups, to governments, to established players like the Gates Foundation—to debias decision-making and create better outcomes for everyone.
Cumulative retractions of science research papers from 2010 to 2018.
Grant culture
In the academic world of grant culture, universities award researchers in line with how much grant money they bring in. Unfortunately, an emphasis on grant money doesn’t necessarily mean the results of the studies will be favored equally by university administrations.3
These environments can be incubators for questionable research practices: some researchers may prioritize quantity of grant money over trustworthiness of results if the two are at odds.3
Lack of generalizability
Historically, behavioral scientists tend to conduct research on one subset of the population - the most convenient being undergraduate students in Western countries.
Behavioral research has a WEIRD problem - we grossly oversample populations that are Western, Educated, Industrialized, Rich, and Democratic.4 Focusing on this subset not only creates bias, but leads to misinformed generalizations when applied to non-WEIRD populations - i.e. the majority of the world population.5,6,4
Lack of replicability
Journals are less likely to accept failed replicability studies - so researchers are less likely to submit them.7
On top of that, labs conducting replicability studies are likely to be the same ones that published the initial paper. This puts them at higher risk of confirmation bias and other questionable research practices.8
Publication bias
Similar to the lack of incentive to publish replication studies, negative or statistically insignificant results are more likely to be ignored by journals.9,10 Researchers are incentivized by their affiliated institutions to be published, and this pressure can lead to the fabrication of false positives or the overestimation of effect sizes.11,12
Publication bias can also emerge given researchers might not want to submit failed replicability studies to journals, and because journals are less likely to accept them if they are submitted.7
There are promising strategies for practitioners
There are immense societal benefits when implementing findings from valid behavioral research. To responsibly implement findings from the field, practitioners should:
- Bring on an expert team that is familiar with behavioral science concepts
- Push for readiness frameworks (just like we see in aeronautical engineering and genetic research)
Just because someone is a behavioral science expert doesn’t mean that they’re familiar with all behavioral science concepts. Though a researcher might have strong analysis skills, their own expertise is likely narrow.
This can be addressed by implementing behavioral science interventions with a team, rather than a sole behavioral scientist.
Effective ways to address uncertainty in behavioral science
Hire a consultancy
Whereas researchers are incentivized to publish, consultancies are incentivized to help clients.
Behavioral science firms are motivated to conduct research that avoids biased or unreplicated studies, in order to create impactful results and maintain their reputation for clients. In the same way that you wouldn’t start working on a space rocket without hiring engineers, it’s a wise strategy to bring on a behavioral science expert to help you avoid any pitfalls.
Push for the use of behavioral readiness frameworks
When the field of genetics was first created, it experienced many of the same crises as behavioral science. However the field has since devoted a significant amount of time developing research workflows, data harmonization, and various other methodologies that improved the accuracy of their measurements.13
Readiness frameworks are a way to categorize the quality of evidence in a given field. They can take many forms, sometimes presented as a scale from least to most ready, or as a flow chart that contains all the necessary factors for success.
Readiness frameworks have buoyed plenty of fields with evidence quality dilemmas. A big proponent of readiness frameworks in aeronautical engineering is NASA - they created TRLs (technology readiness levels) to systematically appraise the quality of evidence used by their teams.14
A proposed model for psychological research, similar to those in aeronautical engineering and genetics, comprises 9 evidence readiness levels (ERLs). It ranges from 1 (stakeholders and researchers have defined the problem) to 9 (the final solution can successfully address a crisis).13
In the same way we wait to launch spaceships into orbit until they’re as close as possible to 100% accuracy, we should push for the same level of rigor in behavioral science in order to mitigate the variety of aforementioned issues. When you hire a behavioral science consultancy, ensure they use studies high on the readiness scale.
Reasons for concern, reasons for encouragement
While issues like grant culture and publication bias are reasons for concern, there are ample opportunities to ensure that your decisions are informed by the best research there is to offer. The development of the PSA, coupled with behavioral science readiness frameworks and the existence of trustworthy behavioral science consultancies, shines hope on the future of the field.
The Decision Lab is a behavioral consultancy that uses science to advance social good. We work with global brands that are at the forefront of technology and innovation, using best research practices to provide robust behavioral insights. If you'd like expert assistance leveraging behavioral science for your business, contact us.
References
- Judd, C. M., Westfall, J., & Kenny, D. A. (2012). Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely ignored problem. Journal of Personality and Social Psychology, 103(1), 54–69. https://doi.org/10.1037/a0028347
- Hantula, D. A. (2019). Editorial: Replication and Reliability in Behavior Science and Behavior Analysis: A Call for a Conversation. Perspectives on Behavior Science, 42(1), 1–11. https://doi.org/10.1007/s40614-019-00194-2
- Lilienfeld, S. (2017). Psychology’s Replication Crisis and the Grant Culture: Righting the Ship. Perspectives on Psychological Science, 12(4). https://journals.sagepub.com/doi/full/10.1177/1745691616687745
- Henrich, J., Heine, S. J., & Norenzayan, A. (2010b). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83. https://doi.org/10.1017/S0140525X0999152X
- Heine, S. J. (2010). Cultural psychology. In Handbook of social psychology, Vol. 2, 5th ed (pp. 1423–1464). John Wiley & Sons, Inc. https://doi.org/10.1002/9780470561119.socpsy002037
- Henrich, J., Heine, S. J., & Norenzayan, A. (2010a). Beyond WEIRD: Towards a broad-based behavioral science. Behavioral and Brain Sciences, 33(2–3), 111–135. https://doi.org/10.1017/S0140525X10000725
- Pashler, H., & Harris, C. R. (2012). Is the Replicability Crisis Overblown? Three Arguments Examined. Perspectives on Psychological Science, 7(6), 531–536. https://doi.org/10.1177/1745691612463401
- Wiggins, B. J., & Christopherson, C. D. (2019). The replication crisis in psychology: An overview for theoretical and philosophical psychology. Journal of Theoretical and Philosophical Psychology, 39(4), 202–217. https://doi.org/10.1037/teo0000137
- Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82(1), 1–20. https://doi.org/10.1037/h0076157
- Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638
- Lane, D. M., & Dunlap, W. P. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31(2), 107–112. https://doi.org/10.1111/j.2044-8317.1978.tb00578.x
- Nuijten, M. B., van Assen, M. A. L. M., Veldkamp, C. L. S., & Wicherts, J. M. (2015). The Replication Paradox: Combining Studies can Decrease Accuracy of Effect Size Estimates. Review of General Psychology, 19(2), 172–182. https://doi.org/10.1037/gpr0000034
- IJzerman, H., Lewis, N. A., Przybylski, A. K., Weinstein, N., DeBruine, L., Ritchie, S. J., Vazire, S., Forscher, P. S., Morey, R. D., Ivory, J. D., & Anvari, F. (2020). Use caution when applying behavioural science to policy. Nature Human Behaviour, 4(11), 1092–1094. https://doi.org/10.1038/s41562-020-00990-w
- Technology Readiness Level. (2021, April 1). NASA; Brian Dunbar.http://www.nasa.gov/directorates/heo/scan/engineering/technology/technology_readiness_level
About the Authors
Lindsey Turk
Lindsey Turk is a Summer Content Associate at The Decision Lab. She holds a Master of Professional Studies in Applied Economics and Management from Cornell University and a Bachelor of Arts in Psychology from Boston University. Over the last few years, she’s gained experience in customer service, consulting, research, and communications in various industries. Before The Decision Lab, Lindsey served as a consultant to the US Department of State, working with its international HIV initiative, PEPFAR. Through Cornell, she also worked with a health food company in Kenya to improve access to clean foods and cites this opportunity as what cemented her interest in using behavioral science for good.
Sekoul Krastev
Sekoul is a Co-Founder and Managing Director at The Decision Lab. A decision scientist with an MSc in Decision Neuroscience from McGill University, Sekoul’s work has been featured in peer-reviewed journals and has been presented at conferences around the world. Sekoul previously advised management on innovation and engagement strategy at The Boston Consulting Group as well as on online media strategy at Google. He has a deep interest in the applications of behavioral science to new technology and has published on these topics in places such as the Huffington Post and Strategy & Business.
Sarah Chudleigh
Sarah Chudleigh is passionate about the accessible distribution of academic research. She has had the opportunity to practice this as an organizer of TEDx conferences, editor-in-chief of her undergraduate academic journal, and lead editor at the LSE Social Policy Blog. Sarah gained a deep appreciation for interdisciplinary research during her liberal arts degree at Quest University Canada, where she specialized in political decision-making. Her current graduate research at the London School of Economics and Political Science examines the impact of national values on motivations to privately sponsor refugees, a continuation of her interest in political analysis, identity, and migration policy. On weekends, you can find Sarah gardening at her local urban farm.
