Foreword
At TDL, our role is to translate science. This article is part of a series on cutting-edge research that has the potential to create positive social impact. While the research is inherently specific, we believe that the insights gleaned from each piece in this series are relevant to behavioral science practitioners in many different fields. As a socially conscious applied research firm, we are always looking for ways to translate science into impact. If you would like to chat with us about a potential collaboration, feel free to contact us.
Introduction
Behavioral science insights can profoundly impact education outcomes — from improving curriculums to making more effective teachers. As a socially-conscious applied research firm, TDL is interested in using empathy, technology, and design-thinking to promote better outcomes in many aspects of society, from health to education to the economic empowerment of disadvantaged groups. To amplify these impacts even further, we reach out to experts currently conducting research in areas that engage behavioral science in the pursuit of socially conscious goals.
In this spirit, we reached out to Dr. Ryan Yeung to understand how his work at in education policy harnesses behavioral science to improve society.
Ryan Yeung is currently the Replication Evaluation Specialist in the Accelerated Study in Associate Programs (ASAP) unit at the City University of New York (CUNY). He has previously taught at Hunter College, SUNY-Brockport, and Rutgers University-Camden. He has a Ph.D. in Public Administration and Policy from Syracuse University. His main research interests are in education policy and public budgeting, and finance.
For this project, Ryan and his co-author evaluated a program to increase enrolment and success in college through texting.
A full version of his project is available here:
https://www.tandfonline.com/doi/full/10.1080/15391523.2019.1683105
Interview
Nathan: How would you describe the focus of your research in simple terms?
Dr. Yeung: We examined how a group of New York City public schools used texting to increase enrollment and persistence in college. The text messages included tips and reminders to fill out the Free Application for Federal Student Aid as well as college tips. The text messaging program was theoretically and empirically grounded in the behavioral science research on nudges, inducements to people to make a specific choice that is in their best interest, for example, filling out the FAFSA after receiving a text message or going to a professor’s office hours.
Nathan: How would you explain your research question to the general public?
Dr. Yeung: We had several research questions in this study. 1) What impact did texting have on college enrollment and persistence? 2) What impact did texting in high school have on college enrollment and persistence? 3) How did response rates to text messages affect enrollment and persistence? 4) How can we improve response rates if we find response rates do indeed affect enrollment and persistence?
Nathan: What did you think you’d find, and why?
Dr. Yeung: There’s a long literature that suggests text messaging programs have had positive impacts on postsecondary outcomes, so we were expecting to find a positive effect of the program itself on college enrollment and persistence. We weren’t sure if the timing of the messages (if they started in college or in high school) had much of an effect. If you think the messages do matter, it is natural to think that responses to the messages matter as well. Responses indicate the students have read the message and felt the need to ask for help.
Nathan: What rough process did you follow?
Dr. Yeung: We created three matched groups, students who received text messages in high school and college, students who received messages only in colleges, and students who did not receive messages at all. These groups were matched to create similar groups of students based on demographics and academic achievement in each group. We then compared how each group performed in terms of college enrollment. We used the same matching procedure to estimate the effect of response rates on enrollment and persistence. Last but not least we ran a multiple regression to predict how different types of messages and demographics affected response rates.
Nathan: What did you end up finding out?
Dr. Yeung: Our multiple regression results suggest that although when the texting began did not appear to matter, the texting program increased enrollment and persistence in college. In addition, response rates were positively associated with enrollment and persistence. Finally, we find that the content of the messages themselves as well as individual characteristics affected response rates.
Nathan: How do you think this is relevant to an applied setting (i.e. in business or public policy)?
Dr. Yeung: Text messaging programs do appear to have positive effects on postsecondary outcomes at a reasonably low-cost, but there are things that can be done to improve their efficacy. Personalization matters. Students were very likely to respond to messages wishing them a happy birthday. Perhaps there are other ways to personalize the messages in the future. Timing matters. You want to send messages that are relevant for students at the specific point in the school year during which they are sent (like during FAFSA time). We also recommend sending out messages early afternoon or on a weekend when students are most likely to see them.
Nathan: Do you see future research stemming from your study? In what directions?
Dr. Yeung: We acknowledge that our study was not a randomized experiment and is instead a quasi-experimental design. Although the results used matched data, we cannot completely rule out potential bias from unobserved factors. An ideal scenario would be an experiment where students are randomly assigned to receive or not receive texts and randomly assigned to when they receive texts (while in high school v. post-high school). The content of the messages may also be randomized to get a handle on the influence of various types of messages. This would have been the best method for controlling for unobservables. Perhaps a future study could use such an approach. Randomization of the content of the messages including personalization would also be helpful to isolate the effects of message types on college outcomes.
