Debiasing Social Impact Investments with Custom Psychometrics

There are thousands of people around the world with big ideas to foster global change. Unfortunately, a lack of funding leads many to give up on their dreams of helping others. 

This is what the Skoll Foundation set out to change when it was founded in 1999 by philanthropist Jeff Skoll, the creator of eBay. Following their mission to catalyze transformation social change, The Skoll Foundation has invested over $1 billion worldwide into social entrepreneurship. 

Investments by Skoll have enabled social innovators to launch groundbreaking projects across a wide range of sectors. Their work is helping to move the needle on some of the most pressing issues facing society today, from inclusive economies, to racial justice, to climate action. 

But distributing that funding is no small feat. Each year, Skoll must sort through hundreds of applications for grants, necessitating some tough decisions. Who is most deserving of funding? Who is likely to have the largest impact? In order to carry out their mission with integrity and justice, it’s crucial that the selection process be as unbiased as possible.

Luckily, de-biasing organizational processes is one of our specialties at TDL. We were asked to review Skoll’s existing tools for selecting funding opportunities and develop new tools to identify bias at all levels of the decision-making process. 

We started with a thorough analysis of Skoll’s existing practices, working closely with staff members to identify key themes. Following an in-depth review of the Skoll Award for Social Entrepreneurship (SASE) materials, we designed a questionnaire to investigate 4 dimensions of the process: sourcing, screening, selection, and general considerations. This spanned the entirety of the reward process, from targeting social entrepreneurs to add to the candidate pipeline to lasting impressions of the selected award-winners. 

Our surveys revealed that a majority of the staff felt there was some degree of bias throughout the SASE award process, based on factors such as candidate gender and the organizational position of their referrer. While these can be relevant factors — after all, people tend to care about issues that most directly affect their communities — the team didn’t want gender and referrals to determine who received funding.

In order to dive deeper into the team’s decision-making, we designed a computer-based experiment to shine light on subconscious bias in the awards process. Since many sources of bias are unconscious, they need to be detected using experimental methods like Implicit Association Tests (IATs).

IATs are a widely used methodology that requires users to categorize two target concepts with a given attribute as quickly as possible, while measuring their response times. We created and deployed two bespoke Implicit Association Tests for Skoll to gauge biases arising from the personal and professional backgrounds of staff members. 

The two IATs were designed to measure gender bias and bias from employees’ focus area — such as a bias for funding global health projects when the team has a strong medical background. We asked employees to partake in these IATs to get a sense of which areas harbored existing bias.

These IAT responses gave us a solid understanding of the existing bias in the award selection process – with an evidence-based foundation of knowledge, we were ready to implement concrete change to further debias the organization. 

We leveraged behavioral change frameworks to translate our insights into actionable solutions, targeting bias throughout the process. The most appropriate for the project at hand were the COM-B Framework and the MINDSPACE Framework, two mainstays of applied behavioral science focused on changing behavior.

Both structural factors and behavioral factors can reduce objectivity in selection processes, so we kept an eye out for both in our research. Based on our empirical findings and questionnaire responses, we identified 2 structural barriers (such as language bias) and 9 behavioral barriers (such as referee influence) to evidence-based decision-making in the selection process.

To tackle these barriers, our team developed 14 design solutions to implement into the SASE decision-making protocol. These ranged from creating region-specific applications in local languages to post-application feedback structures to improve objectivity on their selection panels. Our targeted solutions were based not only on best practice, but on customized research conducted on Skoll employee decision-making.

Bias is a deeply ingrained part of our decision-making processes, particularly in organizations. By using evidence-based practices and a behavioral approach, we can make big decisions — like how to distribute high-stakes funding — more fair for applicants. We believe no one should miss out on realizing their social impact due to unfair selection processes. 

Since our work with Skoll, the organization has distributed an additional $472 million worth of grants. As the group continues to change the world with their funding initiatives, we’re proud to have helped the team better distribute millions worth of social investments.