Survivorship Bias

The Survivorship Bias is harmful due to how commonly it occurs, and how profoundly the bias can influence our choices. Commonly, this bias is linked to financial decision-making, entrepreneurship, gambling, and medical research. When making decisions in these sectors we must make sure to consider both the successes and the failures” if not, survivorship bias can profoundly impact our perceptions and judgments. Without having all the data needed to make rational decisions, individuals will not be able to make the best possible choice for themselves.

Survivorship bias is everywhere we look, as it is a common bias that affects how we interpret data and information when making decisions. Survivorship bias also affects high-level decision-making, which then results in systemic challenges across multiple disciplines. 

Historical Narratives

It is important to consider how the survivorship bias can impact how we look at history, and thus, how we come to understand our world. Depending on the school, the way information is presented and the materials being used can create bias. The focus on certain groups and their successes across history can diminish the stories and struggles of others. Avoiding the discussions of exploitation can give us an inaccurate picture of how several countries came to be and why certain groups seem to have an unfair advantage in the modern age. Looking at the bigger picture, trapping ourselves in the survivorship bias informs our views on systemic racism as well as other inequalities. In order to drive social progress it is important to look at both the triumphs and the great injustices of history.

Epidemiology 

Survivorship Bias has been found in instances of disease diagnoses, specifically concerning survival rates post-diagnosis. For example, patients with the best prognosis are often those with the lowest risk due to their age, previous health history, and fitness level. The more patients display these positive precursors, the better their survival rates. Because patients with a worse health history, do not always survive, their fatality is not included in survival rate calculations. Meaning, patients are disproportionately represented by healthier individuals who have positive outcomes. What should also be taken into account are individuals who die shortly after being diagnosed or those who die prior to being officially diagnosed. By not being included in survival rate calculations, survival outcome is inflated.⁴

During the COVID-19 pandemic, a huge point of question was the survival rate. Many epidemiologists and doctors warn that publicized calculations do not provide a full picture. Patients who die without being tested for COVID-19 cannot be considered part of the virus’ death count, potentially skewing survival rates. In many countries globally, nations and their healthcare systems had issues keeping up with testing, resulting in potential survivorship bias when looking at data generated from the disease.⁵

A significant portion of marketing campaigns involve testimonials – data that the consumer values highly. We often want to know if a product will work, so we may turn to the “clinical trials” and independently funded studies presented in advertisement campaigns. However, there may be more behind the numbers: a flashy “95% of people saw improvement!” doesn’t always tell the full story. When we aren’t made aware of the full parameters of a study, it is easy to get a biased perspective. It’s always a good idea to double check the rigor of a study. For example, what was the sample size? Did people drop out of the study? How long did they use the product for? All of these questions are important in order to determine its validity. When we take these points into account we are actively working against the survivorship bias. 

Incorporating rich data sets and employing rigorous evaluation methods may mitigate the effects of survivorship bias on AI software. However, artificial intelligence is an ever-growing field and widespread use can cause multiple companies to pitch the next big advancement. While technological progress is exciting, pumping out software too quickly could lead to oversight. 

Specifically, it is important not to overestimate the success rates of artificial intelligence, especially the most recent technology. We don’t always consider the multiple failures that allow us to achieve refined systems later on. Underestimating the  limitations of AI can cause us to fall victim to the survivorship bias. This leads to using AI in a way that misinforms our decision-making or researching. 

The Survivorship Bias is a prevalent cognitive bias, which can be attributed to a fundamental misunderstanding of cause and effect, specifically concerning the concept of correlations versus causation.⁶ Though correlation and causation may exist in unison, correlation does not imply causation.⁷ 

Simply because individuals observe a pattern from a dataset, such as the above-mentioned example of successful entrepreneurs and dropping out of school, does not mean that all successful entrepreneurs drop out of school, or that all those who drop out of school will be successful. Causation refers to cases where action A causes B’s outcome, whereas correlation is simply a relationship. The coincidence that many entrepreneurs dropped out of university is a correlation, as the event of dropping out of school, did not necessarily cause their success. The Survivorship Bias, however, causes individuals to believe that the correlation is causation.⁷ 

Being aware of survivorship bias and understanding how it can impact your judgment and decision-making is critical to ensure an individual is practicing critical thinking and making the best possible decisions for themselves. Survivorship bias can impact individuals across several domains; thus, awareness can ensure better product decisions, financial investments, or scientific conclusions. Developing biases is an unavoidable human trait, but taking our time to challenge them is necessary to ensure that we make the best decision we possibly can.

Once they are aware of survivorship bias, individuals can practice avoiding it in several ways. 

Ask yourself what you don’t see

When making a decision, begin by considering what’s missing. What data didn’t “survive,” from an event, or dataset you are using? By asking questions, and taking the time to research these missing data points, you can develop a better understanding before your decision-making moment. Being fully informed, and taking the time to pause, reflect, and research will help ensure the consideration of survivorship bias in your decision-making.⁶

Vet your data sources 

Another method to prevent survivorship bias, specifically in your work and research, is to be selective of the data sources used. By ensuring data sources are crafted to promote accuracy and do not omit critical observations that would change analysis results or decision-making, individuals can reduce the risk of survivorship bias.¹

The term survivorship bias was first coined by Abraham Wald, a famous statistician known for studying World War II aircraft. When Wald’s research group attempted to determine how war airplanes could be better protected, the group's initial approach was to assess which parts of the aircraft had incurred the most damage. Once identifying areas that were in the worst condition, they would then reinforce the aircraft with more protection in those locations. However, Abraham Wald noted that the aircraft that were most heavily damaged were the ones that had not returned from battle. Those same airplanes would also provide the most relevant information regarding which parts of the aircraft would need to be reinforced.⁸

Had this research group been unable to identify this critical fact, the aircraft reinforcements they would have suggested would have ignored entirely a subset of planes that arguably had the most valuable data points regarding the project. The research study results provided an example of how Abraham Wald and his research group at Columbia overcame survivorship bias, saving hundreds of lives.⁸ 

Survivorship bias also impacts financial systems. A typical example of survivorship bias can be seen in mutual fund performance. Specifically, survivorship bias describes the tendency for companies or mutual funds to be excluded from performance analysis studies. The results from these studies assessing financial markets are then skewed in a more positive light, as only companies that were successful and “survived,” were included in the study.⁹

Survivorship bias can be examined more specifically in the case of mutual funds. A mutual fund is a financial vehicle that pools money collected from investors and is managed by a professional money manager The manager then invests in things like stocks, bonds, and other assets.¹⁰ When looking at mutual funds’ investments, it only includes those that are currently successful. Funds that were previously opened and lost money would be either closed or merged with other funds, which hides past poor performances.

Survivorship bias occurs when analysts calculate performance results of groups of investments, such as mutual funds, using only the surviving data at the end of the period, and exclude those funds or companies that no longer exist at the end of a study. For example, in a financial universe where 1,000 funds exist, imagine that 10% of these funds stop existing by year's-end due to poor performance. If an analyst is conducting a performance review of these funds but only begins the study at the end of the year, the analyst would fall prey to survivorship bias and omit the failed funds from their final results. By not including funds that failed, the performance data would indicate a more favorable final result for the theoretical fund universe.⁹

In 1996, researchers Elton, Gruber, and Blake analyzed the relationship between fund sizes and survivorship bias. They found that survivorship bias was more significant in the small-fund sector than in more significant mutual funds. Smaller funds have a higher probability of folding than larger, more established funds, which is why they attributed this to be true for the small-fund sector.¹² The researchers estimated the size of survivorship bias across the United States mutual fund industry as 0.9% annum. Additionally, they defined and measured survivorship bias as the following: 

"Bias is defined as average α for surviving funds minus average α for all funds" (Where α is the risk-adjusted return over the S&P 500. This is the standard measure of mutual fund out-performance).¹²

Another example of survivorship bias can be seen in the medical field and medical research. In 2010 at the Harvard Medical School and Beth Israel Deaconess Medical Center (BIDMC), a study was conducted in hopes of improving patient survival following trauma. A major concern when treating trauma is irregular bleeding. If the patient's blood does not clot properly, the risk of bleeding to death is high.¹³

The Harvard study investigated whether giving trauma patients additional proteins, which naturally occur in our bodies, would encourage blood clotting and improve survival rates. The study targeted patients who had received 4-8 blood transfusions within 12 hours of their initial injury. The trial hoped to recruit 1502 patients, but only recruited 573, and thus was later abandoned.¹³

This study’s failure was due to survivorship bias, as the trial only included patients who had survived their initial injury, and who had then received care in the ER before being transferred to the ICU for 4-8 blood transfusions. Patients who died from their initial injury were not included in the study, making it challenging to find suitable patients for the trial.¹³

What it is

Survivorship bias is a type of sample selection bias that occurs when an individual mistakes a visible successful subgroup as the entire group. In other words, survivorship bias occurs when an individual only considers the surviving observation without considering those data points that didn’t “survive” in the event. 

Why survivorship bias happens

Survivorship bias occurs in many disciplines, professions, and fields of research. Survivorship bias can be attributed to the fundamental misunderstanding of cause and effect and a skewed perception of correlation versus causation. 

Example 1 - Financial systems

The Survivorship Bias occurs in our financial systems, when individuals calculate performance results of groups of investments, such as mutual funds, using only the surviving data at the end of the period, excluding those funds or companies that no longer exist. Typically, mutual funds no longer exist due to poor performance, so omitting them from studies usually skews data in an overly positive light. 

Example 2 - Medical research 

The Survivorship Bias can also be observed in its impact on medical research. The Harvard Medical School and Beth Israel Deaconess Medical Center (BIDMC), attempted to conduct a study on trauma patients and to better their survival outcomes from different types of medical procedures. Due to the specific parameters of the study and skewed survival rates, the trial was only able to count 573 patients, out of their original 1502 participants. This study’s failure was due to survivorship bias, as the trial only included patients who had survived their initial injury, and who had then received care in the Energy Department before then receiving 4-8 blood transfusions.

How to avoid it

Once individuals learn about the Survivorship Bias, they can avoid bias by considering what data points may be missing from a dataset and using accurate data sources that do not omit key observations.

How Biases Can Color Entrepreneurial Decision-Making

This article explores the challenges faced by start-ups when they are venturing into new and unfamiliar territory. When it's time to consider a strategy shift, cognitive biases can creep in to promote fast and efficient decision-making. However, fast doesn’t always equate to better choices being made, sometimes failure to consider alternatives and “going with our gut” can lead to serious missteps.

Great Man Theory 

The great man theory states that leaders are born and not made. Meaning that there is some innate and intrinsic quality that makes certain people so influential. This theory also asserts that because of its inherent nature, those who are leaders deserve to be in that position. This article explores the claims and controversies, as well as how this theory connects to survivorship bias.