Author : Aniket Pathak
Date : 10 May 2023
Have you tried to develop a product or business strategy or an idea ? That dream project of yours, how you want to approach it given a chance ? Will you research or look at similar successful business and implementation and try to design your strategy around it ? if yes then you are likely to become a victim of "Survivor Bias"
The aims of this article is give you some glimpse of how "Survivor Bias" effect your strategies and planning. It's a very interesting field of study, having impact on many domains including product management. So let's dig in ?
What is Survivor Bias
The standard way of explaining the concept of "Survivor Bias" is to Start with below scenario of fighter plane data.
The above picture is the list of bullet marks found on the fighter plan's return to the base. When each fighter plane was returned to the base the engineers use to examine the whole plane and record all the bullet marks. At the end of the day, they consolidate the bullet mark records to create a map of the bullet marks. That's what the red dots represent in the above picture, in simple words each red dot is a bullet hit on that area of a fighter plane.
So the question is - As an engineer if you need to improve your fighter planes, based on the above picture which of the areas you will add more armour to protect the plane?
Take a pause and try to think about this question.
A default process is to look at which areas have the most number of red dots and if that is a critical area like the cockpit or fuel tank then that's the candidate area to improve or to add more armour or you can even choose all the area with red dots should have more armour as it seems like those are the most targeted areas.
This is perfectly natural thinking, but if you look at this picture and think about it carefully the missing values start to appear.
The most important note here is, this data is collected from all the fighter planes returned to the base. Please emphasize "returned to the base", which means all fighter planes with these red dots or bullet marks did come back or returned to the base. This leads to the conclusion that the areas of fighter planes which do not have any red or bullet marks as per the above picture are the most vulnerable, as no fighter plane returned to base if hit in those areas.
Source : https://www.countbayesie.com/blog/2020/11/5/survivorship-bias-in-house-hunting-a-practical-modeling-example-using-jax
This is profound knowledge as it help you see the missing picture. You would have guess where we are going with this. We are focusing on the survived cases but not the failed cases. This is called "Survivor Bias". The official definition is as follows -
A cognitive fallacy in which, when looking at a given group, you focus only on examples of successful individuals (the “survivors”) in the selection process rather than the group as a whole (including the “non-survivors”).
What this has to do with Products
When any product is launched and is out in the market, you will surely get a few successful implementations. This helps to build confidence in your product and showcase the areas where the product is successful and doing well. As a default behaviour especially within many organizations, all these successful implementations get highlighted and build pressure to replicate it to all other areas. Slowly after a point, the only data you have about the product and its implementation is about successful cases.
All the data points and feedback collected is only from the successful sale or implementation. So when you do a product review, publish an article, or look at the areas to improve you will be fully influenced by successful or survived cases. Very similar to the fighter plane picture we saw above. All the red dots you have will tend to mislead you from the areas which need your attention. So when you design the next version or any product improvement all will be based on survived cases, completely ignorant of the failed or non-survived scenarios.
This is "Survivor Bias" in your products. You fail to recognize the critical area which needs care. It could be product features, services, operations, practices, and all respective areas. If you continue to get diverted by your Survivor bias your product tends to focus on a selective market and clients. This will impact product penetration and ultimately lead to stagnant revenue as your product will only cater to survived cases.
Real World example
Remember the story of Kodak, the term survived but not the company. As mentioned in may case studies of how Kodak failed, among all the various reasons, the most important one is 'solely focusing on successful products'. They failed to recognize the emerging trend of digital photography and instead only focused on their successful film products. Kodak failed to see the what is changing and continued to develop their traditional successful products. That's what "Survivor Bias" do to companies.
Nokia had a same fate. Company with around 50% market share was invincible. Nokia was synonym to Mobile, it was suppose to rule the market with new handset launched almost everyday. Still company is no more, some of the new generation don't even know the company. So why did this happen ? and yes Survivor Bias has played a role in this too.
Its true that companies fail to innovate until it comes to survival, when its too late. When companies are too focused on their successful products and fail to see where market is going, they are bound to fail. Survivor bias can impact anyone, not just high-level executives in companies. It's important to constantly look for ways to improve and avoid complacency, or risk reaching a dead end. You can be a victim of it in your day to day life. When you keep doing same thing which worked for you before, you are going to reach a dead end. You tend to become compliant, until you realised its too late. So start looking for the things which are not working and start fixing it.
So is the solution obvious
In a sense yes, you have to look at the missing picture. You have to look at why a product failed for some clients, and what are the pain points that lead to non-Survivor cases. Which areas need more protective armour, which will help you to build strong fighter planes, which can survive any battle.
You have to have a strong feedback loop that collects feedback and data points for all the sales and implementation including failed cases. These data points will give you a realistic picture of your product performance. The successful implementation doesn't need much care cause it's working. If a specific client is really happy with your product then another similar client will also be happy to have the same features. Your product is working very well for those markets. With the normal servicing model, you will be able to retain those clients in your portfolio for a longer period. What you need to care about are the failed cases, as they will lead you to the areas to improve.
Is this easy, maybe or may be not? It all depends on which feedback or data points you collect.
Conclusion
Avoiding survivor bias is crucial for success in product management. By staying curious, analysing both successful and unsuccessful products, and staying aware of changing market trends, companies can avoid complacency and achieve better product development outcomes. Remember, survivor bias is a trap that can happen to anyone, but with the right mindset and approach, it can be overcome.
With survivor Bias there is a likely chance that you will start catering to the same kinds of clients with very similar implementations. This leads to stagnant revenue. Your product may not be able to gather the momentum or the return on investment you like to have. It creates delays in your product journey or worst case scenario your product may become a niche product catering to a small section of your clients.
Imaging 2,3 or 4th version of the picture of the red dots, they will keep getting concentrated on specific areas as more planes failed to return. So it's very crucial to identify survivor bias and focus on the areas which need your attention.
Key Takeaways
Try to acknowledge the biases where ever they are, understands them and ultimately avoid them.
Create a wholistic feedback loop or data points to paint a realistic picture of the product performance
Try to look at the missing picture from all the data points and feedback you collected.
Some helpful articles -
Kodak case study -
Comments