A software engineer has concerns about a recommender system his firm is designing to sell to health and life insurance companies. In a meeting with the sales team leader, he brings up these concerns and argues for extending their development timelines to address them. The sales manager, however, wants to hit the market with their product as soon as possible.
Kenny is a sales manager at a tech firm which designs data management and processing solutions for health and life insurance companies. He is leading the design of a software system that will predict customer product preference based on whether a customer is a smoker or nonsmoker. Specifically, they want their product to compare new customer data with previous customer data, match the previous customer information with the products that they have purchased, and establish associations that connect new customers to existing products. The predictive system is intended to make the process of selling insurance faster, more efficient, and more cost-effective. Kenny wants to move their product to the market as quickly as possible, as there are other firms currently developing similar software to also bring to market.
Charles, one of the members of the solution design team, has vast experience working in the insurance industry. During a late-stages team meeting, Charles raises concerns about the customer-product categorizations they have developed. He claims that they make inaccurate connections between customers and products by assuming too much from the simple smoking/non-smoking designation without considering other types of indicators such as number of children or marital status. As a result, the recommender system will end up matching customers with products that are not necessarily right for them.
Charles continues, “The many nuanced details of a person’s life matter in buying and selling of insurance, and these are left out of these algorithms. Focusing primarily on smoker/non-smoker might make it easier for the insurance company to sell insurance, and sell more of it, but may not benefit the customers themselves in the long run. We need to take a step back in our process and tweak our code to consider additional features from our consumer data. This will achieve more accurate matches and better outcomes for customers without sacrificing the efficiency of the system or missing our deadline to hit the market. We can add these features and establish a more rigorous system without sacrificing run-time performance of the system.”
“Will a more rigorous system produce more accurate risk-assessments of customers?” Kenny asks.
“Yes,” Charles replies, “while the general categorizer of smoker may imply a higher at-risk status, using more classifiers past the smoker/non-smoker distinction will substantiate risk assessment by considering other factors like marital status, age, gender, living situation, etcetera, and better match customers to the appropriate insurance packages.”
“But a recommender system primarily based on the smoker/nonsmoker classifier will group more customers into a higher risk category, and match more customers to a particular type of insurance product?” Kenny clarifies.
“Well, yes…” Charles reluctantly replies.
“Then we should push forward with that.”
Charles again begins to protest the ethics of the system, since it will recommend high risk packages to people who might not need it, but Kenny downplays his critique, “I think you are missing the bigger picture, and what is most important to our firm. At the end of the day we are interested in our product’s ability to increase the revenue of our clients. We don’t need a rigorous recommender system to sell this product, because insurance firms will want customers to stop being classified as soon as they get to the yes that determines the product they want to sell to their customers, which can be achieved with a simple classifier. Our market research has shown that classifier to be smoker/nonsmoker. Insurance companies can generally group their customers into higher risk categories and sell more of the products they want to sell this way.”
“So, in a way, we are only interested in the classifier columns that will scare people into buying a product that an insurance firm wants to sell?” Charles probes.
Kenny replies confidently, “If you choose to look at it that way, that is your choice. The way I see it, a more accurate recommender system does not enhance the profitability of our product, and may in fact diminish its attractiveness to our insurance company clientele. We are not responsible for how our clients choose to provide insurance to their customers, but provide tools that respond to their demand, which is to make more money. Additionally, it’s not as if smoker/nonsmoker is a poor general classifier. There is plenty of data that proves the dangers to someone’s health by smoking. In the eyes of most, this is enough to justify a higher risk health status.”
Kenny continues, “You should not get overly concerned about the current state of the system. Your team has produced a recommender system that can be effectively marketed to our customers, one that accurately addresses the needs communicated by the insurance industry. All indicators suggest this system will be very successful on the market as is, you and your team should be proud of your work. I do not want to overly hype its success, but I would not be surprised to see your team awarded high bonuses this quarter.”
The meeting concludes, and Charles leaves a little frustrated. He thinks that companies should follow the ethical principle of beneficence, meaning that businesses have a moral obligation to act for the benefits of their customers. He has always believed that corporate beneficence— which requires always keeping the customer’s best interests in mind—is not only the most ethical way to run a business, but also the most successful in the long run. After considering what Kenny has said about satisfying their customers’ needs, however, Charles wonders if his concerns are warranted. If what Kenny says is true, his team will be rewarded for producing a product that the firm significantly profits from. Making the unneeded changes to the product might negate some of these rewards, and possibly invite punishments for his team. Further, after the product goes to market, Charles isn’t so sure if he should feel any responsibility for how it is used by their insurance company clientele.
Resources for Further Reading
Dalton George, MS, and Jason Ludwig, MS, are graduates of the Drexel University Center for Science, Technology and Society. June 2017.
In general, when people ask whether a person, group, or organization has acted in a socially responsible manner, they are asking whether it has met expectations for behavior that will benefit society at large. The phrase “social responsibility” holds individuals, social groups or ...