Algorithmic Bias Examples in Hiring, Lending and Healthcare

Contents

Figure 1: Algorithmic Bias Examples in Hiring, Lending and Healthcare — nine documented cases, five mechanisms, and the calibration check that would have caught the most famous one

An algorithm does not need bad intentions to cause harm. It needs a flawed label, a skewed dataset, or a convenient proxy, plus the scale to repeat one mistake millions of times.

That is what the best-documented cases of algorithmic bias show. Each one started with a reasonable-sounding design choice, and each one ended with real people getting worse outcomes.

This guide walks through real cases in hiring, lending and healthcare, explains what went wrong in each, and pulls out patterns you can check for in your own systems. I verified several cases against research papers and legal summaries during this research, and I flag the ones I am describing from memory. Some of these matters involve unproven allegations, and I mark them as such. This is not legal advice.

How Bias Gets Into a Model

A handful of mechanisms explain most cases:

Mechanism	What it looks like	Case that shows it
Biased proxies	The model predicts something convenient, like cost, instead of what you care about, like health	The healthcare cost algorithm
Historical data	Training data records past decisions made under unfair conditions, and the model repeats them	Amazon's résumé screener
Unequal data quality	Some groups appear less often, or their data is noisier	Pulse oximeter measurements
Explicit rules	Someone codes a discriminatory rule into the system on purpose or by accident	iTutorGroup
Feedback loops	The model's decisions shape the data it learns from next	Loan pricing and shopping behaviour

Keep these in mind as you read, because you will see them repeat. Most teams can name the mechanism in their own system once they have a name for it.

Hiring

Amazon's Recruiting Tool (from my knowledge)

In 2018, Reuters reported that Amazon had built an experimental machine learning tool to rate job candidates and then scrapped it. According to that reporting, the system trained on roughly a decade of past résumés, which came mostly from men, and it learned to downgrade résumés that included words like "women's." Amazon reportedly tried to fix specific terms, but could not guarantee the model would not find other proxies, and dropped the project.

I did not re-verify the details of this case in this research, so check the original Reuters reporting before you cite it. The lesson holds regardless: historical data teaches a model to imitate past preferences, fair or not, and patching individual words does not change the pattern underneath.

iTutorGroup: An Explicit Rule

The U.S. Equal Employment Opportunity Commission sued iTutorGroup in a case widely described as its first AI hiring discrimination lawsuit. The complaint alleged that the company's hiring software automatically rejected women aged 55 or older and men aged 60 or older. In 2023, the parties settled: iTutorGroup agreed to change its hiring process and pay a settlement fund to rejected applicants.

This case differs from the others. Nobody needed to explain a mysterious model behaviour, because the alleged rule was explicit. Automation can encode plain discrimination just as easily as a human screener can, and it applies the rule to every applicant without fatigue or second thoughts.

Mobley v. Workday: Vendor Liability (Allegations)

In 2023, Derek Mobley sued Workday, alleging that its AI-driven screening tools disadvantaged older applicants. Mobley says he applied to more than 100 jobs at companies using Workday's platform and got rejected every time.

The case has moved forward. On 16 May 2025, a federal judge in the Northern District of California granted preliminary certification of a nationwide collective action on the age discrimination claim, and later reports describe the court authorising notice to potential plaintiffs. Workday says the claims lack merit, has stressed that the certification is preliminary, and maintains that its technology does not make employers' hiring decisions. These remain allegations, and the case is ongoing, so check current court records for its status.

The case still matters because of its legal theory: that a software vendor, not only the employer, can face liability for discriminatory screening. One law-firm summary also notes that the EEOC supported the plaintiffs' view in earlier stages and has since deprioritised disparate impact cases, which shows how quickly the enforcement landscape can shift.

Lending

Fintech Mortgage Pricing

An academic study by Bartlett, Morse, Stanton and Wallace examined mortgage lending and published its results as an NBER working paper in 2019, followed by a journal version. It found that lenders charged otherwise-equivalent Latinx and African-American borrowers 7.9 basis points more on purchase mortgages and 3.6 basis points more on refinances, costing those borrowers roughly $765 million per year in extra interest.

The finding has two sides. Fintech algorithms still showed discrimination, which the authors link to possible rent extraction and profiling of borrowers who shop around less. But algorithmic lenders discriminated about 40% less than face-to-face lenders in the study, and they showed no discrimination in rejection rates. Algorithms can reduce bias and still fail to eliminate it, especially in pricing, where models can learn to exploit low-shopping behaviour.

The Markup's "Denied" Investigation

The Markup analysed more than two million mortgage applications and reported that lenders were more likely to deny applicants of colour than similarly qualified white applicants: 40% more likely for Latino applicants, 50% for Asian and Pacific Islander applicants, 70% for Native American applicants, and 80% for Black applicants. The investigation also points to the automated underwriting systems that Fannie Mae and Freddie Mac developed, which most lenders use for approval decisions.

Lenders had argued that variables not available in public data would explain the disparities, and the investigation's authors say their analysis undercut that argument. This is a journalistic analysis, and the industry has disputed such findings, so read the methodology and the responses before you cite specific numbers. The lesson: widely shared systems can spread one design choice across a whole market.

Apple Card (from my knowledge)

In 2019, public complaints alleged that the Apple Card gave some women much lower credit limits than their husbands, even when finances looked similar. The New York Department of Financial Services investigated. As I recall, its 2021 report found no unlawful discrimination, and it criticised the lack of transparency in how customers could understand and challenge decisions. I did not re-verify this case in this research, so confirm the details first.

The lesson holds either way: when people cannot see why a model decided something, suspicion fills the vacuum, so explainability and a path to appeal matter even when the model turns out to be lawful.

Healthcare

The Cost-as-Proxy Algorithm

The most cited healthcare case comes from Obermeyer, Powers, Vogeli and Mullainathan, published in Science in 2019. They studied a widely used commercial algorithm that health systems apply to identify patients who need extra care. The finding: at the same risk score, Black patients were considerably sicker than white patients, as signs of uncontrolled illness showed.

The cause was the prediction target. The algorithm predicted health care costs, not health needs. Because of unequal access to care, health systems spend less money on Black patients than on equally sick white patients, so the algorithm concluded that Black patients were healthier. The authors estimated that fixing the disparity would raise the share of Black patients flagged for extra help from 17.7% to 46.5%, and that reformulating the algorithm to stop using cost as a proxy eliminated the bias.

This case teaches the most transferable lesson in the whole field: a convenient proxy can look accurate and still carry bias. The model predicted cost very well. It predicted the wrong thing.

Race Adjustments in Clinical Formulas (from my knowledge)

For years, a common equation for estimating kidney function included a race coefficient that raised estimated function for Black patients, which could delay referrals and treatment. In 2021, a national task force recommended a new equation without race. I did not re-verify this history here, so check the primary sources. Bias can sit inside ordinary clinical formulas, not only in machine learning models, which is why an audit that only looks at your model misses part of the surface.

Pulse Oximeters (from my knowledge)

Research published in the New England Journal of Medicine in 2020 reported that pulse oximeters overestimated blood oxygen levels in Black patients more often than in white patients, which could hide dangerously low oxygen. I did not re-verify this research, so confirm the details. Measurement devices can carry bias that flows into every system built on their data, and no amount of modelling care fixes a systematically wrong input.

Patterns Across the Cases

Step back, and these themes repeat:

Pattern	Where it showed up
The label is the problem	Cost for need, rejection history for merit, past hires for quality
Equal accuracy is not equal fairness	The healthcare algorithm predicted costs accurately and still produced racial bias
Scale multiplies small flaws	One biased rule touches every applicant
Responsibility spreads out	Employers, vendors and data providers each point at someone else
Opacity feeds distrust	Hidden logic makes disputes harder to resolve
Algorithms can help and hurt at once	Fintech lenders discriminated less than face-to-face lenders and still discriminated

How to Check Your Own System

Audit the Label

Ask what your model predicts and what you care about. If they differ, test whether the gap affects groups unequally. The healthcare case would have shown up in a simple check: compare real outcomes across groups at the same model score.

import pandas as pd

def calibration_by_group(df, score_col, outcome_col, group_col, bins=10):
    """Average real outcome per score bin and group. Gaps at equal scores signal bias."""
    df = df.assign(bin=pd.qcut(df[score_col], bins, duplicates="drop"))
    return df.groupby(["bin", group_col])[outcome_col].mean().unstack()

If one group's real outcomes run consistently worse at the same score, your score undercounts that group's need or risk. That single sentence is the cheapest bias test in existence, and it costs one groupby.

The moment this became personal: the first time I ran a chart like that, the overall calibration curve looked fine and the top decile did not. Aggregate metrics hide exactly the region where high-stakes decisions get made, which is why the check has to be per bin and per group rather than one number at the bottom of a report.

A Practical Audit Routine

1. Write down the target and the real goal, and note where they diverge.
2. Compute metrics per group: selection rates, error rates and calibration, with confidence intervals.
3. Check proxies: look for features that stand in for protected characteristics, like location or school.
4. Test pricing and thresholds, not only accept-or-reject decisions.
5. Review explicit rules and hardcoded filters.
6. Examine data coverage: who is missing or poorly measured?
7. Include vendors: ask third-party tool providers for bias testing evidence.
8. Monitor after launch, since drift and feedback loops change outcomes.
9. Offer appeals with human review for high-stakes decisions.
10. Document decisions and their rationale.

Steps one, three and four catch the cases in this article. The healthcare failure was a step one problem, the résumé screener was a step three problem, and the mortgage pricing findings were only visible at step four. The fairness principles guide covers the metric choices behind step two, and the AI risk assessment template shows where this routine sits inside a full register.

Common Pitfalls

1. Assuming accuracy means fairness. The cost algorithm scored well on accuracy, and that was the point: it was accurate about the wrong target.
2. Removing protected attributes and calling it done. Proxies leak the same information back in, which is why outcomes need testing rather than feature lists.
3. Auditing once. Data and populations shift, and feedback loops make the next audit different from the last one.
4. Trusting vendor assurances. Ask for evidence, and test the tool yourself, because the Mobley case shows where responsibility can land.
5. Never checking pricing. Bias hides in rates and limits, not only in rejections.
6. Skipping affected communities. The people harmed often spot problems first, and they are rarely in the review meeting.
7. Treating a legal clearance as a fairness clearance. The Apple Card investigation is the reminder: lawful and understandable are different standards.

Quick-start checklist:
[ ] decisions your models influence are listed and ranked by impact on people
[ ] each model's target is written next to the real goal it is meant to serve
[ ] group-level metrics include confidence intervals and sample sizes
[ ] the per-bin calibration check has been run for every high-stakes model
[ ] features are reviewed for proxies, and hardcoded rules are reviewed too
[ ] vendors have supplied bias testing results, or been asked in writing
[ ] appeals and human review exist where stakes are high
[ ] recurring audits are scheduled, with an owner and a date

Check the edition date before you buy. The cases here move quickly, especially the litigation, so pair any book with current court records and research.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What are the most well-documented examples of algorithmic bias?

Four come up repeatedly. Amazon reportedly scrapped an experimental résumé screener that had learned to downgrade applications containing words like women's, reported by Reuters in 2018. The EEOC sued iTutorGroup over hiring software that allegedly auto-rejected older applicants, settled in 2023. A Science paper by Obermeyer and colleagues showed a widely used healthcare algorithm using cost as a proxy for need, under-identifying Black patients. And the Markup found lenders denying applicants of colour at higher rates than similarly qualified white applicants. Each case is a different mechanism, which is why the set is useful.

What is the difference between a biased label and a biased dataset?

A biased label means the target you trained on is not the thing you care about, so the model is accurate about the wrong question. The healthcare algorithm predicted cost rather than health need, which is why it scored well and still under-referred Black patients. A biased dataset means the examples themselves over-represent or under-measure some groups, as in résumés drawn mostly from men over a decade. The two often combine, but the fixes differ: a label problem needs a new target, while a data problem needs better sampling, measurement or weighting.

Why did the healthcare cost algorithm produce racial bias?

Because it predicted spending, and spending is not the same as sickness. Unequal access to care means the health system spends less on Black patients than on equally sick white patients, so at any given predicted cost Black patients were considerably sicker. Obermeyer and colleagues found this in a Science paper published in 2019, and estimated that fixing the disparity would raise the share of Black patients flagged for extra help from 17.7% to 46.5%. Reformulating the algorithm to stop using cost as a proxy eliminated the bias, which is the most transferable lesson in the field.

Is algorithmic bias illegal?

It depends on the jurisdiction, the domain and the specific practice, and this article is not legal advice. In the United States, hiring and lending sit under longstanding anti-discrimination law, and the Mobley v. Workday case tests whether a software vendor, not only the employer, can be liable for discriminatory screening. In the EU, the AI Act adds obligations on top of existing equality law, with high-risk deadlines now at 2 December 2027 for Annex III systems. Talk to counsel about your own exposure rather than relying on a general summary.

Can removing protected attributes fix algorithmic bias?

No, and relying on it is one of the most common mistakes. Removing race, sex or age from the features does not remove the information, because proxies such as postcode, school, job history or shopping behaviour carry much of it back in. The Apple Card complaints are a useful reminder that outcomes can look unequal even when the attribute was never an input. Test outcomes by group rather than trusting the feature list, and check pricing and thresholds as well as accept-or-reject decisions.

How do you audit a model for bias?

Start with the label: write down what the model predicts and what you actually care about, then find where they diverge. Compute selection rates, error rates and calibration per group, with confidence intervals and sample sizes. Check features for proxies and review any hardcoded rules. Test pricing and thresholds, not only yes-or-no decisions. Ask vendors for their bias testing evidence, monitor after launch because drift and feedback loops change outcomes, and give people a route to appeal a high-stakes decision.

Wrapping Up

The famous cases share a pattern: reasonable-looking choices, a flawed proxy or a skewed history, and scale that turned small errors into systematic harm. Hiring tools learned old preferences, lending systems priced borrowers unevenly, and a healthcare algorithm mistook spending for sickness.

You can catch most of these problems with unglamorous work: question your labels, measure outcomes by group, and let people appeal. Start with the question that would have caught the healthcare case: what does my model actually predict, and is that what I care about?

Which of your models predicts a proxy rather than the thing you care about, and when did you last check it per group rather than in aggregate?

Related Articles

• AI Ethics Principles: Fairness, Accountability and Transparency
• How to Run an AI Risk Assessment: Template and Process
• NIST AI Risk Management Framework Explained for ML Teams
• Interpretability vs Explainability in Machine Learning: What's the Difference?
• Best Explainable AI and Model Interpretability Tools