Back
Amazon's experimental hiring AI
webCredibility Rating
4/5
High(4)High quality. Established institution or organization with editorial oversight and accountability.
Rating inherited from publication venue: Reuters
A seminal real-world example of AI bias in high-stakes deployment, frequently cited in discussions of algorithmic fairness, training data bias, and the risks of automating consequential human decisions without adequate interpretability or oversight mechanisms.
Metadata
Importance: 62/100news articlenews
Summary
Amazon developed an AI hiring tool (2014-2015) that systematically discriminated against female candidates because it was trained on historically male-dominated resume data, teaching itself to prefer male applicants. Despite attempts to patch specific biased terms, Amazon disbanded the project after recognizing the system could develop new discriminatory patterns unpredictably. This case illustrates core challenges in algorithmic fairness, bias from training data, and the difficulty of ensuring interpretability in deployed ML systems.
Key Points
- •Amazon's AI recruiter trained on 10 years of male-dominated resumes, causing it to penalize resumes containing 'women's' and downgrade graduates of all-women's colleges.
- •Attempts to neutralize specific biased terms were insufficient—engineers could not guarantee the model wouldn't develop other discriminatory sorting patterns.
- •The project was ultimately disbanded because executives lost confidence it could be made reliably fair or interpretable.
- •Highlights the core ML fairness problem: biased historical data encodes and amplifies existing societal inequalities when used to train automated decision systems.
- •Serves as a widely-cited real-world case study on the risks of deploying ML in high-stakes decisions like hiring without adequate bias controls.
Cited by 1 page
| Page | Type | Quality |
|---|---|---|
| Erosion of Human Agency | Risk | 91.0 |
Cached Content Preview
HTTP 200Fetched Mar 15, 202615 KB
[Skip to main content](https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G/#main-content)
[Exclusive news, data and analytics for financial market professionalsLearn more aboutRefinitiv](https://www.reuters.com/differentiator/)
SAN FRANCISCO (Reuters) - Amazon.com Inc's machine-learning specialists uncovered a big problem: their new recruiting engine did not like women.
The team had been building computer programs since 2014 to review job applicants' resumes with the aim of mechanizing the search for top talent, five people familiar with the effort told Reuters.
Automation has been key to Amazon's e-commerce dominance, be it inside warehouses or driving pricing decisions. The company's experimental hiring tool used artificial intelligence to give job candidates scores ranging from one to five stars - much like shoppers rate products on Amazon, some of the people said.
Advertisement · Scroll to continue
"Everyone wanted this holy grail," one of the people said. "They literally wanted it to be an engine where I'm going to give you 100 resumes, it will spit out the top five, and we'll hire those."
But by 2015, the company realized its new system was not rating candidates for software developer jobs and other technical posts in a gender-neutral way.
That is because Amazon's computer models were trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most came from men, a reflection of male dominance across the tech industry. (For a graphic on gender breakdowns in tech, see: [tmsnrt.rs/2OfPWoD](https://tmsnrt.rs/2OfPWoD))
Advertisement · Scroll to continue
In effect, Amazon's system taught itself that male candidates were preferable. It penalized resumes that included the word "women's," as in "women's chess club captain." And it downgraded graduates of two all-women's colleges, according to people familiar with the matter. They did not specify the names of the schools.
Amazon edited the programs to make them neutral to these particular terms. But that was no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory, the people said.
The Seattle company ultimately disbanded the team by the start of last year because executives lost hope for the project, according to the people, who spoke on condition of anonymity. Amazon's recruiters looked at the recommendations generated by the tool when searching for new hires, but never relied solely on those rankings, they said.
Amazon declined to comment on the technology's challenges, but said the tool "was never used by Amazon recruiters to evaluate candidates." The company did not elaborate further. It did not dispute that recruiters looked at the recommendations generated by the recruiting engine.
The company's experiment, which Reuters is first to report, offers a case study in the limitati
... (truncated, 15 KB total)Resource ID:
264c7d949adbc0b4 | Stable ID: MjcxODljMD