LLMs Stereotype Job Applicants 65% More Than Humans
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- Princeton and University of Chicago researchers tested ChatGPT, Claude, and Gemini in a 40-round simulated hiring game across 20 jobs and four fictional ethnic groups (Tufa, Aima, Reku, Weki); on a segregation scale where 2.0 means total confinement, humans scored 0.84 while OpenAI's o3 hit 1.83 — roughly 65% higher than human participants.
- LLM training on math and coding rewards generalizing from few examples, an instinct that helps crack logic puzzles but causes models to settle on social stereotypes too early — coauthor Ryan Liu said this is "literally a lot of what they're optimized for."
- Newer reasoning models including OpenAI's o3 and DeepSeek's R1 showed stronger biases than predecessors, with OpenAI and Anthropic declining to comment on the ICML paper presented in Seoul in July.
- Telling the model to be fair barely changed behavior, but promising an explicit bonus for diverse hiring dramatically reduced bias — suggesting the fix lies in redesigning optimization goals rather than appealing to values, Liu said.
- Providing irrelevant personal details like hair color and tattoo shape caused models to fall back on ethnic sorting in a separate resettlement experiment using Canadian cities, while relevant info such as age and education reduced stereotyping.
- Cornell computer scientist Angelina Wang warned the risk grows as chatbots gain memory and personalization, since models can "over-index on the same kinds of behaviors they've experienced before" and form biases from conversation history.
Why it matters: For companies deploying LLMs in hiring, lending, or parole decisions, the 1.83-vs-0.84 segregation gap means today's most capable reasoning models are also the most prone to invented bias. Simple 'be fair' prompts barely work, but adding a diversity bonus to the model's goal dramatically reduces stereotyping — a concrete design lever vendors can pull.
