The large language models (LLMs) that underlie generative AI rely on being trained on vast volumes of data. This may be text, images or video, but in any case, the answers that a fully trained LLM will give to you will be heavily dependent on the nature of that training data. If you are going to train an AI for screening job applications and its training data reflects a particular stereotype, then the AI’s answers will reflect that same stereotype. The AI models themselves can also be influenced by the people who designed them: the weighting of variables or the selection of features will reflect the views, consciously or unconsciously, of the model developers.
Bias in AI models has happened in practice time and time again. Amazon was forced to scrap a recruitment tool that screens resumes for technical jobs, after it was found that it systematically discriminated against women for such positions. A UK makeup artist lost her job at Estee Lauder after software from US company HireVue marked down her body language. This was despite the fact that the applicant had previously been in the role and scored highly, and was reapplying for the same role after furlough; the software company subsequently removed its facial analysis feature. Perhaps three-quarters of resumes (CVs in the UK) never see human eyes these days. One 2025 study found that over 98% of Fortune 500 companies used AI screening.
Various examples of AI bias in such tools have been shown, some of them subtle and surprising. One resume screening tool actively favoured candidates that played lacrosse in high school. The same model also favoured people named “Jared” for reasons that presumably were associated with its training data. As an employment attorney commented, “There was probably a hugely statistically significant correlation between those two data points and performance, but you’d be hard-pressed to argue that those were actually important to performance.” It turns out that your name matters a lot when being hired, even when an AI is assessing it. University of Washington Research found that three different LLMs all favoured ‘white” names over black, and male names over female. Even removing names and gender from resumes does not remove bias from AI models. The book “The Ethical Algorithm,” found that algorithms would instead discriminate based on postal code or even the model of car that they drove. This, quite apart from being unfair, also leaves employers open to legal redress, since racial and gender bias is illegal in most developed countries, and using a biased AI tool does not remove responsibility from the employer that chooses to use one.
The issue of AI bias extends beyond hiring. The city of Detroit used facial recognition to track down a person who stole some watches in 2020, and the system matched a photo of the criminal taken by a security camera at the store to a man called Robert Williams, who was subsequently arrested. The software misidentified Mr Williams, who was black, and who later sued the city for wrongful arrest. The lawyers noted that facial recognition software had been shown to produce biased results among people who were not white skinned. This bias in such a system has been shown years earlier, when facial recognition software from Amazon managed to incorrectly identify 28 members of Congress as matches to a mugshot database; twice as many of these errors occurred with black-skinned congressmen as with white-skinned ones.
There does not seem to be an easy way to eliminate inherent bias in training data used for LLMs. Careful curation of the data used can help, with training datasets being filtered to exclude overtly prejudicial content. LLMS can be adjusted during their training by humans to optimise for equitable outcomes. Ensuring diversity in AI developers and practitioners itself could help identify biases that less diverse teams may miss. However, all of these require active measures and investment.
As we deploy LLMs more widely throughout society, it is important that we increase basic education about the ways in which they work. Many people do not realise that LLMs reflect the biases of their training data, and so may unwittingly introduce bias into situations like hiring or law enforcement, without being aware of the consequences. Addressing AI bias is an important step to ensure that we use technology in a way that benefits everyone in society equitably, rather than just some people. Unfortunately, LLM technology appears to reflect all too well the limitations and biases of our society.
