In August 2025 it was revealed in the press that a report produced by Deloitte for the Australian government contained several entirely fabricated citations to articles and books that do not exist, other than in the fevered hallucinations of generative AI. The report, costing taxpayers AU$439,000, was presumably not offered at a substantial discount to the client due to it having likely been written, at least in part, by an AI.
This is just the latest in a long series of such incidents, many of the best documented ones being in the legal profession. The legal ones are prominently reported and recorded, presumably because case law is one area that can be easily checked for accuracy. Above all, it is a profession where court documents with references to case law are actually routinely checked. This does raise the interesting question of just how common such AI-generated errors are in areas and subjects where there is not the same degree of rigorous checking.
We know that hallucinations in generative AI are common, getting worse and occurring in at least 15% of generative AI answers on average. The more obscure the area under discussion, the higher the chances of hallucination, as the AI has less training data to work with. If you ask an AI “what is the capital of France?”, then it will almost certainly answer “Paris”, because it has seen Paris referred to as the capital of France in countless books and articles that it has been trained on. Indeed, if you asked an AI that same question repeatedly, then it would answer repeatedly and correctly again and again, unless the “temperature” of the AI was adjusted to encourage it to be more creative in its answers. You might be able to ask that question a thousand times of an AI and get the right answer a thousand times, but even with such a commonly referenced fact, there is still a tiny chance that it would answer the question with a wrong answer, one that had a low statistical weighting in predicting the correct answer. Eventually, the AI would give you a different answer if you asked the question enough times. However, now imagine that you are asking an AI something far more esoteric and rarely referenced. The probability of it making a mistake here is dramatically higher.
The very first question I ever asked of ChatGPT in late 2022 was a little test, though I had not intended it to trip up the AI; I was merely curious as to the extent and breadth of its knowledge. I looked up the results of a 1980s chess tournament that I was familiar with (I play regular club chess, in case you are wondering), and opened up the final results table, showing the participants and their scores. When I asked ChatGPT to tell me about the tournament, its answer puzzled me. The answer looked superficially plausible, having a list of players from that era, and a set of scores. However, the scores were off, and some of the players listed did not play in that tournament, while others that did play were missing. This was a classic AI hallucination, presumably caused because its training data was not exactly awash with the details of obscure British chess tournaments.
Now consider a report written, seemingly, by a consulting firm that has been hired to write about some esoteric area of your business. It is fluently written and contains references to books and articles on the subject in question. How confident are you that someone at the client is diligently checking every reference? We have already seen cases of entire books sold on Amazon being AI-written, including ones, alarmingly, as field guides to edible mushrooms. Since September 2023, Amazon has restricted publishers to three books per day (!) being uploaded, which is a likely indication of the scale of the problem. Some unscrupulous “authors” even publish AI-generated books by well-known authors in the hope that fans will buy them, hoping for a new release by their favourite writer. Authors already have plenty of issues with AI companies using their copyright-protected material for LLM training without payment or their permission. Indeed, there are a host of lawsuits underway in this area, with the first already being settled out of court in August 2025.
Clearly, AI has some legitimate uses in the world of consulting. AI can be good at summarising documents and researching large volumes of data, such as research articles. Speeding up desk research by junior consultants has the potential to reduce costs, with nearly half of such tasks potentially being replaced by AI. Consulting firms can potentially make a lot of revenue from providing advice to corporate clients on how to best use AI in their businesses. Accenture alone claims to have invested $3 billion in its AI practice, doubling its AI workforce to 80,000 consultants, almost 10% of its consulting workforce. McKinsey has 5,000 consultants in their AI practice, with over a thousand partners, from academics to start-up companies, and thinks that as much as 40% of its revenue may come from AI-related work.
Along with these opportunities come associated risks and challenges. Using AI for clients means accessing potentially sensitive and confidential data, and generative AI is still at an early stage when it comes to security, with many known issues. Data models need to be trained on data, and this corporate data may have data quality issues as well as potentially have inherent biases that may then be reflected in the model answers. There have been many prior examples of AI applications having to be abandoned due to bias in underlying datasets. Moreover, AI models can deteriorate over time, and need careful monitoring when put into production. All these practical issues are quite apart from the reputational issues that may occur if the consultants use AI to cut corners without disclosing that to clients, which may have happened in the Australian government Deloitte report mentioned at the start of this article.
A further threat to consulting firms is that their customers can easily use generative AI tools for research, and may start to question what value is really added by expensive consultants over just typing a prompt into an AI chatbot. Certainly, the big-name consulting firms have powerful brands, and corporations often use these firms partly because of the reassuring power of these brands. Whereas in the 1970s, there was a corporate mantra that “nobody ever gets fired for using IBM”, in more recent times, much the same could be said for relying on the advice of top consulting firms like McKinsey, Bain, and Boston Consulting Group. However, that brand value has its limits and can easily and quickly be undermined, as auditor Arthur Andersen discovered to its cost with Enron in 2001. Incidents such as using AI in an undisclosed way in consulting reports, or security breaches such as the McKinsey-built chatbot leaking customer details to the security firm Zenity in 2025, may have serious consequences if they turn out to be the tip of the iceberg. After an initial sugar rush of AI consulting work, there are signs that some customers are starting to cut back on their use of consulting firms, and some top firms are reducing their recruitment, at least for junior consultants. Generative AI technology is still a relatively new technology, and its impacts, good and bad, are still being felt. This applies to the world of consulting as much as it does to other industries.
