Deloitte Under Scrutiny: Ai-generated Inaccuracies Found In Aud 440,000 Australian Welfare Report

A report commissioned by Australia’s Department of Employment and Workplace Relations, which was paid AUD 440,000 (around 231,200 Swiss francs), looked into an IT system intended to automate sanctions in the country’s welfare framework. The original report was published in July and was quietly updated on the ministry’s website last week. The updated version featured over a dozen deletions of non-existent references and footnotes, a refreshed reference list, and various typographical corrections.

Report Inaccuracies

Christopher Rudge, a researcher at the University of Sydney, discovered several inaccuracies in the report. He hypothesized that these inaccuracies could be due to instances of AI hallucinations, which occur when generative models fabricate information that appears to be factual.

Findings Deemed Unreliable

While the report now includes a disclaimer that generative AI was utilized to address “gaps in traceability and documentation”, the trust in the report’s findings has been undermined as a result of the reported inaccuracies. “The firm acknowledged the use of generative AI for a significant analytical task but did not initially disclose it,” said Rudge. He expressed concern that the recommendations made in the report could not be fully trusted.

Nonetheless, the ministry maintained that “the essence of the independent review remains unchanged and there will be no alterations to the recommendations”. There are also reports that Deloitte forfeited the final payment for the report, although the amount was not specified.

Impact on Deloitte

This incident does more than just create a financial dent; it also represents a significant blow to Deloitte’s reputation. The firm, which advises corporations, governments, and institutions worldwide on the proper utilization of AI, is now facing scrutiny for the responsible use of these technologies.