The Australian Financial Review reports that Deloitte Australia will offer the Australian government a partial refund for a report that was littered with AI-hallucinated quotes and references to nonexistent research.
Deloitte's "Targeted Compliance Framework Assurance Review" was finalized in July and published by Australia's Department of Employment and Workplace Relations (DEWR) in August (Internet Archive version of the original). The report, which cost Australian taxpayers nearly $440,000 AUD (about $290,000 USD), focuses on the technical framework the government uses to automate penalties under the country's welfare system.
Shortly after the report was published, though, Sydney University Deputy Director of Health Law Chris Rudge noticed citations to multiple papers and publications that did not exist. That included multiple references to nonexistent reports by Lisa Burton Crawford, a real professor at the University of Sydney law school.
"It is concerning to see research attributed to me in this way," Crawford told the AFR in August. "I would like to see an explanation from Deloitte as to how the citations were generated."
“A small number of corrections”
Deloitte and the DEWR buried that explanation in an updated version of the original report published Friday "to address a small number of corrections to references and footnotes," according to the DEWR website. On page 58 of that 273-page updated report, Deloitte added a reference to "a generative AI large language model (Azure OpenAI GPT-4o) based tool chain" that was used as part of the technical workstream to help "[assess] whether system code state can be mapped to business requirements and compliance needs."
Of the 141 sources cited in an extensive "Reference List" in the original report, only 127 appear in the updated report. In addition to the now-deleted references to fake publications from Crawford and other academics, the updated report also removed a fabricated quote attributed to an actual ruling from federal justice Jennifer Davies (spelled as "Davis" in the original report).
Deloitte Australia said it will repay the final installment of its contract with the government, though it's unclear which portion of the total contract that represents. A spokesperson for DEWR told the AFR that "the substance of the independent review is retained, and there are no changes to the recommendations."
But Sydney University's Rudge told AFR that "you cannot trust the recommendations when the very foundation of the report is built on a flawed, originally undisclosed, and non-expert methodology... Deloitte has admitted to using generative AI for a core analytical task; but it failed to disclose this in the first place."
Assassin's Creed Mirage's Saudi-Backed Valley of Memory Expansion Looks a Generous Free Side-Quest — Though Not One Many Fans Would Have Asked For