A breakdown in AI governance at Deloitte Australia has forced the consulting giant to refund part of an AU$440,000 (US$290,000) government contract after AI-generated fabrications were included in a final report, exposing vulnerabilities that analysts said are symptomatic of broader challenges as enterprises rapidly scale up AI adoption.
Deloitte used OpenAI GPT-4o to help produce a 237-page independent review for Australia’s Department of Employment and Workplace Relations (DEWR), but failed to detect fabricated academic citations and non-existent court references before delivery. The firm also did not disclose its AI use until after errors were discovered.
The Australian government told ComputerWorld that Deloitte “has confirmed some footnotes and references were incorrect” and agreed to repay the final installment under its contract. The Secretary provided an update, noting that a correct version of the statement of assurance and final report has been released.
“The refund amount will be disclosed when the contract notice is updated on AusTender,” a DEWR spokesperson said.
“It’s symptomatic of broader challenges as enterprises scale AI without mature governance,” said Charlie Dai, vice president and principal analyst at Forrester. “Rapid adoption often outpaces controls and makes similar incidents likely across regulated and high-stakes domains.”
Despite the errors, “the substance of the independent review is retained, and there are no changes to the recommendations,” the DEWR spokesperson added.
Detection through domain expertise
Dr. Christopher Rudge, a University of Sydney researcher specializing in health and welfare law, discovered the fabrications when reviewing the report. He recognized cited authors as colleagues who had never written the attributed works.
“One of the easiest ways to tell was that I knew the authors personally, and so I knew they hadn’t written the books to which they were attributed,” Rudge told Computerworld. “The works were almost too perfectly tailored and too bespoke for the text, and that was a red flag.”
Sam Higgins, vice president and principal analyst at Forrester, said the incident served as “a timely reminder that the enterprise adoption of generative AI is outpacing the maturity of governance frameworks designed to manage its risks.”
“The presence of fabricated citations and misquoted legal raises serious questions about diligence, transparency, and accountability in consultant-delivered work,” Higgins said.
Shared responsibility for quality control
The vendor and the client share the responsibility, argues Sanchit Vir Gogia, chief analyst and CEO at Greyhound Research. “Clients cannot claim surprise when tools they themselves use internally appear in a vendor’s workflow,” he said. “Why wasn’t that check done before the report went public? Accountability works both ways.”
The revised report, published last Friday, included a disclosure absent from the original, acknowledging Deloitte used the AI tool to address “traceability and documentation gaps.”
Disclosure often lags because firms frame AI usage as “internal tooling” rather than material to outcomes, creating transparency and trust gaps, Dai said.
Higgins said the post-hoc disclosure undermined the Australian government’s own guidance requiring transparency when AI is used in decision-making. “Deloitte’s post-hoc disclosure sets a poor precedent for responsible AI use in government engagements,” he said.
The incident occurred as Deloitte announced a major partnership to deploy Anthropic’s Claude AI to its nearly 500,000 employees worldwide.
Deloitte did not immediately respond to a request for comment.
Modernizing vendor contracts
The disclosure failures and quality control breakdowns in the Deloitte case point to fundamental gaps in how organizations contract with vendors using AI tools.
Rudge argued that organizations deploying AI tools may need to institutionalize subject-matter expert review as a mandatory final quality gate, even as AI promises significant cost and time savings.
“At the end of any AI-assisted project, or any significant project where AI has been dominantly the knowledge-making tool, firms or organizations might still need to employ a human proofreader who is a subject-matter expert in the area to sense-check the documents,” he said.
Rydge suggested that the economics could still favor AI adoption even with expert review built in. “Maybe things will be so much cheaper to produce in the knowledge world that the cost of a subject matter expert who proofreads the paper, report, or product will only be understood as a small cost,” Rudge said. “But vetting by professionals should still continue to be the gold standard.”
Gogia said most current agreements still assume human-only authorship even though automation now underpins much of the work. “When something goes wrong, both sides scramble to decide who carries the blame — the consultant, the model, or the client reviewer who signed it off,” he said.
“Tech leaders should ask explicitly about AI involvement, validation steps, and error-handling processes,” Dai said. “They should also seek clarity on human review, source verification, and accountability for factual accuracy before accepting deliverables.”
Higgins outlined key questions: “What generative AI tools are being used, and for which parts of the deliverable? What safeguards are in place to detect and correct hallucinations? Is there a human-in-the-loop process for validation? How is provenance tracked?”
Building mature governance frameworks
Beyond vendor management, analysts said organizations need comprehensive governance frameworks that treat AI as a systemic risk requiring formal policies and cross-functional oversight.
Dai said CIOs and procurement teams should include clauses mandating AI disclosure, quality assurance standards, liability for AI errors, and audit rights. “They should also pursue alignment with frameworks like NIST AI RMF or ISO/IEC 42001 for risk management,” he said.
Higgins said provisions should require upfront disclosure, mandate human review, define liability for AI errors, and include audit rights.
“IT leaders should treat AI as a systemic risk, not just a productivity tool,” Dai said. “They should implement vendor AI governance, enforce disclosure, and integrate robust QA to prevent reputational and compliance fallout.”
Gogia said he saw an emerging model where joint review boards include client and vendor representatives, ensuring AI-produced content is examined before endorsement. “That is what maturity looks like — not the absence of AI, but the presence of evidence,” he said. “Governance in the AI age will reward collaboration, not confrontation.”
Read the full article here
