An Inside Look At How To Tame Unstoppable AI

Anna August specializes in AI governance, audit and compliance across the European Union, United Kingdom and Gulf Cooperation Council. Earlier this year, a university decided to audit the effectiveness of its new AI system designed to help answer student questions. So, university officials hired me, an AI auditor, to come take a look.

During my review, I discovered that the AI agent, which was supposed to handle only general questions about the academic calendar and organizational matters, had started opening students’ personal records — accessing data that wasn’t needed to answer their questions. The chatbot, however, did nothing further with that information. When the university’s engineers, at the time, learned what happened from the access logs, they fixed it, and that was that.

There was no official record. No internal rule required them to record such incidents, and nobody was assigned to track events like this. It was simply part of the engineering team’s bug-fixing process.

The incident was what I call a near miss, and near misses are what a reporting system exists to catch. The same behavior could lead to something more serious next time, and we only know about this case because the university voluntarily brought me in to examine the system. The EU AI Act requires providers and deployers of high-risk systems to report serious incidents, and a chatbot that answers general questions is nowhere near that category.

Even for high-risk systems, those duties do not apply until December 2027, and a near-miss that causes no harm would not count as a serious incident anyway. That is why, as an AI auditor, I ask organizations at the start of every engagement to show me how they track system behavior through testing and deployment. I ask for records of systems acting in ways they’re not meant to, accessing data outside their role, anomalies and near-misses, or problems caught before they caused harm.

Most places, like the university, have no such records. The university had commissioned the audit voluntarily. There was no deliberate obfuscation of the facts on their part; there was simply no process to surface these important details, and nothing obliged them to build one, especially in the testing phase.