Who Decides When Your AI Agent Goes Wrong? Running an Incident Tabletop Before the Regulator Asks

Picture a claims agent that runs overnight. By morning it has approved several hundred claims, and a growing share of them trace back to synthetic identities. Nothing crashed. No alert fired, because the agent did exactly what its tools allowed. At 8:40 the operations lead notices the pattern. At 9:15 someone asks whether to switch the agent off. At 10:00 a journalist calls.

Who decides to pause the agent? Who is allowed to reverse its decisions? Who tells the regulator, and with what facts? In most organisations I talk to, the honest answer is "we would work it out on the day." That is the gap an incident tabletop is built to close.

Why an AI agent incident is not a normal IT incident

Incident response plans were written for outages and breaches: a system goes down, or an attacker gets in. An agent incident looks different. The system is up, the credentials are valid, and the damage is a stream of business decisions that each looked plausible on its own. The fraud team sees losses, the IT team sees a healthy service, and the business owner sees throughput. Each of them can reasonably assume someone else owns the problem.

The security community already names the root cause. OWASP's Top 10 for LLM applications lists Excessive Agency as a distinct risk: damaging actions taken in response to unexpected, ambiguous or manipulated model outputs, usually because the agent has too much functionality, too many permissions or too much autonomy. Its recommended controls include human approval for high-impact actions and authorisation enforced in downstream systems rather than left to the model. Those controls only help if people know when to use them.

Why now

Three things have changed. First, regulators now talk about agents by name. FINRA's 2026 Annual Regulatory Oversight Report describes AI agents acting without human validation, acting beyond the user's intended scope and authority, and producing multi-step reasoning that is hard to trace, and it asks firms to consider agent-specific controls.

Second, the testing obligation for incident plans already exists. New York's cybersecurity rule, 23 NYCRR 500.16, requires covered entities to test their incident response and business continuity plans at least annually, with all staff and management critical to the response. That rule does not mention AI agents. But if your agents can move money or approve claims, an annual test that never exercises an agent failure leaves out a scenario you are running in production.

Third, the "the AI did it" defence has already been tried and failed. In Moffatt v. Air Canada, a British Columbia tribunal held the airline responsible for wrong information its website chatbot gave a customer, and rejected the argument that the chatbot was a separate entity. It was a small-claims case, but the principle travels: what your agent does is what your company did.

How it fails in practice

From years of building fraud detection for banks, the failure pattern I see most is not technical. It is a hand-off problem:

• No kill-switch owner. Someone can technically stop the agent, but nobody has written authority to do it during business hours without a meeting.
• No reversal path. The agent's decisions flow into payments, claims or account changes. Undoing them needs another team, another system, and often the customer.
• Evidence evaporates. Prompts, tool calls and model versions are not retained in a form an examiner or a lawyer can use.
• The message drifts. Legal, communications and risk each draft a different account of what happened.

A self-check you can run this month

Take one agent that is in production or close to it, put six people in a room for an hour, and answer these out loud:

• Who can pause this agent, and how fast? Name the person and the mechanism. Time it.
• What is the threshold for pausing? A number of bad decisions, a loss amount, a single confirmed case? If nobody can state it, the decision will be made under pressure.
• Can you list everything the agent did in the last 12 hours? Every tool call, every approval, with inputs. If this takes days, your investigation will too.
• Who reverses the damage? Which decisions can be clawed back, by whom, and which are final once made?
• Who talks to the regulator, and on what timeline? Check your notification obligations before the day, not during it.
• Who owns the press line? And does that person understand enough about the agent to avoid saying something untrue?
• What would make you turn it back on? Restart criteria matter as much as stop criteria.

If two people in the room give different answers to any of these, you have found the first thing to fix.

What a good tabletop looks like

A useful exercise is short, specific and uncomfortable. It uses a scenario close to your real deployment, injects new facts at intervals (a second batch of bad approvals, a customer complaint on social media, a regulator's email), and forces named people to make decisions on the clock. Its output is not a slide deck. It is a written record of who decided what, where the plan held, and where people improvised.

It should also run on public and synthetic data. A tabletop does not need your production systems or customer records to be realistic, and keeping them out makes it far easier to run.

How Riskernel runs this as a fixed-scope assessment

The AI Agent Incident Tabletop is a facilitated exercise for your executive and risk team, run as a 3-hour session or an on-site half day. I design the scenario around the kind of agent you actually run (claims, onboarding, servicing, fraud or AML work), and run it on public and synthetic data only.

You get the exercise itself, a scored after-action report showing where decisions held and where they stalled, and a written record of decision rights: who can pause, reverse, notify and speak, and on what trigger. Scope and fees are agreed on a short call.