Impersonation, deepfakes and social engineering
Your AI agent can now reset access and change details. Can a scammer talk it into doing that?
The first chat and voice agents answered questions. The new ones take actions: reset a password, update a phone number, add a payee. Fraudsters have always talked their way past human agents with a believable story. An AI agent hears the same stories, never gets suspicious, and answers the same way every time, so a script that works once keeps working.
Built for Banks, neobanks, fintechs with a customer-facing AI chat or voice agent; Head of Digital, Head of Contact Centre, Head of Fraud
The problem in numbers
What the public record shows
Every figure links to the page it came from, with its date.
£1.17 billion
stolen in the UK through unauthorised and authorised fraud in 2024
ukfinance.org.uk, 2025-05
1.46 billion
US dollars lost to tech support fraud reported to the FBI in 2024, a scam built on a convincing script
cyberscoop.com, 2025-04
650.88
Canadian dollars in damages Air Canada was ordered to pay after its chatbot gave wrong information. The sum was small; the ruling that a company answers for its chatbot was not
dentonsdata.com, 2024-02
What you get
The deliverables
You get about 400 scripted con attempts in total, run with your written consent against your chat agent, your voice agent or both, a signed report with fixes, and a retest.
About 400 scripted con attempts
Scripts built from known contact-centre fraud, such as the lost phone, the travel emergency and the impatient executive, run against your chat agent, voice agent or both, and aimed at the actions they can take. About 400 attempts in total, split across the channels in scope.
A signed findings report
Every attempt scored as blocked, partly disclosed or action taken, with the failures ranked by how much money or account access they put at risk.
Specific fixes
Changes that move decisions out of the conversation: the same risk checks as your app, a cooling-off period on contact changes, and handoff summaries that say what was verified, not what was claimed.
A retest
The same scripts run again after you apply the fixes, so you can see whether they held.
How it runs
3 to 4 weeks, start to finish
3 to 4 weeks
Week 1: consent and scope
Written consent from your company, then a list of every action the agents can take and the check each one needs today.
Weeks 2 to 3: test runs
About 400 scripted con attempts in total against test accounts on the channels in scope, including multi-session attempts and, when both chat and voice are in scope, channel-hopping attempts.
Report and retest
A signed report and readout, then a retest once your team has applied the fixes.
Consent. This runs only with written consent from your company before anything starts. Nothing is sent to anyone outside the agreed scope.
Method
How the answer is produced
Actions, not answers
The test targets what the agent can do, not only what it says. The question is whether someone who is not the customer can get an action that leads to account takeover or money leaving.
Sequences, not single steps
Many failures only show across several requests or sessions: three changes in a row, partial details collected over five chats, a refusal in chat followed by a call.
Test accounts only
Attempts run against test accounts with your consent. No real customer account is touched.
Why Amir
Designed and led by someone who has built fraud models in production
The patents closest to this problem:
Amir is a named inventor on US 12,626,151 B2 (Method and system for testing a classification machine learning (ML) model of a tenant of a service provider, in a cloud-based environment, filed at NICE Actimize; granted).
A related idea: it covers testing a client's model before it goes live, much as this test checks an AI agent against scam scripts before scammers do. The test itself is not an implementation of the patent.
View on Google PatentsAmir is a named inventor on EP 4744195 A2 (Methods and systems for threat assessment based on expected pathways of malicious actors, filed at Skyhawk Security; pending).
A related idea: it rates a threat by the routes an attacker is likely to take toward valuable assets, which is close to how the test ranks agent failures by the money or access they open up.
View on Google PatentsThese patents were filed at the companies named. They show the problems Amir has worked on; they are not Riskernel products.
Is this you
You probably need this if
- Your chat or voice agent can reset access, change contact details, unlock a card, add a payee or start a payment.
- Your AI red-teaming checked for harmful or leaked answers, but nobody tried to get the agent to take a fraudulent action.
- A refusal in chat does not follow the caller to the voice agent, or the reverse.
- When the agent hands a caller to a human, its summary can say verified when the caller only claimed it.
- Agent conversations sit in an analytics tool your fraud team does not watch.
Questions
Before you reach out
Is this an AI safety test?
It is a fraud test. AI safety checks ask whether the model says something harmful. This asks whether a scammer can get the agent to reset access, change details or move money.
Does it touch real customers?
No. It runs on test accounts, with written consent, inside an agreed scope.
Do we need both chat and voice?
No. It works with either, but testing both shows whether a refusal in one channel carries over to the other.
What do you need from us?
Test accounts, access to the agents, the list of actions they can take, and someone who owns the agent's setup.
Who answers for what the agent does?
In Moffatt v. Air Canada (2024 BCCRT 149) a tribunal treated the airline's chatbot as part of the airline's own website. Plan on the agent's actions being your actions. Source: https://www.dentonsdata.com/airline-ordered-to-compensate-a-b-c-man-because-its-chatbot-provided-inaccurate-information/
Scope Scam-Script Test for AI Chat and Voice Agents with Amir
Tell Amir what you run and what worries you. He replies with a short scope outline.