Your LLM May Have Already Seen the Test: Validating AI Models With Held-Out Exams

A model-risk team is asked to validate an LLM before it goes live in a fraud or AML workflow. The vendor sends benchmark results. The scores are high. The validator signs off, and six months later the model performs noticeably worse on the bank's own cases than the benchmarks promised.

Nobody lied. The problem is that the test was not really a test.

The contamination problem

Large language models are trained on enormous scrapes of public text. Public benchmarks are public text. Researchers call the result benchmark data contamination: the model has seen the test, or something close to it, during training. A 2024 survey of the problem defines it as exposure to benchmark data during training that distorts evaluation results, and an empirical study by Singh and colleagues found that contamination may have a much larger effect on reported scores than model releases acknowledge.

For a bank, the consequence is simple. A high score on a public benchmark tells you the model is good at that benchmark. It does not tell you the model can reason about your cases, and it gives a validator nothing to stand on when an examiner asks how the model was tested.

What validation is supposed to do

US banks validate models under the Federal Reserve and OCC's SR 11-7 guidance on model risk management. Its central idea is effective challenge: critical analysis by people who are independent of the model's development, competent to find its weaknesses and with enough standing to force changes. Validation covers conceptual soundness, ongoing monitoring and outcomes analysis.

Effective challenge is hard enough for a scorecard. For an LLM, it is harder, because the validator cannot inspect the model's reasoning the way it can read a logistic regression. That leaves testing as the main lever, and testing only counts if the test cases are ones the model cannot have memorised.

How validation goes wrong in practice

• Reusing the vendor's evaluation. The vendor chose the tests, ran them and reported the result. That is development evidence, not independent challenge.
• Static internal test sets. The bank builds 200 test cases once and reuses them for every model version. Prompts get tuned against them. The set stops measuring anything.
• Average scores hiding edges. A model can be 95% accurate overall and reliably wrong on a narrow pattern an attacker can find, such as a specific phrasing, a currency or a transaction structure.
• No written method. When the examiner asks why these tests and not others, there is no document to point to.

Fraud people know the third one well. Fraudsters do not attack your average case. They probe for the corner where the model is confidently wrong and then scale it.

Self-check: is your validation evidence held out?

• Who wrote the test cases? If the vendor or the development team did, it is not independent.
• Could any test case, or a close variant, be on the public internet? If yes, assume the model may have seen it.
• Is the right answer computed or opinion? Where the answer can be calculated by code (a balance, a threshold breach, a rule outcome), compute it. Human grading drifts.
• Can you generate a fresh set tomorrow? If the test set is fixed, it will leak into prompt tuning. A generator that draws new cases from the same logic each run cannot be memorised.
• Have you probed for edges, not just averages? Small changes to inputs that should not change the answer, and adversarial phrasings, often reveal more than another thousand normal cases.
• Is the method written down and signed? A validator should be able to hand over a document that explains what was tested, how cases were generated and how scoring works.

What a held-out exam looks like for a fraud model

The idea is not exotic. Instead of a fixed list of questions, you write templates: a structured scenario with variables, and code that computes the correct answer for any values of those variables. A template might describe a customer with a payment history, a new payee and a transfer, and the code determines whether a given rule or policy is breached. Each run samples new values, so the cases are new every time, while the logic being tested stays the same and the answers stay checkable.

On top of that sit the probes. For each template you also generate near-identical variants that should produce the same answer (a reordered narrative, a different currency format, an irrelevant detail added) and variants built to push the model towards a wrong one. A model that flips its answer under changes that should not matter has an edge an attacker can use. That is a finding a validator can document, and a developer can fix before go-live rather than after a loss.

None of this replaces judgement about whether the model is fit for its purpose. It gives the validator evidence that was not available to anyone who could have tuned the model against it.

How Riskernel runs this as a fixed-scope assessment

The Held-Out Model Validation Exam is built for model-risk and fraud teams that must validate an LLM or fraud model before go-live. You get held-out test packs generated from code-checked templates, so every run draws fresh cases, plus adversarial probes for blind spots and exploitable edges.

Everything runs under a written test method that Amir Shachar signs, which your validators can attach to their own review. The engagement supports your validation; it does not replace your model-risk function's judgement. Scope and fees are agreed on a short call.