Insights

Testing AI for blind spots: a worked example

What happens when you actually test a model against edge cases, instead of assuming you know where it fails.

Test one: invented scenarios, real model

We wrote five short, invented decision scenarios in a fraud-review style — a transaction flagged by an automated system, with just enough context to make a call — and had a general-purpose AI model decide each one blind, with no ground truth given. We then compared its call against a scripted "correct" judgment.

Result: 3 of 5 matched. The model handled clean statistical anomalies well on its own — an impossible travel pattern, a spoofed merchant ID, a loyalty-adjusted risk call. Both misses shared a shape: the model defaulted to the textbook risk heuristic and missed an equally plausible alternative explanation grounded in specific context (a legitimate transaction tied to a real-world event outside the transaction data itself).

Caveat, stated plainly: n=5, one model, invented scenarios. Enough to demonstrate the mechanism, not a benchmark result.

Test two: real data, real labels

For a second test, we pulled real transactions from a well-known, publicly available, genuinely labeled fraud dataset — real anonymized cardholder data with true fraud/not-fraud outcomes, not synthetic. We gave a model six real transactions' raw feature values, with labels hidden, and asked it to classify each.

Result: 6 of 6 correct.

The honest caveat that matters mostThis specific dataset is famous — it's the subject of countless public tutorials and blog posts explaining exactly which anonymized features tend to correlate with fraud in it. Strong performance here may partly reflect exposure to public commentary about this dataset specifically, not pure pattern discovery. Public benchmarks are structurally contaminated in a way genuinely novel data never is — which is exactly why testing on data nobody has written about is the only test that means anything.

The takeaway

Two honest tests, two different findings — a model can look strong on famous public data and still miss a specific, describable class of case when reasoning from context alone. Neither result is a verdict on "AI and fraud" in general. Both are exactly the kind of finding you only get by actually running the test, rather than assuming the answer.