Danske Bank's fraud-detection AI: the 60% headline was a test figure, production cut false positives 20 to 35%

The problem

Danske Bank’s fraud screening ran on rules. Teradata’s press release describes the original system as “largely based on handcrafted rules that had been proactively applied by the business over time” [source]. Danske’s head of global analytics, Nadeem Gulzar, told Forbes the bank “was picking up 1,200 false positives per day in its transaction monitoring, and 99.5 percent were false positives” [source]. Teradata’s case study adds that the rules engine had “a low 40 percent fraud detection rate”, a figure that appears only in the vendor’s own document [source].

What was built

Teradata’s consulting arm Think Big Analytics “began working with Danske Bank in autumn 2016” to build the engine inside the bank’s existing infrastructure [source]. The engine “includes an interpretation layer on top of the machine learning models, providing explanations and interpretation of blocking activity” ([source]). Constellation Research, an independent analyst firm, describes two layers: machine learning ensemble models that Danske “developed and deployed”, and TensorFlow deep learning models that “have yet to be put into production” as of March 2018 [source].

The outcome

The production result, as reported independently, is smaller than the headline. Constellation Research says the deployed machine learning ensemble models “reduced false positives by 20 percent to 30 percent” [source]. Forbes, paraphrasing Gulzar, put the machine learning reduction at 35 percent, with detection of actual fraud improving “at roughly the same percent” [source]. The two figures disagree, neither source gives a measurement window, and TIN shows both rather than picking one ([source]). A third production-era figure sits in a 2017 Constellation SuperNova award entry filed under Gulzar’s name, which claims “an immediate 20-40% reduction in fraud false positives” and, a few lines earlier, that the bank “already reduced false positives by 50%” [source]. That entry is a self-reported award submission whose author is not disclosed, so it is counted as a conflict, not as corroboration ([source]).

The number most often repeated is bigger, and its status is printed in the vendor’s own small type. Teradata’s case study says the bank was able to “Realize a 60 percent reduction in false positives, with an expectation to reach as high as 80 percent” and “Increase true positives by 50 percent” [source]. The 80 percent is an expectation, not a result ([source]). Teradata’s October 2017 press release then swaps the two numbers: false positives reduced “by 50 percent” and the detection rate up “by around 60 percent”, with the asterisk resolving at the foot of the release to ”* Pre-production measures” [source]. The same footnote sits on Gulzar’s own quoted line, “Using AI, we’ve already reduced false positives by 50 percent*” ([source]). Forbes relayed the deep learning figures from Gulzar, “a 60 percent reduction in false positives and a 50-ish improvement in detecting actual fraud”, without that qualifier [source]. Five months later Constellation reported the deep learning models were still “demonstrating double-digit improvements in fraud detection and further reductions in false positives in a test environment” ([source]).

The weakest load-bearing source is Constellation itself: only its executive summary is public, the full report is behind a membership wall, and whether the study was vendor-commissioned cannot be confirmed from the public page [source]. The Forbes figures are a contributor’s paraphrase of an interview, not a direct quotation ([source]).

Related case files

• The UK’s DWP machine-learning fraud model: a fraud-detection model whose savings figure reached the public through an independent auditor, the evidentiary standard Danske’s production numbers never got.
• Michigan’s MiDAS fraud algorithm: what false positives cost when an automated fraud system’s error rate is measured by the record rather than by its builder.
• Klarna’s AI customer-service assistant: another Nordic financial-services AI headline whose launch figures and later reality had to be read side by side.