Optimism beats evidence in the drive to fingerprint the world
By David Moss • Get more from this author
Posted in Government, 14th August 2009 12:11 GMT
Special report Until the 16th century, educated opinion, as codified by Ptolemy, held that the Earth is at the centre of the universe. Then along came Copernicus.
On 29 June 2009, the Identity & Passport Service (IPS) published their latest paper on the National Identity Service (NIS). According to Safeguarding Identity (pdf), "the vision for the NIS is that it will become an essential part of everyday life, underpinning interactions and transactions between individuals, public services and businesses and supporting people to protect their identity" (para.3.32).
Early adopter
Placing the NIS at the centre of social interaction like this makes IPS about 92 million miles wide of the mark.
How is the NIS supposed to achieve IPS's vainglorious objective? “Our intention is that, at the core of the information used to prove identity will be biometrics, such as photographs and fingerprints” (para.3.6).
It follows that, in the eyes of IPS, the NIS stands or falls on the reliability of the biometrics chosen. If they don’t work, the NIS can’t work.
When he was Home Secretary, David Blunkett told us that biometrics "will make identity theft and multiple identity impossible. Not nearly impossible. Impossible". That is the commonly held view.
It may be the commonly held view, but is it correct?
Ask the Home Office's scientists
Not everyone agrees. Dr Tony Mansfield and Mr Marek Rejman-Greene, for example, opened their February 2003 report to the Home Office by saying the exact opposite: “Biometric methods do not offer 100% certainty of authentication of individuals” (para.4).
Tony Mansfield specialises in biometric device testing at the National Physical Laboratory and Marek Rejman-Greene is the Senior Biometrics Advisor at the Home Office Scientific Development Branch (HOSDB). Who is right? Those two individuals? Or David Blunkett and all the politicians and civil servants and journalists in the UK and abroad who agree with him?
For Copernicans, the answer depends on the evidence.
There follows a review of the biometrics evidence that has come to light over the past six years.
Tony Mansfield and Marek Rejman-Greene’s report makes the distinction between two different jobs for biometrics – identification (section 2.1) and verification (section 2.2).
Identification is the job of proving that each person has one and only one entry on the population register. Professor John Daugman, the father of biometrics based on the iris, demonstrates easily that that job is not feasible for large populations.
uppose that there were 60 million UK ID cardholders. To prove that each person is represented by a unique electronic identity on the population register, each biometric would have to be compared with all the rest. That would involve making 1.8 x 1015 comparisons.
Suppose further that the false match rate for biometrics based on either facial geometry or fingerprints was one in a million (1 x 10-6). It isn’t. It’s worse than that. But suppose that it was that good, then there would be 1.8 x 109 false matches for IPS to check.
It is not feasible for IPS to check 1.8 billion false matches. It is therefore not feasible for these biometrics to do their identification job.
Verification on the other hand, according to Tony Mansfield, is millions of times easier, and requires only that your facial geometry match the photograph recorded on your ID voucher (whether a passport or an ID card or a biometric visa) or that your fingerprints match the templates recorded on the voucher that you proffer to an immigration control officer, for example, or to a bank manager or to a GP, to underpin your transactions and interactions with them.
It may be millions of times easier, but can the biometrics chosen for the NIS achieve even the job of verification?*
Apparently not.
In 2004, the UK Passport Service (UKPS, now IPS) conducted a biometrics enrolment trial. 10,000 of us took part and a report of the trial was published in May 2005.
Under the heading Key Findings (para.1.2), sub-heading Verification Success Rates (para.1.2.1.4), the report says that 31 per cent of people could not have their identity verified using facial recognition technology – they were told that they did not match the photograph of them taken only five minutes before. And that was just the able-bodied participants – for the disabled, the false non-match rate was 52 per cent. And, using flat print fingerprinting technology, 19 per cent of the able-bodied participants could not have their identity verified, and neither could 20 per cent of the disabled**.
Fingerprint verification results from the 2004 trial
With some people, you can give them any amount of evidence, they will continue to believe that the Earth is flat.
Failure rates of 19 and 20, and 31 and 52 per cent clearly scupper IPS’s plans for the NIS. Millions of us would be unable to prove our right to work in the UK if that proof depended on biometrics, we would be unable to obtain non-emergency state healthcare and our children would be barred from state education.
* Verification is a source of some confusion among politicians and the media. If my flat print fingerprints match the templates stored on an ID voucher, then the biometrics have successfully completed their verification job. But was the ID voucher issued by IPS? And even if it was, have I tampered with it since then and inserted my biometrics? The technology needed to answer those further questions and help to make the NIS secure is PKI – the public key infrastructure – and not biometrics. Even David Blunkett gets the two confused, which is surprising considering that he had a job with a PKI company, Entrust, Inc.
** Traditional rolled prints are trusted worldwide and are admissible as evidence in court. But IPS propose to use the new technology of flat print fingerprinting (para.30.86), which is quick and clean, requires no expert in attendance, but appears to fail 19 or 20 per cent of the time and it is not admissible as evidence in court. To give these two different technologies the same name, “fingerprinting”, is literally a confidence trick. According to Professor Daugman, the key to a biometric is the amount of randomness and complexity that it contains. 'Face recognition is inherently unreliable because there isn't nearly enough randomness in the appearance of different faces. Fingerprints are vastly better biometrics than faces,' he says, 'but better still are iris scans'". But note the problem discovered in the UKPS biometrics enrolment trial (para.1.2.1.3). 10 per cent of able-bodied participants were unable to register their iris scans in the first place. That figure rose to 39 per cent for the disabled.
Faced with revolution, the government would have to abandon the NIS. Logic, maths, science, a basic understanding of technology, businesslike common sense, an adult sense of responsibility and simple truth-telling all suggest that the NIS should have been abandoned on the day the biometrics enrolment trial report was published*.
Instead, what did IPS say when the House of Commons Science and Technology Committee confronted them with these failure rates? According to the Committee’s July 2006 report, IPS said that the key findings of their biometrics enrolment trial were not key findings, and that the trial was not a test of the reliability of biometrics, but only a test of their usability (para.88).
If it wasn’t a test of reliability, why are the reliability figures reported as key findings? Why would IPS want to test usability but not reliability? Surely they wouldn’t deploy the NIS with biometrics that are congenial to everyone but just don't happen to work. And what is this distinction between reliability and usability? For 300 pages, the May 2005 report discusses usability almost entirely in terms of reliability.
Unreliable reporting
Despite the polite and sensible entreaties of the Committee, no large-scale field trial of the reliability of flat print fingerprinting has been subsequently conducted by IPS. If the biometrics enrolment trial was not a reliability test, then there is still no evidence to support IPS’s claim that flat print fingerprinting can deliver their vainglorious ambition.
According to the House of Commons Science and Technology Committee report, "on 6 March 2006, we met informally a group of senior policy advisers from the Department of Homeland Security to discuss the identity cards programme. When questioned about the maturity of biometric technologies, the advisers agreed that currently the technology was probably not as reliable or as accurate as it might need to be for a national identity card scheme" (para.81).
Logic, maths, science, etc… all having been abandoned, IPS told the Committee, not quite that the earth is flat, but that the maximum acceptable false non-match rate for flat print fingerprinting is one per cent (para.18) and they pointed the Committee (p.126ff) to a May 2004 report written by the US National Institute of Standards and Technology (NIST).
* The false non-match rate associated with IPS's chosen biometrics varies between 19 and 52 per cent. Really? Is that true? There is an obvious counter-example – schools up and down the country use biometrics to take the register, to manage library-lending and to operate cashless canteens. Why don’t they suffer from 19-52 per cent false non-match rates? The answer is given in NIST's May 2004 report. Please see section 4.3, Trading FRR for FAR, pp18-19. (FRR = false reject rate = false non-match rate. FAR = false accept rate = false match rate.) Schools can calibrate their biometric equipment to operate close to a zero false match rate or close to a zero false non-match rate, one or the other, but not both. There is a trade-off. If the school goes for a low false match rate, they will inevitably get a high false non-match rate and vice versa. They go for a low false non-match rate so that not too many pupils starve. That's why there is no false non-matching problem to report.
But there must be a concomitant false matching problem. According to Figure 9 of the NIST report (pp.16-17), with a low false non-match rate, the false match rate can quickly rise to 10 per cent and even higher. Suppose that a pupil collects his or her lunch from the canteen and is identified only by his or her flat print fingerprint. Then, in a school of 1,000 pupils, it is likely that the computerised biometric canteen system can't identify which of 100 pupils in the school is the luncher and that the school is therefore wasting its money.
The implication is instructive – it is that biometric identity is discretionary. Depending on how the operator calibrates the equipment, the biometrics might say that you are you, or they might not. That is not how we usually understand identity. IPS is using an alien version of the concept of identity.
Following 9/11, the newly established US Department of Homeland Security (DHS) designed US-VISIT, a biometrics-based scheme to protect the US border from infiltration by malevolent aliens. NIST conducted a computer-based trial of flat print fingerprinting to predict the success of US-VISIT. They estimated that the technology would successfully verify identity 99.5 per cent of the time. That is equivalent to a false non-match rate of 0.5 per cent, well within IPS’s one per cent limit.
It may not be immediately obvious how outrageous NIST's forecast is. In the 2004 international fingerprint verification competition, FVC2004, the best algorithm achieved a false non-match rate of 6.21 per cent at a false match rate of approximately zero per cent. Even with a false match rate of one per cent, the best false non-match rate was 2.54 per cent, and IBM promptly formed a partnership with the winning company, Bioscrypt, Inc. No-one in 2004 had ever seen a flat print fingerprint algorithm capable of IPS's one per cent false non-match rate, let alone NIST's 0.5 per cent, and they still haven't.
In December 2004, the US Office of the Inspector General (OIG) reviewed the statistics for the first year of operation of US-VISIT. On average, 118,000 people a day presented themselves to primary inspection at the borders. Primary inspection is largely a biometrics check. If the false non-match rate is 0.5 per cent, you would expect 590 of them to fail and to be referred to secondary inspection by human beings. The actual figure was 22,350 failures or 19 per cent. Just like in the UKPS biometrics enrolment trial.
Round up more fingers
NIST had argued that a true accept rate of 99.5 per cent could be achieved using only two fingerprints. Once US-VISIT had demonstrated that the figure was more like 81 per cent, they teamed up with the US Department of Justice to lobby DHS and the State Department to use 10 fingerprints instead of two.
Would that help? It sounds as though it should but, as Tony Mansfield will tell you, increasing the number of fingers sampled will not lead to an exponential improvement in reliability. Your fingers are not independent events, there are correlations, and if the minutiae on your index fingers are poorly defined they are likely to be poorly defined on your other fingers, too.
Some researchers note that it is hardly worth printing ring fingers and little fingers. NIST found that right index fingers are inexplicably "better" than left index fingers (para.3.1.1).
Moving from two prints to 10 may not help much after all.
NIST provides no support for IPS and the idea that the methodology used in their May 2004 report is a reliable way of forecasting the outcome in the field is thoroughly discredited.*
* Messrs Mansfield and Rejman-Greene note in their report that there are exceptional problems with flat print fingerprinting (Appendix B, p.34). It is obviously hard to enrol people onto the population register if they are missing fingers and/or entire hands. It can also be hard to register older people, they say, manual labourers, East Asians and – that other unimportant minority group – women.
IPS have never explained what alternative arrangements will be made for these cases and the NIST report doesn't consider them. We remain in the dark, therefore, but IPS can't stay silent on the issue forever and when they do propose their alternative arrangements, the question will arise why we can't all use those alternatives and forget about biometrics altogether.
IPS may not have conducted any subsequent trials of flat print fingerprinting, but their cousins the UK Border Agency (UKBA) did start a six-month trial of biometrics based on facial recognition at Manchester airport in August 2008. Eleven months later, no results have been published by UKBA.
But there have been leaks to the BBC and the Daily Telegraph.
"Sources from the UK Border Agency (UKBA) have revealed that the devices are failing to detect when two people pass through them at the same time. The system [smartgates, the Automated Clearance System (ACS)], which replaces traditional passport control measures, is undergoing a live trial at Manchester Airport, where a UKBA worker said it was suffering almost daily malfunctions. He said immigration officers had been able to accompany travellers through the scanners without an alarm being triggered, even though the booths are supposed to detect if more than one person enters at a time. 'Immigration officers have been able to tailgate passengers through the machine, without the machine picking it up,' he said.
"The source said there were malfunctions taking place almost daily in the pilot project, which is thought to have cost the taxpayer several hundred thousand pounds. 'There are five pods and when one breaks down, they all break down.'
"Up until the point of the official launch, it was rejecting 30 per cent of those who tried to get through it,' the UKBA worker said. 'We believe they had to recalibrate it – essentially make it easier to get through the system'".
Unlikely bin Laden doppelgangers
And the Telegraph has tracked down another biometrics expert, like Tony Mansfield and Marek Rejman-Greene:
"In a leaked memo, an official says the machines have been recalibrated to an 'unacceptable' level meaning travellers whose faces are shown to have only a 30 per cent likeness to their passport photographs can pass through. Rob Jenkins, an expert in facial recognition at Glasgow University's psychology department, said lowering the match level to 30 per cent would make the system almost worthless. Using facial recognition software from Sydney airport in Australia set at 30 per cent, he found the machines could not tell the difference between Osama bin Laden and the actors Kevin Spacey or even the actress Winona Ryder, while Gordon Brown was indistinguishable from Mel Gibson".
When asked whether this technology works, instead of referring to the results of their own field trial, UKBA point to a report produced in March 2007 by... NIST.*
This is another one of NIST’s computer-based trials, not a field trial. The conclusion drawn from the report by both UKBA and HOSDB is that biometrics based on facial geometry are now reliable enough for airport security. No earlier report is cited. No later report is cited. This is the single report on which UKBA and HOSDB rely. Are they right to place so much confidence in it?
The trial uses eight different sets of sample biometric data (p.35). Two of them are sets of iris scan data. Iris scans are not on offer in the NIS and those results of NIST’s are therefore irrelevant. One is a set of three-dimensional face data, also not on offer in the NIS and so, again, irrelevant. Of the remaining five sets of data, four of them are taken from very few subjects (257 subjects in the worst case, then 263, then 335, and 336 subjects in the best case). As any GCSE student can tell you, that is too small a sample for UKBA to be able to decide whether the technology would work for 60 million people in the UK.
Which leaves us with just one relevant sample dataset, of 36,000 subjects. And how well did facial recognition verify their identity? According to Figure 20 of the NIST report (p.46), at a false match rate of 0.01 per cent, 100 times worse than Professor Daugman's working figure, the false non-match rate varies between eight per cent and 19 per cent, depending on which supplier’s biometrics algorithm is used.
* In the course of its May 2004 report, claiming that flat print fingerprinting works well, NIST had this to say about the alternative, facial recognition: "Even under controlled illumination, which is not used in US-VISIT, the error rate of face is 50 times higher than the two-fingerprint results discussed here. If the case of uncontrolled illumination is considered, this factor would be 250. This means that face recognition is useful only for those cases where fingerprints of adequate quality cannot be obtained" (para.3.3).
By March 2007, NIST would have us believe, facial recognition algorithms had improved by one or even two orders of magnitude. IPS, UKBA and HOSDB may believe that. But note that the US government seems to be in no hurry to incorporate facial recognition into US-VISIT and certainly not into an ID card scheme for their own nationals.
Once again, NIST provide no support to IPS or UKBA or HOSDB. A false non-match rate of between eight and 19 per cent does not sound like convincing evidence for the reliability of facial recognition as a biometric. And remember that these figures emanate from a methodology which has already been discredited as a predictor of outcomes in the field.
Mansfield and Rejman-Greene note in their report that the reliability of biometrics based on facial geometry falls off a cliff two months after people are first photographed – "even under relatively good conditions, face recognition fails to approach the required performance" (para.52d). For the first two months in the life of any new passport, verification will be erratic. For the last 118 months, it will be impossible – that is the implication. What do NIST have to say about this problem? Nothing.
There is one other piece of facial geometry evidence which it would be useful to see, and that is a report on the results of China’s 10 million faces test, an element of Operation Golden Shield. China, like the UK, is keen on using biometrics. That report is unfortunately not available.
Here at the end of the review, the adventitious question arises of why do politicians and civil servants all over the world continue to advocate the use of biometrics when the evidence simply doesn’t support them? There is no answer. Their behaviour is inexplicable.
One thing is clear, though, and that is that biometrics cannot deliver. Identification is not feasible. Verification is laughably unreliable. And the flat earther David Blunkett is wrong. So is Tony Blair when he says that “biometrics give us the chance to have secure identity”. And so is Gordon Brown when he says that biometrics “will make it possible to securely link an individual to a unique identity”.
The scale of the institutional fantasy which constitutes the NIS is grotesque. Biometrics cannot underpin the NIS and so, by IPS’s logic, the NIS cannot underpin the “interactions and transactions between individuals, public services and businesses”. Safeguarding Identity is a false prospectus – no properly managed stock exchange would allow its shares to be listed. The NIS is guaranteed to fail.*
We have already noted some of the practical implications of NIS fantasy – millions of us would have trouble proving our right to work, getting state healthcare and state education. Here are two more:
• IPS have not even provided a way to collect everyone's biometrics. Italy (population 58 million) has a national network of about 8,000 ID card registration centres. The Netherlands (17m) has – or plans to have – about 4,000 centres. The UK (61m) was recommended by Tony Mansfield and Marek Rejman-Greene to set up a network of about 2,000 centres (para.105), a curiously low number, but not as low as the number IPS came up with: 69. Instead of registering people themselves, IPS expect high street retailers to do the job for them. But which high street retailer, having spent decades growing a trusted brand, will risk the anger of 20 per cent of their customers who, having handed over their fingerprints, are told as a result that they have no right to work in the UK? Fantasy. ®
• If UKBA use flat print fingerprinting to check everyone coming into the country, and everyone leaving the country, UK nationals, other EEA nationals and non-EEA nationals alike, and if the technology performs as well as it does in US-VISIT, then they will have to detain about 8,000 travellers a day. The prisons are full. Where are UKBA going to put all the detainees? Fantasy.
* Anyone not convinced by the facts, figures and arguments presented here may consider the conclusion of the Office of Government Commerce, an independent office of HM Treasury: "This has all the inauspicious signs of a project continuing to be driven by an arbitrary end date rather than reality... I conclude that we are setting ourselves up to fail".
What's more, the UK Passport Agency (UKPA, previously the Passport Office, subsequently UKPS, subsequently IPS) agrees: "I wouldn't argue with a lot of this...".
In addition to the politicians and civil servants driving the NIS, there are, of course, the consultancies, notably PA Consulting. PA give it as their opinion that biometrics is mostly hype.
And beyond the consultancies, there are the biometrics companies themselves. The history of L-1 Identity Solutions, Inc., one of the more financially successful members of the industry, provides some support for PA's view and no support for the NIS.
David Moss has been in IT for over 30 years now and works as an IT consultant. He has failed for over six years to convince the government that we already have ID cards, in the form of our mobile phones, but it's early days yet - the standard gestation period is apparently 12 years. While waiting for the government to have the original idea themselves that we don't need the ID cards the Identity & Passport Service keep writing press releases about, because we already have mobile phones, he is trying to make people confront the evidence before their eyes, that the biometrics emperor has no clothes.
Editor's note: A more heavily annotated version of this document is also available in PDF form, from David Moss himself, or from The Register's library, here.
