3303 - Biometrics: will the Center for Global Development reconsider? by David Moss

SUNDAY, 12 MAY 2013

Biometrics: will the Center for Global Development reconsider?

A recently published report on India's identity management scheme says that: "accurate, biometric-based, identification is quite feasible for large countries, including the US".

The suggestion below is that the conclusion should read: "subject to an annual audit, the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack and as long as we've got our maths right and as long as you realise that it's not identity that's being managed and as long as you're relaxed about the fact that anyone could have any number of entries on the population register and the fact that the discipline of biometrics is out of statistical control".

Will the authors consider issuing a revised version of their report?

-----  o  O  o  -----

On the rare occasions when trials have been conducted, the performance of biometrics technology has been disappointing. For example, when 10,000 of us took part in a UK government-run trial in 2004, about 20% of participants couldn't have their identity verified by their fingerprints.

That's useless. For example, the plan at the time was to use biometrics to confirm people's right to work in the UK. You can't tell 20% of the working population that it's illegal for them to work.

Ever optimistic, the biometrics industry is always announcing that the corner has been turned and that it's safe now to believe their promises. Is that true at last?

Consider Performance Lessons from India’s Universal Identification Program, a 12-page report by Alan Gelb and Julia Clark (Gelb and Clark, G&C).

It's about India's Unique Identification project (UID, also known as "Aadhaar") which relies on biometrics. UID/Aadhaar is the brainchild of UIDAI, the Unique Identification Authority of India. UIDAI are currently trying to register the biometrics of all 1.2 billion Indians.

G&C conclude that:

UID’s performance suggests that accurate, biometric-based, identification is quite feasible for large countries, including the US (p.8). UID shows that countries with large populations can implement inclusive, precise, high-quality identity systems by using existing technology (p.9).

Those conclusions are electric.

If they're correct.

But are they?

Why do G&C conclude that biometrics is now ready for large-scale deployment?

-----  o  O  o  -----

They have identified "160 [biometrics] programs in 70 countries that together cover over 1 billion people and include a wide range of applications – financial access, public payroll management, social transfers [?], health insurance and tracking and voter rolls – as well as national identification systems" (p.1).

Do they say that biometrics is ready for the big time because UIDAI have successfully implemented financial access systems which depend on biometrics? Or public payroll management systems? Or ...

Certainly not.

In fact G&C are at pains to say that:

UID is still at a relatively early stage, and links to the delivery of public programs are only now getting under way (p.2). It remains to be seen how robust the system is against active efforts to spoof it by providing faked fingerprints or iris images, to capture biometric data in transmission or to penetrate the database (p.2). Having a unique Aadhaar number issued by UIDAI itself entitles the holder to no specific privileges or programs (p.3). UID is still at an early stage. Only one fifth of the population has been enrolled and the linkage to public programs is just beginning (p.8).

Their logic doesn't depend on any practical successes of Aadhaar. There aren't any.

What G&C base their conclusions on is the performance of biometrics in the compilation of the Indian population register so far. If we are to answer the question whether their conclusions are correct, we need to look at the UIDAI statistics which measure the reliability of biometrics.

Before we do that, we need to update G&C's conclusions. There's a rider to add. Their p.2 warnings about spoofing and eavesdropping on telecommunications and burgling the population register need to be incorporated – the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack.

-----  o  O  o  -----

Slide rules ready? G&C say (p.5):

How many people would be denied enrolment because of a wrong determination that they had already enrolled? The False Rejection Rate (FRR) of the identity system is critical, especially with a large population. Since every new enrollment has to be checked against every existing enrollment, the number of comparisons increases with the square of the population ... Extrapolating this to our hypothetical Ughana population ...

Wrong.

Think in terms of ice cream. How many unique combinations of two ice cream flavours can you make from a choice of five flavours (A, B, C, D and E)? G&C suggest that the answer is 25, "the square of the population". It isn't. It's 10 (AB, AC, AD, AE, BC, BD, BE, CD, CE and DE), 5!/((5-2)! x 2!).

G&C have a peculiar habit. They're talking about India, with its population of 1.2 billion, but half the time when they use statistics they apply them to Ughana, a country they have invented. Why?

It confuses the readers. It may also confuse the writers. Forswearing Ughana and sticking to India, how many comparisons would have to be made to compare each one of 1.2 billion sets of biometrics against all the rest? Answer, 719,999,999,400,000,000 and not G&C's implied answer 1,440,000,000,000,000,000, which is out by a smidgeon over 100%.

New rider on the conclusions –  the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack and as long as we've got our maths right.

-----  o  O  o  -----

How did G&C get themselves into this bind?

It was in the midst of a discussion about false accept rates and false reject rates.

Aadhaar is all about comparing the biometrics captured by a fingerprint scanner or an iris scanner with the biometrics stored in the population register. Either they match or they don't.

Say your Aadhaar number is 782474317884, that there's an election on and that you have turned up at a voting centre. The biometrics associated with 782474317884 are retrieved from the population register and checked to see if they match your freshly scanned biometrics. If they do, you can vote. It's a one-to-one comparison, an "authentication" process.

Two ways the process can go wrong (among others):

• Either the process says the biometrics don't match, you are not who you claim to be, you are not President Lincoln according to Aadhaar, even though you are in reality. That's a false reject. 
• Or, alternatively, the process can say that, yes, you are who you claim to be, you are President Lincoln, when, in fact, you're not, you're an impostor. That's a false accept.

The False Accept Rate (FAR) and False Reject Rate (FRR) are two measures of the reliability of any biometrics system. They are inversely proportional. This is the "Detection Error Tradeoff" that G&C talk about on p.4. As one goes up, the other goes down. You can't get them both low at the same time.

Take a look at UIDAI's 27 March 2012 report on authentication (p.4). Using one or two fingers to authenticate yourself, UIDAI expect the Aadhaar system to be between 93.5% and 99% accurate. I.e. FRR will be between 1% and 6.5%. That's with a FAR of 0.01%. FRR is high, FAR is low(ish).

Varying FAR from high to low and FRR the other way is achieved by changing the matching threshold. You can set the system to insist on a very high score before asserting that President Lincoln's freshly scanned fingerprints match the set already stored on the population register. That would give a low FAR and a high FRR. Or you can set a very low threshold and achieve the opposite. And all points in between.

This is odd.

In the world we're used to, if you are President Lincoln then you are President Lincoln and that's all there is to it. It doesn't depend on the matching threshold set by some state functionary.

In the world of Aadhaar, depending on the threshold chosen, sometimes you will be President Lincoln (low threshold, easy to achieve a match, low FRR, high FAR) and sometimes you won't (high threshold, hard to achieve a match, high FRR, low FAR). It all depends. At the limit, the functionary could fix it so that no-one was President Lincoln. Or that everyone was.

When we said above that "either they match or they don't", that was a tease. That's the way people imagine biometrics systems to work. All cut and dried. In fact, it's discretionary.

The concept of identity in Aadhaar is different from the concept in the real world. Identity becomes discretionary, something that can be granted or revoked by twiddling the dial on a gizmo.

There's another rider to add to G&C's conclusions – the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack and as long as we've got our maths right and as long as you realise that it's not identity that's being managed.

-----  o  O  o  -----

That's authentication.

Identification is different.

Identification is the process you go through when you are enrolled into Aadhaar. Before identification, you don't exist as far as Aadhaar is concerned. If public services in India ever start to depend on Aadhaar and you don't have an Aadhaar number, you won't get any public services. Why would the state provide benefits to someone who doesn't exist? At the very least you will look very suspicious.

Identification is a one-to-many process. When you first enrol someone in the register, their biometrics have to be checked for uniqueness. Instead of checking them against just one set of biometrics, they have to be checked against every set already registered.

The errors that can be made by the biometrics system are very similar (among others, yes you are already enrolled when really you're not or no you're not already enrolled when really you are) but the process has such existential consequences that it's normal to talk of false negative identification rate (FNIR) and false positive identification rate (FPIR), rather than FAR and FRR, to distinguish it from mere quotidian authentication.

UIDAI talk of FRRs between 1% and 6.5% for authentication using fingerprints whereas, when it comes to identification, their FPIR figure is 0.057%. That's two orders of magnitude different. Identification is a strict process and, by comparison, authentication is flabby.

G&C unfortunately use FAR and FRR for both identification and authentication which obscures the important distinctions between the two processes.

-----  o  O  o  -----

FNIR and FPIR are inversely proportional, like FAR and FRR.

How good are the biometrics UIDAI are using at creating a reliable population register?

It's a problem Professor John Daugman has looked at. Not in connection with Aadhaar in particular. But in general. For any biometrics-based identity management scheme.

Remember, to establish uniqueness for every one of the 1.2 billion sets of biometrics on India's population register, you have to make 719,999,999,400,000,000 comparisons.

Suppose, says Professor Daugman, that there's a mistake 1 time in a million such that a false positive identification is made. Then Aadhaar will throw up 719,999,999,400 false matches.

These can't be resolved by the computer – it's the computer that threw up the false matches in the first place. They have to be resolved by human investigations.

Humans aren't going to complete 719,999,999,400 investigations. It's impractical. The identity management scheme will drown in a sea of false positives, as the professor puts it.

Is there a one-in-a-million chance of a mistake?

Professor Daugman thinks that it's a lot worse than that if you rely on face recognition as a biometric. There's far too little randomness in faces, there are far too few degrees of freedom, for face recognition to support enormous numbers like 719,999,999,400,000,000. (Never mind Ughana, that doesn't stop the UK government wasting money on face recognition.)

Fingerprinting is better in this sense than face recognition, but still not good enough to avoid drowning in a sea of false positives. (That doesn't stop the UK government wasting money on glitzy new fingerprinting systems.)

Irises on the other hand do have enough randomness, he says, there are enough degrees of freedom to stay afloat. Which is good news for UIDAI – Aadhaar uses a combination of both fingerprints and irises.

-----  o  O  o  -----

Is Aadhaar in the clear? Which is it? Sink or swim?

According to UIDAI's report on identification (p.4), on 31 December 2011 when there were 84 million sets of biometrics on the population register, the FPIR was 0.057%, the FNIR was 0.035% and "it is unnecessary and inaccurate to attempt to infer UIDAI system performance from other systems which are ten to thousand times smaller".

It may be unnecessary and it may be inaccurate but it's impossible to resist the temptation – compared to any other biometrics-based scheme known to man, these figures for Aadhaar are astonishing. Certainly no salesman worth his or her salt will ignore it.

It looks as if there would be only 684,000 false positive identifications to investigate by the time the population register is full, and not 719,999,999,400.

684,000 is manageable. As UIDAI say (p.18):

... at a run rate 10 lakhs enrolments a day, only about 570 cases need to be manually reviewed daily to ensure that no resident is erroneously denied an Aadhaar number. Although this number is expected to grow as the database size increases, it is not expected to exceed manageable values even at full enrolment of 120 crores. The UIDAI currently has a manual adjudication team thatreviews and resolves such cases. [1 lakh = 100,000 and 1 crore = 10,000,000]

How do UIDAI know that the FPIR was 0.057% when the register had 84 million entries?

Presumably they had recorded 47,880 cases of false positive identifications to date.

You'd think that. But you'd be wrong. UIDAI tell us that (p.18):

An FPIR of 0.057% was measured when the gallery size was 8.4 crore (84 million) and probe size was 40 lakhs (4 million). The false rejects (legitimate residents who are falsely rejected by the biometric system) were a count of 2309 out of the 40 lakh probes

They did a test. They probed the gallery with 4 million sets of biometrics and they got 2,309 false positive identifications.

Funny way to do it.

Perhaps we shall be told that there's an agreed protocol in the biometrics industry such that this is an acceptable way of determining FPIR. Even so, why not report the actual number of false positive identifications recorded?

That statistic should be available in the case of Aadhaar – G&C tell us that (p.2):

UIDAI places a heavy emphasis on data quality throughout the process. It collects as much operational data as possible, including on the details of each individual enrolment as it is carried out, process by process. This is included, together with biometric and demographic data, in the packet of information sent from the enrollment point to the data center.

Why not tell us how many false positive identifications there were as well as the result of the test probe? Why were there 4 million sets of biometrics in the probe and not 5 million, or 3 million? How were the 4 million chosen?

The questions mount and the answer gradually comes into focus – in order to inspire confidence, UIDAI's figures need to be audited by independent experts and certified like a set of company accounts.

And, like company accounts, they should be audited every year. These figures from 31 December 2011 are getting very long in the tooth.

Another rider – subject to an annual audit, the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack and as long as we've got our maths right and as long as you realise that it's not identity that's being managed.

-----  o  O  o  -----

UIDAI say that the incidence of false positive identifications is manageable and that they expect it to remain manageable. I.e. they're not drowning in a sea of false positives.

G&C have this footnote, #7, on p.5 of their report:

For a huge population like India’s, even this small level of error would result in some 3.1 million false rejections if continued through the program. UIDAI plans to contain the numbers by eliminating some sources of error unearthed by the initial study, and also by relaxing the [FNIR] if needed to further reduce the [FPIR]. Handling false rejections has reportedly been a manageable problem to date.

"UIDAI plans to contain the numbers by ... relaxing the [FNIR] if needed to further reduce the [FPIR]". What? "Relaxing the [FNIR]"?

What does that mean? In order not to drown in false positives, UIDAI will let false negatives go up? UIDAI have got to get the population register completed and if that means tolerating lots of duplicate entries, too bad, so be it, let uniqueness go hang? If that isn't what it means, then what?

How relaxed? Very relaxed? What level does FNIR have to rise to, to keep FPIR down at 0.057%? Do UIDAI even know? Should they change their name to the Multiple Identification Authority of India?

"It is unnecessary and inaccurate to attempt to infer UIDAI system performance from other systems which are ten to thousand times smaller"? On the contrary, it is only sensible to question UIDAI's performance claims.

The riders are piling up now – subject to an annual audit, the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack and as long as we've got our maths right and as long as you realise that it's not identity that's being managed and as long as you're relaxed about the fact that anyone could have any number of entries on the population register.

-----  o  O  o  -----

If a supplicant turns up at an Aadhaar registration centre and is the victim of a false positive identification, you're going to know about it. They're going to demand their Aadhaar number and they're going to stay there and jump up and down until they get it. At least they will if they're legitimate and not impostors.

It's different with false negative identification. If an impostor turns up at the centre and his or her earlier registration is not detected by Aadhaar, then they're not going to tell you. You won't know. Impostors don't have the same desire to keep the performance statistics up to date as upright people do.

The upshot is that you can't measure FNIR. Not in the field.

You can submit a batch of sample biometrics and see how well the system performs. How successful it is at finding these deliberately seeded duplicates on the register. And that's what UIDAI did (pp.18-19):

To compute FNIR, 31,399 known duplicates were used as probe against gallery of 8.4 crore (84M). The biometric system correctly caught 31,388 duplicates (in other words, it did not catch 11 duplicates). The computed FNIR rate is 0.0352%. Assuming current 0.5% rate of duplicate submissions continues, there would only be a very small number of duplicate Aadhaars issued when the entire country of 120 crores is enrolled. Aadhaar expects to be able to increase the quality of all collections as the system matures. Consequently, we expect the potential number of false acceptances to decrease further below this already operationally satisfactory number.

That's fine. But if the actual number of "duplicate submissions" is higher than UIDAI assume and the "false acceptances" are more numerous than they expect, no-one will know. All UIDAI can say is, when we did this test, we got this result. Whether that is an accurate measure of FNIR out there in the operational system in the real world, nobody knows.

What we do know – G&C tell us – is that UIDAI have been "relaxing" the FNIR to keep FPIR low. The confidence we can have in UIDAI's figure for FNIR is severely limited.

-----  o  O  o  -----

It's worse than that.

G&C tell us on p.1 of their report that:

Although there has been extensive laboratory testing of different hardware and software for a variety of biometrics, including fingerprints, iris, face and voice, testing under carefully controlled conditions does not provide adequate information on real-world performance, which can be affected by many factors (Wayman et al 2010).

The paper they cite, Fundamental issues in biometric performance testing: A modern statistical and philosophical framework for uncertainty assessment, is written by three world-class experts – James L. Wayman, Antonio Possolo and Anthony J. Mansfield.

As G&C tell us, the experts conclude that technology tests and scenario tests tell us nothing about how well or how badly a biometrics system will perform in the operational environment. As they put it, biometrics is out of "statistical control".

To put it another way, UIDAI's FNIR and FPIR test probes are a waste of time.

Tony Mansfield is the UK's top biometrics authority and Jim Wayman is the US's. And Antonio Possolo is the top man on measurement at the US National Institute of Standards and Technology (NIST). They're practitioners. They have decades of experience. They advise governments. Their own and others. They know what they're talking about.

And what they're talking about is biometrics being out of statistical control.

That implies many things. Among others, consider the following.

Messrs Wayman, Possolo and Mansfield refer to the USA PATRIOT Act in their paper (p.20). By law, NIST have to certify biometrics systems before they are deployed in the national defence.

That may be the law but, if the technology is out of control then NIST have a problem obeying the law. They could refuse to certify any biometrics systems and then none would be deployed. That's one option. They have chosen another option. The certificate they issue says:

For purpose of NIST PATRIOT Act certification this test certifies the accuracy of the participating systems on the datasets used in the test. This evaluation does not certify that any of the systems tested meet the requirements of any specific government application. This would require that factors not included in this test such as image quality, dataset size, cost, and required response time be included.

That's the best they can manage in the circumstances. The result of the test is the result of the test and that's all we know. How the system will perform in the field is anyone's guess. According to three world-class experts, in biometrics, that is the state of the art.

Final rider – subject to an annual audit, the US could safely deploy an identity management scheme based on biometrics apart from the possibility of cyberattack and as long as we've got our maths right and as long as you realise that it's not identity that's being managed and as long as you're relaxed about the fact that anyone could have any number of entries on the population register and the fact that the discipline of biometrics is out of statistical control.

-----  o  O  o  -----

It is premature to conclude that biometrics have proved themselves in Aadhaar:

• Let's wait and see if any bank is confident enough to authorise payments on the basis of biometrics alone. No password. No PIN. No token. Just biometrics.
• Let's wait and see if legitimate voter participation is increased by Aadhaar.
• India's various food and fuel distribution programmes and its temporary employment programmes for the long-term unemployed are plagued by large-scale corruption. Let's wait and see if Aadhaar reduces the level of corruption or simply automates it.
• And let's wait for an independent audit of UIDAI's results.

G&C have already identified 160 biometrics programmes in 70 countries. This latest report of theirs will be embraced by biometrics salesmen the world over as an unsolicited testimonial from a respected source and will be used to raise funds for more programmes. (G&C driving up the false accept rate?)

G&C work for the Center for Global Development, a Washington-based think-tank and lobbyist which aims to "reduce global poverty and inequality through rigorous research and active engagement with the policy community to make the world a more prosperous, just, and safe place for us all".

It's hard work finding good homes for aid money. There are legitimate doubts about the reliability of biometrics. Aid money isn't necessarily well spent on biometrics systems.

Michela Wrong, a journalist who has covered Africa for two decades, reported on the March 2013 elections in Kenya complete with biometric registration of electors and electronic voting. She had this to say:

I suddenly realised I was watching a fad hitting its stride: the techno-election as democratic panacea ... EU and Commonwealth election monitors hailed the system as a marvel of its kind, an advance certain to be rolled out across the rest of Africa and possibly Europe, too. The enthusiasm was baffling, because almost none of it worked.

The Economist magazine have let down their scepticism guard and become active in the unsolicited testimonials market – please see The Economist magazine sticks its nose into Indian politics, comes away with egg on its face and The Economist magazine's chickens, now on their way home to roost.

That was some time ago. They remain dazzled by technology to this day: "India has registered 275m of its 1.2 billion people in one of the world’s most sophisticated ID schemes (it includes iris scans and fingerprints)". Why do they think that the inclusion of biometrics is ipso facto "sophisticated"?

They should talk to Michela Wrong.

-----  o  O  o  -----

G&C have spotted what the Economist have missed:

• The Wayman, Possolo and Mansfield paper.
• UIDAI relaxing the FNIR.
• The element of smoke and mirrors in biometrics – they talk about the "fiction of infallibility" (p.9) and the "pretense of uniqueness in the ID system" (p.10) and the possibility that "in the longer run, as its mystique evaporates, the identity system will no longer be trusted by anyone, eliminating any value" (p.10).

Above all, quite rightly, G&C call for more countries to release data on the performance of biometrics in the field – "distressingly little data is available on [biometrics] performance, either for identification or for authentication" (p.1) and "there is now no excuse for other countries not to share data—or for donors not to insist on it when financing identification programs" (p.10).

The biometrics salesmen won't like that conclusion of G&C's and they won't mention it, please see UIDAI and the textbook case study of how not to do it, one for the business schools. (Neither will the UK government.)

All that healthy scepticism, and yet G&C conclude that biometrics is ready for large-scale deployment:

• Did they check with NIST or the FBI before publishing their report? Those organisations know quite a lot about biometrics and might have provided some useful input.
• Did they contact Messrs Wayman, Possolo and Mansfield? If G&C believe them when they say that biometrics is out of statistical control, then there's not much point filling up their report with useless statistics, is there? If they don't believe them, why not?
• Would G&C be so generous with their testimonials if Aadhaar was an aeroplane safety system, for example?
• Would they feel qualified to comment if they were dealing with the pharmaceutical industry rather than the biometrics industry?
• Would they be more sceptical if they were dealing with research funded by the tobacco industry?
• Why does biometrics get the kid gloves treatment?
• And what is this fake distinction G&C make between countries with a large population and a small one? The biometrics tested in the UK failed with a trial population of 10,000 participants. Biometrics is a technology. At least it's supposed to be. Either it works or it doesn't. Cars work in the US. And they work in India. If biometrics isn't good enough for the US, it's not good enough for India. Or Uganda or Ghana. Which are two different countries. Ask Michela Wrong.

All that healthy scepticism, and yet G&C conclude that: "UID shows that countries with large populations can implement inclusive, precise, high-quality identity systems by using existing technology".

No.

It shows nothing of the sort.

Is there any chance of G&C reissuing their report with revised conclusions?

Posted by David Moss at 09:37 

2408 - UIDAI and the textbook case study of how not to do it, one for the business schools - David Moss

The Unique Identification Authority of India (UIDAI) came under attack. Its very existence was threatened. Naturally enough, UIDAI decided to defend itself.

It's worked. UIDAI survives for the moment.

But theirs is a Pyrrhic victory. The UIDAI defence could undermine the credibility of every public authority in the world which has nailed its colours to the mast of biometrics – which is most of them – and could destroy the multi-billion dollar mass consumer biometrics industry.

The job of the Unique Identification Authority of India (UIDAI) is to use biometrics to identify every resident of India and to issue them with a unique corresponding number, a so-called "Aadhaar number".

"Aadhaar" means foundation or support and the idea is that, once everyone has an identifying number, it will be easier for the various arms of government to build systems on that foundation to provide social security benefits, for example, and to facilitate national security. And beyond government, the banks will supposedly find it easier to authenticate payments.

UIDAI is not without its critics:

• The Standing Committee on Finance (SCoF), a committee of the Indian Parliament, has considered the National Identification Authority of India Bill, 2010. That Bill would establish UIDAI on a statutory basis if it was ever enacted, but it hasn't been. Meanwhile, UIDAI is operating under executive order only. It's not operating very well according to the SCoF report and it's about time UIDAI came under the control of Parliament.
• And then there's the Ministry of Home Affairs. They're a properly constituted body and not just a creature of the Executive. And they have a competing identity management scheme, NPR (the National Population Register). Result – a turf war, Aadhaar v. NPR.

SCoF and the Ministry of Home Affairs pressed their case with the Prime Minister but UIDAI proved too adept for them. The Chairman threatened to resign, which would be embarrassing for the prime Minister – good move no.1. Good move no.2 – UIDAI arranged some convenient PR with the compliant Economist magazine. And then they published not one but two reports making unprecedented claims for the reliability of the biometrics used in Aadhaar:

• Role of Biometric Technology in Aadhaar Enrollment
• India boldly takes biometrics where no country has gone before

Oops. Bad move. There are five problems here:

1. Both reports are produced by UIDAI only. There is no sign that that they have been audited by any independent expert body.
2. Both reports quote reliability figures. No other public authority in the world does that. Not operational figures – figures measuring the reliability of biometrics in the field, at the border, for example. They should. But they don't. Now, thanks to UIDAI, they will all come under pressure to quote independently audited figures themselves, figures for reliability, to justify their investment of public funds. It is likely that the public are going to be shocked at just how unreliable the biometrics are, that their governments are using. The public will at last understand why their governments have been so reluctant for so long to quote any figures.
3. Why is that likely? Because the figures quoted by UIDAI are hundreds of times better than anything anyone else has ever claimed following tests of biometrics. Hundreds.
4. The second report says that (a) Aadhaar uses flat print fingerprinting and iris scanning, (b) the two biometrics are fused to form one composite biometric, so-called "multi-modal" biometrics, and (c) UIDAI use not one matching algorithm, but three of them. Any large-scale identity management scheme that doesn't do the same, they say – (a), (b) and (c) – is doomed to "catastrophic failure".
5. The suppliers of biometric technology have never had to give public warranties before. Now they will have to.

Great. Now suppose you're the Australian Customs and Border Protection Service. You've spent millions of dollars of public money deploying smart gates at Australian airports as a security measure. These gates depend on face recognition biometrics. Not on UIDAI's list (a). The Australian (and new Zealand) border security system is doomed to "catastrophic failure". Don't take my word for it. Ask UIDAI.

You've spent years refusing to divulge any figures about the reliability of your technology:

Customs refused to disclose the rates at which the system inaccurately identified people. "For security reasons, Customs does not disclose the false positive and false negative rates," a spokesman said.

Now UIDAI have released figures, how are you going to hold the line? You can't.

You could say that UIDAI's figures haven't been audited and may turn out to be false. Now you've got a fight with UIDAI on your hands. And what's the best result you can hope for? UIDAI's figures turn out to be a pack of lies and actually the reliability of Aadhaar is just as appalling as the Australian system. Not what you wanted. It doesn't help to explain why you've been squandering your own citizens' tax money on joke technology.

The same applies to the UK, of course, and our planned deployment of smart gates at airports. Another catastrophic failure? And all those states in the US busy incorporating face recognition biometrics into driving licences. These people – the Australian Customs and Border Protection Service, UK Border Agency, et al – are not going to be pleased with UIDAI. UIDAI have let the cat out of the bag and have almost certainly started a fresh collapse of confidence in public administration as a result.

And neither are the biometrics suppliers going to be pleased. How are Morpho going to sell their products now without giving warranties? They're not.

And how are IBM and CSC going to be able to sign any more nine-figure contracts with credulous governments? They're not.

And how are PA Consulting going to sell any more biometrics assignments? They're not.

UIDAI are going to be persona non grata worldwide. Especially in India, where the Prime Minister may yet regret his decision to carry on funding them. And stop. He may give almost any reason but the big reason, the one several people have pointed out for a long time, is that far from curtailing corruption, Aadhaar was simply going to automate it.

A tragedy with a happy ending, the only people who will be pleased is absolutely everyone else in the world, who can now keep some of their tax money and spend it themselves rather than paying public authorities to waste it for them.

UIDAI's Pyrrhic victory? From now on it's going to be known as an "Aadhaar victory". At least it will when the business schools write it up and teach it all around the world. And when the Economist faithfully report UIDAI's defence, under the heading "Poison pill – that's not the way to do it".

2327 - The Economist magazine sticks its nose into Indian politics, comes away with egg on its face - David Moss

Sunday, 29 January 2012

After 30 years of reading The Economist, you know what to expect.

The correct answers to most questions are found by letting markets operate freely, as far as The Economist is concerned and politically, that rules out any system that pretends to be able to manage control the economy. The magazine is socially liberal. There's not a hint of racism in it, or sexism – "meritocracy" is the name of the game. 

Arguments are conducted logically, preferably they're quantitative, the emphasis is on rational management techniques and evidence-based public administration. The magazine is the opposite of insular, open to new ideas wherever they come from, and always up to speed with new technology.

Given which, what on earth happened in the 14 January 2012 edition? It was out of character. Its Scottish Enlightenment body was snatched by aliens. Did The Economist suffer some sort of editorial stroke?

Take a look at The magic number, a leader on Aadhaar, one of India's many identity management schemes, this one operated by the Unique Identification Authority of India, chaired by Nandan Nilekani:

Armed with the system [Aadhaar], India will be able to rethink the nature of its welfare state, cutting back on benefits in kind and market-distorting subsidies, and turning to cash transfers paid directly into the bank accounts of the neediest. Hundreds of millions of the poor must open bank accounts, which is all to the good, because it will bind them into the modern economy. Care must be taken so mothers rather than feckless fathers control funds for their children ... Mr Nilekani harnessed the genius of Indians abroad, including a man who helped the New York Stock Exchange crunch its numbers and one of the brains behind WebMD, an American health IT firm ... India plainly needs better data-protection laws, but even if the existing rules remained unchanged, the threat to liberty would be dwarfed by the gains to welfare: to people who live ten to a room, concerns about privacy sound outlandish. Some of the resistance is principled, but much comes from the people who do well out of today’s filthy system. Indian politics hinge on patronage—the doling out of opportunities to rob one’s countrymen. [Aadhaar] would make this harder. That is why it faces such fierce opposition, and why it could transform India.

According to The Economist then, Indian fathers are feckless but Indian mothers aren't, the Indians who have left the country are brighter than the ones who have stayed at home, poor people don't need privacy the way western journalists do and UIDAI are clean whereas the other gangs dispensing opportunities to rob their own countrymen are "filthy", a most unEconomist word.

And this, too, is most unEconomist – normally the magazine would instantly spot the problem with the following claims:

The state spends a fortune on subsidised grain for the hungry, but an estimated two-thirds of it is stolen or adulterated by middlemen. The government pays for an $8 billion-a-year make-work scheme for the rural poor, but much of the cash ends up in the capacious pockets of officials who invent imaginary “ghost workers”. Suppose those thieving middlemen were obliged to deliver grain, not to poor people in general but to named individuals who could confirm receipt by scanning their fingerprints? And suppose those ghost workers had to undergo an iris scan before being paid?

UIDAI computerisation may make it harder to steal public money from PDS, the food security programme, and from NREGA, the temporary employment scheme, as The Economist suggest. But equally, it may make it much easier.

Aadhaar could make corruption a much more modern, clean, white collar, highly automated pursuit. It's a lot quicker to use a computer to claim wages for thousands of ghost employees than it is to complete manual requests. If Aadhaar wants biometrics, then a computer will provide them. And if Aadhaar has helped to provide everyone with bank accounts and electronic transfer facilities then, thank you very much UIDAI, the "thieving middlemen" may say, now there's no need to handle any cash and it's easier to launder our ill-gotten gains.

This leader of The Economist's barely rises above the level of sales literature. It is obvious why UIDAI would want it published. But why did The Economist allow it? That is a question for Adam Roberts, the South Asia bureau chief based in Delhi, and for Dominic Ziegler, the London-based Asia editor, and for Patrick Foulis, the India business and finance editor in Mumbai, and maybe for Alpesh Kandoi, to whom all media enquiries should be addressed. 

2315 - True lies of biometric technology in Aadhaar enrolment - David Moss - Money Life

January 27, 2012 01:24 PM  
David Moss

Let’s ask the professors UIDAI cited in its latest report: Do you agree with UIDAI’s assessment of Aadhaar? Do you share their confidence in the project? Did UIDAI ask you in advance, before using your name for their marketing purposes?

The Unique Identification Authority of India (UIDAI) have been accused of making false claims about the reliability of the biometrics that its unique identification number (UID) or Aadhaar scheme relies on. The report released earlier this week by UIDAI is in response to those criticisms.

UIDAI say that “… based on the analysis, it can be stated with confidence that UIDAI enrolment system has proven to be reliable, accurate and scalable to meet the nation’s need of providing unique Aadhaar numbers to the entire population. It is now safe to conclude that the system will be able to scale to handle the entire population”. But that is mere assertion, it begs the question, they would say that, wouldn't they.

They need independent and respected biometrics experts to agree with them, if this report is to boost confidence in UIDAI’s abilities. They mention several names. The casual reader may assume that these named experts all agree with UIDAI’s conclusion that Aadhaar will work. It would be instructive to ring them up and ask them directly for their opinion.

Does Professor John Daugman, for example, agree with UIDAI when they say that “… although [the false positive identification rate of 0.057%] is expected to grow as the database size increases, it is not expected to exceed manageable values even at full enrolment of 120 crores”? It seems unlikely—Professor Daugman is the man who first pointed out that any attempt to prove uniqueness in a large population of biometrics must drown in a sea of false positives, please see

And does Professor Jim Wayman, for example, agree with UIDAI when they say that “… based on the [receiver operating characteristic] model, the UIDAI expects the accuracy of the system to remain within the same order of magnitude as reported above. Hence it can be stated that system will be able to scale to handle the entire population without significant drop in accuracy”? It seems unlikely—Professor Wayman is the lead author of a paper which concludes that biometrics is a discipline out of statistical control, the results gathered so far tell you nothing about what to expect in future, please see
http://biometrics.nist.gov/cs_links/ibpc2010/pdfs/FundamentalIssues_
Final.pdf


If the two professors agree with UIDAI and renounce their earlier statements, well and good.

But if, on the other hand, they say that they have no reason to believe that UIDAI is right, they have not had a chance to assess the evidence that UIDAI claims to have, they do not understand why UIDAI has mentioned their names, then this schoolboy attempt to justify UIDAI’s waste of public money will fall humiliatingly flat on its face.

(David Moss spent eight years campaigning against the UK’s National ID (NID) card scheme, which was finally scrapped by the British government. Mr Moss is an MA in Philosophy from Cambridge University, MSc in Software Engineering from Kingston. With a career spanning of over 35 years, Mr Moss at present works as director at Business Consultancy Services Ltd and can be contacted at bcsl@blueyonder.co.uk.)

2298 - Useful idiots at the Economist - by David Moss

Two articles in this week's Economist, The magic number and Reform by numbers. One response among many:

From the very start, we know that this article of the Economist's must be wrong. Next door to India, Pakistan's NADRA has been issuing biometric ID for over 10 years and that hasn't led to the sunny benefits predicted by the Economist.

Using the brilliant Indians abroad, rather than the presumably dim ones at home, according to the Economist, doesn't necessarily help.

Computerising the welfare system opens the way to computerising its corruption. The managers of ghost employees will find it much easier under UID to automate their claims and to collect payments from the conveniently created bank accounts -- altogether cleaner than the currently "filthy" process the Economist decry.

The Economist believe that Indian politics hinge on patronage. A few old-fashioned idealists may abhor that but, to the extent that the Economist are right, presumably the suggestion is that UID is a new patronage system seeking to dole out opportunities to rob its own countrymen in competition with the more established gangs. The Economist may well be right there.

Skimping on security is acceptable for the Indians, as far as the Economist is concerned, although presumably they wouldn't be too pleased if anyone skimped on the security of their bank accounts and their tax payments. Why is security less important for the Indians?

Time was when the Economist had a strong grasp of technology and a sensible scepticism in the face of snake oil claims made for magic numbers. No more. Now the magic numbers appear in the title of the article and the Economist credulously swallow all the unsupported claims made for the reliability of mass consumer biometrics by its salesmen.

Time was when the actions of the Executive beyond the control of Parliament and outside the reach of the common law would have raised the odd question at Economist Towers. No more. It's been fun knowing you but now it's goodbye, the Enlightenment.

How much did UID pay the Economist to publish this article? Let's hope it's enough to make up for the newspaper's loss of reputation as an intelligent commentator. Welcome to the corps of useful idiots (UID).

India's ID card scheme – drowning in a sea of false positives

Throwing the towel in

----------

David Moss

http://DematerialisedID.com

Business Consultancy Services Ltd

2163 - The biometrics bag no longer contains the cat by David Moss

Date: Sat, 31 Dec 2011 11:18:05 -0000

Subject: The biometrics bag no longer contains the cat

A public servant, Brodie Clark was suspended on 2 November 2011 from his job as Head of the UK Border Force – 20,000 officers whose job it is to secure the border.

He was suspended principally for halting biometric fingerprint checks at the border when there was a danger of crowds of incoming travellers getting out of control.

The Home Affairs Committee have taken evidence from everyone connected with this suspension.

In a six-minute passage of his evidence, between 12:18 and 12:24, Brodie Clark explained just how useful biometric fingerprint checks are: they are the least reliable of the nine checks made, the ninth and bottom priority, he approved of halting the checks, if any check has to be dropped it is "very sensible" to drop the fingerprint checks.

Link to Home Affairs Committee evidence session:

http://www.parliamentlive.tv/Main/Player.aspx?meetingId=9445&st=11:36:43

The HTML below can be embedded in a web page. It works in the UK. It may or may not work abroad:

Not the testimony the biometrics lobby wanted.

Politicians have told us for 10 years since 9/11 that our safety depends on biometrics. The media believe them. The public believe them.

But what do politicians, the media and the public know about the technology? Nothing.

Unlike the politicians, the media and the public, Brodie Clark's staff actually have to use this technology, they have first hand experience. And they're clearly not impressed.

What with the planned public expenditure cuts in the UK, the idea is to make several thousand members of the Border Force redundant and replace them with automated biometrics checks. So much for securing the border.

Brodie Clark's testimony is a new year's present to UK taxpayers. It may help to stop the Executive from wasting any more of our money on defective biometrics technology. And it may force them to concentrate on sensible measures to defend the border.

The taxpayers in other countries may also appreciate his testimony. Please bring it to the attention of your readers and of other campaigners.

Best wishes

David Moss

( David Moss - UK Say No to ID <bcsl@blueyonder.co.uk> )

----------

http://DematerialisedID.com

http://DMossEsq.com

1342 - World renowned academicians say level of uncertainty in biometrics is too high and UID tests prove nothing - Source - Money Life

May 24, 2011 06:33 PM
Moneylife Digital Team

A study argues that there is too much uncertainty in biometrics to predict how well the technology will perform in the real world, much less support investment in this technology

Three scholars who have provided the academic foundation for the biometrics industry, particularly in the Western world, say that the level of uncertainty in biometrics is so great that tests prove nothing.

The academicians have, in a paper titled "Fundamental issues in biometric performance testing: A modern statistical and philosophical framework for uncertainty assessment", ( http://biometrics.nist.gov/cs_links/ibpc2010/pdfs/FundamentalIssues_Final.pdf) argued that the level of uncertainty in biometrics is so great that they cannot be used to predict how well the technology will perform in the real world and therefore this cannot support a valid argument for investment in biometrics.

The academicians are James L Wayman from San José State University, Antonio Possolo, head of the statistical engineering division at the US National Institute of Standards and Technology (NIST), and Anthony J Mansfield from UK National Physical Laboratory, all recognised as stalwarts of the biometrics industry.

However, the Unique Identification Authority of India (UIDAI), which has embarked on a tagging programme that is based on biometrics, is silent on the report. The institution has, till now, been quick to associate with other academic groups.

While UIDAI claims that biometrics will allow it to deliver a unique identification, it has goofed up its own test results while pushing its ambitious Aadhaar project. (Read, 'How UIDAI goofed up pilot test results to press forward with UID scheme'.)

Since its inception, UIDAI has tried to force the use of biometrics for the UID number as the ultimate solution. UIDAI conducted a proof of the concept trial of the Aadhaar project between March and June 2010. The results of the concept trial, or scenario test, suggest that biometrics cannot be reliable and may encounter huge problems while dealing with false positives.

David Moss, who spent eight years campaigning against the UK's National ID (NID) card scheme, sees hard times ahead for the global mass consumer biometrics industry. He said, "Not only has the industry lost its academic support, but governments are starting to abandon ship. President Obama's plans for trusted identity on the web make no mention of biometrics. The same goes for the UK's plans for identity assurance, in which case also there is no mention of relying on biometrics at all."

"The superstitious belief in mass consumer biometrics is like an illness, it's like the tulip mania that affected Holland in the 17th century. And now, perhaps, it is passing. Even in Holland, where they announced last month that they have suspended their plans to develop a centralised population register recording every person's biometrics," Mr Moss said.

There are three types of tests used for biometrics. One is the lab or technology test; the other is the operational or field test; and the third is a scenario test. A biometrics technology test is conducted in the lab and is entirely computer-based. An operational test is conducted in the field, in the real world, with the biometrics package coming under attack from different, unpredictable sets. In a scenario test, researchers recruit a putatively representative sample of the population so that they can test the biometrics packages with real people under still fairly controllable conditions. The UIDAI used the scenario test for UID.

On the scenario test, the three academicians write, "The test repeatability and reproducibility observed in technology tests are lost in scenario testing due to the loss of statistical control over a wide range of influence quantities. Our inability to apply concepts of statistical control to any or all of these factors will increase the level of uncertainty in our results and translate to loss of both repeatability and reproducibility. Test data from scenario evaluations should not be used as input to mathematical models of operational environments that require high levels of certainty for validity."

This exactly is the reason why governments across the globe are not emphasising on biometrics anymore. Last year, the newly-elected government in the UK scrapped the National ID programme citing huge costs, impracticality and ungovernable breaches of privacy associated with the programme. Last month, the US released its 'National Strategy for Trusted Identities in Cyberspace' signed by president Obama and nowhere has the 45-page document used the word 'biometrics'.

Also in April, the Dutch government suspended its plan to develop a centralised flat print fingerprint population register, citing concerns about security and reliability of the system.

However, this is not the case with India, where it seems that there is ample money and nobody cares about security, reliability and privacy, and everybody from politicians, corporates to the media are 'greased', directly or indirectly. This is the reason why UIDAI is forcing the UID number onto gullible citizens.

"Why is India spending billions on Aadhaar, which depends on biometrics whose reliability is, so say the titans, utterly unknowable? And will the UIDAI ever answer my question how they can claim to offer unique identification when, based on their own figures, they would have to perform 18,000,000,000,000 (18 trillion or 18 lakh crore) manual checks to prove uniqueness? And why do they think Aadhaar will eradicate corruption, rather than automate corruption," asks Mr Moss.
You may also want to read…

•    How UIDAI goofed up pilot test results to press forward with UID scheme
•    UK scraps National ID project; Will India's UID face the same fate?
•    UID: Enrolment troubles for helpless residents
•     Fat profit institutions continue to board UID bandwagon
•     Why is UID number being made compulsory through the backdoor
•     UID = more 'consumers' admits Nilekani
•     Now SEBI jumps on the UID bandwagon
•     Right to privacy and biometrics of the UID
•     How Aadhaar or the UID project can get you into deep trouble
•     UID Issue: Numbers Game -1
•     No card only a number despite Rs45000 crore being spent on the UID
•     Even Mahatma Gandhi was against ID cards

1328 - Throwing the towel in by David Moss- Source - Dematerialsied ID

Biometrics – a latter-day tulipmania
 

Biometrics “will make identity theft and multiple identity impossible. Not nearly impossible. Impossible”. That was the view of Rt Hon David Blunkett MP, speaking in 2003, when he was the UK's Home Secretary.

Mr Blunkett is not alone in suffering from that delusion. Many politicians continue to believe that the biometrics emperor is sumptuously dressed when actually he is as naked as the lie told by a snake oil salesman.

It is the view of many Whitehall officials also, who continue to believe, for no reason they can give, that mass consumer biometrics will help to detect and prevent crime and to deliver public services more efficiently and to counter terrorism. So much so that they spend hundreds of millions of pounds of taxpayers' money on biometrics technology.

This tulipmania is not restricted to the UK and it is not restricted to politicians and civil servants. It affects privacy campaigners, the expression of whose fears amounts unintentionally to a powerful unsolicited testimonial, thereby boosting the sales of biometrics technology. It affects journalists. And it affects normal people.

There is a surprising certainty evident in most people – even people who announce proudly that they haven't got a clue how any technology works – that biometrics work. They can get positively tetchy if you suggest that they're wrong and that mass consumer biometrics don't work. Its source is a mystery but this inexplicable certainty of theirs is jealously guarded and seems to survive any number of adverse encounters with reality.

What the doctors say
That has been the case for years but maybe now, at last, just maybe, this inter-continental pandemic is on the wane. Take a look at this:
 

Is there any hope of inductively extending the results of our technical test more broadly to any other algorithms or databases? A Type B systematic uncertainty evaluation after consideration of changes in the unit of empirical significance and statistical controls over its tangible elements might be of value, provided that the specifics of the changes could be given, but we should not sanctify such a “guesstimate” in an emperor’s cloak of imagined analytic rigor.
 

"Inductively extending"? "Type B systematic uncertainty evaluation"? "The unit of empirical significance"? Who writes this elegant prose? James L Wayman, Antonio Possolo and Anthony J Mansfield, referred to collectively henceforth as "WaPoMa", that's who.

Who are, or is, WaPoMa? Three titans of the biometrics industry. Mr Wayman is at San José State University, Mr Possolo at the US National Institute of Standards and Technology (NIST) and Mr Mansfield at the UK National Physical Laboratory. Between them, they provide the academic foundation for the biometrics industry in the Western world, they speak with authority, they have earned the respect they command in academia, in industry and in government. (To all those denizens of academia and industry not listed here who also contribute to the foundations of biometrics, apologies.)

And what is WaPoMa saying?

He's saying that:

• You can't extrapolate from the results of a biometrics technology test. At the end of a technology test, all you know is that the results are what the results are. You can't use the results of one test to predict how well or badly any biometrics package will perform in another test.

• The reason for that is the recalcitrant uncertainty in the test. That's why you can't extrapolate, or "inductively extend". Uncertainty.

• And although to an optimist it "might be of value" to try to measure that uncertainty, no says WaPoMa with his scrupulous scientist's candour, that measure would just be a guess, the emperor would be naked, the figures given would have no "analytic rigor" (or "analytical rigour", as we say here in the UK).

Out of control (statistically) – tests prove nothing ...
And that's not the end of it.
In his paper, Fundamental issues in biometric performance testing: A modern statistical and philosophical framework for uncertainty assessment, WaPoMa is unsparing.

Biometrics technology tests are designed to measure, among other things, the false positive and false negative rates of any number of rival biometrics packages. You can only measure these quantities if they are under "statistical control". And in the world of mass consumer biometrics, they're not, says WaPoMa.

So what? What does that mean? By way of explanation, WaPoMa provides this quotation from another luminary of the world of metrology, Churchill Eisenhart:

... a measurement operation must have attained what is known in industrial quality control language as a state of statistical control ... before it can be regarded in any logical sense as measuring anything at all.
 

Not to put too fine a point on it, according to WaPoMa, given the current state of uncertainty in the field of biometrics, a field which is statistically out of control, researchers don't even know what they're measuring when they perform a technology test.

What politicians want to know (and public servants and the law and the media and the public) is – do biometrics work? Yes? Or no? And what WaPoMa says is that technology tests can't help to answer that question.

He doesn't say that tests can't tell you if biometrics work, whatever "work" means. Much more elegantly, he says that for a given biometrics package, no current test can give you an "intimation of its operational acumen". (I wish I'd said that. You will, Oscar, you will.)

In case anybody missed the point, WaPoMa spells it out:

... technology testing on artificial or simulated databases tells us only about the performance of a software package on that data. There is nothing in a technology test that can validate the simulated data as a proxy for the “real world”, beyond a comparison to the real world data actually available. In other words, technology testing on simulated data cannot logically serve as a proxy for software performance over large, unseen, operational datasets.
... technology tests prove nothing and operational tests prove nothing ...
Let's clarify WaPoMa's "taxonomy". He distinguishes between technology tests on the one hand, and operational tests on the other.
A biometrics technology test is conducted in the lab and is entirely computer-based. An operational test is conducted in the field, in the real world, with the biometrics package coming under attack from the unpredictable torrent of humanity trying to clear immigration at a US airport, for example, or trying to get into an Olympics venue.

Given that researchers don't know what they're measuring even in a technology test, according to WaPoMa, there can be no way to measure the performance of an operational biometrics system, where the level of uncertainty is even greater.

Back in May 2004, NIST reported that they had tested the flat print fingerprinting technology that was to be used in US-VISIT, the border control system non-US nationals go through when they try to get into the US. That is an example of a technology test. So is their March 2007 report on face recognition technology, i.e. iris scans and facial geometry. Both tests were entirely computer-based. So is the International Fingerprint Verification Competition that used to be run by four universities but seems now to have been discontinued, possibly because the results tell no-one anything.

The reports of all those three technology tests have been published. Only IBM's technology trial, conducted to choose the "best" biometrics system for ePassports for UK nationals and residence permits for non-EEA residents of the UK, remains a state secret.

NIST predicted a false reject rate of 0.5% in their May 2004 report. The operation of US-VISIT was later reviewed by the Office of the Inspector General. OIG's December 2004 report revealed that 118,000 people presented themselves to US-VISIT every day on average, most of them got through the primary/biometrics inspection, 22,350 of them were referred to secondary inspection, 1,811 of those were refused entry, and the rest were let in.

That means that (23,500 - 1,811) / 23,500 = 92% of secondary inspections were a waste of time. Not the sort of accuracy you associate with a working 21st century technology.

And it means that the false reject rate was (22,350 - 1,811) / 118,000 = 17.4%, just a tad different from the 0.5% predicted by NIST – WaPoMa knows whereof he speaks: "technology testing on simulated data cannot logically serve as a proxy for software performance over large, unseen, operational datasets".

... and scenario tests prove nothing
In between technology tests and operational tests, you get scenario tests, such as the UK Passport Service biometrics enrolment trial and the Unique Identification Authority of India's proof of concept trial for Aadhaar. ("Aadhaar" is the brand name for India's unique identification scheme.The word means foundation, or support.) In a scenario test, researchers recruit a putatively representative sample of the population so that they can test biometrics packages with real people under still fairly controllable conditions.
WaPoMa has a lot to say about scenario testing. What it adds up to is, don't bother:

We lack metrics for assessing the expected variability of these quantities between tests and [we lack] models for converting that variability to uncertainty in measurands [the quantities intended here are false positives and negatives, failure to acquire and enrol, and throughput].

... each specific recognition technology (iris, face, voice, fingerprint, hand, etc.) will have specific factors that must be within a state of statistical control. This list of factors is not well understood, although ample work in this area is continuing. For example, recent analysis of iris and face recognition test results shows us that to report false match and false non-match performance metrics for such systems without reporting on the percentage of data subjects wearing contact lenses, the period of time between collection of the compared image sets, the commercial systems used in the collection process, pupil dilation, and lighting direction is to report “nothing at all” [c.f. Eisenhart above]. Our reported measurements cannot be expected to be repeatable or reproducible without knowledge and control of these factors. [emphasis added]

... the test repeatability and reproducibility observed in technology tests are lost in scenario testing due to the loss of statistical control over a wide range of influence quantities.

... Our inability to apply concepts of statistical control to any or all of these factors will increase the level of uncertainty in our results and translate to loss of both repeatability and reproducibility.

... Test data from scenario evaluations should not be used as input to mathematical models of operational environments that require high levels of certainty for validity.

Overall, in WaPoMa's own words:

We can conclude that the three types of tests are measuring incommensurate quantities and therefore [we] should not be at all surprised when the values for the same technologies vary widely and unpredictably over the three types of tests.
That's quite enough quotation from WaPoMa's paper.

You get the picture – technology tests are hard to interpret, it's not clear what's being measured, they certainly can't be used to predict the results of scenario tests, which are hard to interpret, it's not clear what's being measured, and neither technology tests nor scenario tests can be used to predict the performance of biometrics systems in operation in the real world, which can't be measured anyway, not least because real impostors don't give themselves up after they've fooled the system and got through, they just don't have that researcher's enthusiasm for maintaining the statistics.

Tulipmania today – SNAFU
Where does that leave us?
If the border control authorities in the UK, Australia and New Zealand are asked why they have spent a fortune deploying so-called "smart gates" at international airports, their answer can't be "because tests show that the technology works so well". They can't say that because no-one knows what it means, the researchers don't know what they're measuring in a test. So what was it? A metrological impulse purchase?

Why are the UK Home Office spending taxpayers' money on the biometrics in ePassports and in residence permits for non-EEA nationals? (That's £650 million of taxpayers' money, split between IBM and CSC.) Why are the Home Office paying VFS Global and CSC to register the biometrics of millions of visa applicants all over the world like so many schoolboy stamp collectors? Why are UK nationals paying three times the correct price for a passport?

Why has Pakistan bothered to register the biometrics of 96 million citizens and to issue 70 million of them with biometric ID cards? All that effort. And the result? Not the harmonious state of law-abiding politically tranquil domestic peace and efficient public services sometimes touted as the automatic consequence of ID card schemes.

Why is India spending billions on Aadhaar, which depends on biometrics whose reliability is, so say the titans, utterly unknowable? And will the Unique Identification Authority of India ever answer my question how they can claim to offer unique identification when, based on their own figures, they would have to perform 18,000,000,000,000 manual checks to prove uniqueness? And why do they think Aadhaar will eradicate corruption, rather than automate corruption?

Why does Safran Group want to spend $1.5 billion acquiring L-1 Identity Solutions Inc., a biometrics company whose technology is statistically out of control? And what rare loss of financial control caused 3M to splash out on buying Cogent Inc.?

The questions keep coming.

Why is it only governments that believe in biometrics? How come the banks and the major retailers seem to be proof against this particular form of tulipmania? (Thank goodness they are proof against it. Any nation that inserted today's mass consumer biometrics into its payments systems would be instantly reduced to barter.)

On what basis does the European Commission spend its member states' money collecting the biometrics of millions of non-EEA visa applicants?

Does China really believe (superstitiously?) that biometrics will provide a Golden Shield against political unrest?

Are Russia about to introduce biometric visas? No-one knows – according to the St. Petersburg Times, not even the Russians.

But that's enough questions for the moment because the answer is the same in each case – there isn't an answer.

The only thing that's certain? Uncertainty
WaPoMa starts with uncertainty and he finishes with uncertainty. One thing you now know for sure is that if a biometrics salesman promises a government that his products can identify everyone in the country uniquely and verify their identity wherever necessary, then that mountebank is talking nonsense. WaPoMa says so. Maybe now governments will stop wasting their taxpayers' money on technology the reliability of which it is impossible to know?
Are WaPoMa right?

If the tulipmania persists, if you still think that there are established and trusted biometrics systems dutifully working away, all day every day, all over the world, helping to provide reliable identity management services to populations of 10 million people, 50 million, 100 million, ..., ask yourself what it is that you know about biometrics, measurement and probability that Jim Wayman, Antonio Possolo and Tony Mansfield don't know. It'll be a short list, but do send it to them.

Tulipmania yesterday – murky
In this mood of unstinting disclosure, WaPoMa chronicles a sad case in the history of science, when the report of a 1993 scenario test was suppressed because it suggested that a particular hand geometry biometrics system performed badly. One disobliging participant in the trial had been practising beforehand and managed all on his own to alter the equal error rate unfavourably by a factor of 25 – such are the perils of scenario testing and, of course, the operational perils in the field. It was poor design if the trial could be so sensitive to one participant. And it was reprehensible to cover up the results. The report has now been restored to the canon.
It is to be hoped that it will soon be joined by IBM's biometrics technology report, the "tulip bulb" that is costing UK taxpayers £650 million, with no known benefit.

WaPoMa finds it necessary to emphasise that researchers must be careful how they describe their results, they must take into account how their non-technical listeners will understand the researchers' words and how they will use – or, in the case of politicians and their officials, almost certainly (Type A) misuse – the results.

That must be the most painful confession in WaPoMa's paper.

Do you really need to do research to discover that you should only say what you mean, that you should say it clearly and that you should only say it if you believe it to be true? Isn't that redundant? Or otiose? Taken for granted?

Apparently not. It's certainly taken NIST a long time to learn the lesson. What on earth did NIST think people would understand from their May 2004 report on flat print fingerprinting when they wrote:

With the proper selection of an operating point, the one-to-many accuracy for a two-finger comparison against a database of 6,000,000 subjects is 95% with a false match rate of 0.08%. Using two fingers, the one-to-one matching accuracy is 99.5% with a false accept rate of 0.1%.
No layman reading that is going to understand that the figures come from a technology test and can't be extrapolated to operational systems. NIST didn't add:

... and by the way we don't know what we've been measuring, that 0.08% false match rate achieved using Cogent products can't be reproduced using another package and it can't be reproduced using Cogent products on another database, forget the 0.1% false accept rate because in an operational system real impostors don't turn themselves in, and please don't get the idea that there's any statistical control in this test of ours – for the best possible reasons, we actually haven't got a clue whether flat print fingerprinting will help to protect the US's borders.
That omission was mendacious. Judging by the WaPoMa paper, NIST should have made the addition suggested or something like it.

One month later they made up for it. A bit.

The USA PATRIOT Act 2001 specifies at section 403(c)(1) that NIST has to certify a technology that verifies people's identity:

The Attorney General and the Secretary of State jointly, through the National Institute of Standards and Technology (NIST), and in consultation with the Secretary of the Treasury and other Federal law enforcement and intelligence agencies the Attorney General or Secretary of State deems appropriate and in consultation with Congress, shall within 2 years after the date of the enactment of this section, develop and certify a technology standard that can be used to verify the identity of persons applying for a United States visa or such persons seeking to enter the United States pursuant to a visa for the purposes of conducting background checks, confirming identity, and ensuring that a person has not received a visa under a different name or such person seeking to enter the United States pursuant to a visa. [emphasis added]
That's what the Act says and, in all honesty, NIST cannot possibly comply. How are they supposed to know if the biometrics used in any particular case are a reliable proxy for someone's identity? It's completely out of their control. They can't put their name to it. So what NIST say in their certificates, according to their June 2004 review of flat print fingerprinting technology, is:

For purpose of NIST PATRIOT Act certification this test certifies the accuracy of the participating systems on the datasets used in the test. This evaluation does not certify that any of the systems tested meet the requirements of any specific government application. This would require that factors not included in this test such as image quality, dataset size, cost, and required response time be included.
There it is, the irreducible inanity of today's mass consumer biometrics is certificated.

WaPoMa's paper was delivered at a March 2010 NIST conference. Rarely has a towel been so well and truly – and elegantly and clearly and precisely and comprehensively – thrown in.

Tulipmania tomorrow – whither biometrics?
 

And since then?
 

In May 2010, a new government was elected in the UK. They immediately cancelled the plans to register the biometrics of all British citizens. (Not that those plans were very far advanced, Whitehall only having had eight years to work them out.) And in December 2010 they repealed the Identity Cards Act 2006. Anticipated volumes for the biometrics industry in the UK are down. Volumes, and political support.

In April 2011, over the signature of President Obama himself, the White House issued its National Strategy for Trusted Identities in Cyberspace. There is not a single occurrence of the word "biometrics" in the whole 45-page document, nor any of its cognates. Volumes and political support – down.

Again in April 2011, the Cabinet Office issued restricted documents describing the UK's proposed Digital Delivery Identity Assurance project. In 75 pages there is not a single occurrence of the word "biometrics" nor any of its cognates. Volumes and political support – down.

The Home Office's biometrics tulipmania may now at last have been shaken off by the rest of the UK government.

And yet again in April 2011, with their first-hand experience of tulipmania, the Dutch government suspended its plans to develop a centralised flat print fingerprint population register: "home affairs minister Piet Hein Donner ... says there are currently too many concerns about the security and reliability of the system". Volumes and political support – down.

Perhaps the tide is going out, the pandemic is receding, ... Governments are starting to peel away. And WaPoMa has pulled the academic rug out from under the biometrics industry's feet. The biometrics companies may be feeling a little lonely with their academic support, their aadhaar, gone. Maybe even a little nervous. If India cancels Aadhaar, where else can these companies ply their trade? Their planet is shrinking. (Ever resourceful, they are now planning to sell biometrics for orangutans.)

The earth may look roughly flat but actually it is roughly spherical. It may feel like the centre of the universe but actually it's not even the centre of the Milky Way. There are medical ailments that leeches can't cure and not all future events can be predicted by inspecting the entrails of a sacrificed sheep, however attractive the astrological symbols on the priest's pointed hat. We know that. And now, thanks to WaPoMa, we know that there is no good reason to invest in mass consumer biometrics.

Thank you
It wasn't just WaPoMa. IBM were at that March 2010 NIST conference as well, delivering a paper on the technique they used to choose the "best" biometrics system. That is the basis on which £650 million of UK taxpayers' money is being spent. Did IBM notice that WaPoMa's keynote speech at the same conference suggested that they were wasting their time, as well as taxpayers' money?
 

"Test data from scenario evaluations should not be used as input to mathematical models of operational environments that require high levels of certainty for validity". The decision to invest hundreds of millions of pounds of taxpayers' money requires high levels of certainty. Otherwise it's unbusinesslike, irresponsible, unscientific and illogical.

What WaPoMa tells us is that, if the investment decision is based on biometrics tests, then the argument is invalid.

WaPoMa's paper hasn't been repudiated. Not by San José State University, not by NIST and not by the National Physical Laboratory, all of which institutions advise governments the world over. There has been no stream of academic rebuttals. Nor has there been any public response from the tottering remnants of the mass consumer biometrics industry. There is nothing obviously wrong with his findings, the intimations of WaPoMa's operational acumen remain auspicious.

Where did David Blunkett and everyone else get the idea that mass consumer biometrics work reliably? Coming from respected institutions like NIST, statements like "using two fingers, the one-to-one matching accuracy is 99.5% with a false accept rate of 0.1%" must have played a part.

WaPoMa didn't write and publish his paper by accident. 

Why did he write it? He must have thought ahead to the effect his words would have, the perestroika that would follow his glasnost. He must have known that publishing his paper would impugn the credibility of the biometrics companies and the politicians and their officials who have let contracts to them.

He went ahead anyway. Why? Was it an act of expiation/atonement for that 99.5% one-to-one matching accuracy? Was it a public service, pointing out that if the anticipated performance of biometrics systems can't provide the basis for investment in mass consumer biometrics, then there is no basis?

Whatever, thank you, Messrs Wayman, Possolo and Mansfield.

David Moss spent eight years campaigning against the Home Office's ID card scheme RIP. Whitehall haven't given up yet – a national identity assurance service has appeared in their G-Cloud Programme. We shall see.

© 2011 Business Consultancy Services Ltd
on behalf of Dematerialised ID Ltd