Posted on April 12, 2011 by kamal
Abstract UIDAI conducted a proof of concept trial of their Aadhaar project between March and June 2010. This paper reviews their report on the trial, ‘UID Enrolment Proof-of-Concept Report‘, published in December 2010.
First UIDAI promised that Aadhaar would deliver unique identification. Then they conducted a proof of concept trial to see if it can.
That’s back to front.
Which quite unnecessarily exposed UIDAI to reputational risks if the trial disproved the concept. And not just UIDAI. The very senior politicians they had involved in Aadhaar could be embarrassed. The Indian people could have some legitimate questions about the proper use of public money and the competence of their government and its agencies.
The trial did disprove the concept and this paper recommends that UIDAI quickly re-establish the logical order of the Aadhaar project.
It should be made clear to politicians and the media and the public that UIDAI’s promises depended on representations made to them by the biometrics industry. Those representations should be published. The biometrics companies’ names should be prominently highlighted in all relevant publicity material. It is the directors of the biometrics companies whose names and faces should be well-known and should be firmly attached to all the promises of unique identification, not UIDAI’s directors.
That way, if Aadhaar starts to unravel, if the sun fails to rise on the “new dawn of equal opportunity” – in short, if the argument in this paper happens to be right – the blame will be placed where it belongs and not, the wrong way round, on UIDAI.
It should be an easy decision to make, to adopt this recommendation. The alternative after all, is for UIDAI to look like the credulous dupes of some over-ambitious salesmen, dupes who wasted billions of dollars of public money while claiming to be “pro-poor”.
–––––––––– o O o ––––––––––
Introduction India has a population of something over a billion people and it is the job of the Unique Identification Authority of India (UIDAI) to enrol them all onto a population register – the CIDR or Central ID Repository – and to issue them with ID cards.
UIDAI have adopted “Aadhaar” as a brand name. Your Aadhaar (denoting foundation and support) is primarily your unique identification number, issued by UIDAI, but it is meant also to denote UIDAI-related personnel, systems, services, and products such as the ID card itself, and it is meant to inspire nationwide trust in them all and rock-solid confidence that the benefits of their project will be delivered.
–––––––––– o O o ––––––––––
The UIDAI Model According to the UIDAI Model: “The existing patchwork of multiple databases in India provides scope to individuals to furnish different personal information to different agencies“. India is not alone in that.
The question is, why should the CIDR database be any different from all the other databases? And the answer is, everyone hopes, biometrics: “The UIDAI has been setup by the Government of India with a mandate to issue a unique identification number to all the residents in the country. A key requirement of the Aadhaar is to minimize/eliminate duplicate identity to improve the efficacy of the service delivery. Biometrics features are selected to be the primary mechanism for ensuring uniqueness … Therefore, it is necessary to create a UIDAI Biometrics Centre of Competence (UBCC) that focuses on the unique challenges of UIDAI“.
The “mission” of UBCC is: “To design biometrics system that enables India to achieve uniqueness in the national registry“.
The CIDR will store and use biographical data, in addition to biometrics: “Registrars will send the applicant’s data to the CIDR for de-duplication. The CIDR will perform a search on key demographic fields and on the biometrics for each new enrolment, to minimise/eliminate duplicates in the database …”
But UIDAI aren’t sure about the accuracy of biographical data, not in the same way they’re sure (with good reason?) about biometric data. At least for the moment, the support and foundation of the CIDR is meant to be biometrics, not biographical data: “While certain demographical information is also provided, UIDAI provides no assurance of its accuracy. Demographic information shall not be used for filtering during the de-duplication process, but this capability shall be preserved for potential implementation in later phases of the UIDAI program“.
–––––––––– o O o ––––––––––
Biometrics Should UIDAI be so sure? How reliable are the biometrics Aadhaar depends on?
Earlier reviews of the chaotic mass consumer biometrics market suggest that UIDAI have taken on an impossible task.
But now UIDAI have conducted their own, up to date, proof of concept trial and, in the Conclusion section of their report, they say: “the biometric accuracy levels necessary for deduplication of all residents of India are achievable”. This follows the claim in the Results section of the report that “we can be confident that biometric matching can be used on a wider scale to realize the goal of creating unique identities”.
In fact, those conclusions do not follow from the evidence reported. Nothing in UIDAI’s surprisingly low quality report suggests that it would be feasible to prove that each electronic identity on the CIDR is unique. Not with a billion+ people on the database. Far from it, India can be confident, from the figures quoted in UIDAI’s proof of concept trial report, that deduplication could never be achieved.
–––––––––– o O o ––––––––––
The sea of false positives It just takes a simple two-step argument to prove the point. Nowhere does the maths involved rise above schoolboy level.
Step 1 – uniqueness
UIDAI must create one electronic identity on the CIDR corresponding to each real person in India. Each electronic identity will include a copy of the person’s fingerprints and irisprints. If UIDAI are to prove that each electronic identity is unique, then each set of biometrics must be compared to, and shown to be different from, every other set of biometrics.
UIDAI know that. As they say in the Results section: “the matching analysis was done on two sets of 20,000 biometrics, for a total of 40,000. However, the number of comparisons was several orders of magnitude more than 40,000, since each set of fingerprints would be matched against every other set of fingerprints in the data set”.
How many unique pairs of biometrics can be chosen from 40,000? Answer: 40,000 x 39,999 / 2 = 799,980,000. UIDAI are right. 40,000 is a number of the order of 104, whereas the number of comparisons which have to be made to prove uniqueness is of the order of 108.
The population of India is of course not 40,000. More like 1.2 billion or 1.2 x 109. So that the number of comparisons between pairs of biometrics that would need to be made to prove uniqueness is 7.2 x 1017.
Step 2 – false positives
It would take a very long time but, in a perfect world, those 7.2 x 1017 comparisons could be performed by computer and it could be proved automatically that there are no duplicates, i.e. each electronic identity is unique.
In the real world, problems arise. UIDAI say quite rightly that they must expect the odd false positive. In other words, on occasion, it will look as though two people have the same biometrics.
There may be hundreds of reasons for that. Here are just four of them:
The equipment used may not be entirely reliable.
An over-worked UIDAI agent may by mistake register Mr Clark’s biometrics against Mr Baker’s name.
Mr Clark may have naughtily enrolled twice, once in his real name and once as Mr Baker.
Mr Clark and Mr Baker may genuinely be two different people who happen to have the same biometrics.
When a false positive arises, it has to be investigated by a team of human beings. It can’t be resolved by computer.
How many false positives should India expect? In the Results section of their report UIDAI define FPIR, the false positive identification rate, and they say “we will look at the point where the FPIR (i.e. the possibility that a person is mistaken to be a different person) is 0.0025 %”. At that point, UIDAI would get 2½ false positives on average for every 100,000 comparisons.
Given that UIDAI have to make 7.2 x 1017 comparisons, how many false positives should they expect? Answer: (7.2 x 1017) x (2.5 x 10-5) = 1.8 x 1013. That’s 18,000,000,000,000 false positives for people to investigate and resolve.
It’s just not going to happen, is it. India has got better things to do with its time than to clean up the mess left behind by today’s unreliable mass consumer biometrics.
And that’s the end of the argument.
To prove uniqueness, every single Indian would have to investigate and resolve 15,000 false positives. Long before they had finished, many of them would be dead, many more Indians would have been born, and the task would remain incomplete. Using UIDAI’s own figures, India can be confident that the proof of uniqueness is not achievable. Not in the real world.
If any journalist asks UIDAI the question “are you sure that all the IDs on the CIDR are biometrically unique”, the only truthful answer is “no”.
UIDAI cannot possibly deliver what they promise. Their own figures prove it. If India is relying on unique identification, then India has a serious problem.
–––––––––– o O o ––––––––––
Feedback How many false positives would be manageable? One million? To achieve that, the FPIR would have to be 18,000,000 times smaller/better than 0.0025 percent. Is that feasible? How many more staff would UIDAI need? How much more would UIDAI have to spend on top quality biometrics equipment to make that improvement? If that is feasible, why didn’t the UIDAI Biometrics Centre of Competence say so? Why did UBCC “look at the point where the FPIR … 0.0025 %” and not at the point where it’s 1.4 x 10-12, which is what it would have to be to reduce the number of false positives to one million?
If the sea-of-false-positives argument above is correct, then biometrics do not provide the foundation needed for Aadhaar, the false conclusions drawn by UBCC in the proof of concept trial report impugn everyone’s trust in UIDAI and no-one can be confident that the benefits of Aadhaar will be achieved.
But is the argument correct? It needs a trusted and independent third party to state their case and deliver the verdict.
Some responses to this paper have been received. More would be appreciated.
One response was to argue that the number of comparisons required to prove uniqueness would be reduced by using multi-modal biometrics. Take another look. The FPIR of 0.0025 percent used in this paper is the multi-modal rate. If the calculations had been based on the FPIRs for either fingerprints or irisprints singly, then the prediction would be that UIDAI would have to perform even more than 18,000,000,000,000 comparisons.
It was also suggested that biographical data used in conjunction with biometric data would reduce the number of comparisons that need to be made to prove uniqueness. That may or may not be true but it isn’t what the UIDAI Model says, “demographic information shall not be used for filtering during the de-duplication process“, as noted above, and it isn’t what the proof of concept trial report says – which is that uniqueness can be proved using biometrics alone, “the biometric accuracy levels necessary for deduplication of all residents of India are achievable“. And on that point, UIDAI are wrong.
Or so it seems. (To repeat, more feedback would be appreciated.)
–––––––––– o O o ––––––––––
13 more questions Presumably the proof of concept trial report is the work of UBCC. They have to say why the sea-of-false-positives argument is wrong, if they can. And here are 13 more questions which could do with a response from them:
1. Over the years, the suppliers of biometric technology have been caught out repeatedly making exaggerated claims for the reliability of their wares. Their marketing material is now a little less gung-ho. UIDAI’s suppliers, L-1 Identity Solutions Inc. and Morpho among others, do not claim on their websites to be able to deliver unique identification in the case of large population registers. Given the sea of false positives, how could they? So why do UIDAI claim to be able to deliver unique identification? It’s easy to see why the suppliers don’t object to being boosted in this way. But why do UIDAI provide this unsolicited testimonial to the historically flaky products of the mass consumer biometrics industry?
2. Should UIDAI change their name? Perhaps they should drop the word “unique” and become simply “IDAI”. Or maybe they should change their name to something more like Pakistan’s “NADRA”, the National Database and Registration Authority. Not that NADRA seem to have brought peace, stability, social justice, universal inclusion and prosperity to Pakistan.
3. How keen will Visa and MasterCard be to proceed with their plans for biometrically verified payment services if unique identification is not available?
4. Many states of the European Union, and Pakistan, and China, and others, use biometrics for their identity management schemes. If today’s mass consumer biometrics are too unreliable to prove uniqueness, are they all, like India, perhaps wasting their time and money?
5. In December 2009 UIDAI published their Biometrics Design Standards For UID Applications. At that stage, apparently under the influence of the US National Institute of Standards and Technology (NIST), they had high hopes of using facial geometry as a biometric. A year later, the support for facial geometry in the UIDAI Model is now tepid, at best: “Multiple modalities such as– fingerprint and iris image will be used for de-duplication. Face photograph is provided if the vendor desires to use it for de-duplication“. And in the proof of concept trial, they dropped facial recognition by computer altogether. Hardly surprising. Facial geometry is traditionally the least reliable of the biometrics commonly considered. In general, people would do better to toss a coin than to rely on facial geometry. Is the International Civil Aviation Organization wasting everyone’s time and money, including India’s, by insisting on facial geometry being implemented in ePassports?
6. … and are the UK, Australia and New Zealand, and Portugal wasting their time and money using so-called “smart gates” for border control at international airports? These machines rely on facial recognition. Does India intend to install them?
7. UIDAI’s identification results (Annexure 3, p.30) are based on 20,000 people chosen from the 60,000 who attended two biometric enrolment sessions. What do the results for all 60,000 look like? Why were the full results not published? How were the 20,000 chosen? What was wrong with the other 40,000? Why don’t UIDAI report the deduplication statistics for the one million people now enrolled on the CIDR, instead of a paltry 20,000 of them?
8. Is a field trial of 20,000 big enough to tell India what to expect when it comes to 1.2 billion people?
9. UIDAI are going to need a lot of different staff using a lot of different biometrics equipment in a lot of different urban and rural locations – how feasible is it to keep the FPIR as low as 0.0025 percent?
10. Most of the participants in the proof of concept trial were adults. UIDAI’s report is not precise on this point, but it looks as though the results for children are based overwhelmingly on a sample taken from just one school. If that is the case, they can tell India so little, why do UIDAI bother to publish the children’s results in the trial report?
11. Why don’t Visa and MasterCard rely on biometrically verified payments anywhere in Europe and the US? If they’re not good enough for Europe and the US, why should they be acceptable in India?
12. The US company Pay By Touch tried to promote biometrically verified payment services. They went bust. Have UIDAI considered this warning?
13. GMAC, the body representing 1,800 business schools in 110 countries, dropped fingerprinting as a way of verifying identity after a two-year trial. If the business schools don’t recommend the technology, why do UIDAI recommend it?
–––––––––– o O o ––––––––––
Identification v. verification This paper concentrates on the problems of identification, i.e. proving that each record on the CIDR is unique.
Some attention must be paid to the separate problems of verification, i.e. proving that your biometrics are the same as the biometrics on the ID card/passport that you are using to cross a state border, for example, or to register with a doctor to obtain state healthcare or to prove your right to work in India.
When it comes to verifying identity, there is a trade-off between false reject rates and false accept rates, they are inversely proportional. The false accept rate must be low to reduce the probability of impostors defrauding the state and the banks. But that tends to push up the false reject rate, more people get wrongly told by a computer that they are not themselves. And when that happens, they can’t cross the border or register with the doctor or get the new job.
The Iris identification ROCs (1:1) for adults and children graph in the proof of concept trial report (Annexure 3, p.31) should probably be labelled “Iris verification ROCs (1:1) for adults and children”. UBCC have some way to go.
It is impossible to tell from UIDAI’s report what the level of false rejection in Aadhaar is. It could be very low. It could be just over 6 percent (Annexure 3, p.32). It could be anything – one 2004 trial in the UK found a false reject rate for fingerprints, using L-1 Identity Solutions technology, of about 20 percent.
But if the entitlement to public services depends on the biometric verification of identity, and if 6 percent of the population find themselves denied their entitlement as a result, that’s 72 million excluded people. They will not be pleased. Neither will Visa and MasterCard be pleased, if they find that they lose 72 million customers because biometric verification is still too unreliable.
72 million rioting people have a way of making their anger and disappointment felt. The result may be that biometrics are no use to India and that all the money spent on Aadhaar is wasted.
–––––––––– o O o ––––––––––
Back to front The proper conclusion from UIDAI’s proof of concept trial seems to be that the concept is not proven, the system design is a failure, its hypothesis is wrong, unique identification is not achievable. Ask any 16 year-old studying science (any logical 16 year-old, come to think of it, not just science students), that should be the signal to halt Aadhaar and think again.
The proof of concept trial report reviewed here is a poor support for Indian confidence, it provides no foundation for trust in UIDAI and it diminishes the Aadhaar brand. The trial results are the opposite of the stated conclusions. UBCC need to raise their game before they conduct their next biometrics trial.
The figures show that unique identification is not possible, the report states that it is. The proof of concept trial is a failure, the Aadhaar project proceeds nevertheless. It’s all back to front. Why?
Because UIDAI’s approach to biometrics is back to front.
First UIDAI assumed that today’s mass consumer biometrics technology is reliable enough to deliver unique identification and adequate verification. They made all their plans accordingly. They hired staff. They contracted with registrars and enrolment agencies and introducers and authenticators (as per the UIDAI Model). They paraded the most senior politicians in the land to give the project their backing. They briefed the press and they ran a nationwide publicity campaign. Global, even. All the while, they were making promises, raising expectations, committing themselves. A lot of hope, wishful thinking, the best of intentions, sackloads of public money, the benefits would be monumental. Then, and only then, they conducted a trial to test the feasibility of Aadhaar. That’s the wrong way round.
As it happens, the UK made the same mistake. For years, between 2002 and 2010, the Home Office were in the undignified position of being quite unable to answer probing questions, whether posed by critics or supporters, about the proposed UK ID card scheme. The facts simply don’t support the claims the Home Office was making – see for example their document ‘Safeguarding identity’ – about being able to “lock” people to a single identity (para.3.29) and their fatuous promise that ID cards would “make life easier” (para.2.1). Public money was wasted on a pipe dream.
There were many problems with the UK scheme. Not just biometrics. But biometrics is the easiest problem to understand and to discuss objectively and on which to reach an agreed decision, it’s quantifiable, there are no difficult value judgements to make, it’s just technology. And not a very good technology – whenever there is a large-scale field trial, as opposed to the mere computer modelling exercises favoured by NIST, mass consumer biometrics prove to be too unreliable for the ID card schemes that depend on them.
By the time the stillborn scheme was finally cancelled, the Home Office had lost all credibility, it was totally demoralised and it is now excluded from discussions of the UK’s new, and still unspecified, Digital Delivery Identity Assurance project.
–––––––––– o O o ––––––––––
Deduping UIDAI If UIDAI wish to avoid the same fate – ridicule, disgrace, ostracisation, … – they had better display a lot more dignity than the UK Home Office did for eight years.
The danger exists that, having given their unsolicited testimonials to the biometrics industry and its unreliable products, UIDAI will be left to clean up the expensive mess left in India as best they can when Aadhaar is cancelled, while the biometrics industry road-show moves on to the next country and repeats the trick.
UIDAI need to make it clear to politicians and the media and the public that the magical claims made for biometric identification and verification were hypothetical. They have been proved to be wrong. And that’s the biometrics industry’s problem, not UIDAI’s.
There are any number of news items in the media like the following article by Amruta Byatnal published in The Hindu of 29 September 2010 …
the recommendation in this paper is that UIDAI should ensure that a lot of news items like the following mock-up are published in addition. And journalists should lose no opportunity to ask the directors of the companies supplying biometric technology to UIDAI to confirm that it is feasible to prove that each electronic identity on the CIDR is biometrically unique:
Pro-poor approach According to the UIDAI Model, India is adopting a pro-poor approach: “The UIDAI envisions full enrolment of the residents, with a focus on enrolling India’s poor and underprivileged communities. The Registrars that the Authority plans to partner with in its first phase … will help bring large number of the poor and underprivileged into the UID system. The UID method of authentication will also improve service delivery for the poor“.
The poor are not helped by UIDAI pretending that a technology works when it doesn’t. Who is?
Abstract UIDAI conducted a proof of concept trial of their Aadhaar project between March and June 2010. This paper reviews their report on the trial, ‘UID Enrolment Proof-of-Concept Report‘, published in December 2010.
First UIDAI promised that Aadhaar would deliver unique identification. Then they conducted a proof of concept trial to see if it can.
That’s back to front.
Which quite unnecessarily exposed UIDAI to reputational risks if the trial disproved the concept. And not just UIDAI. The very senior politicians they had involved in Aadhaar could be embarrassed. The Indian people could have some legitimate questions about the proper use of public money and the competence of their government and its agencies.
The trial did disprove the concept and this paper recommends that UIDAI quickly re-establish the logical order of the Aadhaar project.
It should be made clear to politicians and the media and the public that UIDAI’s promises depended on representations made to them by the biometrics industry. Those representations should be published. The biometrics companies’ names should be prominently highlighted in all relevant publicity material. It is the directors of the biometrics companies whose names and faces should be well-known and should be firmly attached to all the promises of unique identification, not UIDAI’s directors.
That way, if Aadhaar starts to unravel, if the sun fails to rise on the “new dawn of equal opportunity” – in short, if the argument in this paper happens to be right – the blame will be placed where it belongs and not, the wrong way round, on UIDAI.
It should be an easy decision to make, to adopt this recommendation. The alternative after all, is for UIDAI to look like the credulous dupes of some over-ambitious salesmen, dupes who wasted billions of dollars of public money while claiming to be “pro-poor”.
–––––––––– o O o ––––––––––
Introduction India has a population of something over a billion people and it is the job of the Unique Identification Authority of India (UIDAI) to enrol them all onto a population register – the CIDR or Central ID Repository – and to issue them with ID cards.
UIDAI have adopted “Aadhaar” as a brand name. Your Aadhaar (denoting foundation and support) is primarily your unique identification number, issued by UIDAI, but it is meant also to denote UIDAI-related personnel, systems, services, and products such as the ID card itself, and it is meant to inspire nationwide trust in them all and rock-solid confidence that the benefits of their project will be delivered.
–––––––––– o O o ––––––––––
The UIDAI Model According to the UIDAI Model: “The existing patchwork of multiple databases in India provides scope to individuals to furnish different personal information to different agencies“. India is not alone in that.
The question is, why should the CIDR database be any different from all the other databases? And the answer is, everyone hopes, biometrics: “The UIDAI has been setup by the Government of India with a mandate to issue a unique identification number to all the residents in the country. A key requirement of the Aadhaar is to minimize/eliminate duplicate identity to improve the efficacy of the service delivery. Biometrics features are selected to be the primary mechanism for ensuring uniqueness … Therefore, it is necessary to create a UIDAI Biometrics Centre of Competence (UBCC) that focuses on the unique challenges of UIDAI“.
The “mission” of UBCC is: “To design biometrics system that enables India to achieve uniqueness in the national registry“.
The CIDR will store and use biographical data, in addition to biometrics: “Registrars will send the applicant’s data to the CIDR for de-duplication. The CIDR will perform a search on key demographic fields and on the biometrics for each new enrolment, to minimise/eliminate duplicates in the database …”
But UIDAI aren’t sure about the accuracy of biographical data, not in the same way they’re sure (with good reason?) about biometric data. At least for the moment, the support and foundation of the CIDR is meant to be biometrics, not biographical data: “While certain demographical information is also provided, UIDAI provides no assurance of its accuracy. Demographic information shall not be used for filtering during the de-duplication process, but this capability shall be preserved for potential implementation in later phases of the UIDAI program“.
–––––––––– o O o ––––––––––
Biometrics Should UIDAI be so sure? How reliable are the biometrics Aadhaar depends on?
Earlier reviews of the chaotic mass consumer biometrics market suggest that UIDAI have taken on an impossible task.
But now UIDAI have conducted their own, up to date, proof of concept trial and, in the Conclusion section of their report, they say: “the biometric accuracy levels necessary for deduplication of all residents of India are achievable”. This follows the claim in the Results section of the report that “we can be confident that biometric matching can be used on a wider scale to realize the goal of creating unique identities”.
In fact, those conclusions do not follow from the evidence reported. Nothing in UIDAI’s surprisingly low quality report suggests that it would be feasible to prove that each electronic identity on the CIDR is unique. Not with a billion+ people on the database. Far from it, India can be confident, from the figures quoted in UIDAI’s proof of concept trial report, that deduplication could never be achieved.
–––––––––– o O o ––––––––––
The sea of false positives It just takes a simple two-step argument to prove the point. Nowhere does the maths involved rise above schoolboy level.
Step 1 – uniqueness
UIDAI must create one electronic identity on the CIDR corresponding to each real person in India. Each electronic identity will include a copy of the person’s fingerprints and irisprints. If UIDAI are to prove that each electronic identity is unique, then each set of biometrics must be compared to, and shown to be different from, every other set of biometrics.
UIDAI know that. As they say in the Results section: “the matching analysis was done on two sets of 20,000 biometrics, for a total of 40,000. However, the number of comparisons was several orders of magnitude more than 40,000, since each set of fingerprints would be matched against every other set of fingerprints in the data set”.
How many unique pairs of biometrics can be chosen from 40,000? Answer: 40,000 x 39,999 / 2 = 799,980,000. UIDAI are right. 40,000 is a number of the order of 104, whereas the number of comparisons which have to be made to prove uniqueness is of the order of 108.
The population of India is of course not 40,000. More like 1.2 billion or 1.2 x 109. So that the number of comparisons between pairs of biometrics that would need to be made to prove uniqueness is 7.2 x 1017.
Step 2 – false positives
It would take a very long time but, in a perfect world, those 7.2 x 1017 comparisons could be performed by computer and it could be proved automatically that there are no duplicates, i.e. each electronic identity is unique.
In the real world, problems arise. UIDAI say quite rightly that they must expect the odd false positive. In other words, on occasion, it will look as though two people have the same biometrics.
There may be hundreds of reasons for that. Here are just four of them:
The equipment used may not be entirely reliable.
An over-worked UIDAI agent may by mistake register Mr Clark’s biometrics against Mr Baker’s name.
Mr Clark may have naughtily enrolled twice, once in his real name and once as Mr Baker.
Mr Clark and Mr Baker may genuinely be two different people who happen to have the same biometrics.
When a false positive arises, it has to be investigated by a team of human beings. It can’t be resolved by computer.
How many false positives should India expect? In the Results section of their report UIDAI define FPIR, the false positive identification rate, and they say “we will look at the point where the FPIR (i.e. the possibility that a person is mistaken to be a different person) is 0.0025 %”. At that point, UIDAI would get 2½ false positives on average for every 100,000 comparisons.
Given that UIDAI have to make 7.2 x 1017 comparisons, how many false positives should they expect? Answer: (7.2 x 1017) x (2.5 x 10-5) = 1.8 x 1013. That’s 18,000,000,000,000 false positives for people to investigate and resolve.
It’s just not going to happen, is it. India has got better things to do with its time than to clean up the mess left behind by today’s unreliable mass consumer biometrics.
And that’s the end of the argument.
To prove uniqueness, every single Indian would have to investigate and resolve 15,000 false positives. Long before they had finished, many of them would be dead, many more Indians would have been born, and the task would remain incomplete. Using UIDAI’s own figures, India can be confident that the proof of uniqueness is not achievable. Not in the real world.
If any journalist asks UIDAI the question “are you sure that all the IDs on the CIDR are biometrically unique”, the only truthful answer is “no”.
UIDAI cannot possibly deliver what they promise. Their own figures prove it. If India is relying on unique identification, then India has a serious problem.
–––––––––– o O o ––––––––––
Feedback How many false positives would be manageable? One million? To achieve that, the FPIR would have to be 18,000,000 times smaller/better than 0.0025 percent. Is that feasible? How many more staff would UIDAI need? How much more would UIDAI have to spend on top quality biometrics equipment to make that improvement? If that is feasible, why didn’t the UIDAI Biometrics Centre of Competence say so? Why did UBCC “look at the point where the FPIR … 0.0025 %” and not at the point where it’s 1.4 x 10-12, which is what it would have to be to reduce the number of false positives to one million?
If the sea-of-false-positives argument above is correct, then biometrics do not provide the foundation needed for Aadhaar, the false conclusions drawn by UBCC in the proof of concept trial report impugn everyone’s trust in UIDAI and no-one can be confident that the benefits of Aadhaar will be achieved.
But is the argument correct? It needs a trusted and independent third party to state their case and deliver the verdict.
Some responses to this paper have been received. More would be appreciated.
One response was to argue that the number of comparisons required to prove uniqueness would be reduced by using multi-modal biometrics. Take another look. The FPIR of 0.0025 percent used in this paper is the multi-modal rate. If the calculations had been based on the FPIRs for either fingerprints or irisprints singly, then the prediction would be that UIDAI would have to perform even more than 18,000,000,000,000 comparisons.
It was also suggested that biographical data used in conjunction with biometric data would reduce the number of comparisons that need to be made to prove uniqueness. That may or may not be true but it isn’t what the UIDAI Model says, “demographic information shall not be used for filtering during the de-duplication process“, as noted above, and it isn’t what the proof of concept trial report says – which is that uniqueness can be proved using biometrics alone, “the biometric accuracy levels necessary for deduplication of all residents of India are achievable“. And on that point, UIDAI are wrong.
Or so it seems. (To repeat, more feedback would be appreciated.)
–––––––––– o O o ––––––––––
13 more questions Presumably the proof of concept trial report is the work of UBCC. They have to say why the sea-of-false-positives argument is wrong, if they can. And here are 13 more questions which could do with a response from them:
1. Over the years, the suppliers of biometric technology have been caught out repeatedly making exaggerated claims for the reliability of their wares. Their marketing material is now a little less gung-ho. UIDAI’s suppliers, L-1 Identity Solutions Inc. and Morpho among others, do not claim on their websites to be able to deliver unique identification in the case of large population registers. Given the sea of false positives, how could they? So why do UIDAI claim to be able to deliver unique identification? It’s easy to see why the suppliers don’t object to being boosted in this way. But why do UIDAI provide this unsolicited testimonial to the historically flaky products of the mass consumer biometrics industry?
2. Should UIDAI change their name? Perhaps they should drop the word “unique” and become simply “IDAI”. Or maybe they should change their name to something more like Pakistan’s “NADRA”, the National Database and Registration Authority. Not that NADRA seem to have brought peace, stability, social justice, universal inclusion and prosperity to Pakistan.
3. How keen will Visa and MasterCard be to proceed with their plans for biometrically verified payment services if unique identification is not available?
4. Many states of the European Union, and Pakistan, and China, and others, use biometrics for their identity management schemes. If today’s mass consumer biometrics are too unreliable to prove uniqueness, are they all, like India, perhaps wasting their time and money?
5. In December 2009 UIDAI published their Biometrics Design Standards For UID Applications. At that stage, apparently under the influence of the US National Institute of Standards and Technology (NIST), they had high hopes of using facial geometry as a biometric. A year later, the support for facial geometry in the UIDAI Model is now tepid, at best: “Multiple modalities such as– fingerprint and iris image will be used for de-duplication. Face photograph is provided if the vendor desires to use it for de-duplication“. And in the proof of concept trial, they dropped facial recognition by computer altogether. Hardly surprising. Facial geometry is traditionally the least reliable of the biometrics commonly considered. In general, people would do better to toss a coin than to rely on facial geometry. Is the International Civil Aviation Organization wasting everyone’s time and money, including India’s, by insisting on facial geometry being implemented in ePassports?
6. … and are the UK, Australia and New Zealand, and Portugal wasting their time and money using so-called “smart gates” for border control at international airports? These machines rely on facial recognition. Does India intend to install them?
7. UIDAI’s identification results (Annexure 3, p.30) are based on 20,000 people chosen from the 60,000 who attended two biometric enrolment sessions. What do the results for all 60,000 look like? Why were the full results not published? How were the 20,000 chosen? What was wrong with the other 40,000? Why don’t UIDAI report the deduplication statistics for the one million people now enrolled on the CIDR, instead of a paltry 20,000 of them?
8. Is a field trial of 20,000 big enough to tell India what to expect when it comes to 1.2 billion people?
9. UIDAI are going to need a lot of different staff using a lot of different biometrics equipment in a lot of different urban and rural locations – how feasible is it to keep the FPIR as low as 0.0025 percent?
10. Most of the participants in the proof of concept trial were adults. UIDAI’s report is not precise on this point, but it looks as though the results for children are based overwhelmingly on a sample taken from just one school. If that is the case, they can tell India so little, why do UIDAI bother to publish the children’s results in the trial report?
11. Why don’t Visa and MasterCard rely on biometrically verified payments anywhere in Europe and the US? If they’re not good enough for Europe and the US, why should they be acceptable in India?
12. The US company Pay By Touch tried to promote biometrically verified payment services. They went bust. Have UIDAI considered this warning?
13. GMAC, the body representing 1,800 business schools in 110 countries, dropped fingerprinting as a way of verifying identity after a two-year trial. If the business schools don’t recommend the technology, why do UIDAI recommend it?
–––––––––– o O o ––––––––––
Identification v. verification This paper concentrates on the problems of identification, i.e. proving that each record on the CIDR is unique.
Some attention must be paid to the separate problems of verification, i.e. proving that your biometrics are the same as the biometrics on the ID card/passport that you are using to cross a state border, for example, or to register with a doctor to obtain state healthcare or to prove your right to work in India.
When it comes to verifying identity, there is a trade-off between false reject rates and false accept rates, they are inversely proportional. The false accept rate must be low to reduce the probability of impostors defrauding the state and the banks. But that tends to push up the false reject rate, more people get wrongly told by a computer that they are not themselves. And when that happens, they can’t cross the border or register with the doctor or get the new job.
The Iris identification ROCs (1:1) for adults and children graph in the proof of concept trial report (Annexure 3, p.31) should probably be labelled “Iris verification ROCs (1:1) for adults and children”. UBCC have some way to go.
It is impossible to tell from UIDAI’s report what the level of false rejection in Aadhaar is. It could be very low. It could be just over 6 percent (Annexure 3, p.32). It could be anything – one 2004 trial in the UK found a false reject rate for fingerprints, using L-1 Identity Solutions technology, of about 20 percent.
But if the entitlement to public services depends on the biometric verification of identity, and if 6 percent of the population find themselves denied their entitlement as a result, that’s 72 million excluded people. They will not be pleased. Neither will Visa and MasterCard be pleased, if they find that they lose 72 million customers because biometric verification is still too unreliable.
72 million rioting people have a way of making their anger and disappointment felt. The result may be that biometrics are no use to India and that all the money spent on Aadhaar is wasted.
–––––––––– o O o ––––––––––
Back to front The proper conclusion from UIDAI’s proof of concept trial seems to be that the concept is not proven, the system design is a failure, its hypothesis is wrong, unique identification is not achievable. Ask any 16 year-old studying science (any logical 16 year-old, come to think of it, not just science students), that should be the signal to halt Aadhaar and think again.
The proof of concept trial report reviewed here is a poor support for Indian confidence, it provides no foundation for trust in UIDAI and it diminishes the Aadhaar brand. The trial results are the opposite of the stated conclusions. UBCC need to raise their game before they conduct their next biometrics trial.
The figures show that unique identification is not possible, the report states that it is. The proof of concept trial is a failure, the Aadhaar project proceeds nevertheless. It’s all back to front. Why?
Because UIDAI’s approach to biometrics is back to front.
First UIDAI assumed that today’s mass consumer biometrics technology is reliable enough to deliver unique identification and adequate verification. They made all their plans accordingly. They hired staff. They contracted with registrars and enrolment agencies and introducers and authenticators (as per the UIDAI Model). They paraded the most senior politicians in the land to give the project their backing. They briefed the press and they ran a nationwide publicity campaign. Global, even. All the while, they were making promises, raising expectations, committing themselves. A lot of hope, wishful thinking, the best of intentions, sackloads of public money, the benefits would be monumental. Then, and only then, they conducted a trial to test the feasibility of Aadhaar. That’s the wrong way round.
As it happens, the UK made the same mistake. For years, between 2002 and 2010, the Home Office were in the undignified position of being quite unable to answer probing questions, whether posed by critics or supporters, about the proposed UK ID card scheme. The facts simply don’t support the claims the Home Office was making – see for example their document ‘Safeguarding identity’ – about being able to “lock” people to a single identity (para.3.29) and their fatuous promise that ID cards would “make life easier” (para.2.1). Public money was wasted on a pipe dream.
There were many problems with the UK scheme. Not just biometrics. But biometrics is the easiest problem to understand and to discuss objectively and on which to reach an agreed decision, it’s quantifiable, there are no difficult value judgements to make, it’s just technology. And not a very good technology – whenever there is a large-scale field trial, as opposed to the mere computer modelling exercises favoured by NIST, mass consumer biometrics prove to be too unreliable for the ID card schemes that depend on them.
By the time the stillborn scheme was finally cancelled, the Home Office had lost all credibility, it was totally demoralised and it is now excluded from discussions of the UK’s new, and still unspecified, Digital Delivery Identity Assurance project.
–––––––––– o O o ––––––––––
Deduping UIDAI If UIDAI wish to avoid the same fate – ridicule, disgrace, ostracisation, … – they had better display a lot more dignity than the UK Home Office did for eight years.
The danger exists that, having given their unsolicited testimonials to the biometrics industry and its unreliable products, UIDAI will be left to clean up the expensive mess left in India as best they can when Aadhaar is cancelled, while the biometrics industry road-show moves on to the next country and repeats the trick.
UIDAI need to make it clear to politicians and the media and the public that the magical claims made for biometric identification and verification were hypothetical. They have been proved to be wrong. And that’s the biometrics industry’s problem, not UIDAI’s.
There are any number of news items in the media like the following article by Amruta Byatnal published in The Hindu of 29 September 2010 …
the recommendation in this paper is that UIDAI should ensure that a lot of news items like the following mock-up are published in addition. And journalists should lose no opportunity to ask the directors of the companies supplying biometric technology to UIDAI to confirm that it is feasible to prove that each electronic identity on the CIDR is biometrically unique:
Pro-poor approach According to the UIDAI Model, India is adopting a pro-poor approach: “The UIDAI envisions full enrolment of the residents, with a focus on enrolling India’s poor and underprivileged communities. The Registrars that the Authority plans to partner with in its first phase … will help bring large number of the poor and underprivileged into the UID system. The UID method of authentication will also improve service delivery for the poor“.
The poor are not helped by UIDAI pretending that a technology works when it doesn’t. Who is?
