Government databases

There has been, and continues to be, much discussion about the deployment of (UK) government databases. There seem to be three issues: first, can you make them work; second, can you secure them; and third, should they exist at all and, if they should, what limits should be imposed on whose data (and how much of it) they should hold? The last of these is essentially an ethical question and depends on how much you trust the government and its agencies. This is not the forum for such a discussion and I shall not address this third question but I will discuss the first two points.

In terms of whether you can make large databases work the answer is surely that you can, given appropriate design, product selection and implementation skills. However, there are issues to be considered. The first is that there appears to be a view that the correct approach is to construct single monolithic databases. I do not believe that this is always, or even often, the best possible approach, particularly when it is a question of consolidating multiple, existing databases. For example, there was much talk after the Soham murders about constructing a single, national police database because there was no way of linking the different police forces’ databases. But of course you can do exactly that (and you could have done at the time) using federation from someone like Composite Software or IBM.

The second point is that government procurement seems to be perpetually behind the curve: it seems unaware of the latest technology that is available to it or, even, yesterday’s options. Instead, as the Rowntree Reform Trust’s recent report puts it “one noticeable effect is that the UK public sector always appears to get sold whatever technology or methodology is just going out of fashion in the public sector“. Now, I can’t believe that this is the fault of the vendors because they typically want to sell the latest, brightest technology because it’s cool. So, the conclusion must be that it is the system integrators and outsourcers who are advising the government who are behind the curve. And this is either because they simply haven’t done the appropriate research or because they figure they can make more money out of older technology, or both. Probably both.

The second question is with respect to security. This isn’t just about encryption, which is easy enough to handle with proper procedures but also about data quality. For example, the ContactPoint child database has been delayed because there are duplicate records appearing in the database that might help people to identify children at risk. The problem is that children at risk are supposed to be de-identified. And they are. But when the database is updated from external sources such as the child benefit database then duplicate, not de-identified records can enter the system with links created to children on the at-risk register. So, we have a data quality problem. The obvious answer is that the incoming data should be filtered, cleansed, de-duplicated and transformed with appropriate processes before the ContactPoint database is updated.

However, data quality problems will not go away. There is no such thing as perfect data quality. And even if you had perfect data today you wouldn’t have it tomorrow because quality deteriorates. To take a simple example, it is estimated that name and address data deteriorates at 1.4% per month (that’s people moving house, dying, getting married and changing their name, buying a new mobile phone, and so on). So any government database is going to have to take account of data quality issues. In fact, I am surprised that the civil liberties lobby haven’t made more of this point.

While this has been a brief summary, I have to say that, albeit that there have been some successes, I am tired of the expensive failures of the UK government when it comes to be implementing large databases and IT in general. Frankly, this is no more than incompetence: it is perfectly possible to design and implement these systems in a successful manner, given the right tools, knowledge, people and methodologies. The blame lies with the outsourced contractors who clearly do not have (or do not care about) these, and the government for trusting them.