Information hoarding won’t benefit governments in the long run

Better information means better decisions: that’s the theory that makes the rise of machine learning, new technology and algorithmic decision-making so exciting for policymakers and private businesses.

One bit of that revolution that is particularly exciting to UK ministers and officials is Palantir’s Foundry software. The joy of Foundry, from the perspective of the UK government, is its ability to understand, access, monitor and run experiments and projections with different operating systems and methods of data collections. Early on in the Covid-19 pandemic, a system originally developed to allow information to flow seamlessly from agents in the field proved near-perfect for allowing the British government to monitor what was going on across the complex thicket of the NHS’s overlapping health trusts and GP practices.

Civil servants describe Foundry’s use in monitoring the progress of Covid-19 as one of the few things in the UK’s pandemic response that worked as advertised. Many in government now hope that the software might hold the secret to doing something they have failed to do previously: integrate the NHS’s data sources to better understand how to manage treatments and improve patient care.

But better information comes at a cost. It makes it harder for governments or individuals to engage in polite fictions about how services really operate. The UK government’s prisons data dashboard, for example, is a triumph for transparency, but it reveals that Britain’s prison system is generally pretty bad at actually rehabilitating prisoners.

The NHS is more popular than most peer systems despite the fact it broadly delivers similar outcomes to them for similar amounts of money. One reason for this is that the various ugly things healthcare systems have to do — deciding who is treated first and what type of treatments to offer — are kept hidden from view by the health service’s model. A better system-wide view of how treatment outcomes and waits differ for people with learning disabilities, or from different ethnic minorities, is vital in improving how the system operates. But it makes states or governments look better if those realities are kept out of sight.

That is surely why, even as the British state is increasing the number of contracts awarded to Palantir, it is slowing down the public’s ability to see the software at work for themselves. Information collated by Foundry helped build the government’s Covid-19 dashboard, which made much of what was going on in British hospitals publicly available online. The UK government has since tapped Palantir to manage the underlying system for its vaccine rollout and on critical capacity. But the original dashboard risks becoming a historical artefact — even though it could become an all-purpose tool to monitor the performance of UK healthcare infrastructure as a whole.

The use of Foundry and the Covid-19 dashboard are both good examples of how desperation drives governments to innovate. In normal times, the British government would have been much more reluctant to overcome public resistance to allow an American company, and one with deep ties to the intelligence agencies, into the NHS. Nor, in normal times, would it have ever done anything as transparent as use that company’s tools to build a publicly available dashboard measuring the country’s effectiveness in fighting a novel disease.

But now that the acute phase of the pandemic is over, the government has decided it likes the benefits of better data-sharing and modelling, but dislikes the opportunity for better scrutiny that comes with making that information available to others.

Politicians are right to believe that better and more transparent data will mean having to deal with awkward questions. If it’s not clear that the UK is managing to effectively rehabilitate most people it imprisons, say, that raises uncomfortable questions about what, exactly, we hope to accomplish with short prison sentences. And a more transparent NHS may find it harder to engage in the covert rationing the service has practised in the past. But the reality is that you don’t need a government to be transparent to know that UK healthcare is in a bad way or that much of the public realm is in poor repair.

Around the world, the data revolution will make it harder for governments to disguise the truth about how the services they provide work. But they’d be better off becoming enthusiastic advocates for the benefits of transparency and sharing the fruits of better information with the voters, rather than hoarding them for the political class alone.

[email protected]