Government by numbers: How big a problem is gaming?

Governments are under pressure to show their ever more educated and informed citizens that schools, hospitals and other public services are getting better. Traditionally, they have done that by spending money and building things: look, a brand new hospital! Of course, everybody knows that there is more to service quality than dollars, bricks and mortar. But at least we can see and touch bricks and mortar.  How can we put a finger on service quality? 

To answer that question, governments have been turning to the technology of performance management: “key performance indicators” (KPIs) and measuring performance against them. Malaysia is one of the leaders in “government by numbers.” Its service delivery unit, PEMANDU, is proud of the reduction in crime, one of its own KPIs, over the last few years. But here we run into a problem which is posed in a recent blog which focuses on PEMANDU:
 
“Is data really a neutral arbiter? Data can be gamed and metrics can generate perverse incentives.  Since you tend to ‘get what you measure’, if the measure is flawed, it can drive public systems in very extreme ways, with negative consequences for service delivery.”
 
Sure enough, some of PEMANDU’s critics in Malaysia have alleged that the reduction is more apparent than real because the statistics have been manipulated. PEMANDU rejects that. However, any time a government manages a public agency on the basis of performance statistics which the agency itself is gathering – so that the agency is marking its own homework – it is leading the agency into temptation. And if the agency yields to the temptation and gets caught out, public confidence is sure to be undermined. 
 
The British police have been on the receiving end of allegations like those in Malaysia for almost twenty years. When the volume of the allegations became deafening about a year ago, culminating in the UK Statistics Authority withdrawing its triple-A rating from police crime figures, the government asked its police watchdog, Her Majesty’s Inspectorate of Constabulary (HMIC), to have a look.  Its report has just been published.
 
HMIC found no hard evidence of fiddling, but it did find that 21% of a self-selected survey group of 17,000 police staff said they had experienced pressure to misrecord crime data in the previous six months. This corroborates evidence that the practice has got so entrenched in the past that the police have coined their own nicknames for it:

• "Cuffing": officers making crimes disappear by recording them as a "false report" or downgrading their seriousness.
• "Stitching" (from "stitching up"): offenders being charged with a crime when there is insufficient evidence. Police may know that prosecutors will never proceed with the case, but it still appears in police records as a crime which has been "solved".
• "Skewing": police activity being directed at easier-to-solve crimes, at the expense of more serious offences such as sex crimes
• "Nodding" – where convicted offenders are persuaded to admit to crimes they have not committed, possibly in exchange for a lower sentence

The survey evidence has to be read carefully. 61% of the survey group said they had experienced no pressure.  Also, we don’t know how many of the 21% who had experienced pressure yielded to it, let alone to what extent crime data was distorted as a result. But it is clear that UK police forces’ crime data has not been a “neutral arbiter.” Even if the extent of fiddling could be shown to be minor, public confidence would still be undermined – and it has been: we can see that from the  recent UK press coverage.
 
Stating the problem is the easy bit. What’s the solution? That is the subject of my second blog. Stay tuned!