New Zealand’s government says it is the first in the world to produce a set of standards for how public agencies should use the algorithms that increasingly drive decision-making by officials about every area of public life.
The increasing application of algorithms by governments around the world – particularly when they are deployed to profile or generate decisions about citizens by law enforcement, immigration, welfare and health agencies – has proved controversial in recent years. Critics claim that decision-making driven by lines of code can be inaccurate and discriminatory and that their use is often kept secret from the public.
A Dutch court ruled in February that an automated surveillance system to detect welfare fraud was unlawful – a decision that provoked debate about the need for greater scrutiny in other nations.
New Zealand is no exception: a 2019 review of algorithm use in the public sector had found “huge variability as to the extent of the use and how they were being used”, said James Shaw, the statistics minister and co-leader of the left-leaning Green party.
He said algorithms determined matters from the seemingly innocuous – such as when the traffic lights changed on a central Wellington street – to the more serious, including police decision-making for frontline officers, and their use was expanding.
The New Zealand charter, which the Guardian has viewed, was due to launch on Tuesday with 19 government agencies as initial signatories. In it, departments pledge to be publicly transparent about how decision-making is driven by algorithms, including giving “plain English” explanations; to make available information about the processes used and how data is stored unless forbidden by law (such as for reasons of national security); and to identify and manage biases informing algorithms.
Agencies must also consider te ao Māori, or Indigenous, worldviews on data collection – in New Zealand, Māori are disproportionately represented in the justice and prison system – and consult with groups affected by their equations.
“Part of this is driven by the fact that because people are now increasingly aware of how Facebook operates and the extent to which our lives are shaped by the use of and the manipulation of data,” Shaw said, referring to the way social media companies use algorithms to direct users towards content. “This is a really important part of building public trust in government institutions.”
Departments committed to the charter included New Zealand’s accident compensation scheme – which was criticised in 2017 for using algorithms to detect fraud among those on its books – and the corrections agency, which has deployed algorithms to determine an inmate’s risk of reoffending. The immigration agency, found in March to be profiling applicants by algorithm, is also a signatory.
But other bodies that have been the subject of high profile cases in which critics decried the use of algorithms – including the police, who came under fire from privacy advocates in 2019 for introducing facial recognition technology without announcing it, and the country’s spy agencies – were not among the signatories.
A spokesman for Shaw’s office said no agency had ruled out the charter and that others were expected to sign on later.
The standards do not include an enforcement mechanism. Shaw said he hoped that within months agencies that had signed the charter would provide information on their websites explaining how algorithms were used, even supplying source code – and that if they did not, the public could demand it.
Colin Gavaghan, a law professor at the University of Otago, was involved in the 2019 review that suggested an independent regulator to oversee the use of algorithms by agencies – a move the government has stopped short of in its charter. He said there was “plenty to like” in the new standards and he hoped they could serve as a blueprint for other governments.
“As ever, the devil’s in the detail and that means implementation,” he said. “We still feel like some kind of oversight and regulatory body that should have an all-of-government remit seems like a good idea.”
Gavaghan praised a point in the charter that required agencies to provide a channel to appeal against decisions informed by algorithms, but added that “people might not always know” they had been subject to such decisions, even under the new standards.
Shaw said he believed the document was a world first – no other government had produced standards that would cover all algorithms used by every signatory agency.