That might tamp down some of the criticism, but it also undermines Finance Minister Nicola Willis’ message that AI is a key tool in the drive to trim $2.4 billion in public service spending.

The new law has been slammed by one privacy expert who pointed to overseas examples of where AI decision-making has gone badly wrong.

“It beggars belief that a law change with the potential to cause so much harm has been passed under urgency and without public consultation,” Simply Privacy partner Frith Tweedie told Tech Insider.

“The Social Security (Modernisation) Amendment Bill gives MSD very broad legislative authority ‘to approve the use of an automated electronic system to make any decision, exercise any power, comply with any obligation, or take any other related action under any specified provision, with appropriate safeguards’,” Tweedie said, quoting directly from the new legislation.

“We need to learn from the many international examples of welfare automation failures, not rushing ahead without proper scrutiny.”

“There are just so many cautionary tales out there. I looked at them in detail when I drafted the AI Impact Assessment Toolkit for the public service on behalf of Stats NZ, as well as updating it for the GCDO [Government Chief Digital Officer] as part of the Public Service AI Framework,” Tweedie said.

“The Australian Robodebt scandal involved the automated dispatch of welfare debt notices, resulting in more than 400,000 people being wrongly accused of owing money.

“Many suffered financial hardship, severe stress, anxiety and lasting mental health impacts,” Tweedie said. The scheme was found to be unlawful, she said, and was condemned by the Robodebt Royal Commission, endiing up costing the Australian Government over A$2b ($2.42b).

The final wave of settlements was made earlier this year.

Tweedie added: “Even more horrifying was the Dutch childcare benefits scandal, where an algorithm was used to predict childcare benefit fraud. More than 20,000 families were wrongly accused of benefit fraud amid human rights and privacy law breaches.”

The tax authority admitted at least 11,000 people were singled out for special scrutiny because of their ethnic origin or dual nationality, Tweedie said.

“But it’s the human impact that is most shocking – several victims died by suicide and over a thousand children were taken into foster care in connection with the scandal. The Dutch Government was ultimately forced to resign [in 2021].”

Upston responds

“The comparisons these social media commentators make are not comparable with the use of automated decision-making at the Ministry of Social Development,” a spokesman for Upston’s office told Tech Insider.

“The New Zealand MSD automated decision-making standard has been developed with some of the lessons of what has happened overseas. The standard was first introduced into legislation in the Child Support (Pass On) Acts Amendment Act 2023, under the previous Government.”

On Ryan Bridge TODAY, Luxon said: “I can tell you the minister has looked at the Australian situation very closely.”

The Prime Minister said that if a beneficiary had done a 26-week medical check and there was no change in their condition, their benefit could roll over. “That doesn’t need a deep people engagement,” Luxon said.

He compared it to the shift from check-in queues to apps and kiosks at airports during his time in charge of Air New Zealand. Staff were freed up for higher-level work.

This amendment has caught people by surprise. That’s had a clear impact on trust already. One of the consequences of moving so quickly is that people might lose trust, even in systems that might be quite trustworthy.

Automation already here – but now it’s got stronger rules: Think tank director

“It’s worth recognising that this pattern in the legislation – around ‘automated electronic systems’ for making decisions – has been in place in a number of statutes for at least five years now,” Brainbox Institute director Tom Barraclough told Tech Insider.

“There’s an argument that this amendment [the legislation passed last week] strengthens the oversight and governance controls on these systems,” Barraclough said.

The new legislation introduces a requirement to consult the Human Rights Commissioner and makes a published automated decision standard to address issues including discrimination and human oversight mandatory for the first time.

And, notwithstanding Willis’ quip that the civil service is “stuck in the 1980s”, various Government departments have taken advantage of the updated laws and tech.

A recent BusinessDesk round-up of state sector use of AI primarily found it was hard yards to gauge uptake, due to a lack of transparency, but it did uncover several examples, including:

Tweedie sees the AI push coming in the context of a “light touch” regulatory regime, including the latest guidance from the Ministry for Regulation.

Barraclough has highlighted the risks of rushing into dedicated AI legislation that he sees potentially undermining free expression and other human rights, as part of his think tank’s work for the Law Foundation and others on AI and other new technologies.

Things go haywire faster, and further

Barraclough said automated systems have “serious benefits” but also pointed out one of the main downsides of automation: mistakes propagate further and faster.

“The fact they’re made using computers can mean hundreds or thousands of decisions are wrong by the time any errors are discovered.”

He added, “A key insight from where these systems have failed in the past – in cases like the Robodebt system in Australia – is that the way they fail isn’t just a matter of technology.

“It’s really about the governance mechanisms. How do you know the system is any good, and when it’s going bad, how can you fix it as quickly as possible?”

Serious investment had to be made, up front, in oversight systems.

Easier said than done

Tweedie is sceptical about such guardrails.

“Ministers have said that safeguards will be in place, including human oversight and protections against bias,” she said.

“While that sounds promising, the reality is that it is incredibly difficult to build ‘fair’ automated systems to detect welfare fraud.

“The city of Amsterdam spent five years and €535,000 trying to build a ‘fair’ welfare fraud detection system in the wake of the childcare welfare benefit scandal.

“They did everything right, including the use of transparent algorithms, extensive bias testing, community consultation and extensive ethical oversight.

“But the system still produced discriminatory outcomes and was found to be no more effective than human caseworkers.”

Tweedie said the Government has “misread the room on AI” at a time when surveys indicate around two-thirds of Kiwis have qualms about the new technology.

She added: “Part of the problem is that the Government is trying to solve fundamentally political and social problems with technical solutions.

“And we need to understand – and be honest about – what technology can and can’t do.”

Chris Keall is an Auckland-based member of the Herald’s business team. He joined the Herald in 2018 and is the technology editor and a senior business writer.