Critical flaw in farming’s water-quality tool: scientists

The tool many councils are using to try to curb water pollution needs re-building from scratch -- but no-one seems to be listening, says a group of scientists. They say continuing as we are risks getting the wrong picture of  farming's water impacts.

A plan to fix flaws in the Overseer water quality tool is not sufficiently radical, says a group of scientists – with potentially serious consequences for the environment.

Four scientists and mathematical modelling experts want to sound an alarm about what they say is the tool’s use of flawed mathematical modelling, which they fear could result in wildly inaccurate estimates of water pollution.

Former IPCC working group director Martin Manning, Massey University Professor Emeritus of Industrial Mathematics Graeme Wake, Massey agricultural senior scientist Tony Pleasants and a retired associate professor of mathematics, John Gamlen, gave Newsroom a joint statement outlining their worries.

They say proposals to spend millions fixing Overseer’s problems will not be enough unless the underlying mathematical model is ditched and replaced by more sophisticated modelling that can reflect interactions between different biological  processes. And they've asked to be allowed to see inside the tool, and any moves to improve it.

Asked about the group's concerns, the Minister of Agriculture, Damien O’Connor, told Newsroom he supports opening up Overseer to a full and independent peer review – as recommended last year by the Parliamentary Commissioner for the Environment, Simon Upton. Any peer review would include a review by mathematical modellers, O’Connor said.

It is not clear yet when an independent review might happen or whether it would be open to outsiders, or only selected experts.

Overseer may be in line for additional funding in the Budget on May 30,although that hasn't been confirmed.

Newsroom understands officials from the Ministry for Primary Industries are due to report back sometime this month with their response to Upton’s critical report on Overseer, which pointed to serious problems with the tool -- one that regional councils have been increasingly using to measure the nitrogen being lost from farms to lakes and rivers.

Overseer is an online software model jointly-owned by two fertiliser companies, Ballance and Ravensdown, with MPI and a Crown-owned science company, AgResearch. There are plans to calibrate and improve it. However Wake, Manning and the others say the measures announced so far won’t be enough. The foursome support another of Upton’s recommendations to the government -- to consider buying the tool and making it open-source, so that anyone can review its inner workings.

Overseer was originally designed as a commercial tool to help farmers tweak their fertiliser use to maximise the milk or meat they gain for the least expenditure on fertiliser. It is still used to help make farm management decisions, but, because the software estimates how much fertiliser is being wasted from paddocks, it has morphed into a quasi-regulatory tool that several regional councils use to manage nitrogen pollution in some of our most troubled waterways.

Faced with pressure to clean up the country's waterways, councils turned to using Overseer because it was the only tool they felt they had available, Upton said in his report last year. However there have been major questions raised about its transparency and accuracy.

Because Overseer is a commercial enterprise, its source code is secret, making it difficult for outside scientists to properly peer-review it. Even an internal analysis of how accurate it is, was kept out of the public domain – making it something of a “black box”. According to Upton, staff at regional councils who are using the tool for water quality may not fully understand its workings, making it difficult to reassure farmers or the public that it is fit-for-purpose.

O’Connor said Overseer was “an important part” of the government’s plans to improve water quality, though he didn't venture a view on whether it might be purchased outright by the government or given some official status. 

“It’s a useful tool, whether or not its use is stepped up for national water quality regulation," he said. "We allocated $5 million in the 2018 Budget to improve the Overseer model and software and to support Overseer’s operating costs."

"I fully agree that Overseer needs to be more transparent, regardless of how it is used…to ensure that farmers, councils and the public have confidence in the model’s results,” he said. “We’re following up the PCE recommendations and investing in improvements. An uncertainty and sensitivity analysis is already underway.”

O’Connor said he supported Upton’s recommendation of a “full-model peer review”, saying “any peer review would include Overseer’s assumptions and mathematical models. The review would need to be undertaken by independent experts, including mathematical modellers.”

The model uses fertiliser, cattle feed data and other numbers to make its estimations - for example, it will model how much nitrogen is lost in cattle urine by looking at how much a typical animal needs to eat, how much nitrogen is in their food, and how much gets converted to milk or meat to work out how much nitrogen gets lost to the soil beyond the grass' root zone, potentially ending up in waterways.

In their critique, Wake and the others said MPI “lacked understanding” about what other countries were doing to model river pollution and other environmental effects and called for a “proper” peer review. They said their efforts to contribute to improvements had so far been rebuffed.

One of their key concerns is that the model simply adds together the effects of various biological processes, without taking into account the complicated interactions between them. They said New Zealand had good research showing what happens to fertiliser on various soil types and in certain weather, for example, but that information needed to be fed into a better underlying model to get more accurate answers out the other end.

Wake says interactions between natural systems can have a huge impact on the environmental effects from adding just a little more of an input such as fertiliser -- for example, a system may hit a “tipping point" and cause a serious pollution problem from quite a small change in farming practices. That's something you wouldn't expect if you simply added the various effects together, he said. Tweaking the model would not be enough.

“There was $5 million central government funding in 2018 (for Overseer) and they still have not set up a proper peer review. Why not?,” the four critics said in their statement. “The current version of Overseer uses the wrong paradigm and is extremely simplistic.  It is constructed by combining uncoordinated experimental data in a naïve manner and does not address the key interactions that are driving how the system of nitrogen contamination evolves over time...This style of modelling of a complex system is mathematically unsound.”

"New Zealand is at serious risk of losing its credibility in agricultural and environmental management with the public and from our international colleagues….the science does not stand up to peer scrutiny," they said.

Upton's report raised other concerns. Since most of the soil information and rainfall data that was used to build the model came from two regions, Waikato and Southland, estimates for farms with different soil or rainfall profiles might be out by more than 50 percent, according to his report. “In Canterbury, Overseer estimates of nitrogen leaching from dairy farms on light and poorly-drained soils could be anywhere from nearly 40 percent below to 60 percent above the actual leaching rate,” the report said.

The accuracy is important for improving waterways but also for the comfort of farmers – who may face having their compliance with regional water rules judged based on what Overseer says their effects are. On the other hand, cruder measures like limiting stocking rates have been dismissed by farmers and others as being too basic, and not giving farmers any options for finding different ways to shrink their impact. 

Upton’s report said it is hard for farmers to have confidence that the estimates councils were using to make compliance decisions were right.

And, at any time, a software update could change Overseer’s algorithms and put a farm in breach of pollution limits - even if the farmer hadn’t changed anything they were doing.

Despite its shortcomings, six regional councils use Overseer to regulate nitrogen run-off: Bay of Plenty, Canterbury, Hawke's Bay, Horizons, Otago and Waikato. These councils can require farms to use the tool to estimate nitrogen losses and, sometimes, demonstrate how they plan to reduce impacts far in the future. Another three councils, including Southland, require farms to provide nitrogen leaching reports using Overseer but don’t use the tool to impose any limits on run-off.

Upton's report said the councils with the most pressing water quality issues had had little choice but to use the tool. “The severity of the nitrogen problem they face has led them to Overseer. Council staff acknowledge the tool is far from perfect, but blunter tools would be required if Overseer was not available,” it said.

If Overseer was going to be used, it needed to be transformed into an official, open-source tool, the report said.

Yet Ravensdown and Ballance have helped develop and fund the tool, spending close to a million on it in 2017 (roughly the same amount as was spent by the Ministry for Primaries Industries). None of the owners has extracted any profits, other than the goodwill it has generated by building relationships with farmers, said last year's report.

Asked about the latest progress, Overseer's chief executive Caroline Read told Newsroom the company was pleased with Upton's recommendations "as it provides us with some clarity around expectations for use of Overseer in regulatory processes to support our ongoing discussions with regulators."

She listed the work the company was doing already: "We are currently progressing an uncertainty analysis that we will be discussing at an international conference later in the year, we are finalising a calibration exercise that will be submitted for peer review publication later in the year to provide confidence in the nitrogen model for pasture systems, and we are currently securing appropriate expertise to support a calibration exercise for arable systems."

Research had been underway for several years to get data to allow that calibration, said Read. "We are also scoping other activities to support addressing recommendations around transparency and guidance, even ahead of any government response."

"In terms of criticisms of Overseer by mathematicians, we see it as really important to have science debate around the model and we encourage scientists to work with us to ensure that Overseer can incorporate thinking across many disciplines," said Read. She noted resources were needed to allow a multi-disciplinary response.

Help us create a sustainable future for independent local journalism

As New Zealand moves from crisis to recovery mode the need to support local industry has been brought into sharp relief.

As our journalists work to ask the hard questions about our recovery, we also look to you, our readers for support. Reader donations are critical to what we do. If you can help us, please click the button to ensure we can continue to provide quality independent journalism you can trust.

With thanks to our partners