Concerns raised over census IT system
The boss of New Zealand’s census is confident Stats NZ's IT system is up to the job. But internal emails from October paint a shambolic picture of the $121 million project, with contractors pulled in to fix bugs in a key system, delays to crucial testing and blown budgets for individual projects. David Williams reports.
On October 20, a day after Winston Peters announced New Zealand First would form a government with Labour, the head of New Zealand’s 2018 census, Denise McGregor, emailed her staff the management team’s newly drafted list of cultural principles.
“We are one team, not a series of projects,” was the first of 11 principles inspired by a speech from legendary rugby sevens coach Gordon Tietjens. Others included “when things go wrong we collectively fix without blame”, “share the load” and “escalate early”. In a sign of the sweat being expended on Stats NZ’s momentous project, the bigwigs vowed to lead by example by having time off to “ensure we are fit for the job” and “getting out” at lunchtime.
Optimism turned to grim realism, however, in the “IT update” section of McGregor’s email, which focused on the crucial Salesforce software system, used to manage field staff work. “The Salesforce issue continues to provide challenges for resolution and is impacting the delivery of the overall programme,” McGregor’s email said. “There are a number of priority one work packages blocked from completion, and a backlog of testing is starting to build.”
Salesforce testing had improved, she said. The full file of 2.1 million records – all street addresses in New Zealand – had loaded and “a number” of errors were being investigated.
Lurching from success to doom
Two hours later, McGregor wrote an unvarnished assessment to Stats NZ deputy chief executive Teresa Dickinson. The “good news”, she said via email, was that 2 million records loaded. However, 103,000 records failed and performance issues meant 600,000 had to be re-run.
“We lurch back and forth from success to doom all the time and Salesforce is absolutely central to the 2018 census design,” McGregor wrote. “We have to get some stability.”
Stability has been hard to come by for Stats NZ. In fact, nature seems to be stalking the census. The 2011 run was postponed by the Canterbury quakes and preparations for 2018 were jolted by last year’s Kaikoura quake, which severely damaged its Wellington headquarters and left staff without any IT systems, initially, including email. Then, when the re-scheduled field testing took place in April, a state of emergency was declared in flood-hit Whanganui.
Backroom IT systems have never been more crucial to a census. While people have been able to submit online forms to the census since 2006, next year’s iteration – held on March 6 – is “digital-first”. Eighty percent of people will get online codes as opposed to paper forms. Four thousand fewer field staff are being taken on, as there will be fewer forms to collect and process. In the 2013 census, 34 percent of people completed their forms online. Next year’s goal is 70 percent online.
It’s not only a massive undertaking, it’s an expensive one. The multi-year budget is $119 million, including capital expenditure and two census-related projects. It’s a little-known fact that Stats NZ went to the Government’s economic growth and infrastructure committee in July, asking for $5 million extra to “manage the additional risks” after the Kaikoura quake. It was granted an extra $2 million.
Newsroom first heard of possible problems with the census IT system in early November. After writing our Official Information Act request, Stats NZ suggested it was too broad and would likely be rejected, offering us an face-to-face briefing instead.
(Stats NZ deputy chief executive Dickinson wrote to McGregor on November 10: “I saw your email re the OIA. I had heard from [redacted] that, rather than respond directly, we were going to offer the journalist an interview. Is that be (sic) planned?”)
Newsroom politely declined the interview and refined our request, which led to 302 pages of documents and emails being released. What emerged were huge concerns about Salesforce’s problems, rising costs and the delays to important projects and testing.
IT programme manager Joe Meshtrovich wrote to McGregor on October 10, saying six work packages had “slipped” due to the Salesforce performance issue. Testing of recent changes would start the following week, he said, “to see if it meets census’ needs”.
A test on October 15 successfully loaded the operational file for the first time, but had only 611,000 “successful inserts” and 1.5 million errors. Days later that reversed, to 2.1 million files loaded, with 600,000 errors.
Meshtrovich told McGregor and Stats NZ’s chief digital officer Chris Buxton on October 16: “There is a high level of uncertainty as to when will the performance issues with Salesforce be resolved”. Meshtrovich asked Buxton to explore what else could be done to solve the Salesforce problems, including “bringing vendors and experts” to work on them.
On October 20, in between writing emails to Dickinson and her “all staff” update, McGregor shared a draft report with a colleague, which said the census work was “closer to amber/red” because of Salesforce “challenges”.
(Treasury’s definition of amber/red is: “Successful delivery of the project requires urgent action to address major risks or issue in a number of key areas.”)
Recent testing showed improvement, she said, “but a number of work packages have dependencies on this element and this is increasing uncertainty for projects”. She added that Salesforce as a “central and key place in the 2018 census design”.
Earlier in the day, census programme office manager Damien Smith told McGregor the Salesforce development and performance issues were a major concern.
“This system underpins all aspects of the programme and at present is causing the programme and project concern, fixes that have been put in place so far are yet to alleviate the performance issues that face the programme.” He adds: “If not remedied shortly, there will be impact to our end-to-end testing and system accreditations.”
Budgets, Smith said, were tight, with the greatest uncertainties over development work by Stats NZ’s digital business services (DBS) team and “vendor” costs – which are “starting to impact significantly” – as well as print/mail and scanning.
(An October 3 “exception report” put the DBS overspent at more than $500,000, with printing about $60,000. The report didn’t offer alternatives to the decision because “we have no choice”.)
The most scathing assessment of the Salesforce response came from Stats NZ service portfolio manager Mike Forrester. In an email to Smith – which Smith forwarded to McGregor, tagging it “for us only” – Forrester worried that user acceptance testing, end-to-end testing and performance testing would have to be consolidated into a month. “This will sacrifice the quality of our systems and run a high risk of missing production bugs.”
He said it had been a “fight for a long time” for recognition that there was a “large issue” with Salesforce. “We should have had a dedicated team of Salesforce experienced developers on this three-to-six months ago.”
Dropped the ball?
On December 8, Newsroom sat down with McGregor in Stats NZ’s central Christchurch headquarters. We put to her that Forrester’s email suggested Stats NZ had dropped the ball.
“That may be one person’s opinion but we did move as quickly as we could and we brought a lot of people on. And there is a point at which you need to consider whether the best option is to look at the IT system within your own infrastructure rather than necessarily bringing in people who have a good knowledge of the tool but don’t understand the infrastructure that it’s sitting in. I’m confident we’ve got the right people in now.”
What about three to six months ago? “It’s been a process of growing confidence,” McGregor says.
(Forrester’s October 30 email said Stats NZ didn’t have enough in-house expertise to solve the Salesforce issues “in a timely manner”.)
Last week, Salesforce loaded 2.1 million records “with no errors”, Stats NZ confirmed. Salesforce’s 25 performance targets have been met, McGregor says, and the 35 “stories and bug fixes” are done. McGregor says all priority one development work will be finished by the end of the year. She adds: “A lot’s happened since October.”
McGregor says in a perfect world it would not be testing its systems so late. It now has a professional testing firm in, which is planning its integration and end-to-end testing. “We are confident that our public-facing systems have been end-to-end tested and integration tested.”
(This is confusing, as our question was about Salesforce, which is an internal system.)
The huge concerns expressed internally at Stats NZ, particularly over Salesforce, were not reflected in incoming Statistics Minister James Shaw’s official briefing. The briefing said it might seek additional funding for the census “to manage the newly identified risks”. A separate, more detailed briefing, released to Newsroom in the OIA response, didn’t expand on this. Rather it mentioned Treasury listed the project as “amber” – meaning “significant issues already exist, requiring management attention”.
The latest report on Treasury’s website, from April – before Stats NZ’s $5 million request from the economic growth and infrastructure committee – said if further problems with “systems availability and stability” arose, costs could rise because of a “more manual workforce model”. It said a contingency of 10 percent, as opposed to the current 5 percent, “may be prudent”.
McGregor says overall census costs are within budget. “We know that some of those projects are overspent, because we’ve had to bring extra people in, but there’ll be unders in other places.”
In a written response sent by communications and marketing manager Richard Stokes last Friday, Stats NZ said an independent quantitative assessment concluded more contingency funding might be needed. “At this point, however, we are confident that we can deliver the 2018 census within our budget appropriation.”
NZ learns from other countries successes, mistakes
McGregor remains bullish about the census. She says it’s based on a model from Canada, which ran their “best census ever” earlier this year – “and nobody remembers that”.
“It ran very smoothly and they got very high coverage and very high internet – very high self-response.”
(The Ottawa Citizen reported earlier this month that Statistics Canada’s recent census “seriously underestimated” the numbers of Inuit in that state. Interestingly, Stats NZ’s post-census survey in 2014 estimated there was an undercount for Maori, of 6.1 percent, and Pacific People, of 4.8 percent.)
New Zealand gets regular advice from other countries. It’s part of an international census forum and McGregor has a monthly meeting with her Australian counterpart. McGregor is confident New Zealand can avoid the problems encountered by the Australian Bureau of Statistics with its census last year, in which denial of service attacks kept its online form from being accessed for more than two days.
New Zealand’s form has multi-layered security, she says, including double encryption and a web application firewall. The system will be able to cope with 100,000 concurrent users in “peak hour”, as well as 500,000 address look-ups.
“In terms of the difference, our form is purpose-built for 2018,” McGregor says. “It was designed with an eye to security, performance and privacy. It’s been designed from scratch. The Australian form was a continuation of their 2011 form.”
McGregor says her staff have worked hard after the Kaikoura quake and it has taken its toll. “I think we’re all really tired.”
What has it taken to get to this point, considering the concerns raised in the internal emails?
“It’s taken extra money, it has taken a lot of work by people right across the organisation and our contractors, so people have been working very hard and there’s been some weekend work. We’re looking to rest our people as much as we can ready for the operational period.”
“Most” of her staff will get a break over Christmas, she says. “We’re consciously looking at that.”
Do you know more about this story? Email: firstname.lastname@example.org