Crampton: Hard spirits
After attending an "indicators of wellbeing selection event," Eric Crampton does not envy the task ahead of Statistics New Zealand.
It was during the discussions of measuring spiritual capital that the ghost of Sir John James Cowperthwaite hovered near. The shade whispered in my ear, “When I was Financial Secretary of Hong Kong, I refused to collect economic statistics for London. Why? For fear that I might be forced to do something about them.”
I wish more people at last week’s Indicators Aotearoa New Zealand indicator selection event had been able to hear him.
Statistics New Zealand’s Indicators Aotearoa project aims to build a comprehensive suite of (you guessed it) indicators measuring wellbeing. Where SNZ’s 2014 Progress Indicators Tool, now discontinued, included 16 measures ranging from adult educational attainment to distribution of selected native species, the new framework aims to include about 100.
The project is linked to Treasury’s Living Standards Framework and to the Government’s desire for more measures of wellbeing. Treasury has been developing its own suite of wellbeing indicators as part of the Living Standards Framework; whose indicators will reign supreme remains a bit up in the air. If Treasury’s framework and indicators wind up satisfying the Government’s wellbeing needs, then Statistics New Zealand’s indicators project – or at least those indicators that do not make it into Treasury’s framework – might yet follow its predecessor into the bin of discontinued data series.
So the bureaucratic stakes are high, as far as these things go.
Statistics New Zealand took a reasonable approach in trying to figure out which measures to include: they asked New Zealanders what wellbeing means to them in a series of workshops around the country, along with an online campaign. Then data experts tried to figure out which of those might possibly be meaningfully quantified among those that are not already measured.
That led to a long list of indicators to be pared down at last week’s indicator selection event, where I was visited by the ghost of Sir Cowperthwaite.
Inputs, outcomes and A3 posters
There’s a management truism that what gets measured gets managed, and what doesn’t get measured doesn’t get managed. Choosing the right measures then matters, especially if these kinds of indicators get targeted by a Government desperate to be seen to be improving our wellbeing.
The big headline measures should reflect final outcomes (like healthy life expectancy) rather than intermediate outcomes (like cancer rates) or inputs (like the proportion of people using sunscreen appropriately).
If there are lots of intermediate outcomes that are all captured by the final outcome, promoting intermediate outcomes into being final outcomes is a very bad idea. It invites paying more attention to those selected measures rather than to other ones that Government might more usefully target. In the example above, if an intervention targeting diabetes did more to improve healthy life expectancy than an intervention targeting cancer, the latter might nevertheless be preferred if cancer is a headline statistic and diabetes is not.
Turning inputs into headline statistics is at least as bad, for similar reasons. Inputs might be detrimental to some measures of wellbeing, but useful for others. An index of health-related behaviours, counting exercise, diet, and various consumption choices would be a poor one to include in headline statistics on health. To the extent that those behaviours improve health, they are already captured in the healthy life expectancy statistic. But including them as a headline measure risks targeting them for improvement in ways that may hurt other measures of wellbeing.
And, finally, the outcome measures chosen should be things that the Government might plausibly have any business doing something about. We might wish to be careful about what gets measured lest we be managed into improving our wellbeing in ways we might not welcome.
To take an obvious example, I have yet to see a survey of wellbeing or happiness that, if it included satisfaction with one’s sex life as a measure, failed to find that it mattered considerably. If any participant in Statistics New Zealand’s workshops had been honest and reported that sexual fulfilment mattered greatly in their wellbeing, their input did not make it into the workshop. Really, if the number proved to be a bit softer than we might have wanted, what on earth should policy seek to do about it? All options seem atrocious.
Unfortunately, the process of outcome selection at the Statistics New Zealand event was less fruitful than it should have been.
About 150 people attended at the Michael Fowler Centre.
Two hours of introductory remarks included a welcome from the Government statistician, a hand-clapping game, an invitation to check our privilege with the help of a bingo-sheet of privilege indicators like not being red-haired, and an extensive discussion from a PhD candidate who warned (among other things) about consulting with Ngāi Tahu on issues Māori because they are too corporate.
The two hours of introductory remarks did not include any discussion of the difference between final outcomes, intermediate outcomes, and inputs. Nor did it point out that current measures of inputs and intermediate outcomes would continue to be collected regardless of whether they became headline statistics.
After being encouraged to reflect on my privileges in being able-bodied, not hard of hearing, having a loud speaking voice, and not being red-headed, I joined a group of about 50 people who all had to stand for an hour around an A3 poster to discuss the proposed health indicators – before moving on to stand and talk around other A3 posters. Those unable to stand for extended periods were denied effective participation, as were those even of normal hearing if they were not very close to some of the very soft-spoken participants. Microphones and chairs may not have gone amiss, but I am probably in too privileged a position to have standing to comment on that. Our facilitator did his best but had a difficult task.
Because nobody had explained the difference between inputs and outcomes, or that failure to select particular measures did not mean they would cease to be measured, conversations had to keep coming back to those basic points. Every worthy-sounding input measure had its proponent for inclusion in the mix.
And many of the indicators seemed, well, problematic. Some outcomes were listed as “contributing unambiguously to progress” when they seemed manifestly contestable. For example, the response to the Initiative’s report last year lauding the merits of diversity and immigration suggested not everyone agrees diversity is a good thing. But appreciation of diversity was included as a potential outcome variable deemed to contribute unambiguously to progress. It seems a political argument to win rather than an outcome variable to technocratically maximise. If it were unambiguously associated with progress, we would have received less hate mail insisting on the demerits of diversity and that New Zealand needs fewer immigrants from culturally dissimilar places.
A dangerous thing to attempt to measure
Most contentious in the health discussion was the inclusion of spiritual health as a desirable outcome. And it is there that the spirit of Sir John James Cowperthwaite visited me – as I am a very spiritual person in my own utterly non-spiritual way.
Sir John is generally credited with Hong Kong’s transformation from a desolate place at the end of the Second World War into about the richest and most economically free city in the world. He credited some of his success to his refusal to provide Whitehall with the statistics they might attempt to use in applying Atlee-style management to the Hong Kong economy. His policy of positive non-interventionism certainly outperformed the UK’s more scientific-looking approach built on a mountain of economic statistics.
His spirit warned me that spiritual health is a dangerous thing to even attempt to measure as a headline wellbeing outcome, even if it is deeply meaningful to many communities. There is the obvious problem that atheists like me would not have a clue how to begin answering a survey question on whether we are spiritually fulfilled as the area, to those who share my views, is meaningless. But there is the far worse problem that what gets measured may be managed. If a headline outcome measure of spiritual fulfilment were half of what people expected, or double(!), what on earth should the Government ever do about it? The measure invites management. And any attempt at management would be worse than the measured problem.
I do not envy Statistics New Zealand the task ahead of them. The workshop will not have been as useful as it could have been in selecting an appropriate set of indicators. Will they weigh more heavily what makes sense as outcome measures, or what participants claimed to want? Either way, they will be stuck with trying to measure the things – while being unsure whether much of the project will be superseded by Treasury’s work. All this while battling with what seems an ever-worsening problem in getting the Census out.
I wish Statistics New Zealand good spiritual health in the months ahead, as they will need it.
Dr Eric Crampton is Chief Economist with The New Zealand Initiative
Can you help our journalists uncover the facts?
Newsroom is committed to giving our journalists the time they need to uncover, investigate, and fact-check tough stories. Reader donations are critical to buying our team the time they need to produce high-quality independent journalism.
If you can help us, please donate today.