Podcast: The Detail

Polls, rogue polls, and statistics

How do pollsters go about getting a real taste of what the electorate is thinking? We speak to two experts on the methodology behind the numbers. 

They cost tens of thousands of dollars and take tens of thousands of phone calls to get the right people to answer their questions - but can we trust political polls?

This week opinion polls have come under intense scrutiny after the Newshub Reid Research survey put National at a disastrous 25 percent compared with Labour on a historic high of 60 percent.

National campaign manager Gerry Brownlee called it a rogue result and questioned the methodology being used.

The poll has a margin of error of 3.1 percent, and was done between July 16-24 with 1000 people surveyed - the majority by phone and the remainder via an internet panel.

Was it rogue? Maybe, says statistician and Auckland University Professor Thomas Lumley.

"Either this one or the previous one."

He explains to The Detail's Sharon Brettkelly why Brownlee is mixed up when he argues that statistically one in 20 polls are out of kilter, making this one a rogue.

Long-time political pollster Murray Campbell from Baseline Consultancy explains to The Detail how surveys work and why they are so costly.

"You'd be blown away," he says, by the tens of thousands of phone numbers - landlines and mobiles - that are called before the surveyors can "finish off their one thousandth interview".

"Polling companies vary in terms of how many callbacks they make but say I dialled my father, he's on a landline, and say he didn't answer the phone from the first, second, third (try), I would try and contact him at least five times before I substituted that number for another number. And I would make those calls over different times and over different days."

Campbell says a crucial part of the "secret sauce" of political polling is "how many callbacks you make before you substitute for a new number".

He explains how pollsters have a spreadsheet of more than 400 different segments or profiles that need to be ticked off, and why pollsters have to work so hard to reach the "19-year-old man living on the West Coast".

"If someone says they can do a poll in one or two nights, they might get a thousand people but is it the thousand people who are representative of the massive cross section of New Zealanders?"

In today’s podcast we get a real insight into how the numbers are done.

Want more from The Detail? Find past episodes here.

Can you help our journalists uncover the facts?

Newsroom is committed to giving our journalists the time they need to uncover, investigate, and fact-check tough stories. Reader donations are critical to buying our team the time they need to produce high-quality independent journalism.

If you can help us, please donate today.

Comments

Newsroom does not allow comments directly on this website. We invite all readers who wish to discuss a story or leave a comment to visit us on Twitter or Facebook. We also welcome your news tips and feedback via email: contact@newsroom.co.nz. Thank you.

With thanks to our partners