“ONE in ten Australians believes the world will end this year in line with ancient Mayan prophecies,” began a story by journalist Barney Zwartz in the Age last week. It seems like a startling finding. But it’s hard to believe that 10 per cent of Australians are familiar with ancient Mayan prophecies, let alone go along with their prediction that the world will end in the next few months. As research conducted in 2011 by Ewing and Associates found, when a statistic looks ridiculous, 83.2 per cent of the time it is.
Given that I had to talk about sample surveys to a group of postgraduate students later in the week, the Age article looked like the basis for a lively case study. Google quickly turned up the newspaper’s source, a press release from Ipsos (an “independent market research company controlled and managed by research professionals”) titled “One in Seven (14%) Global Citizens Believe End of the World Is Coming in Their Lifetime.”
The exact wording of the key question asked by Ipsos makes things both a little clearer and a little murkier. Respondents were asked whether they agreed with the statement that “the Mayan calendar, which some say ‘ends’ in 2012, marks the end of the world.” That question is so ambiguous that it’s impossible to know what a yes answer really means. It might mean that a respondent believes that the world will end this year, or it might simply mean that he or she believes that the Mayans believed that it would.
My interest, though, was well and truly piqued. I emailed Ipsos asking for more information about their survey. They responded quickly with a short report that did indeed add more information. The research had been undertaken as part of the “Global @dvisor Wave 31 (G@31)” series between 6 and 20 March this year. Usually these surveys are “conducted monthly in twenty-one countries around the world via the Ipsos Online Panel System,” but this month “fielding took place in twenty-six countries.” I had never seen “fielding” used in a non-cricket context – I was hooked.
Ipsos asked respondents the extent to which they agreed or disagreed with the following statements:
• I believe the world will come to an end during my lifetime
• The Mayan calendar, which some say “ends” in 2012, marks the end of the world
• I have been experiencing anxiety or fear because the world is going to end in 2012
• I believe the world will come to an end during the calendar year of 2012.
Respondents could strongly agree, somewhat agree, somewhat disagree or strongly disagree.
Thirteen per cent of Australians agreed that the world will end during their lifetime, 3 per cent “strongly” and 11 per cent only “somewhat” (the discrepancy is presumably due to rounding). It seems unlikely that while one in ten Australians think the world will end this year, only 13 per cent think it will end in their lifetime, but we’ll get to that.
The level of support for the Mayans is somewhat undermined by the fact that only 1 per cent of Australians “strongly agreed” with the statement while 8 per cent “somewhat agreed.” There don’t seem to be too many rusted-on followers of Mayan prophecies. And only 1 per cent of Australians “strongly agreed” that they had been experiencing anxiety or fear because the world was going to end in 2012.
All of this was just a build up really, because the final question from Ipsos asked respondents to agree or disagree with a very clear statement, “I believe the world will come to an end during the calendar year of 2012.” You’d think there’d be a high probability that Ipsos would include the results of that question in their report, but they didn’t. Perhaps the results undermined their finding about people’s attitudes to the Mayan prophecies.
When I emailed Keren Gottfried, the research manager at Ipsos Global Public Affairs, to ask why the responses to that statement were not reported, this was her answer:
Yes, we also asked people to assess this item but the results were very similar and we wanted to keep the factum tight in terms of length so we did not include it. Since we did not release the data I can’t give you the tables for that part of the question, but I can tell you that agreement with the statement “I believe the world will come to an end during the calendar year of 2012” was three points lower, both globally and in Australia. Three points is within the margin of error so the results are essentially the same.
That last sentence is a doozy. What the result does show is that at least a third of the Australian respondents who agreed with the Mayan question misunderstood what Ipsos was getting at. There’s no other explanation for the fact that 9 per cent agreed with the Mayan statement but only 6 per cent agreed that the world would end in 2012. Given that some Australians who think the world will end in 2012 do so on non-Mayan grounds, the proportion made up of those who misunderstood is very probably higher. If we remember that only 1 per cent of Australian respondents strongly agreed with the Mayan statement then it turns out that less than one in a hundred Australians “strongly” believe that the world will end in 2012 in line with Mayan prophecies (the rest of the believers “somewhat” believe). This is not a news story.
If Ipsos really wanted to find out how many people believe that the “supposed end of the Mayan calendar marked the end of the world” they could have first asked if people thought the world would end in 2012 and then asked those who agreed whether their belief was based on Mayan prophecies. They would not have had a lot of Australians agree because not a lot of Australians do agree.
One can only speculate on Ipsos’s motives in all of this. The results from the three unambiguous questions posed in this survey provide (or in the case of the missing question, would have provided) interesting information. The fact that large proportions of the global population believe the world will end in their lifetime is both saddening and worth discussing.
The survey responses in this case are not going to have any real impact, of course. But if Ipsos treat data like this when there’s not much at stake, what does it say about their findings in relation to more important issues? Every time data is collected and analysed in a careless way it can only serve to undermine people’s confidence in such surveys and their uses. This is clearly bad for the industry that collects and analyses sample survey data, and it is ultimately bad for companies like Ipsos. It is also bad for the standard of public debate. Statistics from sample surveys play an important role in helping to shape and inform such debate. At their best they can provide a solid basis on which discussion can take place. If people lose all confidence in data presentation and analysis our public life will be much the poorer.
For the Age the lesson is clear. There are many organisations and individuals peddling statistics in the public domain – Lord knows I am one. Anybody using these statistics needs to be careful; the numbers never speak for themselves. Interpretations need to be based on an understanding of how the data was collected and, in particular, how the questions were framed. This is not always easy to ascertain because private firms are often coy about these issues. To Ipsos’s credit this was not the case here. Although they refused to release the results from one of their questions, they did provide information about how the data was collected and, importantly, the questions that were asked.
Fact-checking by the newspaper would simply have involved a careful reading of the Ipsos press release, which provides enough clues to suggest that the data has some problems. If the Age had asked for the slightly longer report and spent the five minutes or so needed to read it then it couldn’t possibly have published that first sentence in an article featuring prominently on page three. Surely someone, at some stage, would have asked, “Could that possibly be right?” •
