In The Pulse of Democracy, his 1940 defence of the nascent polling industry, George Gallup insisted that polls were important for democracy because politicians needed to understand public opinion, even if they chose not to follow it. The primary purpose of the polls was not to predict an election outcome; it was to “test public sentiment on single issues… when public interest is at its height.”
Testing “public sentiment” in Australia has almost as long a history as in the United States; in September, it will be eighty years since the first Gallup poll, run by Roy Morgan, started gathering Australians’ opinions on a range of issues. Since 1971, when the Australian Sales Research Bureau (for the Sydney Morning Herald and the Age) and the Australian Nationwide Opinion Poll (for the Australian) broke the Morgan monopoly, Australian newspapers have commissioned various polling companies to test opinion when public interest in an issue has been “at its height” but also when public interest has barely been engaged.
What is new this year is the arrival of the Resolve Political Monitor. Until now, issue-based polling has been dominated by the Essential Report, whose findings appear fortnightly in the Guardian Australia. In April, to some fanfare, the company that produces the Monitor, Resolve Strategic, run by Jim Reed, began polling monthly for the Sydney Morning Herald and the Age. Newspoll remains dominant in what it, and most of the political class, sees as the main game — calculating two-party-preferred voting intentions.
Until April, neither the Herald nor the Age had commissioned regular polling since the May 2019 election, when both mastheads — and the Australian Financial Review — predicted a Labor win. All three relied on Ipsos, which estimated that Labor led 51–49 on the two-party-preferred vote, an error slightly less egregious than that recorded by other pollsters, but an error nonetheless.
Resolve, which assures potential clients that it does “the best work,” having been set up “to introduce the advanced research techniques practised by political parties to the communications industry,” wasn’t around for that debacle. Reed insists that survey research questions need to be “understood,” response categories need to be “appropriate,” and there could be “no proxy for proper testing.”
For its latest Monitor, conducted 8–12 June, Resolve was commissioned to “test public sentiment” on Australia’s quarantine capacity, carbon emissions and relations with China, and the uptake of the Covid vaccines. To work one’s way through the Herald’s coverage of the results is to find the odd question without tables or graphs, the odd graph that doesn’t report the response distribution for the sample as a whole, and accounts of the questions that differ between print and online versions if you have sufficient ingenuity to find the two. It is also to become increasingly aware of the poll’s weaknesses (including its polling on individual behaviour around the vaccine, to which we’ll return); its capacity to mislead readers; and, to the policymakers Gallup privileged, its limited utility.
Some of the weaknesses of the poll should be clear to anyone who has even a passing awareness that polls shouldn’t ask questions many respondents will be in no position to answer. Some of the weaknesses might be evident only to a reader who knows something about how questions should be asked. And some of its weaknesses can be illustrated by reference to other polls — the most recent Essential Media, but also the annual Lowy Institute Poll, whose 2021 poll, conducted 15–29 March, was published in the same week as the Monitor.
QUARANTINE
With the federal government under pressure to allow more Australian citizens back into the country and provide alternatives to the hotel quarantine provided by the states, the Monitor saw an opening: “There has been some debate in the media recently about whether Australia should increase or decrease its quarantine capacity to allow more people to enter the country, and if so how this is best handled,” it told respondents. “On this, which of the following comes closest to your own view?” The responses? “I think the number of people entering Australia should be reduced (36 per cent); I think the number of people entering Australia is about right now (19 per cent); I think we should increase hotel quarantine capacity so more people can enter Australia (7 per cent); I think we should increase purpose-built quarantine camp places so more people can enter Australia (30 per cent); Undecided (9 per cent).” For David Crowe, the Herald’s chief political correspondent, those percentages showed that “there is only minority support for increasing arrivals, even if it is done with more purpose-built facilities.”
But piling the various preferences (fewer, the same, more) and possibilities (“purpose-built quarantine camp places,” “hotel quarantine”) into a single question may not have done justice to what respondents actually wanted — or might have been enticed to consider. Those who wanted fewer arrivals might have been happy to accept the present number if more quarantine places (of either kind) had been on offer. Those who wanted “purpose-built quarantine camp places” may have been equally happy with “hotel quarantine capacity,” and vice versa, had they been allowed to say so — and the response may have changed again if “purpose-built quarantine” had not been described as “camp places.” Some may have wanted to increase the numbers entering Australia but not wanted either more “purpose-built quarantine camp places” or an increase in “hotel quarantine” places.
In the latest Essential poll, also conducted online, in this case on 16–20 June, no fewer than 65 per cent favoured “purpose-built quarantine facilities” as “Australia’s long-term approach to safely quarantining international travellers,” compared with 16 per cent who favoured “home quarantine” (a possibility the Monitor did not entertain), and 9 per cent who favoured “hotel quarantine.” While the two questions are not the same, some of the differences — the much clearer preference for “purpose-built quarantine camp places” over “hotel quarantine capacity,” and the reference to “purpose-built quarantine camp places” rather than “purpose-built quarantine facilities” — are instructive.
CARBON EMISSIONS
According to David Crowe’s lead story accompanying the first results of the June Monitor, “A majority of Australians want the federal government to cut greenhouse gas emissions to net zero by 2050 but do not want a carbon price.” Crowe based his conclusion on a question that asked respondents whether their “preferred method for Australia to reduce its carbon emissions” was by putting “a cost on emissions” (preferred by 13 per cent) or by using new technologies (61 per cent). In the print version of the story, the question is prefaced by the words “while both these methods can be used”; but “both methods” was not an option the question went on to offer.
Given the choice, respondents chose “new technologies” over the alternative — the alternative that involved a “cost.” Who would have thought? Not for nothing does the prime minister promote the idea of “technology not taxes.” Surely, he didn’t need the Monitor, or Crowe, to tell him he was “tapping into community sentiment with his vow.” Not many people would choose to pay for a lunch — assuming they might be forced to pay — when they could get one free.
If no one else is paying for your lunch, however, it doesn’t follow that you would not be prepared to pay for it yourself. The possibility that Crowe — and Resolve — had overlooked was that someone who prefers a new technology, especially when their attention is not drawn to any associated costs, may still be willing to have a “cost” put on emissions if new technologies (alone) won’t solve the problem. While the conclusion that people “do not want a carbon price” may have been correct, the reasoning behind it was invalid.
What of the timeline for any “cut”? And how far should the “cut” go? Responding to a separate question — a response that would become the premise for Crowe’s conclusion — 55 per cent of respondents supported “the federal government adopting a 2050 ‘net zero’ emissions target,” a figure revealed in the text of Crowe’s article but not in the accompanying table. The proportion of respondents either opposed to this proposal or “neutral/undecided” (45 per cent) was almost as great as the proportion in favour. In short, there was nothing like the consensus implied by either the front-page headline “Net Zero: Public Is Ready for CO2 Cuts,” or the online headline “Voters Want Australia to Set a Net Zero 2050 Emissions Target, but No Carbon Tax.”
And what did the very large proportion of those who classified themselves as “neutral/undecided” — about a third of the respondents (the report provides no precise number) — understand by words like “adopting,” “emissions targets,” and “net zero” — especially when the prime minister, no less, chooses his words around “net zero by 2050” so carefully? Respondents may have been less clear about what the question meant than the Monitor assumed they would be or the Herald imagined they were.
Perhaps respondents who were reluctant to commit to net zero by 2050 wanted the government to commit to a more modest target but one that could be achieved more quickly. “Asked whether it was more important to concentrate on meeting Australia’s 2030 commitment or to adopt a new 2050 goal [zero emissions?], 42 per cent of voters preferred to concentrate on the earlier target while 29 per cent wanted more importance [sic] on 2050,” Crowe reported; the exact question was published neither in print nor online. The reporting tells us nothing about those who were “neutral/undecided” about net zero by 2050: the 55 per cent who supported net zero may have included those who would have preferred “to concentrate on the earlier target”; but the 42 per cent, for the most part, may have been a different group. As to what, if anything, respondents were told about “the 2030 commitment” — that remains a mystery.
The fact that so many respondents (26 per cent) were “undecided” when asked to choose between “new technologies” and putting “a cost on emissions” may have reflected another problem with this question: it didn’t allow for respondents who did not want Australia to reduce its emissions or didn’t believe that it needed to.
In the Lowy Institute Poll, the majority of respondents (55 per cent) said that the government’s “main priority” in relation to “energy policy” should be “reducing carbon emissions” rather than either “reducing household bills” (32 per cent) or “reducing the risk of power blackouts” (12 per cent). On this evidence, the majority of those who wanted net zero by 2050 may have been prepared to countenance a “cost.” In addition to his reasoning being invalid, Crowe’s conclusion — and Reed’s — that the majority of Australians do not want a carbon tax may have been misleading.
CHINA
Most of the issue questions in June’s Monitor were about China. First, respondents were told that “Australia has taken a number of actions in relation to China in recent years, including those listed below. For each, please tell us whether you support or oppose the action that was taken.” The order of the list, Reed tells me, was randomised or rotated. The options were: strongly support, support, neutral/undecided, opposed, strongly oppose.
Published in descending order of support, Australia’s actions were described in these ways: “Cancelling visas of Chinese citizens suspected of being covert agents” (supported by 71 per cent, opposed by 6 per cent); “Speaking out against human rights issues involving the Uighur” (69–5); “Calling for an investigation into the source of COVID” (66–8); “Launching trade restriction cases against China via the WTO” (63–7); “Reviewing the 99-year lease of Darwin Port” (60–12); “Criticising China on its approach to Hong Kong and Taiwan” (59–10); “Banning Huawei from Australia’s 5G network” (58–9); “Criticising China on its taking over of the disputed Spratly Islands” (56–9); “Cancelling Victoria’s ‘Belt and Road’ agreement” (54–8); and “Warning of the chances of armed conflict with China” (45–19).
The first thing to say about most of these actions is that, unless they were prepared to endorse whatever Australia had done simply because Australia had done it — a point to which we will return — large numbers of respondents would have had little or no basis on which to answer. How many respondents would have heard of or known much about: covert agents and the cancelling of visas; the Uighur; the WTO, even had the acronym been spelled out; the ninety-nine-year lease of Darwin Port; China’s approach to Hong Kong and Taiwan; Huawei or Australia’s 5G network; the Spratly Islands; or Victoria’s Belt and Road agreement, let alone the cancelling of it. And how many would have heard about Mike Pezzullo (secretary of the Department of Home Affairs) or Peter Dutton (defence minister) “warning of the chances of armed conflict”? Most Australians’ knowledge of, or interest in, the finer points of China’s or Australia’s foreign policy is unlikely to be particularly extensive — something that a poll putting words into people’s mouths should not be allowed to disguise.
Just how many respondents had little or no basis on which to answer these questions we cannot say, but the proportions that ticked the box marked “neutral/undecided” provide one clue. The proportions that declined to register a judgement ranged from 24 to 26 per cent (the three actions most widely supported) to 36 to 38 per cent (the three actions least widely supported). These are high numbers; a comparison with the single-digit figure for the “undecided” on the Monitor’s quarantine question is striking.
Those self-identified as “neutral/undecided” in the Monitor no doubt included respondents who knew something of the matter at hand and were genuinely undecided about its merits; but the greater number are likely to have been respondents who hadn’t heard of the matter or given it much thought. And since many respondents will have been unwilling to admit that they knew little if anything about what was being asked, and simply indicated their support for whatever the government had done, the real number of those not in a position to answer is likely to have been much greater than the “neutral/undecided” figures suggest — very likely, over half. Reed himself concluded, on the basis of a quite different survey, that “no matter how inane and ill-conceived your question, and regardless of the inappropriateness of your response categories, a large proportion — perhaps all — survey respondents will try to give you an answer if compelled to do so.”
Had a preliminary question been asked along the lines Gallup once suggested — “Have you read or heard anything about…” — readers (politicians included) would have been much better served; even better, had the substantive question included “Do you have an opinion on this?” or, better still, “Have you thought much about this issue?” Any of these questions may have shown that support for Australia’s actions, not just opposition to them, was the preserve of minorities not majorities; and that the gap between supporters and opponents was narrower than the Monitor figures suggest. Properly pre-tested, these questions may not have been asked at all.
The second thing to say is that the Monitor’s question format lends itself to acquiescence, also known as agreement tendency or yea-saying. Having been told that these were all actions that “Australia” had taken — “Australia” being a cue, for most respondents, likely to carry a high positive affect — and knowing little or nothing about the substance of the actions, a substantial number of respondents are likely to have gone down the list, ticking “strongly support” or “support,” one after the other. Note that in relation to the top nine actions, the range of both “strong support” (33 to 41 per cent) and total “support” (54 to 71 per cent) is quite narrow. Opposition to any of the actions fluctuates even more narrowly (5 to 12 per cent). Both are precisely what we would expect if acquiescence loomed large and cognitive engagement was low.
On what basis would respondents, with little knowledge of these things, dissent? As Reed told the Herald ahead of the Monitor’s launch in April, “What people are thinking about [right now] is how they’re travelling themselves, in their own families and households, as we adapt to life under a global pandemic, and they’re thinking about how our leaders are performing in their response to this extraordinary challenge.”
Had respondents been told that these were the actions of the “Liberal–National Party government,” rather than the actions of “Australia,” respondents would have been given a rather different cue. We might then have expected some respondents to have supported or opposed the ten actions according to whether they were Labor or Coalition voters — more, if Labor objections to any of the Coalition’s actions had been noted; less, if Labor’s support for any of the Coalition’s actions had been noted. Had such a cue polarised the response, it would have narrowed the gap between the proportion that supported and the proportion that opposed the action. While the desire to avoid such a cue is understandable, more thought might have been given to the potential skew introduced by the cue that was chosen in its stead.
Having been taken through this list of Australia’s actions, respondents were then asked: “Do you think Australia should compromise on any of these points if it meant better trade and diplomatic relations with China? Please either pick ‘no’ or choose as many of the options as you like.” Most (56 per cent) of the respondents picked “no.” But “no” wasn’t just one option among many; it was the easiest option — physically, cognitively and emotionally — and in each of these senses the set of options was biased, however unwittingly, in its favour.
Only 12 to 17 per cent went to the trouble of ticking one or more of the other ten boxes, each identifying a different action on which they would be prepared to compromise — the same determined respondents, possibly, ticking more or less all of them. Again, the lack of discrimination — roughly the same low proportion willing to compromise over “criticising China on its taking over the disputed Spratly Islands” and “criticising China on its approach to Hong Kong and Taiwan,” for example — suggests low cognitive engagement. How many ticked no box at all (as many as 27 per cent, potentially, but no doubt fewer) was not reported. The full table was made available online but not in print.
“Some people have said that Australia should not antagonise China as it is a major trading partner with a large military, while others say that Australia should stick to its values, speak up or act against neighbours like China when we feel they are doing the wrong thing. Which of these views comes closest to your own?” This question on China is notable for being the only one that presented Australia’s dispute with China, and what to do about it, in terms of argument and counterargument rather than support for or opposition to a particular government response.
Whether the argument and counterargument were “balanced” is another matter. On the one hand, respondents were presented with a statement about China as “a major trading partner” (an implied risk) with “a large military” (a threat); on the other, they were given a rather longer statement about “Australia sticking to its values” (principled behaviour) against those who are “doing the wrong thing” (unprincipled behaviour). The outcome, surely, cannot have been in doubt: less than a quarter (23 per cent) thought that Australia should “think twice before antagonising” while nearly two-thirds (62 per cent) thought Australia should “stick to values & speak up,” to quote the labels on the pie-chart. On this question, as one would also expect, relatively few (15 per cent) were “undecided.”
Is this a finding worthy of an online headline of this kind — “Australians Wants Nation to ‘Stick to Its Values’ in China Dealings” — in an upmarket daily? Perhaps. But one would have to search quite a way through the annals of polling to find a majority in any country that favoured surrendering to an immoral bully — let alone doing so in the absence of a serious threat of war. Even if respondents imagined that the chances of war were substantial — and the public may be wont to exaggerate such threats — most respondents (75 per cent according to the latest Lowy poll) would have drawn comfort from their belief that “the United States would come to Australia’s defence if Australia were under threat.” Before crafting the question, it might have been a good idea to have considered the pattern of response it was likely to generate, and what — if anything — the pattern would mean.
Having asked about past actions, the Monitor moved on to test the water about future actions. “The Chinese government has said that it may block or place tariffs on other Australian imports in the future. If such trade sanctions were to occur, would you support or oppose the following potential courses of action?” In descending order of support, Australia’s “potential courses of action” were described as: “Focus on finding new export markets outside China” (supported by 79 per cent, opposed by 4 per cent); “Continue to seek a quiet diplomatic solution with China” (63–8); “Take each case to the WTO to try and reverse China’s actions” (56–7); “Restrict or place tariffs on the import of Chinese goods in retaliation” (53–12); “Add export [sic] tariffs on exports to China to compensate affected industries” (53–10); “Push for compensation from China for starting the COVID pandemic” (41–21); “Boycott the 2022 Winter Olympics, to be held in China” (33–30); “Break off diplomatic relations with China, including expelling diplomats” (29–30); “Do nothing so as not to antagonise China and make the situation worse” (15–48).
Here, clearly, respondents discriminated — at least at the top (63 to 79 per cent) and the bottom of the range (15 to 29 per cent). It helped that the two suggestions that were most widely supported referred to actions that no one in Australian public life had opposed: “focus[ing] on finding new export markets outside China”; and “continu[ing] to seek a quiet diplomatic solution with China,” a question that told respondents what Australia was (ostensibly) doing already and essentially invited them to endorse it. It also helped that the two least popular suggestions were actions that no one of any consequence had proposed: “Break off diplomatic relations with China, including expelling diplomats,” and “Do nothing so as not to antagonise China and make the situation worse,” a proposal that might have been understood as rejecting all of Australia’s past actions as well as precluding the search for “new export markets,” and the pursuit of “a quiet diplomatic solution.”
Every proposal that won majority support had to do with trade. Designed to respond to a Chinese tariff wall, proposals that ventured beyond trade — demands for compensation for Covid-19, boycotting the 2022 Winter Olympics, or breaking off diplomatic relations — failed to win majority support.
Noteworthy, too, is that without the comfort of knowing what “Australia” had already done, the levels of support for various future actions were lower at both the top of the range (56 per cent) and the bottom (33 per cent) — ignoring the two most popular and the two least popular suggestions — than they were for past actions (69 per cent and 45 per cent, respectively).
One of the most remarkable features of the Herald’s coverage of the poll is that the high “neutral/undecided” responses — and the exceptions — formed no part of its narrative. Overall, the proportion of respondents who classified themselves as “neutral/undecided” was lower in relation to past actions (an average of 31 per cent) than in relation to future actions (34 per cent); but it was still extraordinarily high. The only question that most respondents could relate to in the list of “potential courses of action” was the one that engaged with the wholly familiar and widely accepted idea of finding new markets; here, the “neutral/undecided” dropped to 17 per cent.
“I think the prejudice,” said Reed, commenting on the results of the questions on future actions in relation to China, “is ‘if this gets resolved and China starts buying our beef and barley again, that’s excellent.’ People see value in the trade relationship and they realise there’s an issue here.” This mercantilist framing of public opinion may be correct, but it is not one that sits particularly well with what the other questions on China purport to show. Nor does it fit well with the findings of the Lowy poll, where “Chinese investment in Australia” was also seen as a negative — and a big one (by 79 per cent) — as were “China’s military activities in our region” (93 per cent). Both were seen as “negative” by substantially largely proportions than in 2016, the last time the Lowy poll checked. And if there were still any doubt, one could look at how China’s favourability ratings have tanked across much of the First World.
DOING THINGS BETTER
All of these questions in the Monitor — the one on quarantine, at a time when the government was pondering whether to move beyond hotels; certainly, the ones on emissions, built around the prime minister’s slogan; but also, those on China — could have been written in the prime minister’s office. The fact that they weren’t tells us that those involved in constructing the poll held strong views of their own; the Herald’s Peter Hartcher, in particular, has just written a book on China. On seeing the results of all the questions, Hartcher wrote of his hope that they would “encourage the federal government in standing against Beijing’s list of 14 demands, and Labor to continue to stand with the government.”
Perhaps that was the point of the polling: to show that public opinion backed the prime minister. In this sense, polling that found majority support for “cancelling visas of Chinese citizens suspected of being covert agents,” for “speaking out against human rights issues involving the Uighur,” for the very public calling-out of China on Covid (though the question wasn’t exactly phrased this way), and so on, while at the same time finding majority support for what the poll, without a hint of irony, described as “continu[ing] to seek a quiet diplomatic solution,” could hardly have been bettered.
If providing cover for government policy wasn’t the point — if the Age and Herald would shudder to think of their polling as a form of propaganda — then the two papers need to reconsider how polls should be done. Crafting questions on matters that are keenly contested — questions that are worth asking in an appropriate manner — means having to take account of more than one view.
An important limitation of the Gallup model, which conceives of polling on an issue as a kind of referendum on that issue — a “sampling referendum” Gallup called it — is that referendums typically involve a single proposition with voters limited to either supporting or opposing it. A question in the Lowy poll, which didn’t follow the referendum model, found that while the majority (56 per cent) supported the proposition that “China is more to blame for the tensions in the Australia-China relationship,” and hardly anyone (4 per cent) agreed that “Australia is more to blame,” more than a third (38 per cent) supported the proposition that “they are equally to blame.” No doubt, had the Monitor asked this question, it would have found something similar.
One way of having polling that acknowledges alternative ways of framing issues is to involve those who hold alternative perspectives in the process of constructing the questions. In the case of China, what the Herald might have done was to have Hartcher sit down with someone like Geoff Raby, whose views on Australia’s relations with China are rather different. The fact that Hartcher and Raby barely reference each other in their respective books might make an exchange between them all the more refreshing. Raby is not necessarily better on China than Hartcher; that question isn’t relevant when it comes to constructing a poll. But Raby is at least as well credentialled. On China, as there are on Covid or on climate policy, there are any number of people who could have helped.
The job of the pollster is to work out how to ask the questions, to advise on the use of argument and counterargument as against approve/oppose, to think about the various assumptions the question makes about respondents or the demands it puts upon them, to pre-test or to build in filters, to contemplate the use of split samples, to organise the sequencing/rotation of questions, and so on.
According to its website, Resolve sees its work as “Always quality,” “Always insightful,” “Always practical” — this last, a dig at “academics, researching for the sake of knowledge or debating theory.” But it’s not just academics who might beg to differ. It would be difficult for anyone concerned with standards in the industry to say that the Monitor’s questions on quarantine, climate or China exemplified “quality” or “insight”; and if they fell well short on either, that the results offered something that was particularly “practical.”
The questions in the Monitor on the Covid vaccine — asking respondents whether they had been vaccinated, whether they were “likely” or “unlikely” to get vaccinated, and so on; and seeking reasons why they may have hesitated — were somewhat better. But these were questions of a different order. First, because asking respondents to report on their own actions, past or planned, is quite different from asking them about issues of public policy, even if people are not particularly good at predicting their own behaviour, especially in unusual circumstances, and response categories can still make a big difference. Second, because following up with a list of fourteen possible reasons, which allows for multiple responses, seems to cover almost all the possibilities, even if respondents are not necessarily very good at explaining their own motivations for doing — or not doing — things; a notable absence from the list is “don’t know.”
The Australian Polling Council, set up in the wake of the 2019 debacle to lift standards in the polling industry, and pollsters’ accountability, is not a body that Resolve wants to join, Reed tells me; apart from not wanting to join a club whose members include some he sees as beyond the pale, he doesn’t want to have to divulge “trade secrets.” If Resolve were to join the APC it might be obliged to lift its standards — if the APC can be persuaded to match the demands of the British Polling Council — and to raise its level of transparency not just by making available its computer tables with the questions, answers and question order but also by revealing some of its other “trade secrets.” The Herald and the Age, endlessly concerned with holding others to account, should insist on nothing less.
“We can’t put all of it in the data centre because of the scale of the results,” Tory Maguire, the Herald’s national editor explained, when announcing the launch of the Monitor, “but we will report on as much of it as our readers find interesting.” As it happens, none of the answers to any of the issue questions (or vaccine questions) polled in the last three months have found their way into “the permanent data centre.” How the Herald judges what its readers find “interesting,” only it would know. But as anyone may judge, there is nothing about “the scale of the results” that would prevent the “data centre” functioning as a repository for every one of the Monitor’s questions and the top-line results. What had sounded promising when it was announced pales by comparison with the repository established by the Lowy Institute. It’s not just the Monitor that needs to reconsider what it does; fifty years after breaking the Gallup monopoly in Australia, and showing that there are other ways of conducting polls, it’s also the Age and Herald that need to reconsider. •
