Here’s a radical notion to inject into the national policy debate: why don’t we try doing what works? It’s easier said than done, of course, even if the parties weren’t caught in their ideological and political straitjackets. But that doesn’t deter Labor’s Andrew Leigh.
The central proposition of his new book, Randomistas: How Radical Researchers Changed Our World is one you might expect an academic to get excited about. Randomised trials, he says, are the best way to establish an evidence-based approach to decision-making. But then Leigh still operates very much within the mindset of his former profession of academic economist: by the time he entered parliament in 2010, he had gained first class honours in law and arts at Sydney University and a PhD in public policy from Harvard and had then, at age thirty-two, been appointed professor of economics at the Australian National University. That makes him an unusual politician or, as Peter Dutton once put it (reflecting his own worldview), “a weird cat,” more interested in ideas and facts than ideology and factions. (Leigh hasn’t joined a Labor faction.)
In their classic form, randomised trials use a randomly selected group of people, whose reactions to a test treatment are compared with a control group that represents the status quo. In medicine, this often involves testing a new drug on one group of patients while the control group is given a placebo.
It’s hardly a new idea, but Leigh argues that it should be applied much more widely and rigorously. He starts by citing the example of scurvy. During the Seven Years War in the mid eighteenth century, 133,708 of Britain’s 185,899 sailors lost their lives to scurvy, compared to 1512 killed in action. Before the war, James Lind, a thirty-one-year-old ship’s surgeon, had tried six different treatments on twelve sailors with advanced scurvy. Surprisingly, the only one that worked was a mixture of oranges and lemons.
As is often the case with unexpected findings, his discovery was largely ignored until half a century later, when lemon juice started being issued on demand in the Royal Navy. Leigh argues that the cure was a key reason for Britain’s naval dominance. “In fact, it is possible that if the scurvy randomised trial had been conducted by another colonial superpower, Australia’s national language might be French, Dutch or Portuguese.”
Of course, we are much more scientific in our approach these days. Or are we? Leigh cites assumptions and commonly held beliefs — that after-school programs help troubled youths, for instance, or that micro-credit solves poverty — that don’t hold up when properly evaluated. “Once you raise the evidence bar, a consistent finding emerges: most ideas that sound good don’t actually work in practice.”
The health sector has made extensive use of randomised trials as part of what’s called “evidence-based medicine.” But the very fact that such a term is used suggests that we can be slow learners. Rare indeed is the kind of frankness displayed by a dean of medicine in the United States who told his first-year class on their first day, “Half of what we teach you here is wrong; unfortunately we don’t know which half.”
Examples abound of what Leigh calls “eminence-based medicine” — cases where the prestige attached to doctors results in their opinions carrying greater weight than justified by the scientific research. He cites one Australian study that identified over 150 medical practices in common use despite being unsafe or ineffective.
Randomised trials remain relatively rare in surgery, for instance, and even then the results are not necessarily acted on. One in a thousand Americans receive spine fusion surgery each year, yet randomised trials show no better results than for intensive rehabilitation. In Australia, Leigh estimates, fewer than 10 per cent of programs for Indigenous people have had any kind of evaluation, let alone been subjected to randomised trials.
The book is full of equally interesting and sometimes disturbing examples. Yet, while Leigh argues for the much wider use of randomised trials, it can be surprising, at least to the outsider, to learn how extensive their reach already is.
The Flybuys loyalty program run by Coles has an inbuilt randomised trial: one in every one hundred customers doesn’t receive any promotional material. This enables the company to test the success of its promotions against a control group. (Judging by the amount of material it pumps out, they must reap very good returns.)
After noticing that its toolbar generated more attention if it was blue, Google split users into forty groups of equal size to establish which shade generated the highest click-through rates. It estimated that applying the result added $200 million to its bottom line.
A Californian winery printed three different versions of a cellar-door price list for a cabernet sauvignon that usually sold for $10, quoting prices of $10, $20 and $40. It then randomly chose one of the three price lists each day for several weeks and found that the wine sold nearly 50 per cent more when priced at $20 than at $10 — a concrete example of the arbitrariness of wine pricing that many have long suspected. Here’s another suspicion confirmed: randomly varying the prices in mail-order catalogues for women’s clothing found that demand jumped by up to a third when prices ended in the number nine, such as $9.99 or $39.
Leigh is such an enthusiast that he runs his own randomised trials, at least in simplified form. He chose the title for this book and a previous one, Battlers and Billionaires, by placing an advertisement on Google for a few days randomly showing different titles — twelve in the case of Randomistas — and then using the most popular.
Some of the greatest potential for making a difference lies in the application of randomised trials to policy-making. Take, for example, the trial wittingly or unwittingly carried out by the Promise Academy in New York’s Harlem district. This is a public charter school that operates extended class hours from 8am to 4pm and after-school activities as late as 7pm. With more applicants than places, it runs a lottery to choose whom it accepts. Whereas average African-American high school students in the United States lag two to four years behind their white counterparts, those who attend the Promise Academy perform equally well.
When it comes to crime, Neighbourhood Watch is Australia’s oldest and best-known community policing program. Despite the claims often made for the scheme, though, randomised trials have shown it to have no impact whatsoever on crime levels.
By contrast, a NSW trial that split those arrested on drug offences randomly between the Drug Court and the traditional court system found that, for every one hundred offenders who went to the Drug Court, eight drug crimes were committed in the subsequent year, compared with sixty-two crimes among the second group. The Drug Court treats addiction outside the jail system, with offenders who successfully complete a year-long program typically put on a good behaviour bond, with the threat of a jail term if they lapse.
And then there is politics. During Barack Obama’s first run for president, his campaign experts judged that a video of him speaking and a message saying “sign up” were likely to be the most appealing to visitors to the Obama website. When they switched between these and a series of different images and messages, they found they were wrong, with the combination of a black-and-white family photograph and a “learn more” message yielding 41 per cent more email addresses, which translated into 280,000 more volunteers and US$60 million more in donations. According to the Republican National Committee, the Trump team tested up to 50,000 different Facebook advertisements every day of the campaign.
In fact, many of the techniques that have become a standard part of election campaigning have been shown not to work, or at least not as well as assumed. Randomised trials of American robocalls, where computers automatically dial telephone numbers and play messages, found they generated one extra vote, at most, for every 1000 people who heard the message.
The results can be so impressive that Leigh sometimes seems to come close to suggesting that randomised trials are the answer to absolutely everything. But they can raise tricky ethical issues.
In one case, a group of surgeons in Houston conducted keyhole surgery on the knees of patients with osteoarthritis, while patients in a control group received an incision in their knee (and, the Houston doctors insisted, were told that was all it was). Two years later, the members of the two groups reported similar degrees of pain and knee function. It’s certainly a useful finding, but it raises the question of how many and what kind of patients would agree to sham surgery when they were told exactly what was (not) happening to them.
In 2012 Facebook conducted a psychological experiment by randomly altering the emotional content in the news feeds of 700,000 users. When people saw more negativity in their friends’ posts, their own posts became more negative and they were less likely to use Facebook the next day.
What amounted to emotional manipulation attracted some strong criticism, prompting Facebook’s chief operating officer, Sheryl Sandberg, to make an apology of sorts: “It was poorly communicated and for that communication we apologise. We never meant to upset you.” Not everything in life can be measured and some things that can should not be.
Randomised trials need to be properly conducted to produce accurate results. Like opinion polling, they have the potential to be misused to achieve a predetermined result.
Those reservations aside, Leigh has produced an absorbing book. Refreshingly, considering it was written by a politician, it isn’t all about its author. We should wish the professor–turned–political evangelist luck. Politics is more constrained than it has been for a long time by what is considered achievable and popular. Experts are routinely mocked and evidenced-backed claims ridiculed. Politicians are focusing more on pandering to emotions and less on persuading voters of the merit of policies that have been shown to work.
Leigh himself is vulnerable to government jibes about prescriptions he advocated as an academic, including an inheritance tax, that he now is obliged to disown as a member of the Labor team. That said, he can fudge with the best of them when put on the spot.
We can only hope that, despite the views of Peter Dutton, he is not regarded as a misfit in politics, too bright for his own good. He deserves a better fate than Barry Jones, also known as the man with two brains, who never advanced beyond junior minister, his expertise often ignored and his prophecies unheeded. ●
Randomistas: How Radical Researchers Changed Our World
By Andrew Leigh | Black Inc. | $29.99 | 256 pages
