Randomly criticizing randomly controlled trials

Are people who design government programmes like doctors who prescribe drugs to patients?

This is the question that Ben Goldacre asks in his recent documentary (which you can listen to here for a while). In fact, he doesn’t so much ask the question as attempt to convince us that the people who design government programmes should use randomized control trials to test the effectiveness of their ideas before they roll out new programmes.

For a limit range of things this seems a perfectly sensible way to go. He features an interview with a senior civil servant who explains that by using a different approach they were able to collect more fines for a magistrate’s court than traditional approaches.

An effective administrator would surely be interested in trialling new methods and reassessing current ways of doing things and randomized control trials are one good way of doing this.

If I was to be critical of the documentary I would make the following points;

1. Little understanding of conflicts in values

In lots of the most important areas of public policy there is not an agreement about goals. This means trials cannot be used to settle debates.

Goldacre discusses sentencing criminals and this is a classic example. Should the court punish the criminal or rehabilitate them? You could set up a trial to find what is the most effective of a number of options for punishing a criminal but you could not set up a trial to find out whether the criminal justice system should punish or rehabilitate.

2. There’s more to life than public services

What makes us healthy, educated and safe? And what makes us sick, ill-educated and victims of crime? It is mostly not public services. While trials are useful for assessing the effectiveness of specific programmes delivered by public services I am not sure they are so useful when thinking about how to change the broader context which public services operate in.

Take the example of education, which is touched on in the documentary. Goldacre talks to the boss of the Educational Endowment Foundation about a new programme to reduce the number of people who cannot read at a certain standard by a certain age. This seems a perfectly sensible time to trial a number of methods for teaching reading.

However, take another challenge. Students from families on low incomes do worse at school than families from high incomes. This is true in ‘good schools’ and in ‘poor schools’. Is this the type of problem that randomized control trials will help us solve?

I am not so sure. At least, I do not think it is the kind of problem that conducting trials in schools will solve. I am open to suggestions here but I am unconvinced.

3. Focuses on results and not institutions

There is certainly something attractive about the idea of proving that a certain policy has a certain effect. But there remains a question of who does the testing, and what happens after the results are in.

We can imagine one direction of travel where a brains trust like the Behavioural Insights Team in the Cabinet Office (featured in the documentary), test out lots of ways of teaching kids how to read. They then find the best way to teach kids to read and tell all teachers in all schools to use this method from now on.

We can imagine another direction of travel in which schools and local authorities become the kind of institutions that are constantly trialling different ways of teaching and then evaluating the results of these trials in partnership with parents.

I am not saying that one outcome or the other is more likely or is being advocated by Goldacre. I am saying that it is important to focus on what type of institutions we build and maintain as what type of programmes are run by these institutions.


Although this piece is a critical take on Goldacre’s show I would not want the reader to think that I thought it was a bad or malign documentary. The idea of using more randomized control trials to establish the relative effectiveness of public programmes makes a lot of sense and is happening more and more.

However, I am afraid that using more randomized control trials will not solve all our problems, partly because we don’t even agree on what our problems are.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s