Let’s Predict The Results of This Survey

We’ve spoken before about how survey questions can be inadvertently leading depending on their structure.

Executive Recap

1. The “primacy effect” shows how lists bore people and they forget the middle options.

2. If you only present people with an ‘either/or’ opinion on a topic, they’re overwhelmingly likely to vote in one direction- and more so if a single alternative is obviously dominant. (“Would you prefer a reduction in taxes or a punch in the face?”)

3. If you make implicit associations between answers to different questions, you’re much more likely to see results that corroborate those associations (here’s Sir Humphrey with more on that).

Bearing this in mind, we wonder whether the Conservative Party’s recent survey of local constituents’ opinions might produce the following ‘answers’.

Exhibit A

Exhibit A

The Nation believes that the biggest concern for the country is the EU.

Something has to come first in a list. But both Questions 1 and 2 use the same list ordering and so the ‘primacy effect’ is reinforced. (The survey designers may have produced multiple versions of this survey, with a randomised list ordering on different versions. If you know, please let us know!)

If you were being cynical, you could say that the advantage of arranging a list in columns is that you can have multiple first and last items. Based on the arrangement in this list, the EU, Welfare Fraud, Taxes, Schools, and Affordable Housing are all likely to score highly.

“Immigration” has been put in the middle of the list, where more people are likely to jump over it. Is someone at CPHQ afraid of negative associations with UKIP, perchance?

The Nation feels good about Conservative approaches to the NHS, Opportunities for the Next Generation, and the A34.

Questions 3 through 7 represent an expert manipulation of the “axis” problem and the problem of dominant alternatives.

For example, the second part of Q3 (on the NHS), asks whether you strongly agree or disagree with increasing the number of doctors on wards… Who in their right mind would think having fewer doctors in the health service is a good idea?

The same can be said for taking action against hospital-related infections. Which, we’re reliably told, have halved. This one is barely even a question: how exactly do you disagree with a fact?

Similarly, nobody is going to answer the question “Do you agree with the policy to increase apprenticeships so that more young people have the skills they need?” with “No, I think fewer young people should have the skills they need.”

Exhibit A, Part 2

Exhibit A, Part 2

UKIP? Is that the BNP?

Interestingly, UKIP are placed next to the BNP in question 9, despite being a larger party than the Greens in terms of popular support. Is this an attempt at fascism by association?

Nick Who?

Also of interest is the omission of Nick Clegg as a third choice in Question 12 (“Thinking about them as people, which of the main leaders would you prefer as Prime Minister?”). Does this paint a picture of the Tories’ views on their coalition partners?

We’d like to point out that we don’t think this survey is manipulative by design. But if you’re designing a survey, everything means something. Of course, we have a lot of respect for Mr. Crosby’s effectiveness. But was this survey drafted by a loyal Conservative with very strong feelings on the topics it discusses? And if so, do the way the questions have been phrased reflect this?

If you need some advice on producing surveys that fully exploit behavioural economic models and text analytics to give more reliable results, email Chris.