Lies, damn lies and statistics – a view on neighbourhood plan surveys
“Right, the first thing we need to do is prepare a survey” he says.
“Why?” I ask.
“So we can understand what people think” he says.
“About what?” I ask.
“Well, about all the issues we want to cover in our plan” he says.
This is a familiar conversation we have with neighbourhood plan groups that are starting to gather evidence for their plans. Whether it is a British thing, a mathematical brain thing or a comfort thing, never under-estimate the penchant for a good old survey. For many, this stems from a belief that all evidence to inform a neighbourhood plan must be statistically robust.
In fact, nothing could be further from the truth. As one very wise person once famously said, “73.6% of statistics are made up”. Surveys have their place but they also have their considerable limitations. This is even more the case when a survey is undertaken very early on in the process. At this stage, a neighbourhood plan group knows very little about what the community thinks about a long list of issues. So it asks them. Common questions include:
- ‘Do you value green space? Would you want more green space in the community?’ – well duh, who doesn’t?
- ‘Do you think that your community needs more housing?’ – so this is directed at you, a homeowner and one of 80% of households in the community that are already homeowners. Does the view of this 80% represent a fair and realistic one or should this question be aimed at those that cannot afford a home here?
- ‘Do you think that new dwellings should be better insulated and more energy efficient?’ – what, resulting in lower energy bills? Hmmm, I’ll have to think about that one.
- ‘Do you think that the Post Office/local shop should be retained?’ – absolutely, even though I only go in there once a year at Christmas to buy those festive stamps.
After 50 or more of these questions – because if you want to know the theory of everything then this is going to take a considerable degree of interrogation – what has this told us? That people like nice things, are less keen on undesirable things and are extremely annoyed at the lack of dog mess bins and the broken paving slab outside 42 Acacia Avenue. This profile of everything and nothing is usually further enhanced by some ‘free text’ questions where people – mainly those with the most time on their hands, which is often a small and distinct part of the population – give their views on life, love and Council bin services.
In short, we have a mass of data which some poor souls have to wade through in order to come to some unreliable conclusions about what the community thinks. And often even these unreliable conclusions are drawn out of questions that really aren’t asking about this.
Here is our view of some important principles about surveys:
- They are notoriously hard to prepare without having ‘leading’ questions.
- Unless you doorstep people and force them to fill a survey out on pain of death, then return rates are rarely more than 25%. If they are more than this, the demographic profile is commonly skewed towards older people.
- A survey is unable to ask the ‘follow-up’ questions to the one asked, which are often the more interesting and revealing ones.
In short, surveys have their place but it is not at the start of a neighbourhood plan process. This is not to say that they do not have a place – they are useful when you have very specific matters that you wish to seek views on. But the survey has to be short (20 questions anyone? No thank you, that’s far too many…) and you have to work very hard to ensure that it is returned by a cross-section of the community. Just putting it online does not cover the population that is under the age of 40.
So what’s the alternative?
Unfortunately, this is where the colour sometimes starts to drain out of the faces of neighbourhood plan volunteers. After years of working in planning and with communities, I can confirm that the best way is ….(drum roll)….to talk to people. Or better still, get them to talk to one another and for you to sit back and observe.
Talk to people and listen to what they say. Ask them those follow-up questions that you couldn’t do in the survey. If you asked them in a survey whether they consider allotments to be important and they said ‘yes’, then through a conversation you can in fact discover:
- that they would never be interested in actually taking an allotment plot themselves because they don’t have the time and would end up harvesting far too much produce; but
- they would in fact be interested in a community gardening scheme which would take up less of their time and was much more convenient to get to
And of course they would know what a ‘community gardening scheme’ was because you would be able to explain and show pictures of one so that they had a proper understanding of what it was before they gave a view.
Talk to people in the right environment. If you want to know about the current provision of sports facilities, then go and talk to the local sports clubs who actually use these facilities on a regular basis. Don’t rely solely on the survey that may largely have been filled in by people with no interest in sport or for whom their sporting days are…ahem…a little way in the rear view mirror shall we say.
Run workshops on specific issues. This will prick the interest of those that care about the particular topic in question. Our experience is that it is far easier to engage people on a specific topic than on broader considerations such as what the ‘place’ serving a whole community should look like in 15 years’ time (succinct definition of ‘sustainable development’ anyone?). This may only bring along a dozen people but that dozen are particularly interested in the future of the topic in question so surely their views must count more than those people quickly ticking a box on page 15 of a survey form in order to keep the very nice but persistent chap from knocking on the door for a third time to ask if they had completed the survey yet.
Good plans come from good engagement. And good engagement is about people, about narratives and about the ability to challenge views back and forth. We often observe that where people round a table disagree on a matter, they often actually ultimately agree on the outcome but differ in their views of the way that the matter should be dealt with. Surveys give none of that richness, that flavour, that human-ness.
So before you start preparing your survey, stop and ask yourselves – what am I trying to achieve and is this survey realistically going to do this? As a guide, if you can’t achieve what you want using a solely multiple choice survey of less than a dozen questions, then the answer is probably no. In which case it is time to get out there and start talking.