Laying the ground for a new way of measuring impact

Today we’ve published the first report in what we hope will be an ongoing series, that generates data from a ‘hyperlocal’ version of the national Community Life Survey which can be used to track and measure impacts in a local area. We are very excited about this research and the potential it has to allow us to objectively measure the local impacts of community businesses.

When we set out to commission this research last year, we weren’t entirely sure if it would work, or what it would tell us. That’s often the case with research, so we didn’t worry overly much. And we had reasons to be confident. We worked with an expert team and designed what we all felt could be a new way of measuring the impact that community business has at a very local level, with the ability to compare community business data to a large national dataset.

One of the things that motivated us was the certainty that most – all? – community businesses will struggle to undertake this kind of research or gather this kind of robust, independent data about the impact they are having on their local community. We are in a position to do this work, and to stick with it over a number of years, so it seemed right that we would start with a ‘baseline’ year of data and test the method. The idea for the research came from many discussions with community business leaders and our other stakeholders. Although we know that this kind of impact measurement is too much to ask of any individual community business, it remains important – for businesses, to understand and to build on their impact, and for sector bodies and policymakers, to support growth in the sector and drive up the impact it delivers.

It is extremely difficult for Community Businesses to accurately measure the often intangible long term benefits their activities have on their community. This research puts a stake in the ground by trying to measure some of those hard to measure benefits & will be an invaluable tracking system for long term benefits.

Sally Anne Watkiss

Homebaked community bakery

So what did we do?

The Community Life Survey (CLS) has been carried out annually by Kantar Public since 2012-13 on behalf of the Office for Civil Society. It provides official statistics on issues that are key to encouraging social action and empowering communities, including volunteering, giving, community engagement and well-being. We felt that this survey would be a good way of measuring these impacts in the areas surrounding community businesses, at a hyperlocal scale. We worked with Kantar to select six established community businesses, and Kantar replicated the national survey in those areas, targeting households in the immediate vicinity of the businesses. We achieved a good response rate (of circa 300 surveys per area) in all six areas, meaning that enough data was gathered for statistical analysis to be carried out. This was our first hurdle overcome, as one of the uncertainties about this research project was whether we would be able to get enough data from such small geographies.

The next step was to construct matched samples from the national dataset, in order to compare the community business data and draw some conclusions. You can read more in the report about how this was done. When the matches had been created, we were able to see the differences between the data coming through from the six community business areas and what we would expect from these areas, based on the national survey data.

What did it show?

The data itself showed a mixed picture, with areas showing both higher and lower results than would be expected. The key individual differences between the six areas and their matched comparison samples are summarised here. It certainly didn’t show us that the areas around these community businesses are performing better in terms of cohesion or wellbeing – but maybe that’s to be expected. Community businesses are often set up in deprived areas, as a response to challenges. No surprise, perhaps, that these challenges are showing through in the data – though six areas is a small sample. There were some interesting results, such as that at least half of the community business areas reported higher levels of satisfaction with local services and amenities compared with the matched comparison sample, and the same proportion reported that the area had gotten better in the last two years. These deserve deeper investigation, and we plan to undertake further detailed analysis of the dataset this autumn.

But the research at present is still only a snapshot – albeit a very interesting one. Now that we know this approach works, we intend to repeat the exercise and build up a longitudinal dataset that can be used to track the rate of change in these areas, so that we can say – for example – whether they are improving (or not), and whether they are doing so more or less quickly than might be expected, compared to the national survey data. When we can do this, then we will be able to get the full value from this research and the rich analysis that it makes possible.