Measuring social impact at the neighbourhood level: A better way
...at Power to Change we want to know if the neighbourhoods directly around community businesses improve in some measurable way over time.
“I have always been optimistic that things will get better. Things will get better, they always do.” – Captain Sir Tom Moore, September 2020
The indomitable spirit of Captain Sir Tom Moore inspired the nation when we most needed it. It is the same spirit that every day inspires charities, social enterprises and community businesses to face an imperfect world with hope, not despair. It is a spirit that can ignite strong passions deep within our soils and yet those blazing passions can also – occasionally – blind us from the truth. As the exonerated directors of Kids Company would no doubt attest, not every well-meant social innovation is successful. And it is the sad and lonely task of the impact evaluator to cast a more objective eye and ask the question, “did things really get better?”
One ‘obvious’ way to measure the social impact of third sector organisations is to look at things before their intervention and then again afterwards and see if there has been an improvement. For example, at Power to Change we want to know if the neighbourhoods directly around community businesses improve in some measurable way over time. Our hypothesis is that the locally-rooted, community-focused nature of their work should have a direct, positive impact on residents’ health and wellbeing, on levels of trust and wellbeing, and so on.
So why not just measure these things at two points in time and see how they’ve changed? If the shift is positive, great! Surely we’ve demonstrated that community businesses make places better? Unfortunately, it’s not that simple. In our rush to judgement, we’ve missed some important steps in the process. Suppose, for example, we were looking at levels of trust in the community. We need to ask two important questions:
- were levels of trust at the time of first measurement higher or lower than you would have expected, all other things being equal?
- was the increase in trust between the first and second measurement higher or lower than you would have expected, all other things being equal?
The reason these steps are important are that we don’t know if our neighbourhood was naturally more (or less) trusting than other neighbourhoods and therefore starting from a higher (or lower) baseline. Nor do we know if there was something else happening right across the country that made levels of trust increase between our first and second measurements.
The best way to answer such questions in the natural sciences is often through the Randomised Controlled Trial. Subjects are randomly split into two groups, one is exposed to the treatment under consideration and the other is not, and the results are measured to see whether or not the treatment has worked. Yet this approach makes no sense when the ‘treatment’ in question is simply a community business sitting in the middle of the neighbourhood with all residents equally exposed to it.
So should we just throw our hands in the air and give up? Not at all! There is a better way. Since 2017, the Power to Change Research Institute has been working with the Department for Digital, Culture, Media and Sport and the survey company Kantar to run ‘hyper-local boosters’ alongside the government’s long-running Community Life Survey to track changes in the neighbourhoods directly surrounding certain community businesses.
Aligning the 350 or so households in these hyper-local boosters with the main survey allows us to use a standard statistical technique called propensity score matching to construct a ‘comparison sample’ for each booster area. These comparison samples are created using respondents from the much larger national Community Life Survey who live in the 10% of English neighbourhoods most similar to each booster area. As the name suggests, comparison samples allow us to compare responses in each area on a like-for-like basis. From this we can determine if a neighbourhood is better or worse than expected, all other things being equal.
By repeating the hyper-local booster surveys in the same areas, we can apply another standard statistical technique, difference-in-differences modelling, to assess how the areas change over time. From this we can determine how a neighbourhood has changed, all other things being equal. Let’s take a practical example. Consider the data represented in the chart below, which is based on Community Life Survey responses to a question about civic participation in the neighbourhood directly around the Bramley Baths community business in Leeds.
- In 2017, 34.7% of respondents in Bramley said they had been involved in some sort of civic participation in the previous 12 months. This was significantly lower than the comparison sample, implying the neighbourhood around the Baths was worse than should otherwise be expected.
- When Bramley was surveyed again in 2019, positive responses to this question had increased by 5.5 percentage points to 40.2%. By contrast, positive responses in the comparison sample had fallen to 33.8%.
- If Bramley had fallen by the same extent (the ‘parallel trend assumption’ used by difference-in-difference models) just 25.3% of the population would have been involved in some sort of civic participation.
- Hence the real, like-for-like change in civic participation in Bramley between 2017 and 2019 was a significant 14.8 percentage point improvement.
In fact, of the 38 Community Life Survey outcome measures that we use measure the impact of community businesses, seven showed statistically significant improvements in Bramley and none showed any detriment. Of course, this is not proof that Bramley Baths was the cause of these changes – but the combination of propensity score matching and difference-in-difference modelling removes the influence of all unobserved characteristics that remain constant across time between the two groups. It is fair to say that the result provides strong support for the hypothesis that ‘community businesses like Bramley Baths make places like Bramley better’.
Sir Tom told us last year that things would get better and slowly but surely, as the vaccines do their vital work, they are indeed getting better. Sadly, the equally vital work done by third sector organisations is going to be needed more not less over coming years. The structural inequalities exposed by the pandemic demand structural solutions, with state and sector working hand in hand to support our poorest and most vulnerable communities.
A key part of this must be a more mature approach to impact evaluation. The world does not need any more ‘single-side-of-A4-word-cloud-style’ theory of change documents. We do not need more ridiculously large SROI ratios that convince no-one. What we need is for charitable foundations and government to follow the example of Power to Change and set aside significant resource for research and hypothesis testing. Money spent on high quality impact measurement and not on the front line is not money wasted. It is in fact the only way to make sure scarce funds are not wasted. It is the only way to make sure things really do get better.