5<\/sup><\/a><\/p>\nHowever robust our approach, the measuring, evaluating, and understanding of place-based change remains challenging, especially where primarily quantitative measures are used. We explore these issues further in the next section.<\/p>\n
[\/et_pb_text][et_pb_text admin_label=”1.4 Limitations and considerations” module_id=”chapter14″ _builder_version=”4.21.2″ _module_preset=”default” custom_margin=”||20px||false|false” custom_padding=”10px||||false|false” global_colors_info=”{}”]<\/p>\n
1.4 | LIMITATIONS AND CONSIDERATIONS<\/h3>\n
Measuring, evaluating, and understanding place-based change can be complex and challenging. It is important, therefore, to treat the findings with some caution:<\/p>\n
\n- Change is not always linear: <\/strong>Change, particularly place-based change, is not necessarily linear, and does not fit into neat patterns that evaluation may require. It is clear from the findings that change fluctuates, but surveys are only conducted at random points in time. Practically, change manifests in complex ways that do not conveniently fit neat timelines for research, or the programmes being evaluated. As this is particularly the case with quantitative approaches, it may still be too early to understand impact over the longer-term, and further fluctuations are possible.<\/li>\n
- High threshold for impact: <\/strong>This methodology was delivered to test whether it was possible to see statistically significant quantitative change amongst respondents living in a local area as a result of long-term investment in community businesses in that area, relative to a comparison group. This means that we are looking for evidence of impact amongst those who may have not<\/em> been directly involved with catalysts or community businesses that have received funding from Power to Change. Additionally, using statistical significance and a comparison group, the methodology imposes a much higher threshold for determining impact than many others commonly used when evaluating place-based change.<\/li>\n
- Methodology and context: <\/strong>Kantar Public\u2019s analysis primarily focuses on using difference-in-difference to assess impact and looking at trends within individual outcomes. However, as the survey provides a comprehensive data set that can also be used in multiple other ways, this report includes additional analysis from Power to Change, such as reviewing combined trends over time.<\/li>\n
- External factors:<\/strong> A myriad of other factors affect the results of all interventions, but particularly locally-led place-based ones. Notably, the research was carried out in 2018, 2020, and 2022, during which the Covid-19 pandemic had a significant impact, and this is likely to be reflected in the data. Although imperfect, using comparison groups aims to mitigate the risk of distortion, as they are constructed to be similar to the Empowering Places areas.<\/li>\n
- Representing disadvantage and marginalisation: <\/strong>The differing experience of the pandemic\u2019s impact, more keenly felt in disadvantaged and marginalised places, also reflects the variety of ways in which any change can be felt and measured. Traditional evaluation methods often favour the majority (for example, by looking for change at a common enough level so as to be considered meaningful and representative), leaving people with experience of discrimination or marginalisation less well served. Although Kantar Public\u2019s approach involved representative samples, accessible, web and paper versions, and incentives for participating in the survey, any quantitative approach will struggle to serve or capture change for everyone, everywhere.<\/li>\n
- Expected change: <\/strong>Not all questions within the Community Life Survey map neatly onto the theory of change informing the evaluation of the Empowering Places programme. As a result, we should neither expect to see clear change across all responses (as change was not intended in these areas), nor treat any lack of findings as an absence of impact. As the survey does not capture the entirety of the programme\u2019s impact, a lack of evidence of change does not mean that change is not actually present in those areas.<\/li>\n<\/ul>\n
To supplement our insight in light of these factors, we also invested in a comprehensive qualitative evaluation across the six areas over the same time period, delivered by Renaisi. \u00a0Focusing on the experiences of those involved in the programme, Renaisi primarily drew from over 100 interviews and video ethnography with catalysts, community businesses, tech leads and stakeholders in the local areas, along with 13 interviews with programme delivery leads at Power to Change and Co-operatives UK. These two complementary approaches help build a more thorough picture of the impact of the programme.<\/p>\n
[\/et_pb_text][et_pb_button button_url=”#section2″ button_text=”SKIP AHEAD TO THE FINDINGS” _builder_version=”4.21.2″ _module_preset=”default” custom_button=”on” button_text_color=”#E6E6E6″ button_bg_color=”#FF3600″ button_border_color=”#FF3600″ button_icon=”E||divi||400″ button_icon_color=”#E6E6E6″ button_on_hover=”off” custom_margin=”||20px||false|false” global_colors_info=”{}”][\/et_pb_button][et_pb_text admin_label=”1.5 Key technical notes” module_id=”chapter15″ _builder_version=”4.21.2″ _module_preset=”default” custom_margin=”||10px||false|false” global_colors_info=”{}”]<\/p>\n
1.5 | KEY TECHNICAL NOTES<\/h3>\n
<\/a><\/p>\n[\/et_pb_text][et_pb_text admin_label=”Text” module_id=”chapter15″ _builder_version=”4.21.2″ _module_preset=”default” custom_margin=”||20px||false|false” global_colors_info=”{}”]<\/p>\n
![](\"data:image\/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==\")
![](\"https:\/\/www.powertochange.org.uk\/wp-content\/uploads\/2021\/12\/subhead-dots.svg\")
![\"\"](\"https:\/\/www.powertochange.org.uk\/wp-content\/uploads\/2023\/11\/difference-in-difference-chart-e1699279276377.png\")