Learning from the nineteenth century

Search, experience, credence: a new framework for social impact measurement?

Gen Maitland Hudson

Head of Evaluation and Impact Assessment

Social impact measurement can be a confusing and contradictory part of a social venture’s planning. Finding a way of integrating collective standard setting might help to cut through some of the confusion, share much of the effort, and better validate outcomes.

At its most ambitious, social impact measurement tends towards theories-of-everything, and the idea that there may be a grand ur-metric that can account for all social value. This metric, or perhaps a small group of them, can – in this grand picture – be made universal in practice as well as in theory through the magic of ‘digital’.

This ambition floats very far above the reality of our day-to-day practices of measurement. History shows that our measures are capricious, embedded, and very hard to shift: “since they operate in societies, they are circumscribed by institutional frameworks, and the institutions in question involve such measures in complex clashes of interests among various social groups.” That is Witold Kula, and the man knows what he’s talking about.

Denying the social side of measurement and concentrating on the theory and the technology is a temptation, but also a mistake. Good accounts of social value need to work with people’s everyday understanding of information, and build on them. They also need to acknowledge where institutional frameworks and approaches have consequences.

An example in the existing practice of social impact measurement, is the way in which consultancy business models and Theory of Change have worked together to emphasise highly individualised conceptions of social programmes. They do this by highlighting the ‘unique’ properties of every venture, in a market in which innovation is highly prized and people prefer to pay for bespoke services, and by focusing on the segmentation of beneficiary groups. That is further exacerbated by the encouragement of the use of ‘validated’ tools, when most of these are geared towards the measurement of personal change in the individual service user.

This has come at the cost of more collective understandings of social value.

It is interesting, and not at all surprising, that many of the more recent approaches to social impact have emphasised the collective. Three of NPC’s eight innovations in measurement and evaluation, for example, are geared towards offering a collective counterweight to inward-looking social programme measurement.

That reintroduction of collective thinking could be supported by an approach to programme planning that explicitly sought to highlight joint standard setting and quality assurance as mechanisms for ensuring the cooperative achievement of outcomes across organisations. Theory of Change does not do that. Nor does it help social programmes to distinguish between the points in their service delivery or product development that are ‘good for’ certain sorts of measurement rather than others. An extra effort is required to fit out a Theory of Change with an evaluation framework and associated indicators, and the tool itself offers no guidance on where, for example, it would be best to invest in feedback rather than, say, data mining.

Is there, then, another approach to planning that might do these things better?

I have found one in the pages of an excellent book by Aashish Velkar on nineteenth century measurements in Britain. Velkar is interested in the governance of the London coal market, the thickness of wire and the grading of wheat, but the classification that he takes from Jean Tirole can be usefully applied to social measurement in the present day.

Tirole’s classification says that: “some standards help in the selection of products by capturing information about their quality or diversity. These are standards based on search attributes that are ex ante, that is, relevant prior to exchange. Experience attributes, in contrast, are evidence ex post, and standards that describe them (e.g. durability) are often relevant post consumption. Credence attributes are trickier to describe and may not be evident even on an ex post basis. Standards that describe these attributes include those that capture information about composition of materials, or presence of (potentially harmful) substances, or manufacturing methods employed and so on. Establishing whether a product possesses the desired credence attributes is difficult and often requires third party monitoring or certification.”

That is a long quote, but it establishes a useful trio of distinctive standards and attributes that could provide straightforward guidance on the points at which social programmes can collect valid data and for what purpose. It also usefully highlights where they should invest in collective standards.

The table below sets out a very simple outline for a community business based on Tirole’s classification:

Attributes What to do Community business example: pub
Search attributes Collect and analyse operational data Pricing; food and drink offer; opening hours
Experience attributes Collect and act on feedback Regular spend / head; simple customer feedback surveys
Credence attributes Set collective standards and quality assure Supply chain standards eg. quality marks and fair trade; paying the living wage; mystery shopping

 

A community pub using Tirole’s classification to plan its social impact would concentrate first and foremost on its core business. This would integrate social impact with business planning, rather than relegating it to a parallel process that focuses on extraneous additional services. Importantly, it would seek to establish the collective standards against which the pub can hold itself, rather than attempting to validate a purely internal set of indicators that require expensive independent verification.

In this example, appropriate collective standards already exist. This will not always be the case. Establishing collective standard setting and mutual assurance as an integral part of impact management, however, supports the development of shared approaches, and should also, not unimportantly, spread the cost of validation.  Given that the day rate of a social impact consultant currently starts at £750, that is an important consideration.

This framework needs testing and development, but it may offer a useful alternative to Theory of Change that helps to set measurement standards, and ensure mutual assurance in ways that can bring real benefit to community businesses.