News

Sharing our approach to Monitoring, Evaluation, and Learning

As we launch our new impact page, we would like to present an in-depth look into how our Monitoring, Evaluation & Learning strategy has evolved in recent years, and the methodology SF undertakes to measure impact.

The goal of Shell Foundation’s (SF) Monitoring, Evaluation, and Learning (MEL) strategy is to track the social, economic, and environmental outcomes of SF’s support and evaluate the foundation in meeting its objectives of lifting low-income populations out of poverty through the provision of energy and transport solutions. MEL is an essential element of good management. It helps SF keep track of resources spent, what outcomes have been achieved, as well as better understand what works and what lessons can be learnt in order to improve future decision making.

SF’s MEL strategy has four guiding principles:

  • Accountability: Meeting our responsibility to our board, partners, and broader stakeholders to measure success in addressing international development challenges and demonstrate the efficiency of its enterprise-based approach.
  • Informed decision making: MEL provides a framework to integrate social, economic, and environmental outcomes into business strategy and operations. This helps SF prioritise where resources are deployed for the greatest development impact as well as innovation.
  • Alignment with the wider energy and transport development sector: Robust monitoring and evaluation processes can help SF maintain thought leadership in its sector and demonstrate to funders that it merits future investments and support.
  • Global learning: Working on complex energy and transportation projects in challenging environments offers SF invaluable insights into what works and what does not. Good MEL data can support SF’s influencing work by improving the evidence shared with other stakeholders to inform their programme design and approach.

Shell Foundation’s MEL is composed of two distinct areas to deliver its strategy: 1) routine performance monitoring; and 2) impact assessments. SF is committed to continue creating new performance monitoring tools, strengthening and rationalising the data, conducting longitudinal studies and identifying and testing new solutions and innovative technology in impact monitoring and evaluation. This will enable SF to build upon the systems it has in place and demonstrate the impact that validates its work.

Routine Performance Monitoring

Shell Foundation’s cumulative impact is achieved through the work of all the various enterprises in its portfolio, and calculated using a comprehensive set of indicators that feed into aggregate numbers. SF tracks and measures changes in performance against pre-defined milestones and targets, which allows for a meaningful analysis of the drivers of impact. SF measures its overall developmental impact by four broad metrics.

Social Impact Livelihoods Improved: Net number of people with access to good quality and cost-effective energy and transportation products and services, skills training, and career support.
Jobs Supported: Net number of people employed directly by partners and indirectly by businesses to which the partner is a core supplier of products, services, human resources, finance or technical support
Environmental Impact CO2 Reduced: Net tonnes of CO2 or CO2 equivalent gasses reduced due to consumer of business adoption of partner products, services and/or industry standards.
Economic Impact Finance Leveraged: Amount of follow-on finance mobilised by partners post-SF support or as direct funding for activities/companies supported by SF programmes.

 

These four broad metrics are underpinned by indicators that are specific to each enterprise, depending on the sector and stage of business. Examples include:

  • Low-income customers served, e.g. through product sales or passenger journeys
  • Environmental benefit, e.g. reduced emissions (aligned with GOGLA’s standardised impact metrics by a number of SF’s energy partners) or water usage
  • Economic benefit, e.g. jobs created, earnings increased, money saved
  • Social benefit, e.g. improved health or time saving

Data collection

As SF has improved and increased the granularity of its performance monitoring, it has had to develop and improve its data management system. In 2018, SF has built a custom application, the SF Enterprise Hub, through which its partners submit their impact KPIs. Having robust systems in place, enables a simple way to collect data and ensure data quality practices are in place. It also gives the ability to analyse and manipulate data in different ways and be able to report impact over time. The Enterprise Hub allows data to be collected from a catalogue of indicators that can are specific to each organisation. These indicators can be disaggregated by different criteria including, type of business, tiers of energy, type of job, gender and geography. The data is time-series and collected bi-annually to allow for impact to be measured over time. The Hub is fully customisable to support changes in requirements to impact measurement.

More than numbers

Consumers are the most important part of any enterprise-based approach. Therefore, understanding their motivations, needs, and behaviours is the key to knowing what impact SF and its partners are having on the ground. SF therefore adopts a consumer research approach to impact measurement. Such an approach is about using a more realistic, behavioural model of impact while keeping a focus on institutions and the context in which consumer decisions are made. This approach is also complimentary to the routine performance monitoring that is undertaken bi-annually and can help refine the indicators collected as part of its monitoring. Consumer research serves to provide insights on a more regular basis and provide a qualitative angle, by sharing human impact stories and giving end beneficiaries a voice.

A consumer research approach to impact measurement can help Shell Foundation:

  • Continue to understand the demographic, socio-economic, and gender profile of the consumers that partners reach and the significance of the benefits that consumers get.
  • Enhance business support to partners. Consumer insights provide countless sources of business advantage to partners that allows them to scale their operations and further reach low-income populations.
  • Strengthen and build an impact narrative that validates their work and attracts and retains stakeholder confidence.

Continuous Improvement

SF will continue to work on creating new performance monitoring tools which improve SF’s ability to assess the impact and cost efficiency of its support, as well as strengthen and rationalise the key performance indicators it collects. One example of this is SF’s support to the Future-Fit Foundation and the development of its Impact Benchmark Tool, a self-guided management tool that helps SMEs and investors understand their contribution towards SDGs. More emphasis will also be placed on conducting longitudinal studies as part of the consumer research work and identifying new solutions to impact evaluation that utilise innovative techniques and technology.