Five dimensions of impact | Contribution

To estimate the outcome counterfactual, enterprises can use many approaches that vary in rigor and costs. Experimental methods such as randomized controlled trials (RCTs) can provide strong evidence that a product or service caused a change in outcome on average across a population, but leave other research questions unaddressed, such as the distribution of effects within the population. 

Other methods such as market research and community engagement can yield valuable insights for (1) understanding what else may be driving the outcome, (2) building a “good enough” counterfactual scenario, and (3) conducting the depth analysis. These methods can be mutually reinforcing and combined to produce complementary insights:

1. Community engagement: Community engagement covers a range of approaches (e.g., Collaborative Outcomes Reporting, Qualitative Impact Assessment Protocol) that involve consulting the people and communities affected by the enterprise’s activities to gain a nuanced understanding of the drivers behind the outcome (e.g., the enterprise’s activities, external factors, government interventions, and cultural practices). If deployed well — covering a large and representative sample inclusive of diverse communities and different points of view — community engagement can suggest possible counterfactuals.
2. Market research: By taking a thorough look at an intervention’s context, market research can be used to identify possible counterfactuals. This method requires a deep analysis of secondary resources (e.g., research papers, government publications, industry reports) to identify what else may be driving the outcome — from other organizations, to government interventions, to external factors (e.g., weather, economic conditions), to individuals’ unobservable characteristics (e.g., motivation, cultural practices).
3. Experimental and quasi-experimental research: Enterprises can estimate depth contribution through quantitative techniques. These include methods such as randomized controlled trials (RCTs) that measure the average difference in outcomes over time among two randomly assigned groups:
   • A treatment group (i.e., receives the product, service, or other intervention); and,
   • A control group (i.e., does not receive the intervention, or receives a placebo or another type of intervention).
   The randomization aims to ensure that the two groups are similar on as many observable (e.g., income) and unobservable (e.g., intrinsic motivation) characteristics as possible. Although a popular method in international development, RCTs typically do not answer questions about the distribution of a change across a population or mechanisms of change. Moreover, experimental and quasi-experimental methods usually involve significant resources and require careful design to avoid research practices that exacerbate power imbalances and ensure the benefits of the research (e.g., learning) outweigh the risks (e.g., excluding people from accessing an innovation from which they would benefit). Experimental methods tend to concentrate attention on positive outcomes intended by the enterprise at the expense of attention to unintended negative outcomes that may be important to people and communities.
4. Theory-based evaluation: This involves critically examining the assumptions underlying the impact pathway, from inputs to outcomes and impacts. Theory-based designs aim to build a credible counterfactual by using a mix of both qualitative and quantitative methods, such as surveys and expert opinions. Theory-based evaluation encompasses many different approaches such as Contribution Analysis that provide a step-by-step framework to inferring causality.

Investors and enterprises aiming to generate insights into their impacts will often favor experimental research methods that provide strong evidence for causality. However, such approaches are often not feasible in the enterprise context due the resources they require and may not be ethical depending on the cultural context (e.g., having a control group may be inappropriate in collectivist cultures that value sharing of benefits).

Enterprises can also involve affected communities when determining how best to assess contribution to avoid overburdening or causing them unintended harm. Regardless of the chosen method(s), any reports of contribution should explain how contribution was assessed. The We All Count Methodology Matrix can help to identify effective and ethical methodologies that are well-suited to various types of research questions.

In practice, practitioners sometimes seek research done elsewhere and extrapolate the results to their own situation. When doing so, they should assess the generalizability of the original research by considering the study’s methodological rigor, population group, country setting, and type of intervention. For example, if an Indian enterprise relied on an estimate from an impact study that took place in Argentina – a country with significantly different socio-economic characteristics – then the generalizability of this estimate is likely to be low, rendering it unusable. In many cases, the data and research being used were not intended to be generalizable at all, and generalizing learnings could even be culturally inappropriate (e.g., extrapolating learnings from Indigenous communities who consider knowledge to be inherently place-based). Research methods may not have been attentive to equity considerations – for instance, the data may only represent men or white people – and thus not be relevant to other subgroups. Enterprises and investors should exercise caution in extrapolating research results to other settings and complement such extrapolation with community feedback and market research.