New Frontiers | Our WorkImpact Ratings Math

This discussion board provides a forum for exploration of different ways to calculate an expected impact rating for prospective investments. Its goal is to build on points made in earlier discussions about impact ratings facilitated by the IMP. (For a synthesis of those discussions, see the IMP Discussion Document “Impact Ratings: Quantified not Monetized,” which provides definitions, use cases, benefits, and risks and challenges of impact ratings.)  

Investors can use impact ratings to identify which investments offer the most of the types of impact they care about as an input to investment decision-making. Impact ratings help to synthesize and simplify the multi-dimensionality of impacts into a common unit in order to compare the expected impacts of different investment opportunities.  

When constructing an impact rating system, investors grapple with a series of choices: 

• Which aspects or dimensions of an investment’s impact they will include in their rating; 

• Which metrics, data sources, and criteria they will use to assign a numerical sub-score to each aspect or dimension of impact; and  
• How they will add or multiply these sub-scores into an overall impact rating 

There are no ‘right’ answers to these questions. Individual investors work out approaches that suit their goals, strategy and context. Part of the value of wrestling with these choices is in prompting investors to articulate explicitly the assumptions, beliefs, and preferences that might otherwise remain implicit (and potentially vague or inconsistent across team members and over time).  

For many investors building their own impact ratings, collaboration to share approaches and discuss common challenges has proven helpful. Among investors in Impact Frontiers cohorts, we commonly see three approaches, illustrated here using the five dimensions of impact for simplicity: 

Approach 1: Weighted Average  

Investors who use the Weighted Average approach score investments’ expected or actual performance on each aspect or dimension of impact on a quantitative scale (e.g., 1-10) and then assign a weight out of 100% (e.g., Enterprise Contribution = 15% of total rating, Impact Risk = 20%, etc.) to each aspect or dimension of impact. They then calculate a weighted average of these scores. Some investors use a normalized scale across dimensions of impact so that similar scores or ranges imply similarly high or low impact performance.  

Example: 

 (6⋅0.10) + (9⋅0.40) + (3⋅0.25) + (5⋅0.10) + (2⋅0.05) + (6⋅0.10) =

 6.156⋅0.10 + 9⋅0.40 + 3⋅0.25 + 5⋅0.10 + 2⋅0.05 + 6⋅0.10 = 6.15

This approach seems to be the ‘just-right’ level of complexity for many investors and it is the most common one among Impact Frontiers cohorts. However, one limitation is the tendency for weighted averages to revert to the mean. Different investments may score highly on different dimensions of impact, such that their weighted averages tend to cluster near a middling score. To avoid this phenomenon, some investors have experimented with giving disproportionate weight to the dimensions of impact that are most important to them or that most effectively discriminate between high and low impact investments within their portfolios. 

Approach 2: Group and Multiply 

Investors who use the Group and Multiply approach also assign a score and a weight to each aspect of impact performance. Rather than take a weighted average of these scores, however, these investors bundle together groups of scores (e.g. [What, Who, How Much, and Enterprise Contribution] [Investor Contribution] [Impact Risk]), and then multiply the summed or averaged scores of these groups to produce an overall rating. 

Example: 

Enterprise Impact Sub-score 

(6⋅0.3) + (9⋅0.3) + (3⋅0.3) + (5⋅0.1) = 5.96⋅0.3 + 9⋅0.3 + 3⋅0.3 + 5⋅0.1 = 5.9

 Investor Contribution Sub-score 

 Impact Risk Sub-score 

𝐼𝑚𝑝𝑎𝑐𝑡 𝑅𝑎𝑡𝑖𝑛𝑔 = (𝐸𝑛𝑡𝑒𝑟𝑝𝑟𝑖𝑠𝑒 𝐼𝑚𝑝𝑎𝑐𝑡) x (𝐼𝑛𝑣𝑒𝑠𝑡𝑜𝑟 𝐶𝑜𝑛𝑡𝑟𝑖𝑏𝑢𝑡𝑖𝑜𝑛) x (𝐼𝑚𝑝𝑎𝑐𝑡 𝑅𝑖𝑠𝑘/10) 

(5.9) ⋅ (2) ⋅ (610) = 7.085.9 ⋅ 2 ⋅ 610 = 7.08

This approach reflects many investors’ intuitions that the total impact of an investment is a product, rather than a sum, of enterprise impact and investor contribution; and that impact risk should act as a ‘discount factor’ that adjusts the score based on the likelihood of those expected impacts.   

This approach tends to generate a wider dispersion of scores than the weighted sum approach, overcoming the ‘reversion to the mean’ challenge.  Some investors shy away from this, as they find it disconcerting to rate some investments’ impact as being tens or hundreds of times greater than that of other investments. Other investors find it intuitive that some investments have orders of magnitude greater impact than others.  

Approach 3: The ‘No Single Rating’ Rating 

Some investors prefer not to aggregate the various scores in their impact rating into a single number, and elect for disaggregated, multi-dimensional comparisons of the expected impact performance of different investment opportunities using visual tools like radar charts. 

Example:  

For organizations working on a smaller number of deals, the disaggregated comparison approach makes room for more case-by-case decision-making. Although this approach may not on its face seem to be an impact rating at all, investors still go through most of the steps of the impact rating approach (i.e., deciding which dimensions of impact to consider, and scoring each dimension in a consistent way across investments). They simply omit the final step of calculating an overall rating. 

. . . 

There is ample room for innovation within and outside of the approaches described above. 

For example, at Impact Frontiers we’ve been exploring a variant on the Group and Multiply approach that more explicitly takes account of enterprise-level counterfactuals:  

 Discussion: 

The goal of this forum is to create a sandbox for consideration of approaches to impact ratings, in which people can discuss the merits of different approaches in different contexts, and offer up new ones for consideration by peers. In early 2023 we will review this discussion and synthesize the insights that emerge. 

1. What formulas are your organizations using in your impact ratings? 
2. What do you see as the advantages and disadvantages of these different approaches? 
3. What other approaches have you seen? 
4. In which context is each approach most appropriate?