We multiply because we want our impact ratings to be in line with the potential scale of our impact as investors. Our general theory of change, if put into an equation, is that:
Investor Impact per dollar = Gross Impact per dollar x Enterprise Contribution x Investor Contribution
Gross impact per dollar is the amount of outcomes per dollar size of the enterprise, without adjusting for the contributions (i.e. additionality). We think of this as another multiplication: ‘Number of stakeholders’ x ‘Amount of change experienced’ x ‘Length of change’ x ‘Importance of the change’. We assess these components individually; we find it is helpful to be more granular and specific than the standard ‘What’, ‘Who’ and ‘How Much’.
So, we are basically all about multiplication! That said, if an investment involves multiple impact pathways (e.g. both from our capital and our engagement, or impacts on different outcomes) we would do separate multiplications for each pathway and sum these results together.
Another key feature of our impact math is what we call ‘scale-sensitive’ scoring. For simplicity, we like to score each dimension on a score of 1-5. But we realize that the actual values 1-5 do not represent how much better we think better scores really are. So, before we multiply we convert our simple scores into scale-sensitive values. The conversion depends on the dimension. See examples in the attached tables.
The result is that our ratings have plenty of dispersion. Both because we multiply and because we use scale-sensitive scores.
In the past, instead of scale-sensitive scores, some of our collaborators have stuck with simple scores. Even if they agree that the actual difference between a “3” and a “4” may be 2x and not just “1 point”. But, we have found that this makes it easy to forget the dramatic dispersion in investor impact that we observe between opportunities. It makes it easy to spend time chasing an opportunity you wouldn’t have prioritized with scale-sensitive scores.
We assess enterprise contribution and investor contribution in terms of dimensions that we call “Scalability” and “Neglectedness”.
Scalability is how productively additional resources can be used right now – it’s no good throwing more money or time at something if there is some other bottleneck.
Neglectedness is the extent to which the supply of resources to an opportunity is limited. It is inversely related to how much money the market is allocating to an enterprise or industry, either because of risk, lack of awareness or other frictions.
For enterprise contribution, we assess these dimensions for the industry or area to which the investee organization is contributing. For investor contribution, we assess them for the investee organization itself.
We aim for our contribution scores to be conservative. We use percentages between 0-100% (though if we thought we were having really large or negative impacts we could go outside this range). This makes the ultimate rating scores often quite small. But we like this because it reminds us to be humble about how difficult it is to generate impact. We could multiply the end ratings by a ‘normalization’ factor if we were worried about legibility (e.g. ratings not fitting into a common range like 0-10).
Note that our ratings math doesn’t require ‘weights’. We like this because then we can focus on debating the scores on each dimension instead. That said, we have also experimented with different team members assessing their own scores for an opportunity and then taking a weighted-average of these scores, weighted by the relevance of each analyst’s expertise – though we don’t yet do this routinely.
We don’t directly include impact risk in our ratings math. Instead we assess the lower & upper overall rating we would plausibly apply to an opportunity. This gives us an uncertainty range for each opportunity. Then when comparing between opportunities we compare both the main estimate, and their uncertainty ranges. If one opportunity’s entire uncertainty range is above another’s, then the decision is easy. If there is a lot of overlap then we need to be more careful. We find using these ranges to be a really useful tool for directing our conversations.
These ranges also highlight one of the fatal flaws in weighted-averages, from our perspective. Because the ‘reversion to the mean’ effect means that not only do great opportunities stand out less, but that with weighted-averages almost all opportunities overlap in terms of their uncertainty ranges.
By the way, we also apply similar ratings math at the industry/sector level to help us prioritize which areas we target in our search for opportunities. We are considering doing this for SDG sub-goals and we’d be happy to talk with you if you’re interested in collaborating on that.
I’m attaching two example tables and a separate visual representation of the difference between multiplying and adding when constructing impact ratings (from an internal exercise).
Excited to see these topics getting discussed!
