Are we really measuring impact?

Antonio and his wife with their new babyHaving read various Duflo and Banerjee papers (see their book Poor Economics), I understand the frustration expressed in this blog post about measuring impact investments. As the post author Anya Kamenetz points out, in measuring our efforts social entrepreneurs incorrectly equate monitoring impacts with measuring the actual impacts created by the initiative.

I’ve always been a bit exasperated by the talk of impacts, and I suspect other social entrepreneurs are, too. It’s tedious to fill out impact performance metrics, especially when those metrics aren’t directly applicable to your project. That doesn’t mean we don’t believe our projects are making a difference or that monitoring isn’t important to do. It’s just that some of the “apples to apples” metrics that allow investors to easily compare projects don’t really quantify the actual impacts.

Over the past few weeks the impact investing world has been buzzing about the launch of GIIRS, the Global Impact Investment Ratings System. We’re hopeful that GIIRS will draw investors who have stood back from impact investing because of its lack of standardization. It’s good for our business.

But will GIIRS enable investors and social enterprises to really quantify their impacts? We are cautiously optimistic, but some of the metrics still fall short. One that we use is the amount of hectares planted. This is easy to measure, but is only quantitative, not qualitative. It doesn’t consider that a 1000 acre monoculture teak plantation might be a net loss for communities and the environment when considering their characteristically high use of chemicals and lack of biodiversity. Context is important. Another common metric is jobs created. Sure, Planting Empowerment generates employment, but people were already laboring on the land we planted, so how many jobs have we actually created?

One of our advisers, Dr. Grace Goodell, taught me the techniques that Duflo and Banjaree use to correctly draw conclusions of impacts generated by specific activities. Planting Empowerment did a small baseline study of our partner and surrounding communities to measure the impacts from our activities. However, our startup budget restricted us from doing it at scale needed to be scientifically valid the way Duflo’s is. Kamenetz notes that effectively measuring impacts would require ~30% of your budget. That would kill the projected returns on our investments.

Impact investors should be mindful that the metrics they’re using to evaluate investments don’t fully capture the impacts. We should be trying to improve this, but for now let’s embrace the fact that to do proper measurement costs more than we are willing to spend.

Ongoing reporting is important, and Planting Empowerment will continue to report on metrics required by investors because it is necessary for the growth of our business. But our partners’ ongoing support for our work is even more important.