Opening Pandora’s Evaluation Box
By Allyson Hewitt, JW McConnell Family Foundation Senior Fellow, Social Innovation at the MaRS Discovery District
April 26, 2016
“Revolutions in science have often been preceded by revolutions in measurement.” Based on the premise from Sinan Aral of the MIT Sloan School of Management
Jason Saul presented to a full-house as part of the MaRS Global Leaders series in April 2016 on his latest venture – the Impact Genome Project (IGP). A public-private partnership to code and quantify the “genes” of what works in social science. The audio of the presentation can be found below.
If you spend 5 minutes in the social impact sector you are sure to be asked, how do you know you are making a difference?
Jason and his colleagues at Mission Measurement have been tackling this question by taking us from the current state: we are spending $400 billion to achieve social outcomes without any standard way to accurately measure ROI; the evaluation industry is in disarray; evidence is unstructured and unintelligible; and yet evidence is growing exponentially – it is just not readily accessible. We have no common language; no benchmarks that allows us to compare social programs; and ultimately no predictive data meaning we can’t forecast before we invest. This is what Mission Measurement calls the black box problem.
Yet other sectors have predictive data and use it to increase their impact: think credit scores, the human genome or even Netflix. The music industry has cracked this code with Pandora and their Music Genome Project, the original inspiration for the Impact Genome Project. Jason approached Nolan Gasser, the architect of the Music Genome Project, and together they embarked on a journey of discovery asking one question: Can we not do for social programs what Pandora did for music?
As it stands the Impact Genome Project comprises 11 total genomes: education; economic development; public health; youth development; international development; human services; criminal justice; sustainability & environment; science & technology; arts; and culture & identity. With 132 common outcomes. The goal of IGP is to produce new benchmarks such as efficacy rate; expected outcomes; and cost per outcome. It is an open data project with advanced analytics available via subscription.
The IGP intends to create a more level playing field by:
- Democratizing evaluation
- Replacing guessing with data
- Learning systematically across the sector
- Unleashing innovation and creating twice the impact with half the cost
It is an audacious goal and yet the future is here. The UK government is already moving to pay for outcomes and have created a What Works Centres, a network of centres to “support more effective and efficient services across the public sector at national and local levels.” Our own governments are not far behind with the Centre of Excellence for Evidence-based Decision Making Support at the Government of Ontario, which was part of Minister Deb Matthews’ mandate letter.
- Un-entrenched philanthropic institutions
- Integrated and collaborative philanthropic sector
- Government prioritizing evidence and value for money
- Institutions willing to lead
- Access to top talent/academic institutions
- Systems-thinking expertise
We are interested in what you think. Does this seem like a way to get ahead of the inevitable move to pay for outcomes? Can we work with funders to make this approach the standard, not the only way forward but one that is “directionally correct”? What are your concerns, if any?
Please let us know and help us determine how we can get to a better place around demonstrating our impact in a world that needs us to use all our talents to tackling our complex challenges.
Consultation has concluded
