FAQ: Measuring Outcomes
The following is a compilation of selected recent postings on the npEnterprise Forum discussing the measurement of social and economic outcomes. If you’d like to remain up to date on these and other social enterprise issues, stay tuned to the listserv: to subscribe, visit www.npEnterprise.net. The npEnterprise Forum is the official listserv of the Social Enterprise Alliance. To become an SEA member, visit www.se-alliance.org.
Since the npEnterprise Forum is operated by volunteers, by publishing this we’re making no commitment to keep it updated as new postings arrive at the listserv.
From the moderators, Rolfe Larson and Andy Horsnell, www.RolfeLarson.com
Contents:
- REDF on Measuring Outcomes
- Social Capital Partners on Social Reporting
- Other Sources on Measuring Impact
- Software for Measuring Outcomes
- Some Longer Discussions
REDF (formerly The Roberts Enterprise Development Fund) is the acknowledged pioneer in the field of measuring social impacts. For that reason, it’s only makes sense to begin with a few postings about their work:
(1) From Joanna Messing <jmessing@redf.org> For the last decade, REDF has been measuring the impact of employment. Our findings and information on our methodologies are available on our website. We encourage you to check out what we have posted at: http://www.redf.org/results-intro.htm
(2) From Kate Sphar <ksphar@olszak.com> The REDF has developed an outcome tracking system called OASIS that it uses with its funded organizations to help them follow through on the SROI model - which is why it can seem overwhelming when one person is trying to implement it. It could be really hairy to attempt to keep track of it all on your own! For a simpler explanation of SROI/evaluation measurement tools and set of more manageable steps for creating one, you might consider reviewing the material put out by the Global Social Venture Competition for use by their contestants. This material is available at the Resource Library on their website: www.socialvc.net
(3) From Cynthia Gair <cgair@redf.org> We're always glad to hear that REDF (formerly The Roberts Enterprise Development Fund) materials are motivating others to engage in social enterprise and measurement and we're also aware of the challenges involved! We came up with our SROI model a number of years ago and in the process we recognized that it has both strengths and shortcomings (one of which is that it can be complex to implement). In our own work since then we've migrated to looking at financial/business and social results separately even though we consider them equally important. We've zeroed in on the key information underlying our SROI analysis - the measurement of social outcomes - and we are eager to find ways of doing this in a more streamlined way. Bravo to you, Jodi, and to all of you who contribute to this work!
(4) From Rolfe Larson <rolfe@rolfelarson.com> "Measurement, Why Bother?" is the title of a recent newsletter article by Melinda Tuan, managing director of the Roberts Enterprise Development Fund. She points out that there are things that nonprofits can and should measure, to evaluate progress in accomplishing their missions and in achieving the full potential of their social enterprises. Visit: www.redf.org/pub_newsletter.htm.
Social Capital Partners
on Social Reporting
Another leader in this field is Social Capital Partners. Here are a couple of postings about their work:
(1) From Joanne Norris <joanne@socialcapitalpartners.ca> Social Capital Partners has been involved in this work for 4 years now. We have our reporting up on our website and have just posted a companion piece that goes into more detail on our social return on investment methodology. I would also be happy to talk to you about our experience.
(2) From Vicki Scully <vickiscully@shaw.ca> Social Capital Partners have produced a document on Social Reporting that is available through their website at: http://www.socialcapitalpartners.ca/ideas.asp
Moderator's note: Vicki is referring to the document at the bottom of that web page entitled "9/28/05 - SROI Reporting Companion Piece." Here's the description that accompanies that title: "We have been doing SROI reporting with our portfolio for the past three years and have now learned a thing or two about the benefits and challenges of measuring social returns. For a detailed look into our methodology, how we do the calculations and our learnings along the way, please take a look."
Other Sources on Measuring Impact
(1) From Laurance Allen <lallen@bateleurmedia.com> There is great online help for free and well worth exploring [including a 24-page “Evaluation Plan Workbook”] at: http://www.innonet.org
(2) The Moderators also suggest the following general outcome evaluation resources:
-
The United Way of America’s “Outcome Measurement Resource Network” at http://national.unitedway.org/outcomes/
-
The “Basic Guide to Outcomes-Based Evaluation for Nonprofit Organizations with Very Limited Resources” at Authenticity Consulting’s Free Management Library at http://www.managementhelp.org/evaluatn/outcomes.htm
-
The “Evaluation & Outcomes” section of Social Venture Partners Seattle’s Resource Library at http://www.svpseattle.org/resource_libraries/Outcomes/Outcomes.htm
(3) From Roland Burkett <rburkert@rolandconsulting.org> Check out my website at www.rolandconsulting.org for samples of economic impact analysis that includes social enterprises.
[Moderator's note: the samples he refers to can be found at the following location: http://www.rolandconsulting.org/modules.php?name=DEMO_Start ]
(4) From Kelly Marie Weiler <KMWeiler@aol.com> The Ansell Casey Life Skills assessments, activities and resources available for free at http://www.caseylifeskills.org/ are exceptional. Even though they were conceived for "youth", they are developmentally appropriate for adults with barriers to employability.
Software for Measuring Outcomes
This is the geek section of this FAQ. More seriously, technology is a necessary part of any strategy for measuring and reporting social impacts. Here are several postings on that topic:
(1) From Annie Betancourt <alord01@yahoo.com> Check out “Efforts to Outcomes” (http://www.socialsolutionsonline.com), “Foothold” (http://www.footholdtechnology.com), and “Evolve”.
(2) From Joe Jurczyk <joe@grassroots.org> This 1998 paper includes a list of then-current software alternatives for outcomes measurement as well as ways of incorporating IT into the measurement process. Collins, Michael. "Using Software Systems to Measure Nonprofit Program Outcomes: Assessing the Benefits and Barriers for Strategic Management." 1998. http://pages.prodigy.net/michael_collins/Outcomes_IT.htm
(3) From Carlos Gasca <communityatwork@comcast.net> I would suggest that you record your current paper process and look at the type of documents you need and are producing to measure outcomes. Identify the essential data that you are looking for and want to report or manage. Having a clear picture of what you are doing and need to do is key to selecting software. I also had to figure out how to use software to measure outcomes. Basically we selected Filemaker Pro and developed our own database. The sources of data were our enrollment forms and customer service requests, customer satisfaction surveys, and annual telephone survey. We organized this data in the database so we could meet reporting requirements and manage our program development (marketing). To develop the database we hired two university students during the summer. Filemaker pro is really easy to use and they were able to develop a very sophisticated product. The key is knowing what information you need and the type of reports you would like to generate. Check your paper trail!
(4) Sri Sridharan <infinisri@gmail.com> I would like to echo Carlos comments. I am an IT professional. Before looking for a software solution, it is really imperative to figure out what you are going to assess, what the data is and how you are going to collect it. If there is clarity in purpose and means, the software needed to impart some efficiency to all that can be configured. It is ok to start with simple spreadsheets, and then mature to databases (File Pro being one) or to look for some packaged solution.
When moving to the latter, be very careful to retain control of your data and don't let it be stashed in proprietary formats without ability to export or migrate it out. I have seen many a time the baneful cries of organizations which could not migrate from one software package to another because they could not carry their past data forward. Database should be transparent. It is your data. Don't let someone else "own" it.
(5) From David Carleton <david@kitchenwithmission.org> I've been involved in two efforts to create databases that can be used from initial application through SROI or longitudinal tracking. Both use MS Access. The first effort with FareStart (Seattle) was overly complicated and resulted in a "heavy" d-base that is cumbersome and hard to work with/adapt. The original creator is no longer in Seattle and we've had a hard time finding someone to work on it. The second effort, with Life's Kitchen in Boise, seems to be headed in the right direction and the goal is to end up with an "as-simple-as-possible" d-base that both gives us what we need and can be easily updated and adapted - by anyone that knows Access. The key is to make sure that the d-base is constructed in a way that folks other than the creator can easily understand...
While the effort in Boise is not complete I'm happy to send screen shots to give an idea of how it is being set up. While we are setting it up to be relevant to our culinary job-training and placement programs, the approach should be that different than any other - the comments from Carlos Gasca were well stated.
(1) From Joe Beckman <joe.beckmann@gmail.com>
Julia,
The Roberts Fund has some very intriguing research on economic and social returns on investment. And it does make immense sense that this kind of evaluation be pursued to identify levers or pry-points where small investments produce large outcomes. Yet, even with the sophistication of the REDF findings - or those of Soros or other social enterprise/venture philanthropies - it seems they under-value or do not attach serious enough significance to some of the social benefits of economic failure. As Arianna Huffington's shadow conventions highlighted, this rising tide has failed to raise all boats, in spite of the rhetoric of Republican and Democratic affluence. What is the economic equivalent, for example, of reducing the number of doubled up families in public housing, or reducing the number of multiple jobs held by low-income people to afford adequate childcare? There is or there ought to be some clear measure or standard, and that standard ought to reflect positively on the local economy and attract new business, etc. Yet without adequate market information, how can new business or investment flow to successful centers, or enrich less successful centers to move to higher levels of productivity?
Does the REDF anticipate producing regional market documentation to encourage this kind of investment? Should it be the role of government to measure this kind of economic and social productivity, and provide a kind of "consumer reports" on localities needing and responding to investment, vs. others not in need and still others misusing that investment? Are these reports equivalent to "enterprise zones" and available already, or are they like the old "Model Cities," and already corrupted?
I find a lot of the literature of venture philanthropy very short on history and very long on a potentially spurious theory that entrepreneurship solves all problems. Over 20 years ago Prince Charles' Prince's Trust was active in investing in social entrepreneurship, and it all evaporated when people got private sector jobs with reliable salaries. The theory underlying Junior Achievement is fine, but the regularity and routine, which enables small business to succeed, is usually unavailable to low income people with higher needs than resources. Single, largely white but often immigrant, entrepreneurs, with extended families for cheap labor but without much other direct parenting responsibility, tend to succeed quite well through start-up phases. It's a very different culture for low-income single parents with inadequate child support or easier access to substance abuse than adequate childcare. (Joe Beckmann)
(2) From Julia Jones <julia.jones@comcast.net>
Joe,
I found your message to be really interesting and thought provoking. You mentioned the REDF results. We have been thinking a lot about issues like the ones you raised. FYI, new information will be coming out by the end of the year.
Here are my personal thoughts on the issues you raised.
First, REDF does not claim to have the total solution to the issues. We do not think that someone should look at a simple return on investment number and make a decision. We are trying to put information out to get reactions such as yours so as a community we can advance our thinking.
That said, REDF's research is not focused on identifying "levers or pry-points where small investments produce large outcomes". Rather, we are trying to find ways to measure how much impact an investment has. This may sound the same to you, but the difference to me is that this is not really about the size of the investment -- but making sure that any size investment has as much impact as possible. Right now many nonprofits measure the impact of their program based on how many people were served, etc. This does not tell what the impact was on the people or on the community. We are trying to really find out the impact of the work of the social purpose enterprise on the individual and the community/public.
We are also trying to find ways to demonstrate the tangible value of the work of the social sector in order to increase access to capital. Historically, many nonprofits have been funded using grants -- which has left them subject to the whims of the funding community and frequently has resulted in the underfunding of organizational infrastructure (many foundations use to only fund programs -- not general funds). Earned income strategies have been around a long time -- but it can be difficult to focus on both the social mission of an organization and running a profitable business. We are trying to quantify the benefits provided so that investing in nonprofits becomes more attractive to a wider range of individuals.
However, a shortcoming in trying to come up with a number is that many things aren't quantifiable. Our work so far has focused on Social Purpose Enterprises -- businesses run by nonprofits to provide supportive employment to people with barriers to employment (like mental illness, homelessness, etc.). Within this field, there are benefits we can quantify and benefits that are more difficult. For example, we can quantify increase in wages for an individual (and the corresponding tax contribution to the community), or the decreased reliance on public assistance. We can measure whether the person is financially better off after participating in the program (do their after tax wages and benefits more than make up for any public assistance/medical that they have lost). However, it is very difficult to quantify the benefits of no longer doubling up families in apartments or the generational effects of being employed. The way we deal with this is by saying that no one should make a decision by focusing on just one number. Look at the quantifiable results but also look at the change in the person’s life -- has their housing situation stabilized, do they have social support, etc. All of this information and more should be considered.
At the same time, I do not believe that entrepreneurship is the answer to all of our problems. Even if getting a job with a social purpose enterprise improves the situation for a person, this improvement only goes so far. For example, in San Francisco, a person would have to make $30 per hour to only spend 30% of their income on rent for an average 2-bedroom apartment (and work a 40 hour week). Many businesses which employ people with significant challenges to employment do not have margins that would support paying $30 an hour plus benefits. It frequently costs more to employ these workers than others without the same challenges. I work at a nonprofit and I don't even make $30 an hour. In the for profit world, people are constantly saying that paying a living wage would make them economically uncompetitive. Non-profit run businesses cannot really have a totally different cost structure and compete with the for profit businesses. If nonprofits pay more, their prices would typically have to be higher -- unless they have reached a point where they are very profitable -- however many businesses (both for and non-profit) NEVER become profitable. So...we need to do more than just create jobs to really improve our communities. Social entrepreneurship, so far, has not created the wealth in inner city communities that I would really like to see. I'm not sure what the answer is right now. I just know that we are not completely there yet. In the San Francisco Bay Area, the gap between the haves and have-nots seems to be growing. The area is becoming less and less livable -- and I am not sure what we do to reverse this trend.
(3) Jeremy Gregg <jeremy@ jeremygregg.com> From Depending on the type of housing that you provide (i.e. affordable housing, emergency shelter, transition for at-risk youth, etc.), you might gather some data about the homelessness of your target population throughout the city and make an argument that, without your programs, there would be increased homelessness. In other words, you are providing housing to people who may not otherwise have access to that housing if it were owned/operated by a for-profit company (which might operate it as a high-rent facility that is inaccessible to the working poor).
Along this same line, you might be able to construct a rather compelling argument for the existence of your business as a viable strategy against gentrification's impact on your neighborhoods. Many cities are finding that the poor are being pushed out of their traditional neighborhoods due to rising real estate prices; developers are not building a lot of housing that is accessible to these populations. If you could measure the amount of supportive/affordable housing in your area compared to the need, you will probably find strong evidence of the need for your business.
Of course, this is all predicated on the assumption that your properties are being operated in a way that complements your non-profit's mission (i.e. providing supportive/affordable housing to families that might not otherwise have it). This won't work if the properties are just being operated as a source of revenue, and the housing is not intended to serve as an affordable housing strategy.
If this is not the case, you might want to consider measuring the economic development aspects of your properties. What was the neighborhood like before you operated the property, and how have you positively changed it? Has your effective management of the properties lifted up the neighborhood? Have non-contributing businesses been replaced by business that support a stronger community?
There might also be some complicated ways of measuring increased employment in the area, increase in tax base, etc., but I think that it would be much better to focus on the changes in your residents.
(4) From Mary Ferguson and Janet
Murray <info@ekonomos.com>
We’re excited to share with the SEA
Network, an electronic publication called The Business of Inclusion, a series of
papers that summarizes our learning from the developmental phase of the Toronto
Enterprise Fund (TEF),
http://www.torontoenterprisefund.ca
Since its inception in 2000, the Toronto Enterprise Fund has supported the development of 15 social purpose enterprises working with homeless and low-income people. TEF has also invested in research and evaluation to document and disseminate our learning about effective practices in developing enterprises and to assess what happens to the low-income people who are involved in those enterprises.
We’ve been following the recent conversation thread that discussed the evaluation of social returns in this work. Over the past five years we’ve been using an asset-based Sustainable Livelihoods framework to explore five critical dimensions of results. We’ve had a great deal of success using participatory and collaborative research methods to document outcomes (particularly the harder to measure qualitative indicators).
The Business of Inclusion explores both sides of the double bottom line. Some papers look at the business side of the equation, supporting feasibility assessment and business planning, but it is the papers that look at the social side of the equation that we think provided the most significant learning.
We encourage those who are interested in learning more about low-income people’s transition out of poverty to have a look at the following papers in particular. We’d be very interested in hearing what you think.
Report 10: Participant Outcomes (http://urlsnip.com/371644)
Report 7: Stages of Livelihood Development through Social Purpose Enterprise (http://urlsnip.com/536427)
(5) From Kath Critchley <kath.critchley@socialfirms.org.uk> SROI & it's various calculation methods & restrictions are the subject of much current discussion in the UK. The New Economics Foundation (nef) (www.neweconomics.org) has been researching the matter quite extensively with input from a pan European Group whose inaugural meeting was in Edinburgh in June.
Also, as part of a program to develop quality & impact tools aimed at 'proving and improving' social enterprise development in the UK, the Social Enterprise Partnership (www.sepgb.co.uk) have been piloting a stakeholder version of the Balanced Scorecard in which the measurement of social return forms a key perspective. Rather than running with a predetermined set of measurements for the BS, Social Firms Scotland is piloting in the Highlands & Islands a more tailored approach to balanced scorecard development, which has much emphasis on the enterprise clarifying its strategy before trying to measure progress towards it. We use adapted strategy mapping techniques, which then lead on to the enterprise devising their own balanced scorecard.
Inevitably, every social enterprise needs to justify its social return. However until there is a standardized way of calculating such return, almost every enterprise will have differing rates for, e.g. personal taxation, welfare benefits saved etc. Consequently, even though the method of calculating SROI may be fairly fixed (if not yet agreed upon) the inputs for the calculations will vary significantly resulting in something, which is useless as a basis for comparison between enterprises. The comparison within an organisation is possible once methods and rates are standardized for a certain organisation but without an identified need (or strategy) to measure the impact, few enterprises actually bother to measure it.
This was our thinking when we took the 'step back' to assist social enterprises in defining their strategy before measuring it. In doing this, the enterprise prepares itself for not only balanced scorecard implementation, but also for 'true' SROI, quality program implementation and both social and financial auditing. See attached diagram of implementation. (WWW = Working with Winners, our strategic management training & balanced scorecard development program).