If only it were this simple

(This blog is co-published with Community Resource Exchange, and is part one of a two-part series)

Ask anybody, and I mean anybody, about evaluating the effectiveness of nonprofit service providers and you will be greeted by winces, whinges, shrugs, groans, gnashing of teeth, sighs, and the burying of faces in flattened palms.  And by anybody I don’t just mean any nonprofit service provider – I mean as well our beloved philanthropic leaders and public sector partners.  After all, we’re talking about an industry that in 2009 paid 9% of all wages, contributed 5.4% of GDP, reported revenues of $1.4 trillion, and held some $2.56 trillion in assets.  Why can’t we find a way to tell how effective this industry is with our money (both public and private dollars)?  How hard can it be?

And yet, after spending 10 years in the philanthropic sector myself, then another 4 years running a major nonprofit, I can tell you that overall evaluation strategies remain fragmented and gap-toothed at best, contradictory and bureaucratic at worst.  In spite of our many admonitions that the nonprofit sector should better emulate its much more handsome and accomplished for-profit sibling – by being more entrepreneurial, more data-driven, more outcome-focused, more scalable, more leveraged, more replicable – the nonprofit sector continues to behave as if it were, well, more folksy, more grass-rootsy, more (you know) messy.  Don’t get me wrong: the past 20 years have seen the emergence of many talented and innovative nonprofit leaders, many highly effective and complex nonprofit service providers, and many have given for-profit competitors in their niche a real run for the money.  But if you were to step back and look at the nonprofit industry in toto and ask yourself the dreaded “but for” question, could you really prove that nonprofit service providers are making a difference – or at least the differences that they claim in their annual newsletters?  Shouldn’t we have hard facts?

Follow the Money

Here’s how capitalism works:  the parts of the market that generate more market activity get more market resources.  That’s the point, right?  Those folks who can provide a clever solution to a defined need at a good price are rewarded with venture capitalists, IPOs, and highly compensated executive managers.  By comparison, most nonprofit providers seem highly inefficient:  they are labor- and cost-intensive, and tangible results (higher literacy, reduced obesity, better job-preparedness) are difficult to observe and measure.  This is especially true in terms of return on investment:  how do you monetize the value of improved attendance in high school, better self-esteem, less street crime?

The Wrong End of Capitalism

And here we come to the crux of the matter:  most nonprofits do not reward capitalism.  Put another way, the goods and services that nonprofits provide make very little money, or cost more than they generate in tangible profits.  We make the mistake, therefore, of attempting to apply market driven measurements and evaluative tools to businesses (and nonprofits are businesses) that operate on the margins of capitalism.  Any by the margins I mean the parts of capitalism where resources are scarce, fragmented, episodic, or plainly NOT driven by an expectation of direct financial return.

Nonprofits choose to operate at the margins of capitalism.  While this should be an obvious fact, I’m impressed by the number of times it seems to go unobserved.  Nonprofits tend to define their very role as working with those most vulnerable, needy, forlorn and forgotten by the market’s drive for revenue growth and capital concentration.

Look, I know I’m sounding pretty old school socialist here.  Well, too bad.  It’s the truth.  And what’s more, in our collective desire for capital efficiency, and in our general distrust for those who are vulnerable, needy, forlorn and forgotten, we fall into the trap of conflating the scale of the nonprofit industry with any other type of profit-motivated industry.  Which it is not.

Why Evaluation Stinks

While there are those who make claims to evaluation models that reduce complex inputs to measurable economic impacts, I’ve seen deep inside those models and (let me tell you) while they are very creative, they are hardly, well, rocket science.  There’s great talk about counterfactuals and weighted variables, but when it comes right down to it you’re still ultimately stuck with making your best guess.  There is some value in this, to be sure, but to claim these ultimately define success through apples-to-apples comparison just ain’t so.

And here’s why:  if we were to round up all the nonprofit service providers who, say, assist the poor in accessing public benefits, we would wind up with a list of groups that are all trying to do one thing, but who accomplish that task in widely different ways.  Some would focus on grass-roots strategies to engage their service population, while others would access clients through referrals from public agencies, while still others would “cross-market” these supports to clients being served by other programs it operates.  Some would combine several of these strategies, along with three or four others I haven’t been clever enough to think up.  All of them would have multiple funding streams from public and private sources supporting this work (government contracts from city, state or federal providers; grants from private foundations and wealthy individuals; large-scale fundraising events; revenues thrown off by subsidiaries developing real estate assets to service or house low-income populations).  Finally, all of them would be working in different communities with tremendous variety in social support networks, economic challenges, and myriad other factors.

In short, they would all look wildly different.

The goal may be the same (e.g., to help an eligible low-income person sign up for food stamps), but the capacities for delivering these supports are radically shaped by the supports received to deliver the services.  Get it?  By operating on the margins of capitalism, nonprofit providers voluntarily submit themselves to the same (cumbersome, inefficient, fragmented, mercurial) process of seeking resources as the very people they are trying to serve.

The Power of Collaboration

Just this morning I attended a very compelling conference put on by Association for Neighborhood and Housing Development, Neighborhood Housing Services of NYC, and my alma mater the Center for NYC Neighborhoods entitled The Power of Collaboration.  The title of the event comes from the very engaging article “Collective Impact,” published by the Stanford Social Innovation Review.  Its authors make the argument that only through collaborative efforts can funders and nonprofit providers rise above the fragmentation of their service niche.  The lay out a helpful framework include the following four components of effective collective action:

1. Develop a common agenda:  Agreeing upon both goals early on is critical, and this is most effective when all levels of the collective are engaged from the front lines, through nonprofit leadership, to stakeholders and funders.  Not only will the agenda have greater depth and strength, but the collective itself will carry greater momentum.
2. Share measurement systems:  Common goals are only as good as the success measures put in place to monitor impacts.  Keep the list of data points short, and if you can piggyback on existing data collection protocols to get your info, all the better.
3. Develop mutually reinforcing activities:  It’s not that every collective member needs to do the same thing, but rather that all participants should take up complementary actions and coordinate to provide comprehensive services.
4. Collective action requires infrastructure:  To do the job right, you frequently need a team dedicated to the task of keeping the collective itself, well, collective.  Communications, data collection, planning and training are best managed through such a hub.

I love this.  Makes complete sense.  But here’s the rub:  it places the onus for collective action on somebody somewhere taking leadership.  And by leadership I mean spending volunteer time and resources to organize partners and engage in planning.

A Thousand Points of Light

One of the great things about operating on the margins of the economy means that we are very used to taking such leadership and creating “something out of nothing” by aggregating the resources we have in greater abundance:  good will, a commitment to mission, the power of the public good, and people who are willing to sacrifice comfort (money, sleep, diet) for all of the above.  But there are two basic challenges:

1. A Thousand Points of Light:  We essentially compete with ourselves to lead, to aggregate, to create collective action.  We end up competing for good will, commitment to mission, the power of the public good, and so on because those are the resources we do have access to (slim as they are).  We mark territories and fear collaboration because we don’t want to lose ourselves or our position in the process.
2. A Thousand Points of Night:  In order to succeed we have to be willing to extend ourselves beyond our usual capacities.  The great thing about having a lot of money and resource is you can always compensate someone to do something for you.  When you don’t, you have to do it yourself or it doesn’t get done.  That means you both push well past your comfort zone, and you probably have to take away from the mission critical work itself to leave room for addressing the bigger picture.  In short, things will have to get harder before they get easier.

Why Evaluation Stinks (part 2)

The problem of creating effective evaluation frameworks goes way beyond the problem of creating effective evaluation frameworks.  Competing for scarce resources on the margins of the market subjects all of us (philanthropy included) to the pressures of never quite having enough.  It requires leadership, true, but leadership that sees beyond itself to a collective action in which its individual role by definition must be diminished simply because it must be shared.  There is no other way to achieve scaled impact when working on the edges of capitalism.

Now, here’s the good news: there’s hope.  I would propose the following baby steps:

1. Instead of trying to create a one-size-fits-all evaluative framework, commit to creating a more open-ended evaluation process that allows individual leaders to share the narrative of their successes.  This still means creating goals and objectives, but ones that recognize the individual capacities and challenges of a particular organization, and encourages systematic growth through succeeding hurdles.
2. Encourage leadership by supporting it wherever you find it.  Nascent efforts to build capacity within sectors, communities, programs and so forth should all be encouraged and supported.  Heaven knows we need it, and if someone is willing to commit the time and energy it takes then get behind them, lending your thinking and whatever resources you can spare.
3. Encourage leadership by taking leadership.  Bring leaders together wherever possible to discuss their strategies, ideas and goals.  Support any effort that connects two or more leaders more firmly to one another through shared goals, measurement, complementary programming or resources.
4. Push beyond your comfort zone in ceding leadership to others.  If someone is on a roll, get behind them rather than trying to outdo them.

Since we have to do this the hard way, it’s going to take time.  Success is built upon aggregated effort.  We must work to always create the environment where good work accumulates.

In Conclusion 

Remember:  vulnerable, needy, forlorn and forgotten.  We are describing not only our clients, our consumers, and our constituents – we are also describing ourselves.  But there’s another important term to keep in mind:  we’re also the majority.  The center is small (and getting smaller), the margins are continuing to widen.  In pushing back against the centrifugal forces of the market, we should never forget that we carry the greater mass by far.  I mean this to go beyond simple attempts at inspiration.  The system will only respond to our ability to throw off the center by creating countervailing forces.  Please, I’m not calling for revolution, but the present system will not re-orient itself toward our work until it can see the direct benefits of doing so.