You need to read Bill Schambra's piece on this.
Over at the Hudson Institute, Bill Schambra has a superb piece on modern philanthropy's faith in what he calls Big Data--the passion for metrics that on the plus side, can demonstrate that you're actually achieving your mission, and on the minus side...well, read on.
There's always a choice, when nonprofits try to demonstrate their value to donors, between data and stories. Stories can bring what you do to life, but a tough foundation or donor is always going to ask how representative that story really is. Data, which is by far the favored approach among institutional donors, can quantify it and try to answer that question, but it often falls short of conveying the truth.
In order to construct the elaborate edifice of Big Data, there simply must be, deep down at its very foundation, a clear, distinct separation of the sheep from the goats, an assignment of 1 (success) or 0 (failure). Just as the vast digital world is built on 0s and 1s, so is the empire of metrics. The most remote and abstract regression analysis depends on that initial, radical division—no hedging, no hemming or hawing, just putting it down on the scoring sheet as a 1 or an 0, leaving the rest to the evaluators at foundation headquarters or at the university.
But this grand divide isn’t as simple as it seems. Alicia Manning, program officer with the Bradley Foundation in Milwaukee, where I once worked, recently took me to meet George Bogdanovich, founder of that city’s Community Warehouse. Among many other things, the warehouse accepts donated surplus construction material, appliances, plumbing fixtures, and so forth, and sells them at reasonable prices to those who are rehabilitating houses and stores in the inner city. They’ve now begun using some of the materials to prefabricate doorframes and sink cabinets, which provides jobs while teaching entry-level carpentry skills. All Warehouse employees are just coming out of prison or off the streets.
As happens so often when I visit front-line nonprofits—okay, sometimes it happens because I raise the question—we started discussing the problem of measurement, the demand that nonprofits classify their efforts into either success or failure. This clearly is a problem that George wrestles with regularly, and without any prompting from impatient funders.
He brought up the example of Joe, a pimp and drug dealer who had been one of his first employees. Things had gone well for a while, but then there had been a couple of thefts. George chose to follow the advice dispensed by the director of the recovery community where Joe lived: “Fire his ass.” Even as we stood in the drafty warehouse talking the question over, though, George was awaiting the call he gets from Joe every year around Christmas. He always checks in, telling him how things are going, and thanking George once again for giving him the opportunity for his first job out of prison—even though George had, indeed, fired his ass.
“So,” George asked, probably more of himself than of us, “was my experience with Joe just a failure?”
“No way you don’t count that a success,” Alicia immediately insisted.
In moments of pridefulness, I refer to Alicia as my student, since twenty years ago I took her around on her first site visits among Milwaukee’s grassroots leaders. But the fact is that she has far surpassed whatever I had to offer her, and now navigates with ease among Bradley’s smallest inner city grantees. Her grant recommendations are based less on numbers than on the “local wisdom” which she gathers with her own eyes and ears and finely tuned BS-detector. She has no problem dealing with the inadequacy of “1s” and “0s,” and the ambiguity that Joe poses to classification. But how many other foundation program officers, academics, or evaluators can claim that hard-won skill?
I encourage you to read the whole piece here. It's a bit long, but it is filled with great stories that admirably demonstrate what this last paragraph does; that sometimes there is no substitute for local knowledge.
This is why I think a social approach to fundraising is so valuable. Individual donors, not foundations, are the lifeblood of most organizations' budgets. And the more intimately connected with your work a person is, the less he will need formal reports (data-driven or story-driven) to demonstrate value. Nonprofits that are the hub of a community that includes their donors have a distinct advantage over ones that are struggling to prove value to strangers.