SOCAP10 ... The Challenge of Moving Beyond Metrics

Dear Colleagues

The SOCAP10 Conference in San Francisco has started. I am disappointed not to be there ... I am in New York, and still trying to do something useful by progressing the development of The Burgess Method (TBM).

One of the tracks at the conference is The Challenge of Moving Beyond Metrics. Below I reproduce some material in one of the blogs about this aspect of the SOCAP10 Conference.

While I am very pleased that the issue of metrics is getting more attention, I am not enthusiastic at all about the progress being made towards meaningful metrics that will actually achieve the objectives that society needs.

From what I know at the moment, TBM has a different approach from what is emerging from the available material I have seen so far. Specifically TBM is aiming (1) to have profit and value adding equally important in progress and performance analysis; (2) to have the impact on the community as the main focus of progress and performance; and (3) to measure progress as a change in state of the community ... where state is similar to a balance sheet in money accountancy except also including quality of life measures.

TBM is conceptually similar to money accountancy ... except with value elements that relate to all quality of life issues as well as merely money wealth.

As far as I can see, the SOCAP10 blog post material below is not moving much towards the concepts of TBM ... but the good news is that it does talk about the need for such metrics. There will be more on this in due course!

SOCAP10: The Challenge of Moving Beyond Metrics – Part I
Posted Fri, 08/20/2010 - 6:28am by Adin Miller

On December 1, 2009, a major shift in assessing nonprofits was announced by Charity Navigator, GuideStar, GiveWell, GreatNonprofits, and Philanthropedia. The announcement aimed to repudiate the myopic focus on nonprofit overhead ratios and executive compensation as metrics to measure their effectiveness. The five oversight organizations also set out a course to establish and embrace new ways to evaluate nonprofits and identify nonprofits.

The announcement was greeted with wide support by those who advocated for such a change. Sean Stannard-Stock, the curator of the Tactical Philanthropy track at SOCAP10, noted that the change would help move more capital to effective nonprofits. Dan Pallotta applauded the change on the Harvard Business Review blog, while also noting that the future challenge of applying new evaluative measures “consistently to hundreds of thousands of organizations."

The new assessment frameworks will be discussed at a SOCAP10 session led by Ken Berger of Charity Navigator, Andrew Wolk of Root Cause, and Elie Hassenfeld of GiveWell. That session will review three analytical approaches to assessing a still unnamed nonprofit. In Ken’s case, for example, the session will allow him to clarify the new approach to rates a nonprofit on its financial strength, accountability, and effectiveness.

While I look forward to learning more the three assessment models, I also want them to reflect on two ongoing challenges. The first challenge deals with the unintended consequences of the past focus on financials indicators. The second challenge, which I’ll address in a follow-up post, examines the ongoing challenges of assessing nonprofit performance in the absence of full information.

The first challenge reflects the reality that we’re still left with the remnants of the previous assessment system that focused largely on overhead ratios and compensation levels. While the blogging community moved to embrace new assessment approaches, many nonprofits are still wrestling to understand how to explain and embrace these new assessment approaches.

Perhaps even more importantly, not all nonprofits are aware that a shift took place. Nor have some of their donors and board leaders – who have in some weird Pavlovian manner become fixated on overhead rations and compensation rates – learned that new assessment models are now emerging.

So, my first set of questions to the panel would be: how do we better inform nonprofits, donors, and nonprofit leaders that a significant change has taken place and that it’s actually OK not to be completely obsessed with overhead ratios and executive compensation? Where are the marketing and dissemination efforts to explain and educate nonprofits and their respective donors and leaders?

Next post: assessing nonprofit performance in the absence of full information.

This is the first post on this subject. The second follows:

///////////////////////////////
SOCAP10: The Challenge of Moving Beyond Metrics – Part II
Friday, September 10, 2010 at 2:31pm

In my first post on moving beyond metrics, I reflected on the challenge of moving beyond the fixation of assessing nonprofits based on their overhead ratios and executive compensation. New assessment frameworks, championed by organizations such as Charity Navigator, GuideStar, GiveWell, GreatNonprofits, and Philanthropedia, will promote and embrace new ways to evaluate nonprofits and identify nonprofits.

These new assessment frameworks will be discussed and used at a SOCAP10 session led by Ken Berger of Charity Navigator, Andrew Wolk of Root Cause, and Elie Hassenfeld of GiveWell. That session will review three analytical approaches to assessing DC Central Kitchen (per this post by Sean Stannard-Stockton). In Ken’s case, for example, the session will allow him to clarify the new approach to rate a nonprofit on its financial strength, accountability, and effectiveness.

The first challenge I noted deals with educating the nonprofit community, its leaders, and donors about the shift in focus away from financial metrics. To my disappointment, I keep hearing about organizations struggling to address overhead concerns. My second challenge deals with assessing nonprofit performance in the absence of full and real-time information.

I’ve been in the business of reviewing nonprofits as potential grant recipients for most of my career. As a grantmaker and steward of public and private funds, I’ve had to make sound recommendations and decisions on how to best allocate scarce resources while maximizing impact. To that end, I’ve had access to a lot of data related to nonprofit performance and financial positions – often data not readily available in the public domain. And yet, I’ve never felt fully comfortable that the data at my disposal provided an accurate and current assessment of the organizations and that the investments I was going to recommend would maximize community impact.

Most financial indicators fail to provide a real-time picture of organization. For example, most financial indicators available online reflect data that are 18 – 24 months old. As such, the data presented online – whether it is an organization’s Form 990 or an assessment of that data – reflects inputs that are somewhat stale. For example, much of the current data online reflect 2008 information. The economic recession’s impact – the loss of donors, the drop in fundraising, the massive staff layoffs and transitions – is not yet fully reflected in the data.

Similarly, performance outcomes are also a look back and don’t provide a full real-time overview of the organization. For example, let’s take an organization that has had a tremendous success in delivering services and has generated a track record of success. One could presume that an investment in that organization will generate additional high community impact. But that organization’s track history and performance outcomes won’t reveal that it is a dysfunctional internal mess that managed to achieve strong performance metrics despite its limitations. Nor will it tell you if those achievements were attained because of a visionary and talented person at the organization who has since left that nonprofit for a new enterprise.

In both examples, I could make a funding recommendation that would end up failing to generate the expected community impact.

How can we reasonably expect that an individual donor looking to make a charitable gift to an organization would be in a position to identify the best performing nonprofit organizations? Whether in looking at only financial metrics or examining a nonprofit’s effectiveness in achieving outcomes or a combination of both, the prospective individual donor still faces three major issues:

1. The data tend to be old;
2. The data don’t provide a complete informative picture of the organization; and
3. They lack simple, digestible information to inform their decisions (see “Money for Good” report (PDF) by Hope Consulting). We still need to translate the performance outcomes into information that can be used easily by individual donors – regardless of their aptitude or interest to research nonprofits.

Without real-time context and information that effectively communicates the organization’s data, financial indicators and performance outcomes will continue to serve as imperfect proxies for identifying high impact nonprofits. To quote an excellent post by Jeff Stanger, “data are not information.” They need context and information to be truly useful.

So, while I applaud the move toward performance outcomes as assessment tool to identify high performing and impactful nonprofits, that move does not represent the ultimate panacea for identifying worthwhile organizations to support. Many individual donors and grantmaking institutions want to reduce the risk exposure of their investments by funding organizations that would most likely have strong performance in delivering expected outcomes. And yet, to paraphrase a famous stock market adage: past performance by a nonprofit is no guarantee of future result.

Perhaps what we ultimately need is a well-informed and real-time crowdsourcing operation to support donor looking to provide funds to high impact nonprofits. Moving from financial metrics to performance outcomes might be a step in that direction, but only a step. Hopefully, the SOCAP10 panel on Beyond Metrics would be able to comment on this challenge as well.

By Adin Miller (Cross Post from Adin Miller Consulting)

Disclaimer: This post is part of a series of posts on the SOCAP10 conference. As part of the blogging coverage for the conference, I was able to register to attend at a discounted rate. A version of this post appears on the SOCAP10 blog.

I am happy to see dialog about movement beyond money metrics ... but really quite disappointed that SOCAP10 is not embracing metrics yet that are along the lines of TBM!

It will happen!

Peter Burgess