The only constant is change. So why don’t we know more about what’s going on?

David Maddox
Sound Science LLC

“How we doing?” asked the Ecological Manager? “Pretty well”, replied the Ecologist. “How much is pretty well?”, asks the Manager, since that’s what s/he needs to know. And not even just how much, but rather how much relative to what they wanted to accomplish in the first place.

Whether they are called “Measures of Success”, “Theories of Change”, or “Impact Monitoring”, or just “monitoring”, organizations should know whether their actions are effective, and whether they are making progress toward their organizational goals. They need to evaluate both the effectiveness of individual projects and the overall impact of their work; that is, they need to seek to understand how effectively their mission is being achieved. Such assessments involve specifically devised measures of success that are matched to institutional goals and rely on objective, transparent, and repeatable measurement methods.

I’ve been curious about and attracted to the way foundations tend to describe what the ecologists call “monitoring and assessment”: “theories of change” and “impact monitoring”. Ecologists often use conceptual ecological models (Sound Science LLC teaches them in our courses) as a way of understanding ecological systems under management.  Bob Unnasch and I, along with Karen Poinani, wrote a book chapter on the topic ten years ago. Such models, once they are accepted as good and informative descriptions of the system under study, should then be used to identify monitoring targets and design assessments.

But they typically aren’t used in such a way. Why is that? If the model is as strong as a causal chain – that is, is a good description of “what’s going on” – then detecting effort and change at the driver side of the model OUGHT to suggest/predict change at the outcome side. I often think that there are four problems that commonly exist:

(1) In the first place, the casual-logic chains aren’t rigorous enough in the models, so there is no really good description of what’s important;

(2) People get too satisfied measuring the implementation of programs that are down in the weeds and too remote from the outcomes;

(3) We often really just don’t want to know the bad news, so we settle for just measuring effort.

And the fourth? Sometimes the goals of the project are too poorly or vaguely defined to create any meaningful measurement of success or theory of change. Entire organizations can suffer from a poorly crafted mission statement and goal statements that prohibit any meaningful measure of impact.

This Blog will be about a variety of topics — conservation, adaptive management, data, politics, non-profit management, polling — but the unifying theme will be how to create scientific information that is useful. A core part of this is understanding the differences between data and information. A big pile of data is not information. It needs to be intimately related to the mission and goals of the organization. It needs to be organized within a system of understanding — a theory of change. It needs to be translated into interpretation that is useful.  It needs to be news that a Manager can use to make good decisions.