Why It Matters
Most evaluations assess a program's effectiveness after its over. Why wait? Rapid-cycle feedback can help aid continuous improvement.
Processing ...

Seeking Continuous Improvement? Try Continuous Evaluation

By Abe Wandersman | Friday, September 22, 2017
Why Wait to Evaluate
The 100 Million Healthier Lives Scaling Community Accelerators through Learning & Evaluation (SCALE) initiative recently won the Outstanding Evaluation Award for 2017 from the American Evaluation Association. Unlike most evaluations, which assess a program’s effectiveness after the program is over, the SCALE evaluation sought to provide rapid-cycle feedback on the program’s performance to aid ongoing improvement. IHI Multimedia Project Assistant Willamina Hadley reached out to the SCALE evaluation’s principal investigator, University of South Carolina Professor of Psychology Abe Wandersman, inquiring what others can learn from his team’s methodology.

What approach did you use for the SCALE evaluation?

We used Getting to Outcomes® (GTO). It’s from the empowerment evaluation approach, of which I was a co-founder. A core tenet of empowerment evaluation is that [the evaluation team] wants to put the tools of accountability into the hands of the people doing the work.

GTO is a ten-step planning, implementation, and evaluation process that help stakeholders consider the elements that are needed to reach and sustain results. The ten steps of GTO are:

  1. Identify needs and resources.
  2. Set goals.
  3. Identify evidence or practice-based innovations to meet those goals.
  4. Consider the fit of those innovations.
  5. Identify and build the motivation and capacity needed to put those innovations into place.
  6. Develop a comprehensive plan to implement, including the who, what, when, and how.
  7. Implement the innovation with quality.
  8. Evaluate the results.
  9. Use a continuous quality improvement process to make changes to improve outcomes.
  10. Put conditions in place to sustain the work. Each of these steps has an extensive literature and evidence-based behind them.

Many of the SCALE evaluation team members come from a GTO-background, so it’s always in our minds when we consider evaluation questions and methods. In the spirit of co-design between our stakeholders, we wanted to consider our common ground.

An interesting development we have observed since first becoming involved in SCALE is the intersection between improvement science, which is IHI’s strength, and empowerment evaluation. Both have similar goals and values that proactively attempt to reach outcomes, rather than wait for a report at the end of a project.

Why isn’t rapid-cycle feedback more commonly used in community health improvement?

Rapid cycle feedback is extremely valuable, but can be difficult to achieve. We know that even in controlled settings there is variability in the quality with which PDSAs are implemented. Despite that, there is a wealth of resources from tapping into local knowledge about practices that lead to improvement, then providing the tools to help community coalitions systematically implement and evaluate their work. We are interested in helping communities succeed and helping them reach favorable outcomes. Rapid-cycle feedback is a potential method to accelerate the progress that can be made on the ground. The SCALE evaluation showed that coalitions are able to learn and use these methods quickly, though there is still a great deal we need to study in order to maximize the effectiveness and efficiency of the methods.

Would you explain the “open nest” metaphor, and how you think it improved the evaluation?

A nest is built from the available materials in the area: the sticks, the straw, the grass. An open nest is one that animals can bring new resources to and visit so they can gain a sense of their surrounding environment. A nest must also be secure enough to provide shelter to a growing life.

Like an open nest, our evaluation drew upon the strengths and resources of those participating. Many people could contribute and add value to our questions and methods. We were all protective of our common core values and purpose: to focus on transformation so we can build the health, well-being, and equity of our communities. It was extremely gratifying to see so many new ideas hatch and thrive here.

What were the challenges during data collection that surprised you?

This work is not easy. If it were, I would be able to fully retire. However, we recognized that we would have to change our methods and focus during SCALE. For example, during the first few months of SCALE we focused primarily on the support system; i.e., the tools, training, and coaching that were provided to communities. We felt this was appropriate because SCALE was a new initiative being tried for the first time. We soon realized that we were not paying enough attention to what the communities were doing. This mattered because the community is where the outcomes ultimately happen. With this in mind, we shifted our focus to the community-level to see how SCALE ideas and tools were implemented, and what outcomes could reasonably be attributed to them.

What are some steps the SCALE initiative will take now that it has this invaluable insight into community health improvement?

We have learned things that will help us support the growth and expansion of SCALE in the future. SCALE was originally unique for the emphasis that it placed on the relationships within and between communities. So, the first step needs to be continued emphasis on these intentional processes to cultivate and nurture such relationships.

We know that relationships are necessary, but not sufficient to reach outcomes. People need to use the available tools and methods on community-based issues. SCALE asks a lot of coalitions, and challenges them to think and act in novel ways. They need to be continuously supported, possibly by coaches, as well as by each other to make sure they work and innovate with quality. This is particularly true now that additional coalitions are joining and will not have access to the intense support and resources given to the first cohort.

Third, we know that local measurement is hard, even in highly controlled settings. So, while we might prefer to wait and build capacity for measurement until coalitions have ample time to build their general capacities, it is better to support the measurement process early on. That way coalitions can begin to track their activities and make changes in the beginning that help them achieve their aims.

Fourth, we need to continually assess and address coalition readiness. We know that readiness can go up and down over time in response to a variety of factors, such as staff turnover. By thinking about readiness and addressing it proactively through coaching and training, we can address the facilitators that make implementation happen.

Equity is “the price of admission” in 100 Million Healthier Lives, and all SCALE communities are working on advancing equity in their contexts. So, we must continue to live that “price of admission” value in our work. SCALE communities demanded (rightfully so) additional tools and content to help address equity in their community. Going forward, we’ve learned that keeping equity at the forefront allows the purpose and methods of SCALE to be crystalized.

Can this evaluation system expand to more community health coalitions in the country?

We’ve been hard at work writing and consolidating our thoughts about that. One team member, Kassy Alia, is responsible for writing a manuscript about how formative evaluation can be used to manage the complexity of community health improvement settings. We strongly believe that formative evaluation is an exemplary approach to help address the complexities in discerning what contributes and what impedes an initiative from reaching its outcomes.

What are you most proud of in the outcomes of the evaluation of the SCALE initiative?

We’ve been extremely fortunate to work with such a talented and dedicated group of colleagues. While we all shared similar goals and values, there was enough divergent thinking that we could capitalize on everyone’s strengths and insights and put a truly fresh and innovative system in place. We often talk about our evaluation approach as “helping to build the boat while we are sailing.” We were lucky to have an outstanding group of sailors. SCALE left the dry dock full of potential parts, and ended up as a catamaran that could navigate challenging waters toward community health.

Editor’s note: 100 Million Healthier Lives is a collaboration of change agents working to improve health, wellbeing, and equity convened by IHI. Learn more about the 100 Million Healthier Lives Measure What Matters platform and read SCALE community transformation stories.

first last

Average Content Rating
(0 user)
Please login to rate or comment on this content.
User Comments