Elizabeth de Soto - Evaluating the Impact of the Humanities: Resources

Elizabeth de Soto is the assistant program director at the Minnesota Humanities Center.

When you work at a humanities center, you realize quickly that the humanities can be surprisingly difficult to explain, be it to your best friend, a fellow conference attendee, or a stranger at the airport. We like First Lady Michelle Obama’s take: “The arts and humanities define who we are as a people. That is their power ― to remind us of what we each have to offer, and what we all have in common. To help us understand our history and imagine our future. To give us hope in the moments of struggle and to bring us together when nothing else will.”

If the humanities are indeed how we understand each other, our experiences, and the world in which we live, then perhaps literature, history, and [insert your favorite humanities field here] could best be described as ‘tools’ in humankind’s evaluation toolbox! Because we believe it is how one goes about building this understanding – and ultimately what one does with it – that define an individual or organization, the Minnesota Humanities Center is filling its toolbox with Utilization-Focused Evaluation (U-FE) tools. This approach, developed by Michael Quinn Patton, is based on the principle that an evaluation should be judged on its usefulness to its intended users (who are actual people - gasp!), and this attention to use and intended users shapes not only the findings but the process itself.

As Nora Murphy and Jennifer Tonko shared in a recent blog post, “A values-driven, relationship-based approach requires different kinds of evaluation.” UFE helps us as internal evaluators not only to understand the impact of the humanities, but also to live into the values and principles of our work. For example, we build and strengthen relationships with our primary intended users -- who in the case of Humanities Center programs, are often colleagues and program participants -- and we help them understand what is working to amplify community solutions for change.

As you strive to understand the impact of the humanities, we hope people are at the center of an evaluation that reminds you what you have to offer, helps you understand history and imagine a future, and, gives hope in moments of struggle. If your interest in the subject has been piqued, there are great resources on U-FE to match any appetite:

• U-FE checklist
  http://www.wmich.edu/sites/default/files/attachments/u350/2014/UFE_checklist_2013.pdf)
• Utilization Focused Evaluation: A primer for evaluators https://evaluationinpractice.files.wordpress.com/2013/04/ufeenglishprimer.pdf)
• Essentials of Utilization-Focused Evaluation
  http://www.sagepub.com/textbooks/Book233973)
• Utilization-Focused Evaluation (Fourth Edition)
  http://www.sagepub.com/books/Book229324)

---

Nora F. Murphy and Jennifer Tonko - How Do You Understand the Impact of the Humanities?

Nora Murphy, co-founder of TerraLuna Collaborative, has a broad set of experiences as an evaluator and researcher. She earned a Ph.D. in Evaluation Studies from the University of Minnesota, an M.A. in Research Methodology from the University of Pittsburgh, and has her B.A. in Education from Earlham College. With strong quantitative and qualitative skills she has conducted evaluation and research in schools, school districts, local community organizations, national non-profits,  and government entities. Murphy’s primary approach to evaluation is to view programs and people as fundamentally interrelated, applying systems theory as a framework for evaluation planning, design, implementation, analysis, and reporting. She’s been working with the Humanities Center since January 2013.

Jennifer Tonko is the Minnesota Humanities Center’s Senior Project Coordinator for Omaha Public Schools.

The Minnesota Humanities Center and TerraLuna Collaborative have been partnering since January 2013 to evaluate the Humanities Center’s Education Strategy with the Omaha [Nebraska] Public Schools (OPS). This relationship-based and partnership-driven strategy is based on the premise that strong relationships between students and educators are essential in increasing student engagement and ultimately their academic success. The Strategy is not a pre-determined model but rather guided by four core values: 1) Build and strengthen relationships; 2) Recognize the power of story and the danger of absence; 3) Learn from and with multiple voices; and 4) Amplify community solutions for change.

A values-driven, relationship-based approach requires different kinds of evaluation. Rather than reducing the evaluation plan to inputs and outputs it calls on us to pay deep attention to educators—their experiences and their relationships. Their stories matter.
TerraLuna Collaborative – a cooperative research and evaluation firm based in Minneapolis, MN – has worked in partnership with the Humanities Center to create a developmental evaluation that supports program development by identifying what is and what is not working, and bringing attention to emerging information. Key elements of the evaluation are:

View Schools as Human Systems. Rhetoric around public education sometimes makes it sound as though teachers are robots delivering pre-determined lessons to passive recipients in an attempt to raise test scores. We justify this by saying that students need to hit certain academic milestones for success. But educators and students are people—people who have relationships, hopes, joys, and fears. Our evaluative process does not reduce people to numbers, describe a single “average” experience, or view the educational system as one that’s static and separate from the people in it. 

Collecting Stories. We use story-based methods to find powerful narratives of change and look for voices that might be missing. Having the stories of learning, implementation, challenge, and success are what help us truly understand what is going on in the offerings.

Reflect the Core Values. We commit to working in and through the core values and principles of the Strategy so participants have a consistent experience when they are working with TerraLuna and the Humanities Center. Because the Strategy is so committed to building relationships, it is absolutely critical that all partners treat participants as the whole people that they are at all times; participants are never just data-points.

Make Meaning Together. We commit to reflective practice and meet regularly both as whole teams and as smaller teams to create a space for shared meaning-making. With project staff spread across multiple states and the quick pace of adaptation, it is critical to convene people to engage in reflective practice. This allows the Humanities Center to continually refine our shared vision and move forward with coherence.

This evaluation approach allows us to learn what this work means for educators and for their relationships with students and with each other. By making sense of stories and experiences together the Humanities Center and OPS can decide how to move forward and a way that best meets the needs of educators, students, family members, community members, and the school district. Through the evaluation process relationships are formed, voices are heard, and solutions are amplified.

---

Jean A. King - How do you measure the impact of the humanities?

Jean A. King is Distinguished Teaching Professor in the Department of Organizational Leadership, Policy and Development at the University of Minnesota where she serves as Director of the Minnesota Evaluation Studies Institute. A long-time writer on program evaluation, she joined the Minnesota Humanities Center Board in 2012 and currently serves on the Program and Development Committees and the advisory group for the Education Initiative, as well as supporting staff in developing evaluation capacity.

There are two reasons why I feel comfortable discussing this topic. First, I am a program evaluator with over 35 years’ experience. Second, I was an English major in college and remain a proud member of Garrison Keillor’s Professional Organization of English Majors (POEM). Plus I have one really good example that demonstrates such impact. Twenty years ago when I lived in New Orleans, I successfully answered the quiz-of-the-day question at an ice cream parlor: “What is the correct version of the following line from Hamlet: ‘Also, poor Yorick, I knew him well.’” Without hesitation I recited the correct line (“Alas, poor Yorick! I knew him, Horatio…”) and earned a free ice cream cone. When I told my mother, she noted joyfully that I had demonstrated the tangible value of an English major and four years of liberal arts college tuition. Talk about impact!
But systematically measuring the impact of the humanities moves well beyond a free ice cream cone. Multiple challenges confound the task, three of which emerge by examining three key words:

• Humanities- An initial challenge stems from the need to be explicit about what, exactly, constitute “the humanities.”  People don’t necessarily agree on what to include under the umbrella nor what should count as a humanities “program” or intervention. The first challenge, then, is being clear about exactly what we are trying to measure the impact of.

• Measuring- A second challenge arises because, even if we can agree on that, from a psychometric perspective, measuring the effects is a daunting task. Quantitative methods generate numbers, and qualitative methods produce thick descriptions and stories, but how can any method document meaningful impact? What instruments and data collection processes will produce credible evidence for those who control humanities budgets--legislators, policy makers, and funders?

• Impact- I routinely encounter well-intentioned people who want to prove causal assertions--to find irrefutable evidence that a certain set of activities leads directly and uniquely to specific measurable outcomes. Herein lies the third and most daunting challenge. While it may seem like a straightforward task, it is simply not possible to measure such impact without making a sizeable number of assumptions about how the multiple systems in which programs operate work. In my experience there is virtually never a straight line connecting program activities, outputs, and short- and long-term outcomes, all of which lead to ultimate impact.

Thankfully, there are two reasons why these challenges honestly don’t bother me. First, we evaluators can humbly design studies that help intended users practice what my colleague Michael Quinn Patton calls “reality testing.” We can provide sound evidence about what is occurring before, during, and after programs. Second, and more importantly, I would argue that some things hold intrinsic value, even if we can’t demonstrate it with 21st century rigor. To my mind, the humanities fall into that category. They define us as human beings and bring us together in community, and that is a good in and of itself, whether or not we can prove it.

---

Ka Vang - Humanities with Impact

Ka Vang is the Director for Impact and Community Engagement with American Public Media Group/ Minnesota Public Radio. She is a recipient of the Archibald Bush Artist Fellowship and several other artistic and leadership awards. Ka is the author of the children's book “Shoua and the Northern Lights Dragon,” a finalist for the 23rd Annual Midwest Book Awards in 2012, published by the Minnesota Humanities Center and Council on Asian Pacific Minnesotans.

Upon my family’s arrival in the United States from a dingy Thai refugee camp, my mother declared two things: First, she didn’t want us to eat at McDonalds. She found Ronald McDonald creepy. Second, she didn’t want us to lose our Hmong folktales. In retrospect, her dislike of McDonalds had more to do with her dislike for junk food than Ronald McDonald’s flaming red hair and nose. As for the folktales, she told me to tell vivid folktales so even the wind would stop to listen.

That was great advice and over the years I have developed an audience happy to hear Hmong folktales. Still, questions plagued me. When audiences listened, how did my folktales affect them, in what unique and meaningful ways? In other words, what was the impact of my folktales?

Gwen Westerman described the humanities in a previous blog as the thing that makes us “human”—languages, music, the arts, philosophy, history, literature, and religion. The humanities foster creativity, critical thinking, communication, collaboration, and reasoning. We are pushed to love and reject, to truly be human. But one of the things I do not see the humanities asking us to do--particularly for organizations whose mission it is to create and promote the humanities--is to measure the impact of our work on people. Historically, there are ‘vanity’ measures like collecting data on how many people attended an art show, visited a website, or even donated money to an organization, but we don’t go beyond that to really understand how and why the humanities matter to people and in what unique ways. Nor do we know with specificity how people really respond to the humanities.

It may not be the cool thing to say that we have to measure the humanities impact on people, but it is a conversation that needs to happen. That is why I am writing this blog to discuss the impact.

Accountability. Most organizations that support the humanities, both small and large, are mission-driven organizations. How do these organizations know with confidence and specificity that their work is making a difference in people’s lives unless they track and measure their goals? More importantly, how do these organizations know they are fulfilling their mission?

Funding. As humanities funders, e.g., foundations and major donors, move towards outcome-focused projects, humanities organizations are expected to prove that they achieved those outcomes. Additionally, funders expect mission-based organizations to use the money they give them in effective and meaningful ways.

The reality of the current humanities community is that it is essential for us to demonstrate we are having an impact. So how do we go about doing that?

Being Intentional. Before starting a humanities project, organizations should be intentional about setting impact goals. Ask critical questions about the difference you want to make with your project -- on individuals, on institutions, and on communities. This backwards design approach will help you set measures to track your impact.

Tracking Impact. Just do it. We need to show the value the humanities have in the community. The only way to do that is to put impact measures in place. There is no cookie cutter measure, no single set of ideal metrics, because your measure of success depends on your goals. The important thing to remember about tracking impact is goal-setting and putting measures in place before you start your project or initiative.

Tell Your Impact Story. Once we learn how and why our humanities project made a difference we have to let the public and funders know also. We have to celebrate it. Again, it was my mother who taught me to celebrate storytelling. We have to tell it to everyone even the wind!

I currently work as the Director for Impact and Community Engagement with American Public Media Group/ Minnesota Public Radio. APMG’s new standard for impact is everything I outlined above: intentional goal setting, measuring, and accountability process where content creators answer the questions: How will the content matter to our audience, and how will you know if it matters to them?

We define impact as ‘an intentional change in the status quo’ as a result of something APMG created for our audience. Examples of the ‘something we created’ can be music or news, digital or live events. We define impact change as something that can happen to an individual, group, organization, and/or system. This change can be a social or physical condition, and is revealed by what appears to be different as a result of our work.

As a news, music radio, and digital media company, we want our audience to know we listen to them and we know them. Knowing our impact on audiences ensures that we will fulfill our mission and play our role in using the humanities to make a difference in our world.

---