What the Hell is Program Evaluation?

class: center, middle, inverse, title-slide

.title[
# What the Hell is Program Evaluation?
]
.subtitle[
## A Forever Question
]
.author[
### Dr. Abhik Roy
]

---

---

class: highlight-last-item
layout: true

---

# Welcome!

Some information about the audio files. Firstly and importantly, I'm no Morgan Freeman but each page will have some audio that will hopefully help you understand a bit more about what's on the page

<div class="centerpiece">
<audio controls preload="auto">
 <source src="audio/criteria/S1_Introduction.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
<audio>
</div>

---

# Defining Evaluation

---

## (Mostly) Official

<center>
<audio controls preload="auto">
 <source src="audio/criteria/S2_Evaluation_Definition.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
<center>
 
.center2[
Evaluation is the act or process of determining the

> merit quality

> worth value

> significance importance

of something or the product of that process
]

---

### A Closer Look

.pull-left[
 
 
 
 
 
Think about this:
 
 
What do merit, worth, and significance mean to you both separately and in combination?
]

.pull-right[
<center>
Then listen to these:
</center>
 
<center>
<audio controls preload="auto">
 <source src="audio/criteria/S3a_Merit_Worth_Significance.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
</center>
 
<center>
<audio controls preload="auto">
 <source src="audio/criteria/S3c_Merit.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
</center>
 
<center>
<audio controls preload="auto">
 <source src="audio/criteria/S3d_Worth.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
</center>
 
<center>
<audio controls preload="auto">
 <source src="audio/criteria/S3e_Significance.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
</center>
]

---

### A Closer Look

.center2[
.pull-left[
Can you think of real world examples of each?
]]

---

## Generally

.center2[
Evaluation answers the questions

> does it work?

> why does it work?
]

---

## What About the How?

.center2[
We typically leave the remaining question how does it work?
to people with grounded knowledge in a particular area

> aka content experts
]

---

# More Terms with the Stem Eval-

.pull-left[

Evaluator 
The person(s) evaluating something

]

.pull-right[

Evaluand 
The thing(s) being evaluated

]

<center>
<audio controls preload="auto">
 <source src="audio/criteria/S5_Evaluand_and_Evaluator.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
</center>

---

# Levels, Performances, and Measures

.pull-left[
<center>
<audio controls preload="auto">
 <source src="audio/criteria/S6b_Criteria.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
</center>
 
<center>
<audio controls preload="auto">
 <source src="audio/criteria/S6c_Indicator.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
</center>
 
<center>
<audio controls preload="auto">
 <source src="audio/criteria/S6d_Standard.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
</center>
]

.pull_right[
.tab[Of the three terms]

.tabdbl[criteria]

.tabdbl[standards]

.tabdbl[indicators]

.tab[which of the three do you believe is harder to identify?]
]

---

# Levels, Performances, and Measures

.center2[
.pull-left[
Can you think of real world examples of each?
]]

---

# Primary Purposes of Evaluation

.pull-left[

Formative 
Conducted with the intent to improve
 

]

.pull-right[

Summative 
Conducted with the intent to inform decision making and/or determine judgment

]

</audio>
</center>
 
<center>
<audio controls preload="auto">
 <source src="audio/criteria/S7_Formative_Summative.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
</center>

---

# Secondary Purposes of Evaluation

.pull-left[
<center>

Developmental
 
To help develop an intervention or program
 

</center>
]

.pull-right[
<center>

Accountability
 
To hold accountable
 

</center>
]

<center>

Monitoring 
To assess implementation and gauge progress toward a desired end

 
</center>

<center style="margin-top: -43px;">
.pull-left[

Knowledge Generation 
To generate knowledge about general patterns of effectiveness

</center>
]

.pull-right[
<center>

Ascriptive
 
Merely for the sake of knowing
 

</center>
]

---

## Developmental vs Formative Evaluation

<center>
Please take a look at the video from an evaluator named Michael Quinn Patton describing the differences
</center>

---

## A Bit More on Developmental Evaluation

.tabdbl[.pull-left[
<audio controls preload="auto">
 <source src="audio/criteria/S10_Developmental.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
 ]
]

.pull-right[
<audio controls preload="auto">
 <source src="audio/criteria/S11_Activity.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
]
 
 
 
<table class="table table-condensed" style="width: auto !important; margin-left: auto; margin-right: auto;">
 <thead>
 <tr>
 <th style="text-align:left;background-color: #212121 !important;vertical-align: middle !important;"> </th>
 <th style="text-align:center;background-color: #212121 !important;vertical-align: middle !important;"> What might the purpose of the evaluation be? </th>
 <th style="text-align:center;background-color: #212121 !important;vertical-align: middle !important;"> What questions might the evaluation try to answer? </th>
 </tr>
 </thead>
<tbody>
 <tr>
 <td style="text-align:left;background-color: #212121 !important;vertical-align: middle !important;"> Formative Evaluation </td>
 <td style="text-align:center;background-color: #212121 !important;vertical-align: middle !important;"> </td>
 <td style="text-align:center;background-color: #212121 !important;vertical-align: middle !important;"> </td>
 </tr>
 <tr>
 <td style="text-align:left;background-color: #212121 !important;vertical-align: middle !important;"> The program has been designed and piloted with a group of 20 principals. What kind of evaluative feedback should the ASDC seek? </td>
 <td style="text-align:center;background-color: #212121 !important;vertical-align: middle !important;"> </td>
 <td style="text-align:center;background-color: #212121 !important;vertical-align: middle !important;"> </td>
 </tr>
 <tr>
 <td style="text-align:left;"> </td>
 <td style="text-align:center;"> </td>
 <td style="text-align:center;"> </td>
 </tr>
 <tr>
 <td style="text-align:left;"> Summative Evaluation </td>
 <td style="text-align:center;"> </td>
 <td style="text-align:center;"> </td>
 </tr>
 <tr>
 <td style="text-align:left;"> The program has been delivered to all principals in the county. What kind of evaluative feedback should the ASDC seek? </td>
 <td style="text-align:center;"> </td>
 <td style="text-align:center;"> </td>
 </tr>
</tbody>
</table>

---

# Introduction to Roles and Styles

.pull-left[
Evaluator Roles:

> Independent

> Topical

> External

> Internal

The one that you take on in an evaluation will determine a lot!
]

.pull-right[
<center>
<audio controls preload="auto">
 <source src="audio/criteria/S12b_Independent.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
</center>
 
<center>
<audio controls preload="auto">
 <source src="audio/criteria/S12c_External.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
</center>
 
<center>
<audio controls preload="auto">
 <source src="audio/criteria/S12d_Topical.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
</center>
 
<center>
<audio controls preload="auto">
 <source src="audio/criteria/S12e_Internal.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
</center>
]

---

## Defining Culture

<div style="margin-left: 100px; margin-top: 240px; position:absolute;">
<center>
<audio controls preload="auto">
 <source src="audio/criteria/S13_Culture.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
</center>
</div>

> What is culture?

> How do we recognize it?

.pull-right[
Culture can be defined as the shared experiences of people, including their languages, values, customs, beliefs, and more. It also includes worldviews, ways of knowing, and ways of communicating. Culturally significant factors encompass, but are not limited to, race/ethnicity, religion, social class, language, disability, sexual orientation, age, and gender.
]

.pull-right[
-- 
<a href="https://www.eval.org">American Evaluation Association</a> (2011)
]

---

### The Iceberg

.pull-left[
<center>
<img src="img/iceberg.png" height="500px" style="background-color:#212121;"/>
</center>
]

.pull-right[
 
 
 
 
 
 
 
<center>
<audio controls preload="auto">
 <source src="audio/criteria/S14_Iceberg.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
</center>
]

---

### Markus & Kitayama (1991)

.pull-left[
<center>
<img src="img/eastvwest.png" height="480px" style="background-color:#212121;"/>
</center>
]

.pull-right[
 
 
 
 
 
<center>
<audio controls preload="auto">
 <source src="audio/criteria/S15_Self.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
 
 
 
 
Download the <a href="docs/%4D%61%72%6B%75%73%20%26%20%4B%69%74%61%79%61%6D%61%20%28%31%39%39%31%29%2E%70%64%66" target="_blank">article</a>
</center>
]

---

### Cultural Influences Activity

.pull-left[
 
 
<center>
How might each of the following influence an evaluation’s design and implementation?
</center>
 
 
 
 
 
 
<center>
<audio controls preload="auto">
 <source src="audio/criteria/S16_Cultural_Activity.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
</center>
]

.pull-right[
> Language of participants

> Role of food and eating

> Evaluators attire when interacting with participants
 
> Concept of time

> Communication styles

> Importance of family relationships

> Values/norms re: individualism, independence, conflict

> Understanding of social structure
]

---

## Increasing your Cultural Sensitivity

<div style="margin-left: 380px; margin-top: 340px; position:absolute;">
<center>
<audio controls preload="auto">
 <source src="audio/criteria/S17_Sensitivity.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
</center>
</div>

> Assess your own attitudes, beliefs, values

> Incorporate culturally diverse groups and perspectives throughout evaluation process

> Build in sufficient time to negotiate access and build trust in community / organization

> Demystify evaluation, avoid jargon (please!), build buy-in

> Be flexible in choice of evaluation methods, use multiple data sources, multiple data collection methods, multiple perspectives

---

# Meta-evaluation

<div style="margin-left: 380px; margin-top: 280px; position:absolute;">
<center>
<audio controls preload="auto">
 <source src="audio/criteria/S18_Meta_Evaluation.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
</audio>
</center>
</div>

<center>
A &nbsp;meta-evaluation&nbsp; is an evaluation of an evaluation or evaluations
</center>

---
 
## Classifications

.pull-left[

According to the role
the evaluation plays 
Formative or Summative
 
 

]

.pull-right[

According to the content or the program phase evaluated 
Design, Process, or Results
 
 

]

.pull-left[

According to the moment of the evaluation 
After the fact - aka ex-ante or before the event -aka ex-postfacto

]

.pull-right[

According to the evaluator's role 
Internal or External
 

]

.footnote[Concentrate on the idea of a formative and summative evaluation this week. There is no need to worry if you don't know what any of remaining terms mean and/or imply (and yes that includes this idea of a meta-evaluation)! You'll have plenty of opportunities to learn about them as we progress through the term.]

---

# Considerations

---

## Things to Evaluate

.pull-left[
.tabtpl[
> Evaluation criteria

> Evaluation standards

> Performance

> Program costs

> Program goals
  ]
]

.pull-right[
 
 
 
<center>
<audio controls preload="auto">
 <source src="audio/standards/S1 Things to Evaluate.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
<audio>
</center>
]

---

## Things to Assess

.pull-left[
.tabtpl[
> [Evaluation Standards](https://evaluationstandards.org/program/)

> Location

> Stakeholder Needs

> Sponsor Needs

> Your costs
 ]
]

.pull-right[
 
 
<center>
<audio controls preload="auto">
 <source src="audio/standards/S2 Things to Consider.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
<audio>
</center>
]

---

# A Look at Goals

---

## Include

<div style="margin-left: 240px; margin-top: 220px; position:absolute;">
<center>
<audio controls preload="auto">
 <source src="audio/standards/S3 Goals.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
<audio>
</center>
</div>

.pull-left[
Preparatory conditions
]

.pull-right[
+ What does it take to be ready to target a goal?
  
+ Progress toward that state is a goal
]
  
--

.pull-left[
Intended operations
]

.pull-right[
+ What a program tries to implement to achieve a goal is a type of goal.
]
  
--

.pull-left[
Targeted outcomes
]

.pull-right[
+ Short-term (initial)

+ Medium-term (intermediate)
  
+ Long-term (distant)
]

---

## Whose are Important?

<div class="centerpiece">
It's not always obvious!
 
<center>
<audio controls preload="auto">
 <source src="audio/standards/S4 Whose Goals.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
<audio>
</center>
</div>

---

## How to Decide Which Matter Most?

<div style="margin-left: 220px; margin-top: 200px; position:absolute;">
<center>
<audio controls preload="auto">
 <source src="audio/standards/S5 Deciding on Goals.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
<audio>
</center>
</div>

.pull-left[.tabtpl[
&#10003;
 ]
]

.pull-right[.revtabdbl[
Talk to stakeholders/sponsors; explicitly ask their perspective on recipient needs, program goals, and evaluation goals
 ]
]

.pull-left[.tabtpl[
&#10003;
 ]
]

.pull-right[.revtabdbl[
Figure out other ways to find needs
 ]
]

.pull-left[.tabtpl[
&#10003;
 ]
]

.pull-right[.revtabdbl[
Read all available program documentation
 ]
]

.pull-left[.tabtpl[
 
&#10003;
 ]
]

.pull-right[.revtabdbl[
Develop a logic model connecting needs through program activities, outputs and outcomes associated with goals
 ]
]

---

# Logic Model Parts

<div style="margin-left: 580px; margin-top: -60px; position:absolute;">
<center>
<audio controls preload="auto">
 <source src="audio/standards/S6 Logic Model Parts.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
<audio>
</center>
</div>

.pull-left[
.tabdbl[

INPUTS

]
 
.tabtpl[

Program Investments

 ]
]

.pull-right[
 
what resources go into a program - aka 

investments


]

.pull-left[
.tabdbl[

OUTPUTS

 ]
]

.pull-right[
what is produced through those activities
]

.pull-left[
.tabtpl[
 

Activities

 ]
]

.pull-right[
.tabtpl[
the activities the program assumes - i.e. what actually occurs

 ]
]

.pull-left[
.tabtpl[
 
Participation

 ]
]

.pull-right[
.tabtpl[
who is part of the program and how are they served - i.e. who is served like stakeholders and sponsors and what engagement they have

 ]
]
 
---

# Logic Model Parts

.pull-left[
.tabdbl[

OUTCOMES

 ]
]

.pull-right[

changes or benefits that result from the program

]

.pull-left[
.tabtpl[
 

Short

 ]
]

.pull-right[
.tabtpl[

Learning like awareness, aspirations, attitudes, knowledge, motivations, opinions, skills, ect.

 ]
]

.pull-left[
.tabtpl[
 

Medium/Intermediate

 ]
]

.pull-right[
.tabtpl[

Actions like behaviors, decisions, policies, practice, etc.

 ]
]

.pull-left[
.tabtpl[
 

Long

 ]
]

.pull-right[
.tabtpl[
Consequences like those that are civic, economic, environmental, political, professional, social, etc.

 ]
]

---

### A Basic Model

.pull-left[
<center>
<img src="img/Blank Logic Model 1.png" style="background-color:#212121;"/>
</center>
]

.pull-right[
 
 
 
 
<center>
<audio controls preload="auto">
 <source src="audio/standards/S7 Blank Logic Model.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
<audio>
]

<center>
.pull-left[
 
 
Download a 
<a href="docs/Blank Logic Model.pdf" target="_blank">high quality PDF</a>
]
</center>

<center>
 
 
.pull-right[
Look at other 
<a href="https://fyi.extension.wisc.edu/programdevelopment/logic-models/" target="_blank">types of logic models</a>
]
</center>

---

### An Example

.pull-left[
<center>
<img src="img/REDACTED Logic Model.png" style="background-color:#212121;"/>
</center>
]

.pull-right[
 
 
 
 
<center>
<audio controls preload="auto">
 <source src="audio/standards/S8 Example Logic Model.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
<audio>
</center>
]

<center>
.pull-left[
 
 
Download a 
<a href="docs/REDACTED Logic Model.pdf" target="_blank">high quality PDF</a>
]
</center>

<center>
 
 
.pull-right[
Look at additional <a href="https://fyi.extension.wisc.edu/programdevelopment/logic-models/bibliography/" target="_blank">examples of logic models</a>
]
</center>

---

# Basic Factors of an Evaluation

---

## Identifying Criteria

.pull-left[
 
 
What matters enough to measure given what you know about all the other factors?
]

.pull-right[
> Any relevant past performance data

> Costs

> Existing literature and standards

> Needs and Goals
]

<center>
<audio controls preload="auto">
 <source src="audio/standards/S9 Identifying Criteria.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
<audio>
</center>

---

## Estimating Cost

.pull-left[
 
 
.tabdbl[Often very hard to do and do well]
]

.pull-right[
Almost always addressed in a relative way

+ Relative resource intensity of alternative programs (including development and/or retraining costs)
  
  + Proportion of resources allocated to different program activities (logic modeling can be very helpful for thinking about this)
]

<center>
<div style="margin-left: 65px; margin-top: -140px; position:absolute;">
<audio controls preload="auto">
 <source src="audio/standards/S10 Cost Analysis.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
<audio>
</center>
 
---

## Measuring Performances

.pull-left[
> Sponsor perception

> Staff performance / perception

> Stakeholder performance / perception
]

.pull-right[
In a nutshell

> Who is best positioned to really know?

> Multiple times, multiple perspectives
]

<center>
<audio controls preload="auto">
 <source src="audio/standards/S11 Measuring Performance.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
<audio>
</center>

---

## Evaluation Statements

---

### Developing a Purpose

.pull-left[
 
Step 1
]

.pull-right[

Critique sample evaluation purpose statements1 to better understand what might go into crafting a good one

]

.footnote[1&nbsp;More about this coming up later]

.pull-left[
 
Step 2
]

.pull-right[

Write preliminary evaluation statement for the program you'll evaluate

]

.pull-left[
 
Step 3
]

.pull-right[

Write 3 - 7 potential evaluation questions relevant to the evaluation purpose statement you wrote for step 2

]

---

### Thinking About Statements

<div style="margin-left: 220px; margin-top: 230px; position:absolute;">
<center>
<audio controls preload="auto">
 <source src="audio/standards/S13 Evaluation Purpose Questions.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
<audio>
</center>
</div>

In the following slides, take a look at the sample evaluation purpose statements and *for each* try to identify the following four things

.pull-left[.tabtpl[
&#10003;
 ]
]

.pull-right[.revtabdbl[
what is being evaluated?
 ]
]

.pull-left[.tabtpl[
&#10003;
 ]
]

.pull-right[.revtabdbl[
how will results be used?
 ]
]

.pull-left[.tabtpl[
&#10003;
 ]
]

.pull-right[.revtabdbl[
what kind of evaluation will be implemented (developmental, formative, or summative)?
 ]
]

.pull-left[.tabtpl[
&#10003;
 ]
]

.pull-right[.revtabdbl[
how could the statement of purpose be improved?
 ]
]

---

#### Example of Evaluation Statements

.pull-left[
<center>
<img src="img/Purposes of Evaluation 1.png" width="100%"/>
</center>
]

.pull-right[
<center>
<img src="img/Purposes of Evaluation 2.png" width="100%"/>
</center>
]

---

### Some Tips

> Be confident but never demeaning or overpowering

> Be clear

> Be concise

> Be truthful

> Don’t overpromise1

> Get buy in

> Write in a way that's meaningful to your sponsor(s)/stakeholder(s)

.footnote[1&nbsp;Under-delivering can destroy careers!]

<div style="margin-left: 700px; margin-top: -200px; position:absolute;">
<center>
<audio controls preload="auto">
 <source src="audio/standards/S14 Tips.mp3" type="audio/mpeg">
 Your browser does not support embedded audio.
<audio>
</center>

---

## Thats it!

If you have any questions, please reach out

<center>
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a> This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License</a>
</center>