Course Description

The purpose of this course is to provide students with the theoretical and practical bases for program evaluation.  Students will develop basic skills in a variety of approaches to evaluation, including techniques that are particularly suitable for evaluating health promotion, community health improvement, and related health and social services programs.  Course learning will be synthesized through design of an evaluation framework and methodology for a relevant program.

Prerequisites

There are no formal prerequisites for this course.   It is assumed that students have some familiarity with health services delivery and the organization of the health system.  If you feel you do not have this knowledge, please contact me to discuss the suitability of this class for you.

Learning Competencies (OMPH Approved)

At the completion of the course, students will be able to:

1. Explain how theory informs evaluation design and methods
   • Describe the linkage between theory and evaluation
2. Solicit and integrate input from program stakeholders in the development of an evaluation plan
   • Communicate with program stakeholders about their programs and about evaluation
   • Obtain program information from stakeholders and other sources
   • Integrate input from program stakeholders in the development of an evaluation plan
3. Critically assess evaluations and interpret evaluation findings
   • Evaluate major components of articles presenting evaluation results
   • Identify limitations of published evaluations
4. Design and prepare an evaluation plan
   • Describe the major types, concepts, strategies, and methods for evaluating programs
   • Formulate evaluation questions appropriate for a specific program
   • Design an evaluation appropriate for answering particular evaluation questions
   • Work in a collaborative team to develop the evaluation plan
5. Explain how ethical principles apply to evaluation
   • Explain procedures for protecting human subjects in evaluation
   • Describe potential ethical considerations for program evaluation
6. Critically analyze how power differences play a role in evaluation
   • Identify issues of power differences between and among evaluator, community and program stakeholders
   • Articulate role of economic and political power in influencing evaluation
   • Incorporate principles of cultural competency into evaluation design, methods, and language used

At the conclusion of the course, students will be asked to determine the extent to which they have accomplished each of these competencies through their own learning.

Methods of Evaluation

There will be multiple methods of evaluation that will determine both your course grade and the evaluation of the course itself.

1.  Evaluation of Students

The course grade will be determined as follows:

Short assignments 40%

Evaluation project 50%

Class participation 10%

2.  Evaluation of Course and Professor

I welcome your feedback on the class, and will conduct brief process evaluations periodically during the course to invite your input.  On the basis of your comments, we will make “mid-course corrections” as necessary to ensure that the class meets your needs and is responsive to your suggestions, while still fulfilling the course objectives set out above.  A final evaluation of the course and the professor will be conducted during the last class session.

Required and Supplemental Readings

There are two required resources for this course:

Daniel L. Stufflebeam and Anthony J. Shinkfield. Evaluation Theory, Models, and Applications. San Francisco:  Jossey-Bass, 2007.

Sherril B. Gelmon, Anna Foucek and Amy Waterbury.  Program Evaluation:  Principles and Practices.  2nd edition.  Portland:  Northwest Health Foundation, 2005.  (Copies may be downloaded from http://nwhf.org/resources/.)

Many other useful books and journals on program evaluation exist.  You are encouraged to reference these books and related materials, available at the PSU and OHSU libraries as well as at other public and private libraries, for additional information and relevant examples as well as for background information for your assignments.  I will provide assistance if you wish specific direction with respect to identifying particular resources.

Description of Major Assignments

There are three components to the grade for this course:  two short assignments, a team evaluation project, and class participation.

1.  Short Assignments (40% of course grade)

There will be two short assignments, each worth 20% of the course grade.

Assignment #1:  Critique of an Evaluation Article

This will be due at the beginning of the class session on January 31^st.  A late assignment will result in a reduction in course credit.  Graded assignments will be returned on February 7^th.  The purpose of this assignment is to reflect on foundational material covered in the first sessions of the course, and to use your learning from these sessions to critique an article reporting evaluation/research findings.  In class you will be given a set of guidelines for completing the critique.  You will have a choice of one of several articles from the American Journal of Public Health, available in the PSU, OHSU or other health-related libraries (or through on-line access).  You should obtain a copy of your selected article.  The list of articles is provided in Appendix 1; they have been selected to reflect a variety of topics as well as a range of uses of evaluation strategies and methodologies.

The critique should be completed independently, and may be no more than 5 pages in length (references are additional to the 5 pages).  It must be typed, double-spaced using a 12 point font, with standard 1″ margins, and proofread for spelling and grammatical accuracy.  No references other than the textbook, course materials and the article itself are necessary. Any citations should be presented using a generally accepted citation format (such as APA).  Please do not attach a copy of the article to your submission.

Assignment #2:  Critique of an Evaluation Model

This will be due at the beginning of the class session on February 14^th.  A late assignment will result in a reduction in course credit.  Graded assignments will be returned on February 21^st.  The purpose of this assignment is to consider and critique one of four commonly recognized models of evaluation (or approaches).  The four models are:

• Stufflebeam’s CIPP Model/Improvement-Accountability Approach (Chapter 15)
• Scriven’s Consumer-Oriented Approach (Chapter 16)
• Stake’s Responsive/Client-Centered Approach (Chapter 17)
• Patton’s Utilization Focused Evaluation (Chapter 18)

You should read all four chapters, and then select one of the models.  Using material covered in the class to this point, compare and contrast your selected model with the other three models.  Frame your analysis by drawing upon personal experiences with evaluation as well as any of the 26 approaches described in Chapters 6-10 that are relevant to your discussion.  Address the following points in your paper:

• Description of the core elements of the model/approach (no more than a ½ page)
• Benefits of this model/approach for health program evaluations (with comparisons to at least two of the other three)
• Limitations of this model/approach in health program evaluations (with comparisons to at least two of the other three)
• Questions or concerns you would have about using this model (with comparisons to all of the other three) (no more than a ½ page)

The critique should be completed independently, and may be no more than 5 pages in length (citations are additional to the 5 pages).  It must be typed, double-spaced using a 12 point font, with standard 1″ margins, and proofread for spelling and grammatical accuracy.  Any citations should be presented using a generally accepted citation format (such as APA).

2.  Evaluation Project (50%)

A major part of your learning in this class will be achieved through the application of theory and concepts to an actual program evaluation project.  Through this project, you will also participate in community-based learning (also known as service-learning) — working with a community organization to develop an evaluation framework that will respond to the organization’s needs and assets, and will be of future use to that organization. The objectives of this experience for the students are to:

• gain knowledge about the program/organization for which the evaluation design is being prepared;
• meet with program/organization leadership to identify program objectives, intent of the evaluation, and anticipated outcomes;
• design a realistic and comprehensive evaluation plan that is compatible with program/organizational assets, needs and resources; and
• present the evaluation plan to the program/organization, the professor and the class.

An important aspect of community-based learning is to provide service to the community partner (the organization or agency for whom you will design the evaluation).  Therefore, there are also objectives for the community partner; these are to:

• inform students about the program/organization;
• meet with students to identify program objectives, intent of the evaluation, and anticipated outcomes;
• provide consultation to students throughout the process as necessary, and provide relevant background information; and
• review the evaluation plan and offer periodic feedback to the students.

Other objectives may be defined between the students and the community partner at their initial meeting.  The intent of the project is not to conduct the evaluation, but to design an evaluation framework that the community partner can then use (frequently a student returns to the organization to conduct the evaluation during a subsequent field experience or special project).  It is anticipated that students will meet with their community partners at least three times — an initial information gathering meeting of the entire student team with the partner representative, a meeting to discuss preliminary ideas and collect additional information, and a final meeting for the entire student team to present the evaluation plan.  The second meeting could be a telephone discussion.  Additional contact is often necessary, either by e-mail or telephone.

A description of the projects will be provided at the first class session, students will rank their individual preference for the evaluation projects, and I will determine the assignment of teams to the various projects during that class.  A list of this year’s community partners is included in Appendix 2.  Each group will receive a contact name, with phone number and email address.  Within two weeks, each group must meet with its community partner to initiate the evaluation design process, and will report on this meeting to the entire class on January 24^th.  The protocol for the final evaluation paper, which identifies the expected components of your paper, is appended to this syllabus (Appendix 3).  The exact content will vary with the nature of the organization and the intent of the evaluation design.

Each evaluation team will make a brief presentation of its evaluation framework; presentations will occur in class on March 7^th and March 14^th.  Each presentation will be up to 15 minutes in length, followed by 10 minutes for class discussion facilitated by the presenting evaluation team.  The presentation should provide a brief overview of the organization and the intent of the evaluation, and then focus on the proposed framework and methodology.  All members of the group must participate in the presentation.  Teams should provide handouts for all class members of your evaluation methodology and instruments (e.g. survey, interview protocol, focus group script, etc.) so that the discussion may provide feedback on possible ways to strengthen the methodology.  Teams should invite your community partner to attend the presentation (this may take the place of the formal presentation to the partner, unless they request a presentation at their organization); we will schedule the timing of the presentations to accommodate the partners’ availability.  Teams should email the PowerPoint slides for your presentation to me by 6:00 p.m. of the day before you are scheduled to present.

After your presentation you may wish to make minor changes in your paper prior to submission. One integrated paper will be submitted for each project team, following the guidelines in Appendix 2.  The paper should be typed, double-spaced (using a 12 point font), proofread for spelling and grammar, and no more than 20 pages in length (appended materials, sample evaluation instruments, and references are additional to the 20 page limit). All citations should be presented using a generally accepted citation format (such as APA).  Each student will also append an independently written 1-2 page reflection statement on lessons learned from working in the community (see Appendix 2).  Evaluation papers for those groups presenting on March 7^th are due no later than Wednesday, March 9^th; papers for those presenting on March 14^th are due no later than Wednesday, March 16^th.  Papers are due by 4:00 p.m. in hard copy to my office.

The 50% of course credit allocated for this assignment will be distributed as follows:  25% for the presentation and 25% for the written paper.  Each student on a team will receive the same credit for the written paper; the presentations will be judged individually by the professor (10% group evaluation, 10% individual evaluation), with 5% allocated for peer evaluation.  There is no grade assigned for the reflection statement, since this is a personal observation; nonetheless it is required and should be given careful attention.  Since much of the grade is a group evaluation, it is particularly important that you give careful attention to the quality of the group products, ensuring that the collective work is well integrated, the presentation is focused and within the time limits, and the paper reads as a coherent single work.  If you experience difficulties within your group, please contact me.

3.  Class Participation (10% of course grade)

Ten percent of the grade will be allocated to class participation; this includes preparation for each class session, active participation in class discussions, and general involvement in class activities.  Participation in the class sessions is an important part of your learning — to reflect on the reading and thinking you are doing related to the course content, engage in discussions with your classmates and professor, share your experiences with respect to your evaluation project, and ask questions and seek answers from all participants in the class.  You are expected to attend all class sessions; the proportion of course grade allocated to class participation will reflect your level of participation and demonstrated learning.  If you must miss a class, please let me know in advance so that we may discuss how you may make up missed material.

Inappropriate use of technology during class will result in loss of the entire 10% for class participation.  Inappropriate use includes:  texting, using your laptop for games or Internet surfing during class, cell phone/Bluetooth usage, Tweeting, updating Facebook pages, etc.

Expectations

A course syllabus can be considered as a contract between the professor and the students.  This syllabus includes all expectations for performance in the class, and you should now understand what is required of you, and the deadlines for assignments.  If you have questions about any of these expectations, I encourage you to discuss them with me sooner rather than later.  Any changes in the course requirements or schedule will be communicated in class.

Student Code of Conduct

In a graduate level course, students are clearly expected to do their own work, as stated by PSU policy.  Plagiarism and other forms of academic dishonesty will result in the grade of zero for the work involved and may, if in the judgment of the instructor that the particular case warrants it, result in the grade of “F” for the course and/or referral to the University for further action. To learn your rights and responsibilities as a member of the Portland State Community, please review the Student Code of Conduct that describes behavior for which a student may be subject to disciplinary action (http://www.pdx.edu/dos/codeofconduct).

Accommodation for Disabilities

If you have a disability and are in need of academic accommodations, please notify me immediately to arrange needed supports.  If you are registered with the Disability Resource Center, please provide me with the paperwork they have given you regarding accommodations.

Course Schedule

The following is the anticipated schedule of class topics, readings, and assignments.  All readings are from the required textbooks (other than where noted), and should be completed before the class for which they are assigned.  NOTE:  Please bring both of the resource books with you to each class, as well as electronic or hard copies of other assigned readings.  Any other readings will be distributed/announced in class the week prior to their use.

Session 1:  January 3

Introductions and course overview

Introduction to program evaluation

Fundamentals, theory, standards, logic of evaluation

Overview of group projects and determination of groups for projects

Readings: Stufflebeam & Shinkfield, Chapters 1, 2, 3; Gelmon, Foucek & Waterbury, Chapter 1

Session 2:  January 10

Approaches to evaluation

Feasibility and value of evaluation, evaluability and needs assessments

Creating an evaluation matrix

Readings: Stufflebeam & Shinkfield, Chapters 5, 6, 7, 8, 9, 10, 11 (skim); Gelmon, Foucek & Waterbury, Chapters 2, 3

B.A. Israel, C.C. Coombe, R.R. Cheezum, et al.  “Commnity-Based Participatory Research:  A Capacity-Building Approach for Policy Advocacy Aimed at Eliminating Health Disparities.”  American Journal of Public Health 100 (November 2010): 2094-2102.

Session 3:  January 17

No class — Martin Luther King holiday

Session 4:  January 24

Evaluation designs (experimental, quasi-experimental, randomized control, case study)

Issues of reliability and validity

Ethical issues and protection of human subjects in evaluation

Readings: Stufflebeam & Shinkfield, Chapters 13, 14; Gelmon, Foucek & Waterbury, Chapter 4; Review “Research Integrity: Human Subjects” with specific reference to guidelines for determining nature of review at http://www.rsp.pdx.edu/compliance_human_process.php

Centers for Disease Control and Prevention.  “The Tuskegee Timeline.”  Available at < http://www.cdc.gov/tuskegee/timeline.htm>

Centers for Disease Control and Prevention.  “How Tuskegee Changed Research Practices.”  Available at < http://www.cdc.gov/tuskegee/after.htm>

United States Holocaust Memorial Museum.  “The Doctors Trial (the Medial Case of the Subsequent Nuremberg Proceedings).”  Available at < http://www.ushmm.org/research/doctors/twoa.htm>

Groups:  Report on first meeting with community partner

Hand out short assignment #1 instructions

Session 5:  January 31

Methods used in evaluation #1:  Surveys

Readings:  Gelmon, Foucek & Waterbury, Chapters 5, 6D.H. Odierna and L.A. Schmidt.  “The Effects of Failing to Include Hard-to-Reach Respondents in Longitudinal Surveys.”  American Journal of Public Health 99 (August 2009): 1515-1521.

Groups:  Bring evaluation matrices for class review

Due:  Short assignment #1

Session 6:  February 7

Methods used in evaluation #2:  Focus Groups, Interviews

Readings: Gelmon, Foucek & Waterbury, Chapters 5, 6

Groups:  Bring sample surveys for reviewShort assignment #1 returned

Session 7:  February 14

Class critique of evaluation approaches

Readings: Stufflebeam & Shinkfield, Chapters 15, 16, 17, 18

Groups:  Bring sample focus group and interview protocols for review

Due:  Short assignment #2

Session 8:  February 21

Methods used in evaluation #3:  Observations, Documentation, Use of Experts

Evaluation budgets

Readings: Gelmon, Foucek & Waterbury, Chapters 5, 7; Stufflebeam & Shinkfield, Chapter 22

Groups:  Bring final questions on evaluation design and methods

Short assignment #2 returned

Decide on group presentation schedule

Session 9:  February 28

Evaluation information collection, analysis, synthesis and reporting

Managing, completing, using and communicating evaluations

Pitfalls of evaluation

Readings: Stufflebeam & Shinkfield, Chapters 20, 24, 25, 26; Gelmon, Foucek & Waterbury, Chapter 8

Groups:  Bring evaluation budgets

Session 10:  March 7

Student presentations (four groups)

Due:  Final papers for these presentations to my office by 4:00 p.m. Wednesday, March 9th

Session 11:  March 14

Student presentations (two groups)

Due:  Final papers for these presentations to my office by 4:00 p.m. Wednesday, March 16th

Synthesis of course themes

Final course evaluation

Graded papers will be available for pickup from the PA office on March 28th.

APPENDIX 1

Articles for Short Assignment

T. Dumanovsky, C.Y. Huang, M.T. Bassett, and L.D. Silver.  “Consumer Awareness of Fast-Food Calorie Information in New York City after Implementation of a Menu Labeling Regulation.”  American Journal of Public Health 100 (December 2010): 2520-2525.

R.C. Klesges, D. Sherrill-Mittleman, J.O. Ebbert, G.W. Talcott, and M. DeBon.  “Tobacco Use Harm Reduction, Elimination, and Escalation in a Large Military Cohort.”  American Journal of Public Health 100 (December 2010): 2487-2492.

A.P. Martinez-Donate, J.Z. Zeliner, F. Sanudo, et al.  “Hombres Sanos:  Evaluation of a Social Marketing Campaign for Heterosexually Identified Latino Men Who Have Sex with Men and Women.”  American Journal of Public Health 100 (December 2010): 2532-2540.

A.E. Maxwell, R. Bastani, L.L. Danao, et al.  “Results of a Community-Based Randomized Trial to Increase Colorectal Cancer Screening Among Filipino Americans.”  American Journal of Public Health 100 (November 2010): 2228-2234.

T.N. Nguyen, J.H. Tran, M. Kagawa-Singer, and M.A. Foo.  “A Qualitative Assessment of Community-Based Breast Health Navigation Services for Southeast Asian Women in Southern California:  Recommendations for Developing a Navigator Training Curriculum.”  American Journal of Public Health 101 (January 2011): 87-93.

A.M. Novoa, K. Perez, E. Santamarina-Rubio, et al.  “Impact of the Penalty Points System on Road Traffic Injuries in Spain:  A Time-Series Study.”  American Journal of Public Health 100 (November 2010): 2220-2227.

S.E. Samuels, L. Craypo, M. Boyle, et al.  “The California Endowment’s Healthy Eating, Active Communities Program:  A Midpoint Review.”  American Journal of Public Health 100 (November 2010): 2114-2123.

T.G.M. Sandfort, J.M.W. Bos, K.L. Collier, and M. Metselaar.  “School Environment and the Mental Health of Sexual Minority Youths:  A Study among Dutch Young Adolescents.”  American Journal of Public Health 100 (September 2010): 1696-1700.

T.K. Takaro, J. Krieger, L. Song, D. Sharify, and N. Beaudet.  “The Breathe-Easy Home:  The Impact of Asthma-Friendly Home Construction on Clinical Outcomes and Trigger Exposure.”  American Journal of Public Health 101 (January 2011): 55-62.

J.F. Thrasher, R. Perez-Hernandez, K. Swayampakala, et al.  “Policy Support, Norms, and Secondhand Smoke Exposure Before and After Implementation of a Comprehensive Smoke-Free Law in Mexico City.”  American Journal of Public Health 100 (September 2010): 1789-1798.

R.R. Wing, M.M. Crane, J.G. Thomas, R. Kumar, and B. Weinberg.  “Improving Weight Loss Outcomes of Community Interventions by Incorporating Behavioral Strategies.”  American Journal of Public Health 100 (December 2010): 2513-2519.

P.A. Wyman, C.H. Brown, M. LoMurray, et al.  “An Outcome Evaluation of the Sources of Strength Suicide Prevention Program Delivered by Adolescent Peer Leaders in High Schools.”  American Journal of Public Health 100 (September 2010): 1653-1661.

APPENDIX 2

Community Partners for Winter 2011

1. Growing Gardens, Youth Grow Program – www.growing-gardens.org

2. Ladder to Leadership, Patient Navigator Program

3.  Friends of Zenger Farm, Healthy Eating on a Budget Program – www.zengerfarm.org

4.  Oregon Foundation for Reproductive Health, One Key Question Program – www.prochoiceoregon.org

5.  Preventive Medicine Residency Program, Oregon Health & Sciences University, Training Residents for Prevention and Policy Program – http://www.ohsu.edu/xd/education/schools/school-of-medicine/departments/clinical-departments/public-health/education-programs/preventive-medicine-residency/index.cfm

6.  Portland Police Bureau, Crisis Intervention Training Program – http://www.portlandonline.com/police/index.cfm?c=30680

APPENDIX 3

Final Evaluation Paper Protocol

The group evaluation paper should be prepared as follows:

1.  Descriptive information about the organization where the program/activity is located

2.  Program description, goals and objectives, and scope of activities

3.  Purpose of the evaluation

4.  Literature review, documenting relevant evaluations that provide the basis for your design and choice of methods (include complete citations in a bibliography of all references)

5.  Conceptual matrix and discussion of core concepts and key indicators

6.  Evaluation design, including choice and justification of selection of design, other designs considered, costs/benefits of selected design, methods to address threats to validity

7.  Measurement methods and data sources identified in conceptual matrix; discussion of selection and design of data collection instruments; drafts of sample instruments (appended); issues of sample size (where applicable)

8.  Discussion of involvement of human subjects, including anonymity, confidentiality and socio-economic-cultural issues; need/procedures for Institutional Review Board review/approval

9.  Proposed methods for synthesis and analysis of data, as well as strategies for presenting results and findings

10.  Budget for the evaluation (budget statement and narrative), illustrating major categories of expenditures (human resources, operations, supplies, travel, etc.); indicate sources of funds (confirmed or potential) and alternatives for funding

11.  Anticipated uses of the evaluation; suggested alternative methods of reporting and dissemination of results to multiple audiences.

12.  Appraisal of the likelihood of completion of the evaluation by the community partner; resources needed for completion

Appended to the group paper from each individual group member:

13.  Reflections on community-based learning and lessons learned – each team member should independently write and submit a 1-2 page statement; these may be appended to the final paper

The copy of the paper you give to the partner should not include #12 or 13.

NOTE:  Since you will hand in one final paper per team, it is important that you allow sufficient time to prepare the paper so that your submission is an integrated, consistent report written in a single voice, rather than a series of fragments pasted together.  This should be a well-edited, error-free, professional report that the community partner will be pleased to receive.