by Chuck Lennox
Principal (Consultant)
Cascade Interpretive Consulting LLC
Program evaluation is a valuable process to determine the efficacy of programming being offered to the public by an organization or agency. Sometimes the process can feel intimidating. How do I know we made a difference? How do I get started? Logic models help us develop an evaluation process in an organization or agency. They are also a useful tool to organize information in a succinct but comprehensive manner for grant applications and funding proposals. Using logic model formats encourage us to ask difficult questions that focus our goals and prioritize our efforts.
Introduction
Program evaluation can be challenging for front-line staff and their managers. Can someone who develops and presents a program effectively evaluate it themselves? What is the impact of a program on the public and target audience? How does a manager know if a program contributes to an agency’s or organization’s goals in a cost effective manner? First and foremost in many professionals’ minds, how do I get started in this process?
Logic models are an excellent tool to gather information needed in an evaluation. Foundations, government agencies (i.e. National Oceanic and Atmospheric Administration’s B-WET grants and Environmental Protection Agency’s EE grants) and other funders are beginning to even require grant applicants to submit logic models as part of the application process. Logic models help us gather information, organize it and answer critical questions about our goals and priorities.
Getting Started with Logic Models
The first step is to identify your organization or agency’s mission statement, goals and objectives. Many employees may not even know this information. The evaluation process should be linked back to this information.
Next, determine your work group’s goals and objectives. Do you have an annual work plan? An annual plan determines what management expects of you in the course of a year. Is your personal work plan linked to the organization or agency plan? Good evaluation should always link back to the mission and work plans developed for the organization or agency. If your work is not relevant to the mission, why is it being done?
Components of a Logic Model
The second step is to identify and clarify the different elements of the program you want to assess. Use Figure #1 below and the description provided to fill in the logic model. Information is generally inserted into the logic model to read left to right.
Inputs are all of the resources you need to carry out your activities and accomplish your Outcomes such as staff members (from the registrar to the maintenance worker – list the number of projected hours or FTEs – Full-time Equivalents), funding (grants, budgets, in-kind donations), volunteers, partners, supplies and materials, handouts, etc. Inputs should be quantified in as much detail as known.
Outputs are divided into two categories – Activities and Participants. Activities are what you do with the Inputs or resources you have to achieve your Outcomes. Examples include outreach events, classroom visits, public programs, festivals, forums or speaking engagements. Quantify (actual or anticipated) the Activities to show information such as frequency, duration and participation.
Participants are those individuals or groups that are your target audience. Keep in mind the breadth and depth of an audience. For example, school groups generally are accompanied by a teacher (or teachers) and adult/parent chaperones. In this case, all three are participants. Volunteers supporting the program might also be considered participants if your goal is to increase their knowledge or support of your operation.
Outcomes are the changes or benefits you are want to see that effect individuals or groups during or after participating in Activities. Examples might include less litter, reduced vandalism, increased numbers of volunteers, greater understanding of a watershed or reduction in feeding wildlife. Outcomes are NOT number of participants, viewers or their satisfaction levels. Generally, short-term outcomes occur within a time frame that allows you to measure them such as the length of a grant-funded program. Medium-term outcomes might extend out beyond your initial program length and could be measured if you had the time and resources available.
Typically, you may not be able to measure long-term outcomes due to lack of resources (time and budget) but can reflect what you hope would be the ultimate goal(s) accomplished (i.e. greater appreciation for wildlife, support for historical preservation or clean surface waters). Time frames between short, medium and long-term outcomes will vary depending on activities or the type of program.
Final Thoughts
Logic models are the first step of an evaluation process and an excellent tool for organizing information, resources and questions during program development, preparing for a grant and beginning an evaluation process. Once a logic model is complete, an evaluation plan is developed that asks us the questions of what do you want to know, how will you know it, who will have the information and how will you gather the data. Developing a logic model is the first step on the path to program evaluation.
References
Publications
Ernst, Julie A.; Monroe, Martha C.; and Simmons, Bora. Evaluating Your Environmental Education Programs: A Workbook for Practitioners, Washington D.C.: North American Association for Environmental Education. 2009
Diamond, Judy. Practical Evaluation Guide: Tools for Museums & Other Informal Educational Settings, Lanham, MD: Altamira Press — American Association for State and Local History Book Series. 1999
Wiggins, Grant and McTighe, Jay. Understanding by Design. Alexandria, VA: Association for Supervision and Curriculum Development. 1998
Web Resources
Designing Evaluation for Education Projects A great resource from NOAA’s Office of Education and Sustainable Development available in a PDF format. http://wateroutreach.uwex.edu/use/documents/NOAAEvalmanualFINAL.pdf
Measuring the Success of Your Environmental Education Programs Another great on-line resource that will help you develop an evaluation plan for your EE program available in a PDF format.
http://www.peecworks.org/PEEC/PEEC_Inst/I00052276.0/ee-success.pdf
My Environmental Education Evaluation Resource Assistant An on-line “evaluation consultant” created to assist you in your evaluation needs. http://meera.snre.umich.edu/
National Extension Water Outreach Education Resources for program evaluation with an emphasis on the use of Logic Models. http://wateroutreach.uwex.edu/use/Evaluate.cfm
Chuck Lennox
507 27th Avenue
Seattle, WA 98122
chuck.lennox@interpconsulting.com
http://www.interpconsulting.com