By Pamela Jull, PhD
Applied Research Northwest
t’s easy to get caught up in the excitement of planning the activities of an environmental education program. Programs are neat. They are often fun and innovative and they usually give participants a memorable, positive experience.
Sometimes in the thrill of innovation, the concrete goals of a program get lost. By planning your evaluation as you plan your program, you can better insure that you produce the results you intended when you came up with your great program idea.
An evaluator thinks of program planning this way:
• What are the specific skills and knowledge you intend your audience to acquire?
• How will you know if your audience has acquired the skills and knowledge you intended for them? What will it look like? What will they be able to do? What will they be thinking?
• What systematic information can you observe or collect to verify your belief that they have learned what you expected?
The best program design makes evaluation possible without it being an added burden to the program staff, and it provides useful information that the staff values in a timely way. Asking these questions early in the planning gives you the opportunity to embed the evaluation in the design. If this happens too late, the program agenda is often too firmly set and structured, and you may lose the flexibility to integrate good assessments into your program.
For example, perhaps you have a registration form for participants. That form could contain a few well-crafted questions that enable you to set a baseline or capture some information about your learners. Maybe as a result of planning, you will decide to add an initial reflection to show program staff their learners’ attitudes or understanding of the topic. Do you want to know if they used what they learned? Contacting people after their educational experience is not difficult if it’s planned for in advance, for example by collecting email addresses and priming participants to look for a follow up from your organization.
Perhaps one reason evaluation is sometimes stuffed to the margins is the fear factor. People often ask me “What if we find out our program is not working?” Failure is a depressing prospect, but my response is clear:
• Environmental education programs are designed by thoughtful highly motivated people. There is always success to be found in at many elements of the program
• Even negative findings nearly always produce a positive sense of efficacy among staff. It is really nice to know what’s working as well as what is not rather than guessing.
• Program staff almost immediately start puzzling through potential solutions, highly motivated to improve upon what they’ve accomplished.
The motivation to improve generates optimal program results. Staff might ask: Was there something unexpected about the group of learners that would account for the outcomes? Do the results point to an opportunity to make an easy adjustment, for example by giving a specific learning module more emphasis, or better clarifying an important concept? Reviewing program implementation is also helpful: Did something in the program delivery change from the plan due to some unforeseen constraint that may have removed an effective component?
Getting help is a good move. A good evaluator will help program managers think about their program as a system. Systems thinking can lead to a stronger sense of cause and effect, and a stronger program design. Solid evaluations are recognized by funding agencies as the hallmark of thoughtful program developers, and a great evaluation can set you up for more easily obtaining more program funding. The American Evaluation Association (www.eval.org) has a list of evaluators by region. Find someone close by who has an interest in your area.
Dr. Jull is a Cornell-trained sociologist and applied social scientist with more than 15 years experience in research and program evaluation. Her clients have included school districts, educational service districts, non-profits and local and state agencies engaged in environmental and K-16 education.