 |
 |
|
|||||||||
RSS
 |
Home > Program Evaluations: A Synthesis of Lessons Learned by Child Neglect Demonstration Projects > Lessons Learned for Evaluation Management
Program Evaluation: A Synthesis of Lessons Learned by Child Neglect Demonstration Projects
Grantee Lessons Learned
2. Lessons Learned for Evaluation ManagementSeveral factors related to management of the evaluation process contributed to these grantees' ability to overcome numerous challenges and produce high quality program evaluations:
In its fiscal year 1996 funding announcement, the Children's Bureau made its expectations clear. Applicants were asked to provide a logic model and propose an evaluation plan for their projects, including methods, data, and criteria that would be used to evaluate the proposed project's processes and outcomes. As a result, all of the programs established clearly defined and measurable implementation and outcome objectives as they started their program planning. These objectives, along with operational definitions and specific measures for each outcome, helped programs assess their progress accurately and objectively. (For more information about how individual programs structured their evaluations, see Appendix B. For a sample logic model, see Appendix D.) 2.2 Sufficient Evaluation Budget and Project Duration Grantees were expected to invest 10 to 15 percent of their budgets (or approximately $15,000 to $22,500 per year) in program evaluation. This helped the programs leverage the resources needed to support an in-house evaluation or contract with an external evaluator. (See Section 2.4, Strong Evaluation Teams.) Many of these evaluators were able to invest sufficient time to become very familiar with the project. In several cases the evaluator also supported graduate students who assisted in data collection and processing. These grantees had 5 years to implement and evaluate their projects. This allowed sufficient time to develop and implement strong program evaluation plans. 2.3 Strategies to Address Staff Turnover Eight of the 10 programs reported that addressing staff turnover was a crucial part of managing the program evaluation. Staff turnover sometimes resulted in uneven data collection and analysis. One program found that its pre- and post-test evaluation design did not work well, because high staff turnover resulted in missing discharge data and inconsistency from pre- to post-testing. Another found that clients with a consistent caseworker had higher posttest scores and showed greater improvement than clients who experienced caseworker turnover. Several programs minimized the impact of staff turnover on the evaluation process by focusing on training. Some projects provided their staff with evaluation training and manuals to ensure consistency despite staffing changes. Others used videotape training to reduce errors. One program found that an intervention manual, weekly seminars, and supervision were helpful in keeping the evaluation on track. One program found using the evaluation itself as a learning opportunity helped minimize the impact of turnover. They designed their evaluation to be empowering for staff so they could learn while doing their work. Project staff reported that they enjoyed and benefited from the evaluation feedback provided about the types of success their efforts were supporting. All 10 projects assigned program evaluation responsibilities to qualified individuals or organizations. Universities operated five of the projects, and a medical center operated one. Four of these projects used their own in-house program evaluators. All but one of the remaining programs used nonuniversity outside evaluators. Several of the evaluation teams also performed or assisted with data collection and entry. In several instances, the evaluation team also assisted with providing training and developing manuals. Most of the program evaluators stayed with the program for the full 5 years, although one of the grantee programs changed its lead evaluator three times in 5 years and two projects using graduate student data collectors experienced schedule conflicts and high turnover due to graduation. All the evaluators maintained close regular contact with the programs. Grantees reported that having evaluators with in-depth knowledge of the program and its participants, and having staff willing to implement a program with an extensive evaluation component, were key components in the evaluation's success. In particular, one program reported that placing research staff in the agency during the delivery of services to monitor recruitment of families, random assignment to treatment groups, timely completion of data packets, and service parameters (such as length) may have improved targeting, sample size, and adherence to the intervention's original definitions. 2.5 Federal Evaluation Supports: Grantee Cluster and Technical Assistance From the beginning, these projects were intended to operate cooperatively as a grantee cluster. The Children's Bureau facilitates this cooperation within clusters of discretionary grantees by providing technical assistance and by encouraging the development of common evaluation criteria, data elements, and measures to maximize comparability of evaluation findings. Grantees reported that operating as a cluster led to stronger and more credible results. Project staff were able to refine their evaluation plans and implementation processes at annual grantee meetings and by keeping in touch via a listserv. Together with their Federal Project Officer and the evaluation technical assistance provider, grantees developed a uniform final report format that emphasized implementation and outcome evaluation strategies and results and, most importantly, helped them think through the relationship between program implementation and participant outcomes.
This material may be freely reproduced and distributed. However, when doing so, please credit Child Welfare Information Gateway. |
||||||
|
|||||||
Email
Order (Free)
Print (PDF 236 KB)
Share
Rate This
