|
Home > Program Evaluations: A Synthesis of Lessons Learned by Child Neglect Demonstration Projects > Challenges and Successful Strategies for Evaluation Design and Methodology
Program Evaluation: A Synthesis of Lessons Learned by Child Neglect Demonstration Projects
Grantee Lessons Learned
Author(s): United States. Children's Bureau.
|
| Year Published: 2005 |
1. Challenges and Successful Strategies for Evaluation Design and Methodology
All 10 grantees encountered challenges related to design and methodology. These challenges, and the strategies grantees reported to be successful in addressing them, are summarized below. They relate specifically to the processes of:
1.1 Selecting Instruments
All 10 programs used standardized instruments, and many used the same ones. The table in the appendix shows which instruments each program used. The most commonly used were:
- Child Well-Being Scales (6 programs)
- Adult Adolescent Parenting Inventory (4)
- Child Behavior Checklist (4)
- Home Observation for Measurement of the Environment (4)
- Family Support Scale (2)
- Maternal Social Support Index (2)
- Parenting Stress Index (2)
Grantees did not recommend using standardized instruments alone for the evaluation of these programs. One grantee felt that these commonly used standardized instruments were designed for individual therapy sessions or for populations with less pervasive problems and therefore did not work well for measuring outcomes with the grantee's target population. The majority of projects found the Child Well-Being Scales to be the most appropriate and useful instruments, but one program found that these scales were not sensitive to detecting risk and were more appropriate for higher functioning clients.1 Another grantee reported that its interventions simply did not address some of the domains that the standardized instruments assessed and, thus, these instruments would not detect change.
Administering the instruments also proved challenging. One program reported that participants' low literacy levels, test anxiety, absenteeism, and hostility made it difficult to obtain valid and reliable pre- and posttest scores. Another grantee reported that in-home assessments were difficult to administer (these were not CPS-mandated cases, so home access was voluntary). Some participants were afraid to allow clinicians into their homes due to substandard living conditions, overcrowding, domestic violence, or drug dealing. Those who did allow home visits were self-selected and may have had fewer of these concerns. Many young mothers lived with family members and were not free to invite clinicians into the home without the owner's permission. When access was possible, some grantees' staff also encountered scheduling and safety issues.
Some of the strategies suggested by grantees to address these challenges include:
- Modify existing instruments to meet your needs. For example, recommendations for use of the Child Well-Being Scales included clustering items to match program objectives and adding a seriousness scale to make the findings more interpretable.2
- Develop your own instrument. The Parent Empowerment Program believed its own instrument, the Personal Goal Achievement Measure (PGAM), assessed outcomes more accurately and meaningfully than the available standardized instruments. When program participants and program staff worked together to develop this instrument, participants were empowered by selecting their own goals and working toward meeting established criteria for successful achievement.3
- Use multiple strategies to collect data. All programs collected qualitative data to help them interpret and clarify the results of the standardized instruments. These data collection tools and techniques included:
- Referral and discharge forms
- Mentors' pre- and post-intervention surveys, weekly reports, and in-service meeting notes
- Surveys of presenting problems (types and severity), focus groups, life history research, and satisfaction surveys
- Staff notes, service documentation sheets, family case logs, activities notes, meeting minutes
- School interviews
- CPS verification
- Exit interviews with families and staff
1.2 Establishing Comparison Groups
Six of the 10 projects reported program evaluation challenges related to the establishment of treatment and comparison groups. A number of factors affecting the use of comparison groups may have compromised evaluation results, including voluntary participation, statutory requirements that made a "no-treatment" group impossible in at least one site, inappropriate referrals from community partners, difficulties achieving random assignment to groups, and unintended variations in service intensity.
In one program, only a few families were able to successfully engage and graduate, so it was not possible to compare treatment and comparison group service outcomes as planned. At least one grantee felt that the process of assigning some families to a comparison group and the fact that some families could not receive full services competed with the program's goals. While not denying the value of experimental design for determining program effectiveness, they noted that without this constraint their services would have reached and benefited more families, resulting in better outcomes overall.
Strategies used or suggested by grantees include:
- Use service enhancements. Several programs avoided having to deny services to families in their comparison groups by providing core services to these families and adding service enhancements (such as a mentoring component) or longer-term services (such as 9 months rather than 3 months) for treatment group families.
- Establish and communicate clear participation criteria. Several grantees recommended establishing clear eligibility criteria for entry into the program to screen out families whose needs were greater than the program was designed to address. It was also recommended that care be taken during the process of assigning families to treatment and comparison groups to ensure that eventual outcomes will not be biased because of initial differences between groups in their risk of future neglect.
1.3 Collecting Data
At least seven grantees reported difficulties related to data collection. Challenges in this area varied widely, and related to:
- Establishing baselines. Several grantees reported difficulty accurately evaluating families at intake. One program reported that this intake process took 6 weeks, often due to family crises. Families drifted in and out of the program and moved in and out of crises, so their progress could not be represented as a straight line from intake to exit. Another program indicated that intake scores were unrealistically high, probably because families were not comfortable telling a stranger how bad things were or because their chaotic lives seemed normal to them. While thorough assessments were considered essential, the intake process was often found to be burdensome for large families. It sometimes took them 4 to 6 hours to complete the numerous standardized measures and provide demographic data.
- Collecting follow-up data. Grantees found many of the families were difficult to locate or contact. Follow-up interviews with caretakers sometimes were delayed due to scheduling difficulties. One program reported that posttest data were more difficult to collect simply because clients were reluctant to exit the program.
- Involving children. Children presented unique data collection challenges, because they often were unavailable or too tired to complete the process. One program found screening preschool-aged children nearly impossible, as parents were reluctant to risk having their children labeled at an early age.
- Using self-report measures. One program reported differences in findings from observational versus self-report measures. In several instances, self-report measures identified significant depression and child behavior concerns that were not observed by student interns.
- Generating large enough samples. One program reported that use of assessment instruments by the various intake workers was inconsistent, resulting in significant gaps in data. Another grantee reported that due to project extensions to maximize sample size, the 12-month follow-up period was not reached for all families at the time of the final report, so findings for some of their evaluation questions could not be included at that time. A third program reported that their sample size was only half of the projected size. Another program reported a high attrition rate: 438 families were referred, 250 enrolled, 195 started the program, and only 87 graduated.
Strategies used or suggested by grantees included:
- Streamline data collection. Programs recommended striving for a balance between the need to conduct an in-depth initial assessment and the need to streamline initial demographic data collection. One recommendation was to use Bavolek's Adult-Adolescent Parenting Inventory (AAPI) as a clinical tool to initiate conversation while gathering baseline data. The AAPI is a brief agree/disagree survey designed to assess parenting attitudes. Staff reported that using this tool was an efficient and effective way to gather data while engaging the family. (See Appendix C for more information on the AAPI.) Another program found it helpful to limit the amount of information families were asked to provide during initial meetings with agency staff.
- Use multiple data collection methods. Programs recommended using many sources of qualitative and quantitative data and gathering data more often than just when they enter and exit the program. One grantee recommended getting releases from families for access to Department of Social Services data to track what happens to families once they exit the program.
- Offer incentives. Providing incentives, such as cash or "graduation" parties, to families for participating in baseline and follow-up interviews was recommended.
- Gather child data. Most of the programs collected data on children as well as families. One program had success assessing children while in child care during parent group meetings.
1.4 Analyzing the Impact of Individualized Services
Programs frequently customized services, provided resources tailored to individual needs, changed plans when families were in crisis, and varied the level of service to meet families' unique needs. At least 8 of the 10 grantees agreed that providing multifaceted and flexible services was good for meeting families' needs. However, this strategy also makes comparisons across families more difficult and makes it very challenging to describe a replicable service model.
Strategies employed by grantees to address this challenge included:
- Document service variations. Several grantees opted to closely monitor the intervention to ensure it was being implemented as planned and to docuent variations as they occurred.
- Work closely with the program evaluator. Several grantees believed that having the program evaluator participate in program activities and attend staff meetings deepened their understanding and helped ensure that the evaluation evolved along with the program.
1 Magura, S., and Moses, B. S. (1987). Outcome measures for child welfare services: The Child Well-Being Scales and rating form. Washington, DC: Child Welfare League of America. Available online at www.cwla.org/pubs/pubdetails.asp?PUBID=3062. back
2 There is a risk, in adapting standardized tests, of nullifying the reliability and validity of the instruments. However, creating or modifying a risk assessment instrument to use in the interpretation of the results can have merit. back
3 Data from site-specific instruments should be interpreted with caution. If reliability and validity have not been established, the ability to generalize their results is limited. back
This material may be freely reproduced and distributed. However, when doing so, please credit Child Welfare Information Gateway.
|
