The purpose of these projects was to develop, implement, evaluate, and disseminate training curricula designed to strengthen the supervision provided to child welfare staff working with older youth in foster care and/or in independent living programs. Six grants were awarded under this funding opportunity announcement to universities across the United States. Summaries of project activities and findings are provided here. This synthesis was a collaborative effort by Child Welfare Information Gateway and James Bell Associates.
NOTE: Projects will be identified by the name of the State in which they are located (e.g., "University of Houston" will be referred to as "Texas") for ease of reading.
|Lead Agency||San Francisco State University (California)|
|Project Title||YOUTH Training Project|
|Target Population||Child welfare supervisors and key managers|
|Contacts||Jamie Lee Evans
|Lead Agency||University of Iowa (Iowa)|
|Collaborating Partners||Iowa Department of Human Services|
|Project Title||Improving Outcomes for Youth in Transition|
|Target Population||Iowa child welfare supervisors|
|Lead Agency||University of Louisville, Research Foundation, Inc. (Kentucky)|
|Project Title||Evidence-Based Supervisor-Team Independent Living Training|
|Contacts||Anita Barbee, Ph.D.
|Lead Agency||State of Massachusetts, Department of Social Services (Massachusetts)|
|Collaborating Partners||University of Massachusetts Medical School|
|Project Title||Supervisory Training to Enhance Permanency Solutions (STEPS)|
|Target Population||DCF social work supervisors, equivalent personnel in other State agencies, and staff of contracted program providers|
|Contacts||Gretchen Hall, Project Director
|Lead Agency||Hunter College School of Social Work, CUNY Research Foundation (New York)|
|Collaborating Partners||Child Welfare League of America, National Foster Care Coalition, Oregon Department of Human Services, New York City Administration for Children's Services, Mississippi Department of Human Services|
|Project Title||Preparation for Adulthood – Supervising for Success (PASS)|
|Target Population||Public child welfare agency supervisors in MS, NY, and OR|
|Contacts||Gerald Mallon, D.S.W.
Joan Morse, L.C.S.W.
|Lead Agency||University of Houston (Texas)|
|Project Title||Preparation for Adult Living: Supervisor Training and Empowerment Program (PAL-STEP)|
|Target Population||CPS supervisors statewide|
|Contacts||Maria Scannapieco, Ph.D.
Kelly Connell-Carrick, Ph.D.
In the funding opportunity announcement (FOA), the Children's Bureau (CB) described the primary target population as "Public Child Welfare Agency Supervisors with supervisory authority over caseworkers assigned to work with youth in Independent Living Programs." Each program targeted public child welfare agency supervisors, but descriptions of grantees' primary target audiences varied across projects. For example:
- Several projects worked with staff at different levels within contracting organizations.
- At least one project provided training for supervisors for the first day and then included caseworkers on the second day.
- One project (New York) worked with its partner organizations to train public child welfare agency supervisors in MS, NY, and OR
- Each project worked with some current and former foster youth, teaching youth tangible skills to facilitate their participation in training development and delivery.
CB offered grantees a high degree of flexibility in designing and delivering curricula that would best meet the needs of public child welfare supervisors in their target communities. As described in the FOA, the overarching activity was "to develop, implement, evaluate, and disseminate a training curriculum for public child welfare agency supervisors" that would strengthen supervision of staff interventions to help older youth in foster care and/or independent living (IL) programs make a successful transition to adulthood. Major program activities and curriculum features are summarized in Table 1, Summary of Program Features and Activities.
Each grantee developed a curriculum for supervisors; however, there was wide variation in how these curricula were developed and implemented.
- Most projects began curriculum development with focus groups that included various stakeholders such as supervisors, caseworkers, foster youth, former foster youth, foster parents, and representatives from various State agencies and community groups.
- All of the projects utilized face-to-face trainings, although one grantee required completion of web-based training before attending the face-to-face training.
- Trainings ranged from a single day to 6, 6-hour sessions spread over 12 months.
- Training sessions most often enrolled child welfare supervisors and in some cases, child welfare workers.
- One project (Iowa) supported 8 all-day "Community Days" in different regions of the State to encourage collaboration across agencies such as child welfare, adult social services, and the Department of Education.
- Kentucky conducted a daylong Youth Summit.
In 2005, CB published an FOA titled, "Training of Child Welfare Agency Supervisors in the Effective Delivery and Management of Federal Independent Living Services for Youth in Foster Care." CB received proposals from applicants seeking funds to develop, implement, evaluate, and disseminate training curricula designed to strengthen the supervision provided to child welfare staff working with older youth in foster care and/or in IL programs. In the FOA, CB emphasized four basic principles that were identified by the Muskie School of Public Service at the University of Southern Maine and the National Resource Center for Youth Development at the College of Continuing Education, University of Oklahoma, as being associated with successful program designs regardless of the types of services provided:
- Positive youth development
- Cultural competency
- Permanent connections
The FOA charged applicants with designing curricula that increased child welfare supervisors' ability to provide guidance and oversight to workers in:
- Assessing a youth's readiness for IL services, support, and training
- Identifying culturally competent IL services and activities
- Utilizing positive youth development principles for involving youth in decision-making about, implementation of, and evaluation of training and program activities
- Identifying areas of stress and their impact on youth in foster care
- Helping youth deal with crisis situations and assess the results of interventions
- Working with youth to develop and maintain permanent connections
- Collaborating with both inter- and intra-agency resource people to achieve positive outcomes for youth transitioning to adulthood
This FOA built on the work of an earlier cluster of 12 grantees funded by CB in the fall of 2000 that focused on the development of training curricula for child welfare practitioners. This grantee cluster found that in order for supervisors to support their staff's casework efforts, they needed training on youth development and the unique developmental and service needs of youth in out-of-home care.
|FOA Title:||Training of Child Welfare Agency Supervisors in the Effective Delivery and Management of Federal Independent Living Services for Youth in Foster Care|
|Approved Project Period:||September 29, 2005 ï¿½ September 28, 2008|
|Funding Instrument Type:||Grant|
|Anticipated Total Priority Area Funding:||###36;1,000,000 per budget period|
|Anticipated Number of Awards:||0 to 4|
|Ceiling on Amount of Individual Awards:||###36;250,000 per budget period|
|Floor on Amount of Individual Awards:||None|
|Average Projected Award Amount:||###36;250,000 per budget period|
|Length of Project Periods:||36-month project with three 12-month budget periods|
|Match:||Grantees must provide 25###37; of the total approved cost of the project (cash or in-kind).|
- State governments
- County governments
- City or township governments
- Special district governments
- Independent school districts
- State-controlled institutions of higher education
- Nonprofits having a 501(c)(3) status with the IRS, other than institutions of higher education
- Nonprofits that do not have a 501(c)(3) status with the IRS, other than institutions of higher education
- Private institutions of higher education
To be eligible to apply for a grant, institutions of higher education had to have an accredited social work education program or other accredited bachelor's- or graduate-level programs leading to a degree relevant to work in child welfare, while State and local government entities had to be child welfare agencies. Collaborative efforts involving multiple organizations were required to designate a primary applicant to administer the grant.
Grantees identified common challenges in the design, implementation, and operation of their IL training programs. These challenges fell into several categories:
- Workload/caseload demands
- Participant recruitment
- Youth involvement
- Staff turnover
All six of the projects identified workload/caseload demands as a significant challenge to project activities. This challenge manifested itself in two ways:
- Massachusetts, California, and Kentucky identified existing workloads as significant barriers to supervisors signing up for and attending the training. This made meeting participation targets more difficult.
- Supervisors across all projects also identified high workloads as challenges to implementing lessons learned and transferring training knowledge and skills to their staff when they returned to their home agencies. Massachusetts, Texas, and New York noted that high workloads, competing workplace demands, and a lack of resources made it difficult for supervisors to provide formal training to their staff in new casework knowledge and skills. Many felt that the training taught an "ideal" approach to working with youth that their real-world workloads could not support.
(Back to Top)
Several sites identified a challenge in persuading prospective participants to see the value of the training and to commit time and resources for participation:
- Kentucky noted that getting supervisors and their teams to sign up for training was a challenge because of a lack of institutional support.
- California reported that supervisors and workers had many different training requirements and limited time for training, so mandatory trainings tended to be prioritized over optional ones. Also, initial marketing for the training left many potential attendees with a negative impression that the training would focus more on criticizing their current work rather than offering a youth-informed perspective on potential service improvements.
- New York characterized some caseworkers as "jaded" and therefore more resistant to changes in case management and workplace practices. Some supervisors felt that their staff could more effectively acquire youth development skills in a workshop setting than in individual case supervision.
(Back to Top)
Youth involvement, while key to all six projects, presented specific challenges, including:
- Texas staff noted that the youth with whom they worked required a great deal of support and training to feel comfortable with the project team. They worked intensively with youth before and after their participation in each training to prepare them for audience questions and feedback and to process the experience in a positive manner.
- Massachusetts reported that it was particularly challenging to help youth incorporate both positive and negative feedback from their experiences in foster care into the training in a manner that maintained the training's strengths-based approach.
- Iowa worked closely with youth to balance the many demands on their time, particularly as their participation in trainings grew in popularity and requests for their participation increased. The project also trained more youth participants than originally intended to keep pace with demand without overburdening the youth. Finally, project staff in Iowa were careful during training sessions to monitor the questions asked of youth in order to maintain appropriate boundaries. Part of their work with youth was supporting them when they did not wish to respond to certain questions, and to help them feel comfortable sharing only certain personal information with audiences.
(Back to Top)
Texas and Iowa both identified staff turnover in child welfare agencies as a barrier to project implementation. High levels of turnover meant that newly trained staff often left positions shortly after the training, limiting its impact. New York and California reported related issues, in that staff turnover created higher workloads for existing staff, leaving less time for staff participation in training or to apply lessons learned from the training in their daily work.
(Back to Top)
Several projects reported logistical challenges to implementing their projects:
- California noted challenges in obtaining valid email addresses to send participants follow-up information and evaluation surveys. That project also relied on the counties to provide meeting space, which was sometimes not available.
- Two of the projects used a "Museum of Lost Childhoods" (described further in Table 5), which required shipping and transporting large and heavy items; these were key components of the training, and project staff had to take particular care to make sure they were not lost or damaged.
(Back to Top)
Grantees noted several issues that affected their ability to produce more conclusive findings from the projects:
- Texas reported that supervisor training ran concurrently with several other systemic changes to the State's overall approach to working with youth; therefore, it was not possible to attribute changes in supervisory knowledge and practice to the CWS/ILS curriculum alone.
- California and Kentucky both struggled with research design and implementation issues, including securing Institutional Review Board approval and resistance to the use of a quasi-experimental design methodology that limited their ability to implement evaluations as originally designed.
- California and Iowa both reported low levels of survey completion among participants, even when incentives were offered.
(Back to Top)
Several successful strategies were identified across CWS/ILS grantees related to program planning, implementation, and evaluation, including:
- Program planning that was informed by multiple stakeholder groups
- Use of multiple training modalities
- Emphasis on peer-to-peer learning
Program planning informed by multiple stakeholder groups
All of the projects engaged in planning processes that identified successful training and dissemination strategies currently in use in the child welfare field, as well as areas for improvement. These planning processes included the perspectives of administrators, supervisors, caseworkers and youth; in addition, some agencies included other stakeholders such as contracted services agencies and foster families. For example:
- Texas conducted focus groups with foster youth, former foster youth, foster parents, and IL caseworkers and supervisors as part of its curriculum planning process. That project also conducted a literature review to identify evidence-based approaches to curriculum development.
- New York worked with a mix of city, State, and rural partners in MS, NY, and OR to test whether the training materials would be effective in different settings. New York also engaged child welfare leadership early on in an effort to increase support for the finished product.
- California held three focus groups with child welfare supervisors, along with four youth focus groups throughout the State, to gather information on the topics to include in the training as well as the best methods for delivering the content.
- Massachusetts conducted focus groups with DCF staff, youth, providers, and foster parents to inform curriculum development.
(Back to Top)
Use of multiple training modalities
Each of the projects moved beyond a traditional lecture-based curriculum and employed multiple training modalities that accessed different learning styles and engaged participants on many different levels. These approaches included:
- Digital stories (Texas, New York, Massachusetts)
- Youth participation (Texas, Kentucky, Iowa. Massachusetts)
- Experiential exercises (Texas, California)
- Use of electronic media (Texas, New York)
- Small- and large-group activities (New York, Kentucky, Iowa)
Table 2, Training Modalities, provides more detailed information about the training modalities used by CWS/ILS projects.
(Back to Top)
Emphasis on peer-to-peer learning
Many of the projects emphasized the value of peer-to-peer learning and incorporated this element into their trainings:
- In Texas, a "teach back" component allowed participants to practice teaching the material within a safe setting and receive peer feedback to improve training delivery.
- New York's Learning Circle model provided program participants with a consistent group of peers throughout the 6-month training period to facilitate the development of strong relationships and build trust and cohesion.
- Kentucky offered some training modules for supervisors and caseworkers together, in order to strengthen those relationships and establish a joint commitment to incorporating positive youth development principles into casework practice.
- Massachusetts noted that peer-to-peer learning was very powerful in helping participants engage in problem solving, especially around issues related to specific cases and/or workers.
(Back to Top)
Each of these projects developed a comprehensive approach to integrating essential positive youth development principles and strategies into child welfare service delivery systems. Using the four core principles for the development of successful adolescent transition programs as a framework for supporting older youth in foster care, the projects sought to augment the traditional child welfare focus on the acquisition of tangible skills deemed necessary for survival after aging out of foster care. Process and outcome evaluation findings from these training programs point to several important lessons that can serve as guidelines for future work in connecting adolescent development theory with transitional planning services. Several of the most significant lessons learned are discussed below.
It is essential to include and engage foster youth in designing and implementing CWS/ILS training programs.
Youth voices in the form of digital stories and youth panels played a powerful role in the supervisor training programs. The incorporation of a youth perspective was very well received across all programs, with supervisors often noting the potential of digital stories to influence their casework practice. Although it created some logistical challenges, the inclusion of youth in the transitional living planning process has proved to be an effective strategy for both supervisors and caseworkers. For example:
- New York supervisors described digital stories as having one of the most significant impacts on how they engage with youth
- Digital stories were a key component of the curriculum in Massachusetts, and were used as an additional means to include the youth perspective and encourage reflection on the youth experience in care.
- Kentucky noted that focus groups with youth were instrumental in designing a training curriculum that effectively addressed youths' transitional needs
- Iowa and California also noted the importance of involving youth as trainers.
Building relationships with child welfare administrators and community partners can facilitate the implementation process.
Grantees found that strong administrative and community support is needed to encourage supervisors and caseworkers to participate in the training programs. Kentucky noted challenges in recruiting supervisors and caseworkers primarily due to a lack of support from child welfare service administrators. Some supervisors believed that previous administrations did not value training or that they regarded it as a low priority. Thus, engaging and building strong relationships with child welfare leadership might have strengthened their ability to recruit supervisors and caseworkers.
Providing online access to training materials is beneficial in facilitating ongoing training and disseminating materials to a broader population.
Each training program developed a project website that allowed participants and those who could not attend the training to use the training materials. Project websites served as an effective tool in disseminating information on youth development principles in the context of foster care and child welfare services.
Expanding the target audience for IL services training can further address the needs of youth transitioning out of foster care.
Many key stakeholders across multiple systems are involved in the lives of youth in foster care, particularly youth preparing to transition out of the child welfare system. As such, supervisors recognized the value of widening the audience for training in youth development and IL services. For example:
- Supervisors in Iowa participated in planning and implementing training sessions for community partners as well as caseworkers
- Kentucky decided to broaden the scope of its training program after recognizing the need for more collaboration among key stakeholdersï¿½including ILS coordinators, youth specialists, community providers, clinicians, foster parents, and youthï¿½involved in the lives of youth in out-of-home placement.
Rigorous methods of measuring the transfer of knowledge and skills must be developed for future evaluations of supervisor training programs.
It is important to understand how the knowledge gained from a training program moved from theory to practice. The effective measurement of knowledge and skills transfer from supervisors to caseworkers, and the subsequent impact of this transfer on youth outcomes, poses significant methodological and logistical challenges. While grantees attempted to measure this construct through a variety of surveys, tests, and interviews, more rigorous experimental studies that quantify and measure longitudinal changes in worker behaviors and youth outcomes may provide more conclusive evidence regarding the ultimate impact of future supervisor training initiatives.
The theory of change underlying the CWS/ILS grantee cluster is: "Increasing knowledge and awareness among child welfare supervisors and workers through new materials and trainings will result in increased utilization and application of skills and knowledge, which should, in the long-term, result in organizational changes that support youth-focused frontline practice that incorporates IL components" (Lyon and DeSantis, 2008). Each grantee implementing a supervisor training program evaluated a range of outcomes expected to occur in supervisors, caseworkers, and foster youth. Evaluation findings reported in the following sections reflect the specific outcomes measured and reported by each grantee.
Adhering to the Children's Bureau's evaluation requirements, grantees conducted comprehensive evaluations that included both process and outcome components. Process evaluations focused on a variety of constructs; however, most addressed participant recruitment and training, participant characteristics, and participant satisfaction. Grantees also addressed similar sets of questions for their outcome evaluations by exploring changes in supervisors' knowledge of key aspects of youth development and the four core principles of youth transition planning: positive youth development, collaboration, cultural competence, and permanent connections. In addition, grantees' outcome evaluations measured changes in supervisory practice and the transfer of knowledge and skills from the supervisor to the caseworker.
Table 3, Overview of CWS/ILS Grantees' Evaluation Designs, summarizes each grantee's approach to evaluating its program, including research designs, key process and outcome measures, and data collection tools and methods. Four grantees implemented pre-post test research designs, with the remaining two grantees implementing comparison group and time series designs. As a core process evaluation concept, all grantees examined participant satisfaction using a post-training survey consisting of Likert scales and open-ended questions. In addition, many training participants were asked a set of questions about the content and format of the training and the quality of training instructors.
Outcome evaluation constructs were measured using quantitative and qualitative methods. All grantees used some type of pre-training and follow-up survey to measure the degree of change in supervisor (or caseworker) knowledge in the area of youth development, particularly as it related to the four transition planning principles noted above. For example:
- Texas, Kentucky, Iowa, and New York utilized multiple-choice tests to collect baseline and follow-up data regarding the influence of their curricula on knowledge of positive youth development theory and its applications.
- In a slightly different vein, Massachusetts and California examined this construct by developing a pre-post test to measure perceived competency in the concepts taught during the training.
- Iowa, Massachusetts, New York, and California directly measured how training influenced changes in supervisory practice methods as a component of their outcome evaluations.
- While Massachusetts and New York relied primarily on qualitative interviews with a random sample of supervisors, Iowa utilized a paper pre-post test and California conducted an online survey.
Several grantees measured the extent to which supervisors transferred transition planning skills and knowledge gained from the training to their caseworkers:
- Texas used a comparison group design to examine differences in how caseworkers engaged youth in transition planning, as evidenced by case record documentation.
- California surveyed caseworkers whose supervisors attended training to assess the transfer of knowledge and skills.
- Iowa measured this construct from a youth perspective by comparing the transition planning services provided to two cohorts of youth leaving the foster care system at different times.
- Massachusetts and New York did not directly examine the transfer of skills and knowledge as part of their outcome evaluations.
Caution should be used in drawing direct causal inferences when considering the findings reported. The grantees did not use experimental research designs, which limits the ability to identify cause and effect relationships between changes in behavior and the supervisor training programs. In addition, several training programs were implemented in the context of broad changes throughout the child welfare system. Thus, moderate positive gains alone may not be sufficient evidence to conclude that the training activities directly influenced changes in knowledge, supervisory practice, or the transfer of skills. Nonetheless, the grantees reported promising findings, which warrants further research into ways to strengthen the supervision provided to child welfare staff working with older youth in foster care and/or in IL programs.
The following section summarizes key process findings across all six programs. Process evaluation findings were analyzed along several dimensions, including:
- Participant recruitment and training
- Participant characteristics
- Participant satisfaction
Participant recruitment and training
Table 4, Number of Participants, provides an overview of the target number of supervisors/caseworkers trained and the actual number trained. As evidenced from the following bullets below, most states exceeded their initial target numbers for supervisors trained:
- Texas cited support from agency administrators and managers a key factor in its ability to train 34 percent more supervisors than originally projected.
- Kentucky had to abandon its original plan to implement a quasi-experimental research design due to challenges with keeping supervisor and frontline caseworker teams together during the recruitment process. Instead, Kentucky recruited supervisors and caseworkers across different teams, a change that the grantee credited with increasing the number of caseworkers trained to within 95 percent of its original goal.
- New York trained only 23 percent of the supervisors in MS, NY, and OR than originally planned. They experienced challenges in recruiting and training supervisors at their Mississippi site due to significant staff turnover.
(Back to Top)
Table 5, Characteristics of Supervisors in CWT/ILS Training Programs, summarizes the available demographic data from each grantee. Findings of note include:
- The majority of supervisors in Texas, Massachusetts, and California were female.
- While Kentucky and Massachusetts supervisors were predominantly White, the participants in Texas and California reflected a more diverse population.
- Among grantees that reported this information, the average number of years working in the child welfare field ranged from 7 to 13 years, while the average years of supervisory experience varied from as little as 2 to almost 8 years.
- Massachusetts supervisors had the most experience working in child welfare and in a supervisory role, while Texas supervisors had the least experience on average.
(Back to Top)
Using a 5-point Likert scale to measure training satisfaction, all grantees reported overwhelmingly high levels of satisfaction among training participants, with training curricula generally rated as useful and practical. Specific grantee findings include:
- Texas supervisors viewed the trainings as organized, interesting, and interactive, and reported a mean satisfaction score of 4.7.
- Iowa reported mean satisfaction scores for four individual training sessions that ranged from 4.12 to 4.55 for supervisors and from 4.03 to 4.47 for caseworkers.
- Kentucky supervisors and caseworkers viewed the training materials, exercises, and lectures as useful and aligned with their level of knowledge. They rated the utility of the training curriculum with a mean of 4.0, while 100 percent of the participants reported that the difficulty level of the training was "just right."
- In Massachusetts, between 70 percent and 96 percent of the participants stated they would recommend the training to other colleagues.
- Supervisors in the New York project gave the training a very high mean satisfaction score of 4.99.
- California participants reported that the training topics were relevant to their work, met their expectations, and were well organized, with mean satisfaction scores for each item of 4.58, 4.56, and 4.49, respectively.
Overall, the trainees expressed that the training content and format accurately reflected their job duties, improved their knowledge, and that training staff were highly knowledgeable. The trainers' preparation and organization were viewed favorably, with New York reporting a score of 4.98 in this domain and a score of 4.89 for teaching effectiveness. Trainer presentation skills, such as encouraging participation and interaction, were among the most highly rated items for the Texas supervisors.
Integrating a youth perspective in the training curriculum appeared to have a strong influence on trainee satisfaction ratings:
- Texas supervisors viewed youth as a "powerful component" in training on IL services.
- Participants in Massachusetts viewed the youth panels within the curriculum as one of the most valuable features.
- New York's digital stories were rated overwhelmingly positive, with 100 percent of the participants enjoying the stories and 94.5 percent finding them informative. In addition, 80 percent of the supervisors in New York's program stated that viewing youth stories had an impact on their attitudes toward working with youth.
(Back to Top)
Outcome findings are reported in this section within the domains of:
- Changes in supervisory knowledge
- Changes in supervisory practice
- Transfer of skills and knowledge
Changes in supervisory knowledge
Test scores across all six grantees suggested that supervisors experienced moderate positive knowledge gains in the areas of youth development and transition planning that may have been influenced by their participation in the training program. Many of the programs reported statistically significant increases in knowledge as measured by changes in scores on pre- and post-tests completed by participants:
- California (N=337) reported knowledge gains within key competency areas, with total average scores increasing from 2.80 at pre-test to 3.29 after training.
- In Iowa, between 93 percent and 100 percent of supervisors across four training sessions demonstrated knowledge gains.
- On a 12-item knowledge survey, Kentucky's (N=1478) test score means increased from 9 to 12, an increase of 33 percent.
- In Massachusetts (N=35), the average response values for supervisors increased for 96 percent of the survey items, and statistically significant positive value changes were identified in 6 out of 10 competency domains.
- New York (N=8) reported data from supervisors who completed both the pre- and post-test youth development survey in the program's final Learning Circle. With a possible high score of 20 correct answers, the test mean grew from 7.9 to 14.5, which equates to a test score increase of 84 percent.
- Texas (N=154) reported a pre-test mean of 9.16 and a post-test score of 11.23 on a 15-item knowledge test, which indicates a test score increase of 23 percent.
(Back to Top)
Changes in supervisory practice
Grantees reported positive changes in the ways supervisors engaged with their staff after participating in training, particularly in the areas of transitional planning services and positive youth development. While Iowa and California primarily used quantitative data to illustrate how the training curriculum influenced supervisors' workplace practices, New York and Massachusetts relied on qualitative interviews. Kentucky indirectly addressed changes in supervisory practices through other components of their evaluation.
Using baseline surveys administered at the start of training, Iowa found that between 50 percent and 66 percent of supervisors already used the skills taught in training; 6 months after training, between 65 percent and 74 percent of supervisors reported using the new skills learned during the training program. California reported data from an online survey of trained supervisors, which found:
- 92 percent of supervisors are integrating the knowledge and skills they acquired during training into their workplace practices.
- 92 percent have encouraged their caseworkers to understand the importance of using positive youth development principles.
- 91 percent have motivated caseworkers to pursue resources for youth who are transitioning out of the system.
- 64 percent have collaborated with other key stakeholders to discover additional resources for youth.
Several themes involving supervisory practice emerged from the qualitative data reported by New York and Massachusetts. For example:
- Supervisors reported increased feelings of empathy as they worked with youth aging out of foster care.
- The training provided supervisors with opportunities to learn from and share information with other supervisors both within and outside of the training setting.
- The training provided supervisors with tools to enhance and improve their supervisory skills.
- Supervisors became more aware of important resources for youth who are aging out of the foster care system.
- The training generated a renewed focus on adolescents and permanency, which encouraged workers to support the inclusion of youth voices in their case plans.
New York supervisors stated that they were able to integrate different activities from their Learning Circles into their practices, for example, by incorporating the concept of "Permanency Pacts" (pledges from supportive adults to establish a lifelong, kin-like relationship with youth transitioning out of foster care) into casework practices. Massachusetts supervisors reported that the training fit well into the larger picture of child welfare services and acted as a refresher on "good social work practice."
Three grantees implemented strategies that, while not directly measuring changes in supervisory practice, gave supervisors an opportunity to reflect on concrete steps for improving services to youth in transition:
- Kentucky conducted a Youth Summit to provide supplemental training and facilitate collaboration among supervisors and other key figures in the lives of youth transitioning out of foster care. Some of the items in the Youth Summit evaluation survey reflected potential changes in supervisory practice. For example, participants pledged to apply the knowledge they had gained by creating action cards that detailed the tasks they would complete, such as providing mentors for youth, being a mentor to youth, using the newly discovered resources from the Youth Summit, and increasing collaborative efforts with key stakeholders.
- New York used the concept of "action plans" in their interviews to encourage participants to integrate knowledge gained from the training into their workplace practices.
- In California, a post-training survey revealed that 79 percent of supervisors said they plan to be more effective in supporting their caseworkers' capacity to provide services to transition-age youth in foster care.
(Back to Top)
Transfer of skills and knowledge
Texas, California, and Iowa measured the transfer of skills and knowledge from supervisors to caseworkers by assessing the ways caseworkers used training tools and concepts or engaged in transition planning with youth.
Texas compared the case documentation of workers whose supervisors attended the training (N=14) to the case documentation of those whose supervisors did not attend (N=16). Using a numeric rating scale created specifically for this project, an independent evaluator assessed the degree to which each group's case documentation reflected evidence of knowledge and skills transfer. Although no significant differences were identified, Texas noted that a small sample size and variations in how cases were documented may have made it more difficult to detect significant changes.
Through qualitative interviews, Texas also identified various methods supervisors used to deliver the information to their staff. They found that supervisors transferred what they learned from the training curriculum in unit meetings, case supervisions, and casual conversations. In these settings, supervisors discussed youth development issues, explored the idea of involving youth in case decision-making, and used case examples from the training manual to help their caseworkers make decisions regarding appropriate services for older youth in foster care.
California examined knowledge and skill transfer from the caseworker's perspective, surveying 58 workers whose supervisors attended training. Their results were mixed:
- Several months after training, 82 percent reported that their supervisors encouraged collaboration among workers to identify youth resources. Thirty-five percent of those individuals indicated an increase in encouragement since the training, while 81 percent reported that supervisors allowed caseworkers to share youth success stories during unit meetings.
- Caseworkers also reported that supervisors encouraged them to focus on the child's future by anticipating needs and setting goals; 53 percent stated that this behavior had increased since the training.
- California reported some concern over the discrepancy between the percentage of workers who stated their supervisors have focused on the importance of positive youth development (68 percent) and the percentage who indicated that they knew the principles underlying this concept (58 percent). Thirty-five percent of caseworkers suggested they were "not sure" what positive youth development means. California hypothesized that caseworkers may simply be unable to explain the concept or that supervisors are not effectively teaching caseworkers how to integrate the idea of positive youth development into their casework practice.
Iowa was the only grantee that measured the transfer of skills and knowledge from a youth perspective. This grantee compared interview data from a 2007 cohort of youth who were in the process of transition planning before the implementation of CWS/ILS training with a 2009 cohort of youth who began transition planning after the training. Compared to the 2007 cohort, youth interviewed in 2009 reported that the transition planning process was more helpful and that they received more information from a variety of sources. Youth in 2009 also described having more contact with their workers, being listened to more often, and being challenged to develop their unique skills and goals. Youth within this cohort described an eagerness among adults to empower and include them in decision-making concerning their transition to adulthood.
Overall, the grantees' evaluations appear to reveal improvements in the youth transition planning process following CWS/ILS training, with positive changes reflected in individual attitudes and practices as well as at the agency level. For example:
- Texas reported a child welfare agency in which a caseworker in each unit was identified as a "permanent care worker" whose duties involved assisting youth who are exiting the foster care system.
- Interviews with agency directors in California highlighted the emergence of "e-conferences" in Fresno County, which facilitated collaboration between youth and child welfare professionals in casework planning and decision-making.
(Back to Top)
The following products have been developed by the projects in this cluster and are available to the public. Links are included where available. For unlinked items, refer to Contact under the Grantees tab. Some items may be available through the Child Welfare Information Gateway Library: email firstname.lastname@example.org.
- Through the Eyes of Youth: How Child Welfare Supervisors Can Positively Impact the Lives of Foster Youth: Youth-developed and -delivered training (California)
- Dating Violence: http://cwte.louisville.edu/IL/home/ilmodules.htm (Kentucky)
- Mentoring: http://cwte.louisville.edu/IL/home/ilmodules.htm (Kentucky)
- Motivational Interviewing: http://cwte.louisville.edu/IL/home/ilmodules.htm (Kentucky)
- Reconnecting With Birth Parents: http://cwte.louisville.edu/IL/home/ilmodules.htm (Kentucky)
- Time Is Ticking: Tools for Transitioning Youth: 2½-day IL training curriculum (Kentucky)
- Working With Challenging Youth: workshop curriculum (Kentucky)
- The STEPS Curriculum: six 6-hour modules (Massachusetts)
- Learning Circle competencies, agendas, discussion guides, digital stories, and all of the training materials and tools for the six learning circles (New York)
- Web-based training, Trainer's Manual, and Supervisory Toolkit of teaching strategies (Texas)
- Improving Outcomes for Youth in Transition: Participant Manual (Iowa)
- Improving Outcomes for Youth in Transition: Resources for Community Collaboration (Iowa)
- Improving Outcomes for Youth in Transition: Supervisor Training Participant Manual (Iowa)
- Strength-Based Resources for Working With Youth in the Commonwealth of Massachusetts: Updated manual of State-specific resources for youth, information on promising practices (Massachusetts)
- Detailed discussion guides for each of the six segments of the Learning Circle (New York)
- Participant's Handbook (Texas)
- Supervisor Behavioral Competencies tool. Helps supervisors self-assess their behaviors, break tasks down into categories, and operationalize their competencies. Can also be used as a pre- and post-test or for assessing staff to target caseworker education. (Iowa)
- Adolescent Implicit Association Test: Supervisors identify unconscious biases they may have about youth in foster care and examine the practice implications of those biases for their work in child welfare. https://implicit.harvard.edu (Massachusetts)
- Digital stories: www.silencespeaks.org (California)
- Desire to Inspire CD produced by Elevate youth (Iowa)
- Digital stories created by foster youth, former foster youth, child welfare supervisors, and child welfare workers www.hunter.cuny.edu/socwork/nrcfcpp/pass/digital-stories/index.htm (New York)
- Antle, B., Barbee, A., & Sullivan, D. (2010). Evidence-based supervisor-team independent living training: Kentucky development and implementation. Training and Development in Human Services, 5(1), 53-66. (Kentucky)
- Antle, B., Johnson, L., & Barbee, A. P. (2009). Fostering interdependent versus independent living in youth aging out of care through healthy relationships. Families in Society, 90, 309-315. (Kentucky)
- Barbee, A., & Antle, B. (Eds.). (2010). Independent living [Special issue]. Training and Development in Human Services, 5(1). (All 6 projects)
- Barbee, A., Antle, B., & Johnson, P. (2010). Introduction, improving outcomes for youth in transition. Training and Development in Human Services, 5(1), 5-7. (Kentucky)
- Barbee, A. P., Antle, B. F., Sullivan, D. J., & Johnson, L. (Under review- submitted 7/3/10). Focus groups inform curriculum development on topic of independent living. Child Welfare. (Kentucky)
- Barbee, A. P., Taylor, J., Antle, B. F., Sullivan, D. J., Landsman, M., & Gilbaugh, P. C. (2010). Matrix of the key components in each of the original independent living grantee curricula. Training and Development in Human Services, 5(1), 8-20. (Iowa and Kentucky)
- Dodd, S., Morse, J., & Mallon, G. (2010). Preparation for adulthood: Supervising for success. Training and Development in Human Services, 5(1), 43-52. (New York)
- Estafan, S., Evans, J., & Lum, R. (2010). Y.O.U.T.H.full intelligence for child welfare supervisors. Training and Development in Human Services, 5(1), 94-103. (California)
- Hall, G., & Coakley, J. (2010). Supervisory training to enhance permanency solutions: The Massachusetts experience. Training and Development in Human Services, 5(1), 30-42. (Massachusetts)
- Landsman, M. J., & D'Aunno, L. (2010). Improving outcomes for youth in transition. Training and Development in Human Services, 5(1), 80-93. (Iowa)
- Scannapieco, M., Connell-Carrick, K., & Steinberg, C. (2010). Texas training project: Preparation for Adult Living Supervisor Training and Empowerment Program (PAL-STEP) improving outcomes for youth in transition. Training and Development in Human Services, 5(1), 67-79. (Texas)
- Scannapieco, M., Connell-Carrick, K., & Painter, K. In their own words: Challenges facing youth aging out of foster care, Child and Adolescent Social Work Journal 24(5), 423-435. (Texas)
- Final reports from these six projects are available by contacting email@example.com
- Lyon, K., & Desantis, J. (2008). Planning, implementing, and evaluating training projects for public child welfare agency supervisors: The application of logic models and theory of change. Arlington, VA: James Bell Associates.
- National Network for Young People in Foster Care (Foster Club) (2009). Permanency Pact. Retrieved January 21, 2010, from www.fosterclub.org.
- U.S. Department of Health and Human Services. (2005). Program announcement: Training of child welfare agency supervisors in the effective delivery and management of Federal independent living services for youth in foster care. Funding opportunity no. HHS-2005-ACF-CW-0009.