Methodology

As interest among researchers and policy makers has grown in the last ten years, racial disproportionality has been examined and documented empirically using administrative data sets, primarily. The advantage of an empirical analysis of disproportionality is that it provides objective findings regarding such important issues as the incidence and prevalence of disproportionality, or can document trends over time. The limitation of using administrative data to understand disproportionality, however, is that it does not provide evidence of the context in which the phenomena occur. Specifically, administrative data sets do not allow for the possibility of linking outcomes of interest, for example, the number of minority children in foster care at a given time, to the processes that influence them, such as worker perceptions and agency practices. This study was designed as a first step in understanding the context in which disproportionality occurs from the perspective of those making the decisions.

The research team comprised researchers from Caliber Associates and Howard University, a historical black college. Caliber Associates chose to work with faculty from Howard University's School of Social Work because its mission and the expertise of the faculty in the welfare of African-American children. In addition, a research fellow with the Administration for Children, Youth and Families' Child Outcomes Research and Evaluation division with expertise in African-American child welfare served as an assistant to the Federal Project Officer during the term of her fellowship but chose to stay as a team member after her fellowship terminated. She is currently an Associate Professor at the University of Maryland, College Park.

As an exploratory study and one of the first major efforts in the child welfare field to explore the attitudes and perceptions of the child welfare community concerning racial disproportionality, a qualitative approach was chosen as the primary method of inquiry. In new fields of study, such as this one, where little work has been done, few definitive hypotheses exist, and little is known about the nature of the phenomenon (e.g., the field's perception on over-representation), qualitative inquiry is a reasonable beginning point for the research. Exploratory work of this kind is the way that new fields of inquiry are developed (Patton, 2002).

The project comprises two separate but related tasks. The first effort is to gain input from the child welfare field on the issue of over-representation of African-American children and families in the child welfare system. The study is designed to explore the issue of disproportionality from the perspective of administrators, supervisors, and direct-service staff from child welfare and child welfare related agencies, and their perceptions regarding its origins, prevalence and persistence.

In line with the study's goal to gain insight into the issue of racial disproportionality, the second effort is to examine the strategies that child welfare and child welfare related agencies are implementing to meet the needs of children and families of color. In combination, these efforts are designed to begin to provide guidance and assistance to the child welfare field and the Federal government relating to such issues as policy, practice, and targeted resources associated with reducing disproportionality, and the delivery of appropriate and effective services to child welfare involved children and families of color.

To meet the goals of the study, the team developed a comprehensive qualitative information gathering system for on-site implementation with nine child welfare agencies. Using this approach allowed the team to gather detailed information about each agency, its staff, and partner agencies, and also to explore the similarities and differences among agencies in terms of the variables of interest. Site visits were the primary means of collecting qualitative data from each of the participating child welfare agencies, and were focused on conducting in-depth individual and group discussions with child welfare and partner agency staff, including administrators, supervisors, caseworkers and other direct service providers. Information gathered through on-site discussions was supplemented by written documentation, including policy and procedural documents, organizational charts, annual budgetary and operational reports, and program-specific brochures, manuals, and evaluation reports, provided by agency administrators or their representatives prior to the on-site visit. This information was intended to provide the team with a context for each agency, including such important characteristics as its mission and philosophy, operational policies, and organizational structure.

The following sections of the report present the details of the team's approach to completing these five major tasks.

(Back to Top)

3.1 DEVELOPING THE RESEARCH QUESTIONS

The development of the research questions was an iterative process. That is, the questions were reviewed and refined by the team over a period of three months. The questions were designed after both a thorough review of the literature on over-representation, and discussions with several experts, both researchers and practitioners, on the issue. In the end, there were three overarching research questions, including:

• How do child welfare and child welfare-serving agencies, organizations and communities address the issue of racial disproportionality?
• How do different child welfare and child welfare-serving agencies, organizations and communities operating at different levels and at multiple decision points along the child welfare continuum, address the specific needs of children and families of color?
• What types of services do children of color receive after entry into the child welfare system? What factors influence decisions about the nature and type of services minority children and their families receive?

The final set of questions was used to guide the development of the information gathering plan, including the criteria by which sites would be selected for participation, potential discussion topics and questions, and the characteristics (e.g., job title, position, years of experience) of individuals who would participate in on-site discussions.

(Back to Top)

3.2 SELECTING AND RECRUITING THE CASE STUDY SITES

Because of the exploratory nature of the study, the project team felt it was not methodologically necessary for the sample to be representative of child welfare agencies nationwide. As one of the first efforts to engage the child welfare community in open discussions regarding issues related to disproportionality, the project team felt is was more important to identify and recruit sites that were willing to participate and where there were activities of interest rather than those that were representative of the population of agencies nationally.

Nine sites were selected for participation in the study. As was the case with the research questions, the site selection process was iterative. That is, there were several steps taken before the final selection was determined, with each step producing additional, relevant information that helped guide the final selection.

The first step was to organize a meeting with the project team, several key Federal stakeholders, including the Federal project officer and five nationally recognized experts in the field of disproportionality. The Federal project officer, in collaboration with other ACFY stakeholders, identified the expert panel. The expert panel included Jestina Richardson, President, Black Administrators in Child Welfare; Gretchen Test, National Association of Public Child Welfare Administrators Mark Testa, Director, Children and Family Research Center, School of Social Work, University of Illinois at Urbana-Champaign; Clarice Walker, Howard University and the Black Child Development Institute; and Carol Wilson Spigner, University of Pennsylvania.

During the meeting eighteen sites, and contacts for each, were recommended for consideration, including county- and state-administered child welfare agencies and private and non-profit child welfare-serving agencies. The private and non-profit child welfare-serving agencies included both stand-alone agencies and agencies that held a contract for services with the local child welfare agency. These groups were not mutually exclusive. While the selection criteria varied somewhat across sites, at the minimum, sites were known to be implementing initiatives, reform efforts, or programs, activities, and projects that were aligned with the study's goals (e.g., to reduce disproportionality and meet the needs of children and families of color.) In addition, the sites were thought to have data available regarding disproportionality and program outcomes, and a willingness to participate in the study.

The project team developed several key participation criteria for assessing the suitability of the sites for inclusion. These criteria included:

• The overall mission and philosophy of the agency
• The demographic breakdown of service areas and client populations
• The types of efforts that were being implemented to meet the needs of children and families of color or children and families, particularly African-American children and families, including systems reform, and special programs, practices, and activities
• The availability of descriptive and evaluative data, including operational manuals, organizational charts, annual reports, program brochures, and evaluation findings, among others
• The nature and extent to which the agency collaborated with other agencies to provide services to children and families of color, specifically in relation to public child welfare agencies
• The interest and ability of the agency to assist in the planning and implementation of the site visit.

Letters informing the sites of the study and their potential inclusion in the study were sent to the eighteen sites, and approximately two weeks later, the project team began conducting screening calls. During these calls, project staff first referenced the letter and then briefly assessed the situation to determine if the contact person was someone who could provide the information of interest. In many cases, once the purpose of the call was determined, project staff were redirected to other senior agency staff persons, such as administrators or directors. The screening process took place over a three-month period. At the end of the screening process eleven sites remained. Three sites had been eliminated because they had not returned telephone calls after three attempts. An additional site was eliminated because the child welfare system was being reorganized at the state level and the agency administrator felt the system was under too much strain to participate. Finally, three other sites were deemed too small in terms of client population and staff to meet the study's goals.

At several points in the screening process, the team held discussions to review the information being gathered and discuss its implications for the study. Over the course of these discussions, it became clear that the focus of the selection process needed to shift to the public child welfare agencies. This decision was made for several reasons. First, because the study was focused on gaining insight into the issue of disproportionality from those in the field, specifically, child welfare agency administrators, supervisors, and direct service staff, the team felt it was important to focus their efforts on those individuals. Second, the team felt that, in the end, they would not be able to make generalizations or recommendations regarding the child welfare system's response to children of color without fully involving the system. Finally, as the primary, child welfare-serving agency in a given area, the public child welfare agencies were serving the largest numbers of children and families of color. As a result, they were more likely than any other agency to influence the over-representation of minority children in the system. Many of the private and non-profit agencies were simply not serving large enough numbers of children to affect the nature and extent of over-representation. However, one issue remained; the small private and non-profit agencies were often implementing innovative programs or practices with child-welfare involved families that were of interest to the study and for that reason would be important to include.

In the end, the team decided to select nine public child welfare agencies from the initial list, including an agency in Georgia, Illinois, Michigan, California, Virginia, Texas, North Carolina, and two agencies in Minnesota. As part of the revised selection plan, the team asked the public child welfare agencies to identify and include local private and non-profit child welfare-serving agencies (partner agencies) in on-site discussions so that their perspective could be included and their practices explored. These included agencies that held contracts with the public child welfare agency or were serving minority children and families in the same service area as the public agency.

Once the sites were chosen, another letter was sent to the public child welfare agency administrators to inform them that they had been selected to participate and that someone from the project team would be contacting them within two weeks to speak with them further. Initial telephone conversations were focused on presenting the study in greater detail and in discussing how the on-site visits would be planned and implemented. In all cases, the agency administrator appointed a representative to work with project staff to plan and implement the site visits. The process by which site visits were planned and implemented is presented in detail in the section 3.4, below.

(Back to Top)

3.3 DEVELOPING THE DISCUSSION TOPICS

Given the goals of the study, the primary focus of information gathering efforts included individual and group discussions with public child welfare agency and related staff, and parents of child-welfare involved families (parents). Child welfare agency staff included administrators and directors, supervisors of CPS, adoption and foster care workers, and direct-service staff. In addition, staff members (administrators, supervisors, and direct service providers) from partner agencies also were asked to participate in individual and group discussions. This multi-level sample was designed to provide the team with the most comprehensive cross-section of the child welfare community as possible.

In total, three sets of discussion topics were developed. Once the team had identified the various groups of participants, they used the research questions to guide the development of a pool of discussion topics, one for each group of participants. Over the course of several weeks, the team reviewed, discussed, and revised the discussion strategies. In addition to the content of the discussions, length had to be considered as well. Because child welfare agency staff and direct service providers generally have very little time for activities other than those required by their jobs, it was necessary to develop strategies to keep the discussions brief. In the end, discussion topics ranged from eight to four topics, with the former intended for child welfare agency staff and the latter for parents. Once the final pool was agreed upon, the procedures were pilot-tested in two different settings at two time points.

The first set of discussion topics, those for agency staff, was designed to elicit opinions and attitudes regarding the issue of over-representation of minority children in the child welfare system, including Federal, state, agency, and individual factors that might be associated with disproportionality, either directly or indirectly. To accommodate time limitations, the team prioritized the final list of topics. Priority one topics were deemed the most important and would be asked first. The set of priority one topics addresses the following questions:

• What is your perception of over-representation? That is, why do you think children of color are over-represented in the child welfare system?
• How have Federal policies like the Multi-ethnic Placement Act (MEPA) and the Adoption and Safe Families Act (ASFA) changed the way in which your agency serves children and families of color?
• What has your agency done, if anything, to improve the delivery of services to children and families of color?
• What types of services, programs, or policies do you think are necessary to reduce the over-representation of children of color in the child welfare system?

A second set of discussion topics was deemed priority two. It was decided that priority two topics would be addressed if time allowed. While these topics were not necessarily of less interest to the team, pilot test findings suggested that this set of topics shared some similarities with the set of priority one topics and so were frequently answered within the context of the other topics. In addition, in situations where there was not enough time to address all eight topics, it was necessary to focus the discussion on those topics that were likely to elicit the most relevant information. The set of priority two topics addressed the following questions:

• What kind of training does your agency provide, if any, to assist you in working with children and families of color?
• What policies, procedures or practices would assist your agency to serve children and families of color better?
• As a child welfare worker, what do you do to increase the effectiveness of your work with children and families of color?
• What else do you think is important to consider in reducing the over-representation of minority children and families in the child welfare system? For reducing the number of children of color in out-of-home placement?

In some cases, all eight topics were covered. In other cases, most often due to time restrictions, discussions were focused on priority one topics only.

A second set of discussion topics, for program staff, was designed to gather information regarding the history, components, factors associated with implementation, and outcomes of agency initiatives, practices and programs focused specifically on children and families of color. Program staff included those who were employed by either the child welfare or partner agency to implement or direct the activities of specific initiatives, practices and programs of interest. For example, program staff might include a child welfare worker that is responsible for overseeing the implementation of a special program to recruit minority foster and adoptive families. Topics for program staff elicited the following information:

• How was the program developed? That is, what is the history behind the development of the program?
• What are the major components of the program and how are they implemented?
• How is the program different from what you or your agency used to do?
• How has your involvement with this program changed the way you think about how to work effectively with children and families of color?
• For new programs: What are your expectations for this program? What do you think this program will change for children and families of color?
• For existing programs: What difference has this program made for children and families of color?

A final set of discussion topics was designed to gain insight into parents' experiences as recipients of child welfare services. Topics for parents targeted the following information:

• Can you talk about your history and experience with the child welfare system? Specifically, how did you come to be involved in the system and what has been your general experience?
• How is your experience with this agency (or program) different from other experiences you have had in the child welfare system?
• As a minority family, do you think your experience has been different from non-minority families involved in the child welfare system? If so, how do you think your experience has been different?

Because the project team had very little control over selecting the parent sample, there was more variation in the parent group than in any other. Often, parents were selected based on availability and convenience (they were at the agency for an appointment at the time of the site visit) or because their experience with the agency had been a particularly positive one, with positive outcomes. As a result, this set of issues often required modifications. For example, in some cases the team talked with parents who had adopted children through the child welfare agency. In others, they talked with parents whose children had been removed and were in the process of working towards reunification. In others, parents were not African-American and so could not talk about their experiences in that way. As with modifications made to the other sets of discussion topics, these were made at the discretion of the senior team member responsible for conducting the discussion and noted accordingly.

The study procedures were pilot-tested in two different child welfare agencies, one urban (Baltimore City) and one suburban (Montgomery County). The pilot sites were chosen for reasons of geographic convenience and ease of access; when approached, administrators from both agencies agreed to serve as pilot sites. Second, the two sites provided the team with the opportunity to pilot-test the discussions in two very different agencies. The Baltimore City agency is located in a poor, urban setting, and serves a primarily African-American population. The Montgomery County agency, on the other hand, is located in an affluent, suburban community, and serves a more diverse population. Given the nature of the study and the content of the discussions, it was thought that these differences (e.g., race and class) might affect the manner in which agency staff responded to the issues and, as a result, would be important to explore during the pilot-tests. In general, the pilot tests were conducted to assess the discussion strategies on three factors, including content, clarity, and response.

(Back to Top)

3.4 CONDUCTING THE SITE VISITS¹

As a first step in conducting the site visits, the team worked with agency administrators to identify a contact person at each site who could assist the team to plan and implement on-site discussions. In most cases, this person was an employee of the child welfare agency. All communications regarding site visits were coordinated through the identified contact person and the team member assigned to a particular site. Through telephone conversations, team members worked with each contact person to identify the individuals and groups most important to include in on-site discussions, and then to plan proposed discussion groups, and other activities to be conducted during the site visit. To the extent possible, across sites, the team attempted to focus discussions on similar topics and to talk with individuals of similar title and position. In the end, the discussions commenced as follows:

• Individual discussion sessions with state-level child-welfare officials and agency administrators as well as partner agency directors and other high-ranking individuals (i.e., Attorneys General and judges)
• Individual or group discussion sessions with child-welfare agency supervisors and direct service workers, and partner agency supervisors and direct service providers
• Individual or group discussion sessions with child-welfare agency and partner agency program staff
• Individual discussions with parents.

Opportunities for gathering additional information, such as observations of program activities or court hearings, were identified by the contact person and discussed with team members prior to scheduling. In two sites, discussions were held with researchers who provided evidence regarding the effects of particular service delivery strategies on children and families of color.

At the beginning of each site visit, the team met with the agency administrator to review the purpose of the site visit, discuss the intended goals of each planned discussion session, and to resolve any issues or concerns related to the team's efforts. In many instances, this first meeting included agency administrators as well as management staff. At the conclusion of the site visit, a brief meeting was scheduled with the agency administrator or his or her assigned representative. The meeting was designed as an opportunity for the administrator to ask questions but also for the team to give feedback regarding the issues of interest.

In total, nine site visits were conducted. At the conclusion of each site visit, audio-tapes were transcribed and analyzed using both traditional qualitative techniques (e.g., content analysis) and text analysis software, specifically, IN-VIVO, a qualitative software package that allows the analyst to store documents, create text categories, code text segments, and generate reports.

¹ A summary of the site visit procedures is described in this section. A more thorough presentation of site visit procedures can be found in the Appendix. back

(Previous) (Next)

(Back to Top)

This material may be freely reproduced and distributed. However, when doing so, please credit Child Welfare Information Gateway.