Supporting Statement_Part B

B. Collection of Information Employing Statistical Methods

B1. Respondent Universe and Response Rates

The target population for each data source to be collected in this evaluation varies, thus these populations will be described individually. Table B-1 includes sample estimates for each data source. These estimates are based on our prior experience conducting the evaluation under OMB Control Number 0970-0288.

Table B-1: Expected Response Rates by Strata
	Response Offers	Expected Response Rate	Expected Number of Responses by Strata			Total Responses
			Individuals/ Households	Private Sector	State/ Local/ Tribal Gov’t
Stakeholder Survey	415	65%	20	60	190	270
Child Welfare Agency Survey	750	80%	0	0	600	600
Supervisor Interviews	100	90%	0	0	90	90
Stakeholder Interviews	100	90%	10	30	50	90
Project Director Interviews	23	100%	0	0	23	23
Focus Group with Family Members	170	60%	102	0	0	102
Parent Partner Interviews	40	60%	0	0	24	24
Child-Welfare Agency and Partner agency group interviews	467	60%	0	80	200	280
Community Description Form	9	100%	0	0	9	9
Organizational Structure Form for Case Study Sites	3	100%	0	0	3	3
Organizational Structure Form for Non-Case Study Sites	20	100%	0	0	20	20
Collaborative Membership Form	23	100%	0	0	23	23
Training and Technical Assistance Quality Assurance Assessment	23	100%	0	0	23	23
Training and Technical Assistance Conference Call Feedback Forms	185	65%	24	0	96	120
Total						1,677

Surveys

Stakeholder Survey: The target population for the stakeholder survey includes all members of each grantee’s local implementation collaborative board, and the survey will be administered to all members. An estimated response rate of 65 percent was identified based on our prior experience conducting the evaluation under OMB Control Number 0970-0288. With this response rate, we can expect the final sample of completed surveys to include 30 individuals in each implementation site, for a total sample of 270 respondents across all nine grantees.

Child Welfare Agency Survey: The target population for the child welfare agency survey includes all direct service workers and supervisors in the child welfare agencies across all implementation sites. Our power calculations (discussed in detail below) indicate a need for a sample of at least 67 individuals per grantee site (for a total of 600 across all nine grantees). An estimated response rate of 80 percent was identified based on our prior experience conducting the evaluation under OMB Control Number 0970-0288.

Training and Technical Assistance Quality Assurance Assessment: The target population for the Training and Technical Assistance Quality Assurance Assessment includes twenty-three project directors and project coordinators at nine grantee sites. Given the discussions the evaluation team has had with the grantees and federal project staff to date during which there has been agreement on the instruments of choice as well as the timeline for completion, we expect to achieve a 100 percent response rate for the completion of this data collection effort.

Training and Technical Assistance Conference Call Feedback Forms: The target population for the Training and Technical Assistance Conference Call Feedback form includes project directors, project coordinators, and local evaluators that participate on monthly conference calls with technical assistance staff. An estimated response rate of 65 percent was identified based on our prior experience conducting similar data collection efforts under OMB Control Number 0970-0288. With this response rate, we can expect the final sample of completed conference call feedback forms to include a total sample of 120 respondents.

Interviews

Supervisor Interviews: The target population for the supervisor interviews includes all supervisors in the child welfare agencies who oversee staff working with the targeted child and family population in the site. Based on estimates of cost and time, ICF International has determined that interviews can be conducted with approximately ten supervisors in each of the nine grantee sites, for a total of 90 supervisors across all sites. ICF International estimates that approximately 90 percent of the supervisors recruited for the study will agree to participate based on our prior experience conducting similar data collection efforts under OMB Control Number 0970-0288. Thus, ICF International will recruit 11 supervisors per grantee site (approximately 100 supervisors will be recruited across all sites).

Stakeholder Interviews: The target population for the stakeholder interviews includes all stakeholders who are part of the collaborative boards at each implementation site. ICF International aims to achieve a sample of ten stakeholders in each of the grantee sites, for a final sample of 90 stakeholders across all sites. ICF International estimates that approximately 90 percent of the stakeholders recruited for the study will agree to participate based on our prior experience conducting similar data collection efforts under OMB Control Number 0970-0288. Thus, ICF International will recruit 11 stakeholders for interviews in each grantee site (approximately 100 stakeholders will be recruited across all sites).

Project Director Interviews: Evaluation activities are within the job responsibilities of the project director and project coordinator. As one of the major duties of the project directors and project coordinators is to participate in the local evaluation, we anticipate a 100% response rate from these individuals. Thus, we anticipate interviewing 23 project directors and project coordinators total.

Parent Partner Interviews: Parent partners will be recruited to participate in interviews during each site visit. ICF International aims to achieve a final sample of 24 parent partner interviews across all nine grantees. We anticipate that approximately 60 percent of parent partners will agree to be interviewed for this study based on our experience conducting similar data collection efforts. Thus, to achieve our target final sample, ICF International will recruit 40 parent partners for interviews.

Focus Groups and group interviews

Focus Group with Family Members: The target population includes family members of the children being targeted by each grantee’s systems of care efforts. The interview population may include foster parents, birth parents, or adoptive parents, depending upon the specific population of interest to the grantee. Given monetary incentives and other anticipated recruitment strategies, we estimate that 60 percent of the family members recruited will agree to participate in the focus group. Thus, in order to achieve a final sample of 102 family members, we will recruit 19 family members per site (a total of 170 family members will be recruited across all grantee sites).

Child-Welfare Agency and Partner Agency Group Interviews: The target population for the child welfare agency group interviews includes child welfare caseworkers at each local implementation site. ICF International will recruit an average of 13 direct service workers from the child welfare agency at 18 implementation sites (a total of 234 child welfare direct service workers will be recruited across all grantees). Assuming a 60 percent response rate, this recruitment should yield the desired eight direct service workers per implementation site.

The target population for the partner agency group interviews includes all partner agency staff who work with the grantee’s target population. The sampling for the partner agency staff groups will be identical to that for the child welfare agency focus groups, with a goal of including an average of eight partner agency staff in each focus group. ICF International will recruit an average of 13 partner agency staff at 18 implementation sites (a total of 234 partner agency staff will be recruited across all grantees). Assuming a 60 percent response rate, this recruitment should yield the desired eight direct service workers per implementation site.

Process Evaluation Forms

Process Evaluation Forms include a Community Description Form, the Organizational Structure Form for Case Study Sites, the Organizational Structure Form for Non-Case Study Sites, and a Collaborative Membership Form. It is expected that the project director and project coordinators at 18 local implementation sites will complete each form and submit it to the evaluation team on an annual basis. Given the discussions the evaluation team has had with the grantees to date during which there has been agreement on the instruments of choice as well as the timeline for completion, and the support we have from the federal project staff, we expect to achieve a 100 percent response rate for the completion of these forms.

B2. Procedures for the Collection of Information

B2.1. Statistical Methodology for Sample Selection

For grantee sites in which the systems of care efforts are being implemented in multiple local sites, the sample for the child welfare survey will be stratified based on the percent of target population children being served in each of these implementation sites. As a result, the sample at the grantee level will be representative of the target populations across the local implementation sites.

We do not anticipate any unusual sampling problems associated with the design laid out above. However, a sampling statistician will be consulted throughout the course of this evaluation to ensure that the data collected are as representative of the grantee sites as possible.

The sample sizes for other data collection instruments were determined by the size of the universe. Given the content of the stakeholder survey, the process evaluation forms, and the project director and project coordinators – as well as the small universe in the target population for these instruments – our sampling plan requires us to collect information from all individuals in the target population, to the extent possible.

The sampling plan for the interviews with supervisors, parent partners, and focus groups with family members was developed based on the cost of conducting and analyzing interviews and focus groups, as well as the cost of providing incentives for family members. Given the high cost of conducting face-to-face interviews, ICF International decided to interview enough individuals from the target population to provide a broad overview of activities in the implementation sites, while staying within the evaluation budget. After cost calculations, we determined that interviews or focus groups could be conducted with approximately 90 supervisors, 24 parent partners, and 102 family members. Although this sample will not be representative of all supervisors, parent partners, and family members, these qualitative data are not meant to be representative of the target population. Rather, these interviews are meant to provide illustrative examples of the experiences of individuals within these populations as the sites implement their systems of care efforts.

Finally, the sample sizes for the child welfare agency and partner agency direct service worker group interviews were determined based on recommendations about the minimum and maximum number of participants in a focus group. Like the supervisor and family member interviews, the sample for these group interviews will not be representative of all direct service workers, but is feasible given the financial resources available to conduct this evaluation. Again, however, group interview data are not meant to be representative of a specific population, but instead are used to gain in-depth information regarding particular experiences.

B2.2. Statistical Power

As described above, a number of factors were considered in identifying the sample size for each of our data collection instruments. These factors include statistical power analyses, the size of the target population (i.e., small universe), methodological requirements of the data collection approach, cost, and burden. The only data collection instrument for which the sample size was determined solely by a power analysis was the child welfare agency survey. For this survey, there is minimal added cost of administering the instrument to greater numbers of individuals. Thus, in order to identify the sample size, we calculated the number of individuals necessary to achieve a minimal effect size of .20 at an alpha of .05 with a power of .80 (using a two-tailed T-test). A power estimate of .80 is consistent with the accepted norms for statistical power (Cohen, 1988). A total sample of 600 completed surveys will be necessary to achieve this level of power. The number of child welfare workers to be sampled from each site will be stratified based on the percent of target population children being served in each of these implementation sites.

B3. Maximizing Response Rates and the Issue of Non-Response

As our data collection activities pose different challenges in regard to response rates, the methods we will use to maximize response rates vary depending upon the data collection instrument and/or population. We anticipate the greatest challenges to maximizing response rates for the family member focus groups. Thus, for this data collection source, we will provide financial incentives for focus group participants ($35 per participant).

For interviews and surveys involving the child welfare agency and its partner agencies (i.e., child welfare agency survey, interviews with supervisors, group interviews with direct service workers), our efforts to boost response rates will rely predominantly upon the support of project directors and agency administrators. Prior research in this field has demonstrated the importance of gaining “buy in” from agency administrators in collecting data from these sources. Thus, we will help develop email messages that agency administrators will send to sample members encouraging them to participate in the survey administration, and interviews. We will also ask administrators to encourage participation through other modes of contact with their employees (e.g., in staff meetings, asking supervisors to encourage participation during unit meetings). Further, for both the group interviews and the child welfare agency survey administration, we will provide – and advertise – food for participants, an incentive that has proven to be effective in prior research with agency populations.

While we anticipate high response rates for the stakeholder surveys and interviews, we will make efforts to further maximize response rates. For the stakeholder survey, we will work with the project directors to send letters or emails to each stakeholder noting that the survey will be administered at the next regularly scheduled Systems of Care collaborative meeting. The letter will highlight the importance of stakeholders’ contribution to understanding how to implement a system of care. Local evaluators will administer and collect surveys and mail them to ICF International, with pre-paid postage provided. For the stakeholder interviews, ICF International will again work with the project directors to draft letters or emails informing the stakeholders that they have been selected for an interview and encouraging them to participate. The project directors will schedule the interviews with stakeholders while ICF International staff are on site. Those stakeholders who cannot participate during the site visit will be contacted for a telephone interview.

We do not anticipate problems in achieving full response for the completion of the process evaluation forms, technical assistance quality assurance forms, and the interviews with the project directors and coordinators. In our experience, project directors and coordinators complete all interviews and forms requested of them. We will send reminder emails to project directors and coordinators a few weeks prior to the submission date for the forms, and also, will work closely with the participants to schedule interviews at times that are convenient for them.

The response rate for the Child Welfare Survey will be calculated by dividing the number of completed surveys (numerator) by the number of direct service workers invited to participate in the survey (denominator). The number of workers invited to take the survey is determined by the power analysis as described above (p. 15).

The response rate for the Stakeholder Survey will be calculated by obtaining the total number of active collaborative members from each site’s project director (denominator) and dividing the number of completed surveys (numerator) by this number for each site.

B4. Pre-testing of Procedures or Methods

All instruments for this data collection effort were tested for usability with ICF International staff who had previous child welfare agency experience. The child welfare agency survey was pretested with a population of child welfare caseworkers and supervisors in the Fairfax, Virginia area. This session identified a number of issues that necessitated revisions to the survey. The stakeholder survey was initially developed for another, but similar, study of systems reform involving child welfare and has been shown to have adequate psychometric properties. Thus, pretesting was determined to be unnecessary for this instrument.

The local evaluation teams reviewed both the child welfare agency survey and the stakeholder survey, and minor revisions were made based on the teams’ feedback. Finally, the process evaluation forms were created from forms that have been successfully implemented in other systems reform evaluations conducted by ICF International, thus it was determined that usability testing within ICF International was sufficient for these documents.

The interview protocols and focus group protocols were deemed too specific to systems of care implementation to be appropriate for pretesting with a population not involved in this demonstration project. Thus, experts in child welfare and systems reform at ICF International reviewed the protocols and recommended minor revisions to ensure that they are effective in achieving their purpose.

B5. Contact Information

Should you have any questions about the contents of this OMB submission package, please contact one of the following individuals:

• Jan Shafer, U.S. Department of Health and Human Services, Children’s Bureau, 202-205-8172, [email protected]

• Raymond Crowel, ICF International, 703-385-3200, [email protected].

17