Supporting_Statement_Part_ARev 8-6-08

Department of Health and Human Services

Administration for Children and Families,

Children’s Bureau

Data Collection for the Evaluation of the Improving Child Welfare Outcome through Systems of Care Grant Program

Office of Management and Budget

Clearance Revision Package Supporting Statement

and Data Collection Instruments

July 2008

TABLE OF CONTENTS

Section Page

Executive Summary i

A. Justification 1

A1. Circumstances Requiring the Collection of Information 1

A2. Use of the Information 1

A3. Use of Improved Technology 3

A4. Duplication of Effort 3

A5. Burden on Small Businesses 3

A6. Consequences of Less Frequent Data Collection 3

A7. Special Data Collection Circumstances 3

A8. 5 CFR 1320.8: Public Comment and Consultation 4

A9. Respondent Compensation 4

A10. Confidentiality 4

A11. Sensitive Questions 5

A12. Respondent Burden 6

A13. Annualized Cost for Respondents 7

A14. Annualized Cost to the Federal Government 7

A15. Justification for Program Changes or Adjustments 7

A16. Schedule and Reporting 8

A17. OMB Expiration Date Requirement 8

A18. 5 CFR 1320.9: Exceptions to Certification for Paperwork Reduction Act Submissions 8

B. Collection of Information Employing Statistical Methods 9

B1. Respondent Universe and Response Rates 9

B2. Procedures for the Collection of Information 14

B3. Maximizing Response Rates and the Issue of Non-response 16

B4. Pre-testing 17

B5. Contact Information 18

LIST OF TABLES

Number Page

A-1. Annual Burden Estimates 6

A-2. Annualized Cost to the Federal Government 7

A-3. Project Schedule 8

B-1. Expected Response Rates by Strata 10

LIST OF APPENDICES

Appendix A: Child Welfare Act Public Law 96-272

Appendix B: Informed Consent Forms

Appendix C: Confidentiality Agreement

Appendix D: IC1, IC2, IC3 - Stakeholder Survey

Appendix E: IC4 - Child-Welfare Agency Survey

Appendix F: IC5 - Supervisor Interviews

Appendix G: IC6, IC7, IC8 - Stakeholder Interviews

Appendix H: IC9 - Project Director Interviews

Appendix I: IC10 - Focus Group with Family Members

Appendix J: IC11 - Parent Partner Interviews

Appendix K: IC12, IC13 - Child-Welfare Agency and Partner Agency Group Interviews

Appendix L: IC14 - Community Description Form

Appendix M: IC15 - Organizational Structure Form for Case Study Sites

Appendix N: IC16 - Organizational Structure Form for Non-Case Study Sites

Appendix O: IC17 - Collaborative Membership Form

Appendix P: IC18 - Training and Technical Assistance Quality Assurance Assessment

Appendix Q: IC19, IC20 - Training and Technical Assistance Conference Call Feedback Forms

Executive Summary

The Systems of Care Demonstration Project is the Children’s Bureau’s response to the findings of the Child and Family Service Reviews (CFSR), which, regulated by Congress in March 2000, mandated changes in the monitoring of State child and family service programs funded under Titles IV-B and IV-E of the Social Security Act in relation to child safety, permanency and well-being, the goals articulated by Congress in 1980 in the Adoption Assistance and Child Welfare Act (P.L. 96-272) (Appendix A).

The national evaluation will be conducted using a comprehensive, quasi-experimental data collection effort that has both process and outcome evaluation components. It will incorporate a multi-method, multi-respondent approach to data collection and will include quantitative and qualitative components.

The evaluation of the Children’s Bureau Systems of Care grants is not duplicative of any other evaluation that has been conducted. To date there have been no systematic evaluations of the Systems of Care grants issued by the Children’s Bureau of the Department of Health and Human Services (DHHS).

Responding to this data collection effort is completely voluntary. Individuals recruited for this effort will be given the opportunity not to respond at all, and to respond whenever it is convenient for them during the data collection period. In all cases, informed consent will be obtained from participants prior to data collection. All respondents will be informed that participation in the data collection efforts is voluntary, that they have a right to decline participation in the study, and that the data supplied will be kept confidential by the research team.

We estimate the annualized costs to the Federal government to be $135,000. This cost estimate is based on the work plan projected for completion under the contract for this effort. The current OMB inventory estimated 1,652 hours as the reporting burden; however, the proposed evaluation estimates 1,672 hours as the reporting burden. The increase in the annual burden is due to program change, which includes minor revision of all existing interview and survey protocols and changes in sample sizes.

Supporting Statement for the

Paperwork Reduction Act of Submission

A. JUSTIFICATION

A1. Circumstances Making the Collection of Information Necessary

The Systems of Care Demonstration Project is the Children’s Bureau’s response to the findings of the Child and Family Service Reviews (CFSR), which, regulated by Congress in March 2000, mandated changes in the monitoring of State child and family service programs funded under Titles IV-B and IV-E of the Social Security Act, the goals articulated by Congress in 1980 in the Adoption Assistance and Child Welfare Act (P.L. 96-272) (Appendix A). The evaluation of the Systems of Care Demonstration Project will provide critical data regarding if and how well this type of approach can result in better outcomes for child welfare-involved children and families. Following includes a description of proposed revisions to our existing OMB approval number 0970-0288. Proposed revisions include the addition of survey questions to existing instruments to allow the evaluation team to capture sustainability efforts planned by demonstration communities. This package also includes adjustments to the expected burden hours based on response rates obtained from 2005-2007.

A2. Purpose and Use of the Information Collection

The research questions for this evaluation project include:

1. What is the nature and extent of activities the grantees engage in to plan, implement, evaluate and refine their systems of care?

2. Do these activities create improvements in service delivery and child and family outcomes?

3. What is the process by which these activities result in improvements in service delivery and child and family outcomes?

4. What were the patterns of change both within and across sites?

Information gained from this evaluation will be used by the Children’s Bureau in determining whether and under what circumstances a child welfare driven Systems of Care improves service delivery and child and family outcomes. This will assist the Children’s Bureau in future decisions concerning funding child welfare reform efforts.

A3. Use of Improved Information Technology and Burden Reduction

No automated or technological data collection techniques will be employed in this evaluation. This decision is based on prior experience with reduced response rates when electronic submission of responses was presented as an option for certain protocols.

A4. Efforts to Identify Duplication and Use of Similar Information

Based on extensive contact that ACF program staff have with organizations in both the private and public sectors, it was determined that the evaluation is not duplicative of any other evaluation that has been conducted.

A5. Impact on Small Businesses and Other Entities

Not applicable, small businesses are not part of the universe.

A6. Consequences of Collecting the Information Less Frequently

The current evaluation plan includes periodic data collection methods and utilizes the minimum amount of data collection necessary to obtain valid, reliable, and useful information. Reducing data collection further would jeopardize the quality, integrity, and utility of the evaluation results.

A7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

Responding to this data collection effort is completely voluntary and respondents can respond whenever it is convenient for them during the data collection period. In all cases, informed consent will be obtained from participants prior to data collection. Informed consent forms (Appendix B) will include language regarding voluntary participation, confidentiality, and risks and benefits to participation that is consistent with the Privacy Act of 1974 and ICF International’s Institutional Review Board (IRB) requirements.

A8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

Individuals within and outside of the Children’s Bureau have been repeatedly consulted about all aspects of data collection, analysis, and reporting for this study. Consultation has included meetings, telephone calls, and e-mail communications. Individuals who have participated in this consultation include:

• Jan Shafer, 202-205-8172, [email protected]

• Pam Johnson, U.S. Department of Health and Human Services, Children’s Bureau, 202-205-8172,[email protected]

• Janet Griffith, ICF International, 703-279-4662, [email protected]

• Gary DeCarolis, Center for Community Leadership, 802-863-9132; [email protected]

As required by CFR 1320.8, a Federal Register notice appeared on May 21, 2008(Volume 73, Number 99) Page 29518. The first notice received no public comment.

A9. Explanation of Any Payment or Gift to Respondents

As written in the program guidelines and documented in the cooperative agreements between the Children’s Bureau and its nine grantees, fiscal agents in each of the nine sites were required to set aside $50,000 each year for the five-year term of the grant program for local and national evaluation activities. These funds are considered “compensation” for the local sites’ participation in the national evaluation, with one exception: the family members. Family members who participate in focus groups will be compensated for their time individually. Each family member focus group participant will receive $35.00, as well as reimbursement (or upfront compensation) for travel to and from the focus group site, if necessary. Finally, child welfare and partner agency staff will be given refreshments during the data collection activities in which they participate to increase response rates.

A10. Assurance of Confidentiality Provided to the Respondents

All respondents will be informed that participation in the data collection efforts is voluntary, and that they have a right to decline participation in the study (Informed consent statements are included in Appendix B). The confidentiality of the hard data, including interview tapes, transcripts, and surveys, will be protected by excluding names from all hard data. Also, in order to preserve the confidentiality of personally identifiable information, all staff persons engaged in the collection, management, and/or analysis of data will be required to sign a confidentiality agreement (see Appendix C).

The Privacy Act does not apply to this study because no unique or identifying information will be collected or reported. Signed consent forms are not linked to the data, and we are not collecting dates of birth, social security numbers or addresses from participants. All data will be reported in aggregate form only.

A11. Justification for Sensitive Questions

During focus groups, family members will be asked to reveal some detailed personal information about potentially sensitive elements of their lives. Our experience and the experience of other researchers in several studies interviewing similar populations, suggests that the type of focus group questions we are proposing to use does not usually lead to psychological distress. We are using focus group protocols that have been developed with input from a family advocate as well as ex-child welfare agency staff. These safeguards were taken to minimize the sensitivity of the questions being asked. However, in the event that the focus group arouses emotions that require a response, focus group facilitators will be trained to manage these situations and, in the event that additional help is needed, the evaluation team will work with representatives from each site to procure an emergency number to which respondents can be referred. We also will get contact information for at least one local therapist, should follow-up be necessary. In addition, respondents will be told at the outset that they can stop the focus group at any time or decline to respond to any specific question.

It is necessary to ask family members about their experiences with social service agencies in order to learn about improvements in service delivery and child and family outcomes. Information obtained from family members involved with the child welfare systems of the demonstration sites is critical to the evaluation of the impact of the System of Care approach.

A12. Estimates of Annualized Burden Hours and Costs

Estimates of response burden are outlined in the following table. These estimates were developed using response rates from data collection approved under OMB Control Number 0970-0288.

Table A-1: Annual Burden Estimates

Instrument	Affected Public	Number of Respondents	Number of Responses per Respondent	Average Burden hours per Response	Total Burden Hours
Stakeholder Survey	Individuals/ Households	20	1	.5	10
	Private Sector	60	1	.5	30
	State/Local/ Tribal	190	1	.5	95
Child-Welfare Agency Survey	State/Local/ Tribal	600	1	1	600
Supervisor Interviews	State/Local/ Tribal	90	1	1	90
Stakeholder Interviews	Individuals/ Households	10	1	1	10
	Private Sector	30	1	1	30
	State/Local/ Tribal	50	1	1	50
Project Director Interviews	State/Local/ Tribal	23	1	1	23
Focus Group with Family Members	Individuals/ Households	102	1	1.5	153
Parent Partner Interviews	State/Local/ Tribal	24	1	1	24
Child-Welfare Agency and Partner agency group interviews	Private Sector	80	1	1.5	120
	State/Local/ Tribal	200	1	1.5	300
Community Description Form	State/Local/ Tribal	9	1	2.5	23
Organizational Structure Form for Case Study Sites	State/Local/ Tribal	3	1	2	6
Organizational Structure Form for Non-Case Study Sites	State/Local/ Tribal	20	1	1	20
Collaborative Membership Form	State/Local/ Tribal	23	1	1.5	35
Training and Technical Assistance Quality Assurance Assessment	State/Local/ Tribal	23	1	1	23
Training and Technical Assistance Conference Call Feedback Forms	Individuals/ Households	2	12	.25	6
	State/Local/ Tribal	8	12	.25	24
Estimated Total Annual Burden Hours:					1,672

A13. Estimates of Other Total Annual Cost Burden to

Respondents and Record Keepers

There are no monetary costs to respondents other than their time to participate in the study. They are not asked to keep records.

A14. Annualized Cost to the Federal Government

We estimate the annualized costs to the Federal government to be $135,000. This cost estimate is based on the work plan projected for completion under the contract for this effort. As outlined below, the estimated annual Federal costs associated with this effort include costs to implement, manage, and monitor the data collection activities outlined in this application.

Table A-2: Annualized Cost to the Federal Government

Manage the project: $45,000 per year

Implement data collection efforts: $50,000 per year

Monitor efforts: $40,000 per year

T otal costs: $135,000 per year

A15. Justification for Program Changes or Adjustments

The current OMB inventory estimated 1,652 hours as the reporting burden; however, the proposed evaluation estimates 1,672 hours as the reporting burden. The increase in the annual burden is due to program change which includes minor revision of all existing interview and survey protocols and changes in sample sizes. Changes in sample sizes are based on average response rates obtained during fiscal years 2005-2007. Interview and survey questions were modified based on information gained conducting these surveys and interviews over the past three years. We have changed individual interviews with family members to focus group interviews in attempt to increase the number of family members who participate in the study. We eliminated two data collection instruments (Major Activities Form and Policy Changes Form) because we can obtain the same information through interviews. We have added parent partner interviews because they have played a key role in most of the grant communities and their perspective will be crucial to the study.

A16. Plans for Tabulation, Publication, and Project Time Schedule

Table A-3 presents the data collection and reporting schedule:

Table A-3: Project Schedule

Submit OMB package	June 2008
Ongoing data collection	October 2008 – September 2011
Data Analysis	Ongoing
Final Report	September 2011

Analysis of survey data will include basic descriptive statistics such as frequencies, means, and percentages. Additional analyses of survey data may include ANOVAs, t-tests, and chi-squares as appropriate. Qualitative site visit data and process data will be coded and analyzed to highlight recurring themes and key contextual factors.

A17. OMB Expiration Date

The OMB clearance number and expiration date will be displayed on all data collection instruments that will be provided to study participants.

A18. Exceptions to Certification for Paperwork Reduction Act

Submissions

No exceptions are requested.