Supporting Statement Part A

SUPPORTING STATEMENT

FOR OMB CLEARANCE

PART A

COMMUNITY EMPOWERING YOUTH EVALUATION

FOLLOW-UP DATA COLLECTION

ADMINISTRATION FOR CHILDREN AND FAMILIES

OFFICE OF PLANNING, RESEARCH AND EVALUATION

February 4, 2009

Part A: Justification

A.1 Explanation of Circumstances That Make Collection of Data Necessary

The Office of Community Services in collaboration with the Office of Planning, Research and Evaluation contracted for an evaluation of the Communities Empowering Youth (CEY) component of the Compassion Capital Fund program in December 2007. The evaluation will document changes in organizational capacity achieved by faith-based and community organizations participating in the CEY program from an initial point in time, measured through a baseline survey (OMB #0970-0335), to a period approximately 12-months later for the full sample and again about 24-months from baseline for the 2007 grantee cohort. In order to assess change from one period to the next, the baseline and follow-up survey instruments cover the same areas of organizational capacity. About ninety percent of the questions on the follow-up survey are the same as those on the baseline survey; modules have been added for organizations brought into CEY partnerships after baseline data collection and for organizations that left the partnerships.

This is a request for approval of the follow-up information collection instrument for the CEY Evaluation. As described in more detail below, this instrument will be used to gather information on change and progress made in various domains of organizational capacity among the faith-based and community-based organizations engaged in CEY supported activities.

Background: The Compassion Capital Fund (CCF) was established by Congress in 2002 and is administered by the Office of Community Services (OCS), Administration for Children and Families (ACF), within the Department of Health and Human Services. The primary purpose of CCF is to help faith-based and community organizations (FBCOs) increase their effectiveness, enhance their ability to provide social services to serve those most in need, expand their organizations, diversify their funding sources, and create collaborations to better serve those in need. One of the three major programs under CCF is the Communities Empowering Youth (CEY) program.

In keeping with their commitment to program performance, OCS awarded a contract for a descriptive outcomes study of the CEY program in December 2007. This information collection request is for a follow-up survey of faith-based and community organizations to assess change in domains of organizational capacity associated with participation in the CEY program.

Funded CEY projects consist of a lead organization (the grantee) and partnering faith-based and community organizations with a history of collaboration. Together, they form a partnership that is intended to improve services to youth within a specific geographical area by increasing the organizational capacity of lead and partner organizations. CEY grants are made to faith-based and community organizations working on issues of gang involvement, youth violence, and child abuse and neglect. In 2006 OCS awarded 3-year grants to 100 CEY partnerships; in 2007 the agency funded an additional 31 partnerships with 3-year grants. The CEY Evaluation includes 50 of the 2006 grantees and all 31 of the 2007 grantees.

The proposed data collection is part of a multi-method evaluation of CEY. Information collection instruments for two other components of the evaluation have already received OMB approval:

• Baseline survey of all 2007 and half of the 2006 cohort of grantees and their partners (OMB #0970-0335)

• Longitudinal case studies of selected partnerships (OMB #0970-0352)

The data collection covered by this request is for follow-up survey data collection that will provide the basis for analyses of outcomes in multiple domains of organizational capacity within individual faith-based and community organizations as well as the outcomes related to the growth of partnerships’ organizational capacity. The evaluation is an important opportunity to examine the extent to which the Compassion Capital Fund Communities Empowering Youth program meets its objective of improving the capacity of faith-based and community organizations.

The information collected through this proposed data collection activity is critical to the evaluation. The results of the evaluation will document the extent to which the broader CCF and specific CEY program objectives are achieved and will provide useful information to OCS to, if necessary, make adjustments to the CEY grant parameters or their approach to technical assistance to CEY grantees.

A.2 How the Information Will Be Collected, by Whom, and For What Purpose

How Information Will Be Collected and by Whom

The proposed survey will be self-administered and is to be completed by a program manager, executive director or other senior manager at the lead organization and at each partner organization. The survey will be Web-based and accessed through a secure Website hosted the evaluation contractor. Experience with administration of the baseline survey indicates that the organizations in the study have access to the Web for this purpose. However, if Web access is not available or a respondent prefers to complete the survey in hard copy, the survey will be made available in paper format to be mailed back to the contractor. The evaluation contractor will establish a help-desk system that will be available to answer questions and facilitate completion of the survey.

This information collection request is for a single survey instrument that will be administered in two waves of follow-up. The first wave will include all of the organizations in the study. The second wave will be administered only to the 2007 cohort whose grants are continuing until 2010.¹

For What Purpose

Data gathered through the survey instrument will be used to assess the CEY-funded lead organizations’ and FBCO partnership members’ progress in capacity building as well as changes in capacity for the partnership as a whole. The instrument includes questions related to the CEY program’s four capacity building priorities: leadership development, organizational development, program development, community engagement, and partnership management and effectiveness.

The first follow-up will be administered about 12 months after grantees and their partner organizations completed an initial survey at baseline. For the 2006 cohort, the first follow-up will occur near the end of their 3-year grant cycle; for the 2007 cohort, this initial follow-up will occur about at the midpoint of their grant. This information will be compared to the data collected through the baseline survey to measure changes in organizational capacity between the two periods of data collection. An interim report on the findings will be prepared.

The second follow-up will be conducted with the 2007 grantee cohort about 12 months after the first follow-up survey or about 24 months after the baseline survey (near the end of their 3-year grant cycle). A final report on the findings from the evaluation will be prepared.

In order to document changes over time on similar measures of organizational capacity, most of the content of the follow-up survey is the same as questions included in the baseline instrument, with separate modules for lead organizations and partner organizations. Ninety-one percent of the questions included in the follow-up survey come from the OMB approved baseline survey. The areas of new content include questions related to changes in the composition of the CEY partnership, opinions about their experience with the CEY program, and two additional modules -- one for new partners that joined the partnership after administration of the baseline survey and one for partners that exited the CEY partnership. Exhibit 1 shows the specific question numbers that are new under this OMB information collection request.²

Exhibit 1 Differences Between Baseline and Follow-Up Survey Instruments
	Same As Baseline	New for Follow-Up
Contact Information
Questions 1-9	X
Organizational Profile
Questions 10-15	X
Leadership Development
Questions 16-24	X
Organizational Development
Questions 25-34a	X
Program Development
Questions 35-38	X
Community Engagement
Questions 39-40	X
CEY Partnership
Questions 41-42		X
Questions 43-44	X
Questions 45-49		X
Questions 50-54	X
New Members of the CEY Partnership
Questions 55-63	X
CEY Activities
Questions 64-65d	X

Research Questions

The evaluators will use follow-up survey data to address the following key research questions.

• What capacity building activities were provided?

• How extensive were these activities?

• To what extent and in what domains was organizational capacity increased?

• How do outcomes vary by characteristics of the lead agency, the partners and partnership?

As summarized in Exhibit 2, the CEY evaluation involves a total of 543 respondents in the first survey wave. This is made up of 81 CEY grantees/lead organizations and approximately 462 FBCO partnership members.³ The second survey wave will include 31 lead organizations and their 134 partnership members, awarded grants in 2007.⁴ The average annual number of respondents will be 354.

Exhibit 2 Summary of Proposed Data Collection Activities
Survey	Design	Funding Cycle	# of Respondents	Data Collection	Survey Administration	Timing
Follow-Up Survey Wave 1	Pre/Post	2006/2007	543	Follow-up	Web-Based/ self-administered	June 2009
Follow-Up Survey Wave 2	Pre/Post/Post	2007	165	Follow-up	Web-Based/ self-administered	June 2010
Average per year			354

A.3 Use of Improved Information Technology to Reduce Burden

In order to reduce burden, a Web-based format will be used to collect data. Using a Web-based format allows for automated “skip” patterns limiting the number of questions respondents have to read/answer. For some users, this may reduce the burden by up to 25%. Any organizations in the sample that lack access to the Web or prefer to complete a paper version of the survey will be mailed a hard copy of the survey. Contractor staff will be available by phone and email to answer questions and facilitate completion of the survey.

A.4 Efforts to Identify and Avoid Duplication

While similar information about changes in individual-level organizational capacity among faith-based and community organizations is being collected for the evaluation of the Demonstration Program component of the Compassion Capital Fund, the CEY program’s unique focus on partnerships, rather than intermediaries, to improve the organizational capacity of youth-serving organizations requires additional measures to assess performance under CEY. Further, CEY grantees will likely report on activities and progress on organizational capacity building under the grant but such reporting is not systematic.

A.5 Efforts to Minimize Burden on Small Business or Other Entities

No small businesses will be involved as respondents. Respondents are nonprofit faith and community-based organizations. The web-based survey will apply skip patterns based on responses to minimize burden. Further, where appropriate (e.g., address), responses from the baseline survey will be used to pre-populate the follow-up survey and require only those with changes to enter data.

A.6 Consequences of Less-Frequent Data Collection

The first follow-up survey will provide the only opportunity to collect outcome data from the 2006 grantee cohort in the study prior to the end of their 3-year grant period.

Administering two follow-up surveys to the 2007 cohort allows for the measurement of interim and longer term gains near the end of the 3-year grant period. Measurement at the mid-point for the 2007 cohort will provide valuable information about incremental changes in organizational capacity that may not be recalled if only one follow-up survey were administered 24 months after baseline. Further, because of the high frequency of staff turnover in small nonprofit organizations, key respondents who were initially involved and could address questions of organizational change and the activities in which the organization participated may often not be available 24 months later. Two data collection points for the 2007 cohort will increase the validity of the evaluation findings.

A.7 Special Circumstances Requiring Collection of Information in a Manner Inconsistent with Section 1320.5(d)(2) of the Code of Federal Regulations

There are no special circumstances associated with this data collection.

A.8 Federal Register Comments and Persons Consulted Outside the Agency

In accordance with the Paperwork Reduction Act of 1995, ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of data collection activities. The notice was published on September 11, 2008, in volume 73, number 177, pages 52860-52861, and provided a 60-day period for public comments. One individual requested a copy of the proposed instrument but no comments were received.

The follow-up instruments utilize many of the same measures included in the baseline survey developed by the Abt and Branch Associates (sub-contractor) research team, comprising: Howard Rolston, Project Quality Advisor; JoAnn Jastrzab, Principal Investigator, Project Director; Rebecca Zarch, Project Manager; Kristina Kliorys; Suzanne Klein; and Dr. Nancy Burstein, Economist. William Ryan, a research fellow at Harvard University’s Hauser Center for Nonprofit Organizations, played a key role in the development of survey questions to measure changes in organizational capacity. Finally, three of the 2006 CEY lead organizations and six 2006 CEY partnership members offered feedback on the burden, clarity and utility of the baseline survey instruments.

The follow-up survey instrument involved additional work by Jacob Klerman and Howard Rolston, co-Project Quality Advisors; JoAnn Jastrzab, Project Director; Ryoko Yamaguchi, Deputy Project Director; Lianne Fisman, Maura Simpson, and Philip Gordon. Feedback on the surveys was solicited from staff at ACF, the research team at Abt Associates responsible for the CCF Demonstration Program Evaluation, and William Ryan of Harvard University.

A.9 Payments to Respondents

There will be no payments to respondents for the follow-up surveys.

A.10 Assurance of Confidentiality

Every effort will be made to maintain the privacy of respondents, using several procedural and control measures to protect the data from unauthorized use. Other than name, no personal identifying information is proposed to be collected. The procedures to protect data during information collection, data processing, and analysis activities are as follows:

• All respondents included in the study sample will be informed that the information they provide will be used only for the purpose of this research. Individuals will not be cited as sources of information in prepared reports.

• To ensure data security, all individuals hired by the contractor are required to adhere to strict standards and sign an oath of confidentiality as a condition of employment.

• Hard-copy data collection forms will be delivered to a locked area at the contractor’s office for receipt and processing. The contractor will maintain restricted access to all data preparation areas (i.e., receipt, coding, and data entry). All data files on multi-user systems will be under the control of a database manager, with access limited to project staff on a “need-to-know” basis only.

• Individual identifying information will be maintained separately from completed data collection forms and from computerized data files used for analysis. No respondent identifiers will be contained in public use files made available from the study, and no data will be released in a form that identifies individuals.

A.11 Questions of a Sensitive Nature

The questions included on the data collection instruments for this study do not involve sensitive topics.

A.12 Estimates of Respondent Burden

Exhibit 3 presents estimates of the annualized reporting burden for the survey. The survey will be administered twice, over a 2-year period. The average number of annual respondents is presented in Exhibit 3. Time estimates are based on experience with the baseline survey and other similar instruments in studies of similar organizations.

Exhibit 3 Estimates of Annualized Burden Hours and Cost
Data Collection Sources	Number of Respondents	Minutes Per Respondent	Response Burden in Hours	Estimated Cost Per Hour a	Total Cost Burden
Follow-up Survey	354	28	165	$42.31	$6,981.15
Notes: a Estimated cost per hour is calculated based on median income of $88,006 for nonprofit program directors in 2005. As such, it represents a conservative estimate of the cost to respondents.

A.13 Estimates of the Cost Burden to Respondents

There is no annualized capital/startup or ongoing operation and maintenance costs associated with collecting the information. Other than their time to complete the surveys, which is estimated in Exhibit 3, there are no direct monetary costs to respondents.

A.14 Estimates of Annualized Government Costs

This information collection activity has been developed in the performance of the Contract Number: 233-03-0088 (Task Order HHSP233200600002T). The two rounds of follow-up survey data collection, including instrument development, Web design and programming, data collection, analyses, and report writing, are estimated to cost about $213,023 annually.

A.15 Changes in Hour Burden

The baseline survey burden was estimated to be 810 respondents and 608 hours of time burden annually. The change in burden reflects more detailed information about the number of partner organizations associated with each lead/grantee and experience with administration of the baseline survey. In addition, since data collection is complete for the baseline instruments, ACF is not requesting their continued approval. As a result of discontinuing the baseline instruments, there is an overall reduction in burden.

A.16 Time Schedule, Publication, and Analysis Plan

The schedule shown in Exhibit 4 displays the sequence of activities required to conduct the information collection activities and includes key dates for activities related to data collection, analysis, and reporting.

Exhibit 4 Time Schedule
Activities and Deliverables	Cohort	Date
Follow-Up 1 Data Collection	2006/ 2007	May–July 2009
Data Analysis for Follow-Up 1	2006, 2007	August– October 2009
Reporting First Follow-Up	2006, 2007	Fall 2009
Follow-Up 2 Data Collection	2007	May–July 2010
Data Analysis for Follow-Up 2	2007	August–October 2010
Reporting Second Follow-Up	2007	Fall 2010

To answer the research questions, the follow-up survey will be utilized to measure changes in organizational capacity and partnerships. The main analyses for the follow-up data will be descriptive statistics and gain scores (difference between the baseline and the follow-up) to determine change over time. The analytic approach is designed to track three broad issues: the growth in the capacity of lead organizations; the growth in capacity of the FBCO partner organizations; and the improved organizational capacity of the community partnership itself.

A.17 Display of Expiration Date for OMB Approval

ACF is not requesting a waiver for the display of the OMB approval number and expiration date on the data collection instruments.

A.18 Exceptions to Certification Statement

This submission does not require an exception to the Certificate for Paperwork Reduction Act (5 CFR 1320.9).

1 The 3-year grant period for the 2006 cohort ends in September 2009 and in 2010 for the 2007 cohort. The first follow-up survey will be administered in the last year of the grant period for the 2006 cohort and the second year of the grant period for the 2007 cohort. The second survey will be administered to the 2007 cohort in the last year of their grant period.

2 In some instances, minor changes in the format of questions have been made based on feedback from baseline respondents and staff working on the helpdesk.

3 The partner sample size is approximate due to changes in the partnerships where partner organizations exit or enter throughout the grant period. Therefore, the number provided is the number at baseline.

4 The second wave of follow-up may also include any 2006 grantees that receive grant extensions that result in their continuing operations at the time of the second follow-up.

Administration for Children and Families