CEY Evaluation - OMB Part A 9-5-07

Historically, faith-based and community organizations have played valuable roles in providing social services to individuals and families in need. Recently, the federal government has focused attention on the unique position of faith-based and community organizations (FBCOs). Recognizing the critical role faith-based and community organizations play in their communities, President George W. Bush launched a national initiative to expand opportunities for FBCOs through the establishment of the White House Office of Faith-Based and Community Initiatives and Faith-Based Centers in ten federal Cabinet agencies, including the Department of Health and Human Services (HHS).

The Compassion Capital Fund (CCF), a key component of the President’s Faith-Based and Community Initiative, was initially established by Congressional appropriation in 2002. The CCF is intended to build the capacity of FBCOs to serve needy individuals and families. The Office of Community Services (OCS), within the Administration for Children and Families, has administrative responsibility for the CCF program. OCS has established three components of the CCF program: the Demonstration Program that funds intermediaries to provide capacity building assistance to FBCOs, the Targeted Capacity Building Program that provides direct capacity building grants to FBCOs, and the Communities Empowering Youth (CEY) Program that funds capacity building grants to partnerships of organizations that focus on reducing youth violence and gang involvement and supporting positive youth development. Grants were awarded under the CEY component for the first time in fiscal year 2006. In 2006 OCS awarded $30,000,000 in funds to 100 CEY partnerships with the expectation of additional funding for the next two years. The 2007 CEY grantee cohort is expected to be awarded in late September 2007 and include about 30 partnerships, with a similar three-year grant period.

Given the agency’s interest in evaluating its programs and documenting performance, OCS awarded a contract for an evaluation of the CCF Demonstration Program that includes an outcome study and an impact study. The evaluation of the Demonstration Program is currently underway. While the newer Communities Empowering Youth component is also designed to support organizational capacity building of individual FBCOs, it includes a unique feature of working through coalitions or partnerships. Because of this major difference in approach to capacity building, OCS believes a separate evaluation of the CEY program is warranted to assess the capacity building outcomes achieved through the CEY model.

The proposed data collection included here is intended to support a study of the capacity building accomplishments within the partnership and of the designated “lead” organization and the other individual members of the partnership/coalition. The study will also document the range of capacity building activities undertaken. This evaluation provides ACF/OCS the opportunity to document the extent to which organizations operating within the new model developed for the CCF Communities Empowering Youth program meets the program objective of improving the capacity of faith-based and community organizations.

This evaluation is expected to include the collection of data at a baseline period to document the level of organizational capacity at the start of the CEY grant and at follow-up intervals to measure accomplishments over time. This submission requests approval for the initial survey instrument to be used in the study with the 2006 cohort and the 2007 cohort of CEY grantees and partnering organizations. A future submission will include follow-up information collection instruments.

A.2 How the Information Will Be Collected, by Whom, and For What Purpose

We propose to collect information through questionnaires (survey instruments) to be completed by CEY lead and partner organizations electronically on a web-based form or on a hard-copy form, based on their preference. A web-based application will be developed following OMB approval of the information collection.

The questionnaire has modules that are relevant for specific types of respondents. For example, some questions will only be asked of the “lead” organization; some questions will only be asked of the partner organizations; and some questions will only be asked of the organizations that are a part of the 2006 cohort. These questions are marked in the hard copy of the instrument included with this submission.

The questionnaire would be sent electronically to all the organizations for which we can confirm current email addresses and via mail to others. We propose to collect data at three points in time for the 2007 grantee cohort: within a few months of their grant award from ACF (baseline), at a mid-point of the three-year grant period (about 18 months after award), and shortly after the end of the three-year grant period. For the 2006 grantee cohort, they will have begun the second year of their grant at the time of the planned baseline data collection for the 2007 cohort. We propose to have the first data collection with the 2006 grantee cohort address some “baseline” measures and measures related to their accomplishments and activities during the first year of the grant. The baseline measures asked of the organizations in the 2006 cohort are related to topics for which recall is expected to be reliable (e.g., legal status, governance). We would then also conduct another follow-up data collection with the 2006 grantees at the end of their three-year grant period. This submission is for the initial information collection activity for both the 2006 and 2007 cohort.

The self-administered survey will seek information on a variety of topics/measures used to identify the capacity building status of the lead organizations and partnership members. The survey instruments will also mirror much of the information asked of faith-based and community organizations receiving services from Intermediary grantees under the Compassion Capital Fund Evaluation, focused on the CCF Demonstration program. We rely heavily on the survey questions previously approved by OMB in that study.

The initial survey instrument will be used to assess the level of organizational capacity in multiple domains for both the 2007 and 2006 grantee cohorts. As stated above, the 2006 grantee cohort will be asked a limited set of questions for which responses are to reflect the period at the start of the CEY grant in September 2006 and which can be expected to be reliably recalled. The instrument for each cohort includes questions related to leadership development, organizational development, program development, community engagement, and partnership interaction. We anticipate that the follow-up surveys will address a similar range of questions as included in this initial survey to allow for assessment of changes in organizational capacity in each domain over time. In addition, we plan that the follow-up surveys will include some additional questions about the types of capacity-building assistance received and partnership activities instituted during the period preceding the follow-up survey(s).

This survey is to be self-administered and completed by a program manager, executive director or other senior manager at the lead organization and by the executive director or other senior manager at each FBCO partnership member organizations. Surveys are to be completed through a secure website, or, if access to the Web is not possible, the survey may be completed by the organizations in paper format.

Research Questions

The CEY Evaluation is intended to address the following primary questions:

To what extent was capacity of the lead and partner organizations increased in the capacity domains identified in the program announcement?
To what extent was the capacity of the partnership increased?
What capacity building services (e.g., technical assistance, training, financial assistance) were provided to/received by lead and partner organizations?
How do these measures and outcomes vary by characteristics of the lead agency, the partners and the partnership?

The study is designed to answer these questions for individual lead agencies and partner organizations and the partnership as a whole by tracking capacity improvements over time. Once the initial data collection process is complete, a descriptive analysis will be conducted to report on the characteristics of the organizations participating in CEY and the partnership typologies. This information will be used to inform ACF officials and other interested parties about the types of organizations participating in the CEY grant program and to inform follow-up data collection efforts to better assess and measure the capacity building achievements of the organizations participating in the CEY program individually and at the partnership level. The Appendix to Part A includes a list of capacity building and partnership constructs and the associated questions in the survey instrument that address the constructs.

A.3 Use of Improved Information Technology to Reduce Burden

We propose to utilize a Web-based format to collect data to the extent possible. The Web-based format will reduce respondent burden and facilitate completion of the survey by CEY organizations by skipping questions that are not necessary/appropriate for specific respondents given their responses to prior questions and reduce the time required to transmit the completed survey back to the contractor. While we expect that most organizations will be able to utilize a Web-based format, a hard-copy form similar to that provided with this submission will be provided for organizations that lack access to the technology needed to complete a survey electronically or otherwise prefer to complete a paper survey.

A.4 Efforts to Identify and Avoid Duplication

While CEY grantees will be required to submit reports to ACF on their activities and progress, the information typically included in such reports is not uniform across grantees nor reported in sufficient detail to allow for assessment of small as well as large changes in organizational capacity. In addition, even though similar information is being collected within the evaluation of the CCF Demonstration program, we cannot assume the findings from the CCF Demonstration Evaluation will be applicable to the CEY program and its participants since the two programs operate in very different ways: the CEY program requires that grantees work through a partnership or coalition to build individual and partnership capacity and the CCF Demonstration program requires that an experienced intermediary provide capacity building technical and financial assistance to smaller organizations.

A.5 Efforts to Minimize Burden on Small Business or Other Entities

No small businesses will be involved as respondents. Respondents will be nonprofit faith-based and community-based organizations. The survey will not have a significant economic impact on a substantial number of small entities.

A.6 Consequences of Less-Frequent Data Collection

The initial instrument, included in this submission, is to be administered to each of the organizations in the 2006 and 2007 CEY grant cohorts only once to establish initial or early levels of organizational capacity. The initial collection is needed to document the range and level of organizational capacity among a diverse group of faith-based and community organizations.

To measure change in organizational capacity achieved during the period of the CEY grant, additional information collected at a later point is time is needed. We expect that there will be one additional follow-up survey of the lead and partner organizations in the 2006 grantee cohort at the end of their grant period and two additional follow-up surveys of the 2007 cohort, one at the mid-point and one at the end of their grant period. We will seek OMB approval of the follow-up information collection instruments in the near future.

A.7 Special Circumstances Requiring Collection of Information in a Manner Inconsistent with Section 1320.5(d)(2) of the Code of Federal Regulations

There are no special circumstances associated with this data collection.

A.8 Federal Register Comments and Persons Consulted Outside the Agency

In accordance with the Paperwork Reduction Act of 1995, ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of data collection activities. The first notice was published on June 4, 2007, in volume 72, number 106, pages 30805-30806, and provided a 60-day period for public comments. The second notice was published August 21, 2007, in volume 72, number 161, page 46634. No comments were received by ACF in response to the publications. Copies of the Federal Register notices for this information collection are included in Appendix A.

The instruments were developed by an Abt/Branch Associates research team comprising: Howard Rolston, Principal Investigator; JoAnn Jastrzab, Project Director; Rebecca Zarch, Project Manager; Kristina Kliorys; Suzanne Klein; and Dr. Nancy Burstein, Economist. Dr. William Ryan, a research fellow at Harvard University’s Hauser Center for Nonprofit Organizations, also played a key role in the development of survey instrument.

Input and feedback on the instrument was also obtained from the team at Abt Associates who have analyzed baseline data collected from the initial survey used in the CCF Demonstration program evaluation. Finally, feedback on the burden, clarity and utility of these survey instruments was sought from three of the 2006 CEY lead organizations and six partnership members. In general, the individuals from these organizations reported that the questions were clear and the burden was reasonable and within the estimates provided.

A.9 Payments to Respondents

We are not proposing to make payments to respondents.

A.10 Assurance of Confidentiality

Every effort will be made to maintain the privacy and/or confidentiality of respondents, to the extent possible. We will inform respondents that we will do everything we can to keep others from learning about their participation in the research study.

No sensitive or personal identifying information is being sought through this information collection, other than the name of the person completing the form. Data from the surveys will be reported in the aggregate, not by individual organization. Nevertheless, the contractor responsible for conducting the survey and extracting data will apply the same standards of data protection within their organization as they do with other data. The procedures for this study during data collection, data processing, and analysis activities will include the following:

All individuals hired by the contractor are required to adhere to strict standards and sign an oath of confidentiality as a condition of employment.

Hard-copy data collection forms will be delivered to a locked area at the contractor’s office for receipt and processing. The contractor will maintain restricted access to all data preparation areas (i.e., receipt, coding, and data entry). All data files on multi-user systems will be under the control of a database manager, with access limited to project staff on a “need-to-know” basis only.

Individual identifying information will be maintained separately from completed data collection forms and from computerized data files used for analysis. No respondent identifiers will be contained in public use files made available from the study and reports developed for the study will report aggregate, not individual level, findings.

A.11 Questions of a Sensitive Nature

The questions included on the data collection instruments for this study do not involve sensitive topics. The survey requests information about organizations; other than the name of the individual completing the form, no personal information is asked.

A.12 Estimates of Respondent Burden

The total burden for the initial data collection is estimated to be 607 hours. Exhibit 2 presents estimates of the reporting burden for the initial survey. Time estimates are based on experience with similar instruments in other studies of comparable organizations.

Exhibit 2 Estimates of 2007 Annualized Burden Hours and Cost
Data Collection Sources	Number of Respondents	Minutes Per Respondent	Response Burden in Hours	Estimated Cost Per Hour^a	Costs per Respondent	Total Burden (Costs)
Initial CEY Evaluation Survey	810	45	607.25	$42.31	$31.73	$19,268
Notes: a Estimated cost per hour is calculated based on median income of $88,006 for nonprofit program directors in 2005. As such, it represents a conservative estimate of the cost to respondents.

A.13 Estimates of the Cost Burden to Respondents

There are no annualized capital/startup or ongoing operation and maintenance costs associated with collecting the information. Other than their time to complete the surveys, which is estimated in Exhibit 2 above, there are no direct monetary costs to respondents.

A.14 Estimates of Annualized Government Costs

The information collection activity and associated form have been developed in the performance of the U.S. Department of Health and Human Services Contract Number: HHSP233200600002T. The period of performance is from September 20, 2006 through September 29, 2008. The total cost to the Federal government for the Communities Empowering Youth – Performance Measurement and Evaluation Design Study under which the initial instrument has been developed is $1,668,966. Of that total, approximately $123,953 (or 7 percent) will be used for the development of the data collection for which clearance is requested.

ACF will be issuing an RFP for a separate contract to carry out the CEY evaluation, including the administration of the initial and follow-up surveys, data extraction and analyses and reporting. The costs associated with this future procurement and associated tasks have not been determined at this time.

A.15 Changes in Hour Burden

This submission is for new collection.

A.16 Time Schedule, Publication, and Analysis Plan

The schedule shown below in Exhibit 3 displays the sequence of activities required to conduct the information collection activities and includes key dates for activities related to instrument design and estimates of date for actual data collection, analysis, and reporting. This submission address only the initial round of data collection.

Exhibit 3 Time Schedule
Activities and Deliverables	Cohort	Date
Instrument Design	2006, 2007	July 2007
Initial Data Collection	2006, 2007	January 2008-Febraury 2008*
Mid-Point Data Collection	2007	April-May 2009*
Follow-up Data Collection	2006	October-November 2009*
Follow-Up Data Collection	2007	October-November 2010*
Data Analysis	2006, 2007	TBD
Reporting	2006, 2007	TBD

* ACF will issue an RFP to contract for the conduct of the CEY Evaluation; the dates provided here reflect current estimates of the timeline for that study. Final dates will be determined within the final contract for the CEY evaluation.

A.17 Display of Expiration Date for OMB Approval

ACF is not requesting a waiver for the display of the OMB approval number and expiration date on the data collection instruments.

A.18 Exceptions to Certification Statement

This submission does not require an exception to the Certificate for Paperwork Reduction Act (5 CFR 1320.9).

File Type	application/msword
Author	USER
Last Modified By	USER
File Modified	2007-09-06
File Created	2007-09-06