CEY Evaluation - OMB Part B 9-5-07

A contractor (TBD) will collect information for the Communities Empowering Youth (CEY) Evaluation study on behalf of ACF. ACF will issue an RFP to procure the services of a contract firm for the conduct of the CEY evaluation. The survey included in this submission was developed through ACF’s current contract with Abt Associates Inc., based in Cambridge MA. This existing contract allowed for the development of an initial survey that could be used if ACF determined to undertake an evaluation of the CEY program. ACF has determined that it is in the best interests of the agency to undertake a separate evaluation of the CEY program because the findings from the ongoing evaluation of the CCF Demonstration program are not likely to be applicable to the CEY program.

The analysis for the CEY evaluation study will assess changes/improvements in various domains of organizational capacity among CEY lead and partner organizations and for the partnerships as whole units. Results will focus on changes in various domains of organizational capacity of the organizations and partnerships in the time frame between the initial and follow-up data collections. The analysis will be based on two time-point measurements for the 2006 grantee cohort and three time-point measurements for the 2007 grantee cohort. We will examine the organizational capacity changes in leadership development, organizational development, program development, community engagement, and partnership functioning and interaction. We will conduct sub-group analysis where sample sizes allow, such as reporting findings by type of organization (faith-based vs. community-based), background and expertise of the lead organization (capacity building vs. direct service), and partnership model (governance structure, past experience working together and capacity building approach).

B.1 Respondent Universe and Sampling Methods

Sample Selection

CEY 2006 Grantee Cohort: ACF proposes to include 50-60% of the 100 2006 CEY grants in the evaluation study, equaling a total of about 450-540 organizations (one lead and an average of eight partner organizations in each grant). Although the 2006 grantee cohort will be in the early phase of the second year of their grant at the time the initial information collection instrument is fielded, ACF believes it is important to include a representative sample of the 2006 cohort in the CEY evaluation study. The 2006 cohort is more than three times the size of the 2007 cohort (expected to be about 30 grants). Relying solely on findings from the smaller 2007 cohort would not provide sufficient information on the CEY program to document program performance or sufficiently support the identification of potential changes in program design or management within ACF that may benefit current and future grantees under the CEY program.

Based on a review of the approved 2006 grant applications, it is clear that there are differences among the lead organizations that may be associated with achievements within the partnership and among the partners and that would be of interest from a policy and program perspective. The key differences are: whether the lead organization is faith-based or not and whether the lead agency is primarily a direct service provider, primarily a capacity building organization, or a combination of the two. We propose that the 2006 grantee cohort sample be stratified based on these factors to ensure that each type of organization is represented in the sample in proportion to their numbers in the universe of 2006 grantees.

CEY 2007 Grantee Cohort: The approach for the CEY evaluation study assumes the 2007 CEY grantee universe (all lead organizations and all partners) will be included in the study. As noted elsewhere, based on the ACF grant announcement for 2007, the universe is estimated to include 30 new CEY grants – with each having at least one lead organization and a varied number of partner organizations. Assuming the average number of partners is the same for the 2007 cohort as in the 2006 cohort, we expect to include a total of about 270 organizations in the study. Given the small size of the CEY 2007 grantee cohort and the modest contract burden and resources required, we believe it is appropriate to include all of them in the evaluation study.

The stratified random sample of 2006 grantees and the universe of 2007 grantees will allow for descriptive analyses of the initial (baseline) characteristics of organizations to be compared across sub-groups. Comparisons will likely include faith-based vs. secular organizations; cohort in which funded; and characteristics of the lead agency. This sampling plan will support future analysis of follow-up survey data which will involve the use of descriptive statistics such as frequencies and cross-tabs and basic significance tests (t-tests and chi-squares) to determine whether responses related to changes in organizational capacity differ significantly for various sub-groups of organizations.

B.2 Information Collection Procedures

As noted in section A, we propose to collect information through questionnaires (survey instruments) to be completed by CEY lead and partner organizations electronically on a web-based form or on a hard-copy form, based on their preference. The questionnaires would be sent electronically to all the organizations for which we can confirm current email addresses and via mail to others.

We propose to collect data at three points in time for the 2007 grantee cohort: within a few months of their grant award from ACF (baseline), at a mid-point of the three-year grant period (about 18 months after award), and shortly after the end of the three-year grant period. For the 2006 grantee cohort, they will have begun the second year of their grant at the time of the planned baseline data collection for the 2007 cohort. We propose to have the first data collection with the 2006 grantee cohort address some “baseline” measures and measures related to their accomplishments and activities during the first year of the grant. We would then also conduct another follow-up data collection with the 2006 grantees at the end of their three-year grant period. This submission is for the initial information collection activity for both the 2006 and 2007 cohort.

B.3 Methods to Maximize Response Rates

Compliance with the CEY evaluation is a condition of the CEY grants which are awarded as cooperative agreements. The program announcement under which applications are submitted specifies: “Lead organizations selected to receive a CEY award will be responsible for:

Complying with requirements associated with a Federal evaluation of CEY. This may include, but is not limited to, providing administrative data, cooperating with survey data collection, interviews, and site visits involving the lead and partner organizations.”

The study and expectations regarding the survey will be included on the agenda of the combined annual and orientation meeting for the 2006 and 2007 grantees, respectively, conducted by the funding office. This will allow key staff of the organizations to receive detailed information about the evaluation study and the survey instrument, as well as hear from the funding agency about the importance of their participation through survey completion. As she has for the other CCF evaluation study, we will ask that the Director of the Office of Community Services sign the cover letter that will accompany the email/mail transmitting the survey instrument to the grantees and partner organizations stressing the importance of their input. Given that we will be surveying currently active grantees in the initial survey, we expect a very high response rate very near 100 percent. The Web-based format is also expected to increase the response rate because it will ease administration of the survey.

We will have a number of documents to use to help address non-response bias. ACF and contractor staff will review existing grant case files for non-respondents to extract available characteristic information in order to assess non-response bias. To the extent such data are included in the files, we will obtain information on basic characteristics of the organizations, such as type of organization, age, legal status, size (budget and staffing), and geographic location, to compare with the characteristics of respondents. We will conduct appropriate tests (e.g., t-tests and chi squares) to determine whether there are significant differences between responding and non-responding organizations on those variables we are able to extract from the files. We believe, however, that non-response bias is likely to be minimal with a high response rate.

B.4 Test of Procedures

In order to obtain a true estimate of the burden associated with these surveys and to assess the efficacy of elements selected for inclusion, the instrument was pre-tested with three representatives of lead organizations within the 2006 grantee cohort and six FBCO partnership members. These pre-test interviews included debriefings in which respondents provided their feedback on the time associated with the surveys and the extent to which they were able to answer the survey questions. Modifications to the length, content and structure of the surveys were made based on the results of the pre-test interviews. Several questions were shortened or deleted to minimize respondent burden, ensuring that the surveys can be administered in 45 minutes or less.

Many of the survey questions are identical or similar to questions included in the baseline survey approved by OMB for the ongoing CCF Evaluation study (OMB Control Number: 0970-0293). The CCF Evaluation baseline survey has been completed by over 1200 FBCOs that received organizational capacity building services from CCF intermediary organizations. We have not identified any problems with specific questions being answered appropriately nor received any negative feedback from respondents about that survey or specific questions.

B.5 Individuals Consulted on Statistical Aspects of Design

The plans for statistical analyses for this study were primarily developed by Abt Associates Inc. The team is led by JoAnn Jastrzab, Project Director; Howard Rolston, Principal Investigator; and Rebecca Zarch, Project Manager. Economist, Nancy Burstein, also provided consultation on the statistical aspects of the design. Contact information for these individuals is provided below.

JoAnn Jastrzab, Abt Associates Inc.,

55 Wheeler St., Cambridge, MA 02138

617-349-2372

Howard Rolston, Abt Associates Inc.

4550 Montgomery Ave.,Bethesda, MD 20814

301-634-1820

Rebecca Zarch, Abt Associates Inc.

55 Wheeler St.,Cambridge, MA 02138

617-349-2628

Nancy Burstein, Economist, Abt Associates Inc.

4550 Montgomery Ave.,Bethesda, MD 20814

301-634-1820

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods

File Type	application/msword
File Title	Compassion Capital Fund: Communities Empowering Youth Evaluation
Author	USER
Last Modified By	USER
File Modified	2007-09-06
File Created	2007-09-06