Supporting Statement Part B

SUPPORTING STATEMENT

FOR OMB CLEARANCE

PART B

COMMUNITIES EMPOWERING YOUTH (CEY) EVALUATION

FOLLOW-UP DATA COLLECTION

ADMINISTRATION FOR CHILDREN AND FAMILIES

OFFICE OF PLANNING, RESEARCH AND EVALUATION

January 29, 2009

Part B: Collection of Information Employing Statistical Methods

This study will use statistical methods to assess changes in organizational capacity outcomes in the following major areas covered in the survey instruments: leadership development, organizational development, program development, and community engagement. The analysis for this study will assess changes in capacity for CEY lead organizations, partnership member organizations, and the CEY partnerships as a whole. Results will focus on changes in the capacity in the time frame between the initial data collection and each follow-up data collection.

The analysis will be based on two time-point measurements for the 2006 grantee cohort and three time-point measurements for the 2007 grantee cohort. Due to changes in partnership composition (the addition or exit of partner organizations), modules will be asked of newly added or departed partner organizations.

We will conduct subgroup analysis where sample sizes allow, reporting outcomes by type of organization (faith-based vs. community-based), expertise of the lead organization (capacity building vs. direct service), and partnership model (governance structure, past experience and capacity building approach), for example.

B.1 Respondent Universe and Sampling Methods

We used a two-step cluster sampling strategy. In the first step, we selected a sample of grantees (lead organizations). In the second step, we selected the population of partner organizations associated with the selected lead organizations. The selection process for each of the two cohorts, 2006 and 2007, is described below.

CEY 2006 Grantee Cohort

The 2007 CEY program/grant announcement made clearer than did the 2006 announcement that applicants must have existing partnership relationships and, thus, be able to move quickly to work as a partnership. As a result, the agency was interested in including all of the 2007 grantees in the evaluation and a sample of 2006 grantees, as could be supported within the resources available. Available evaluation resources allowed for 50 of 100 2006 grantees to be included in the evaluation. In order to select the sample of 50, a stratified random sampling scheme, discussed below, was utilized. In a second step, we then selected all 328 partner organizations associated with the sampled lead organizations.

In selecting 50 out of 100 lead organizations (grantees), the stratified sampling reflected two substantive interests: (1) How outcomes differ between lead organizations that are faith-based and lead organizations that are community-based; and (2) How outcomes differ between lead organizations that provide only capacity building services and lead organizations that provide capacity building services and also direct services.

Exhibit 1 shows the number of lead organizations/grantees in each stratum (to the right of the slash), and the sample by strata (to the left of the slash). Because of the small number of faith-based lead organizations (10), we selected all of them for the sample.

Exhibit 1 Distribution of 2006 Grantees and Sample by Strata
Type	Capacity Building Only	Both Capacity Building and Direct Service	Total
Faith-based	2/2	8/8	10/10
Community-based	12/18	28/72	40/90
Total	14/20	36/80	50/100
Note: <number of grantees> / <number of sampled organizations>

The goal was to sample 50 organizations. We selected all ten faith-based organizations, leaving 40 community-based organizations to sample. Below, we discuss the methods used to select the remaining 40 organizations.

Stratum 1: Community-based organizations providing only capacity-building services. Twelve out of 18 lead organizations were selected within this stratum. First, the sample was allocated proportionately to the population. In this stratum, proportional allocation resulted in a small sample size (8 organizations). Therefore, the second step was to adjust the sample slightly to ensure reliable estimates for each stratum. This resulted in oversampling in this stratum, selecting 12, instead of 8, organizations.

To select 12 grantees in our sample from the 18 available in the community-based and capacity-building stratum, the list of grantees was sorted by rural, urban, and mixed, and numbered from 1 to 18. This sorting before selection and using systematic sampling method for selection of the sample assures proportional representation of rural, urban, and mixed grantees in the sample.¹

Once the list of grantees was sorted by urbanicity, an equal probability systematic sample was selected. The sampling interval for the selection of the sample was determined by taking the ratio of the population size and sample size. In this case, the sampling interval was 18/12 = 1.5. Using a random start (derived by selecting a random number between 0 and 1 and multiplying this number by the sampling interval) and the sampling interval of 1.5, 12 grantees were selected.

Stratum 2: Community-based organizations providing both capacity-building and direct services. From Stratum 1, we filled 12 of the 40 “slots” available. This left 28 “slots” to fill. Therefore, our goal was to select 28 out of 72 lead organizations within this stratum. First, we allocated the sample proportionately. This resulted in 32 organizations within this stratum. In the second step, we adjusted the sample due to previous oversampling discussed above. For this stratum, we decreased, or undersampled, the sample size slightly. In sum, the two strata comprise 40 community-based organizations. To select the specific lead organizations, we proceeded as with the organizations providing only capacity building. The list of 72 grantees within this stratum was sorted by rural, urban, and mixed and numbered from 1 to 72. The sampling interval was 72/28 = 2.5714. Again, using a random start and a sampling interval of 2.5714, 28 grantees were selected in the sample.

CEY 2007 Grantee Cohort

For the 2007 cohort, we selected all 31 lead organizations (i.e., there was no sampling) and then all of their approximately 134 partner organizations (again, there was no sampling).

Description of the Study Sample

Exhibit 2 shows the total number of organizations included in the evaluation. The sample sizes for the partners are based on baseline data. We anticipate that some additional partner organizations may have joined the partnerships and some may have left the partnership. Therefore, the partner sample size could be different based on changes in the partnerships since the baseline collection.

Exhibit Description of Study Sample
Data Sources	Sample Sizea	Universe of Organizations
2006 CEY lead organizations	50	100
2006 CEY FBCO partners	328	800
2007 CEY lead organizations	31	31
2007 CEY FBCO partners	134	134
Total	543	1270a
a Based on baseline data collection information.

B.2 Information Collection Procedures

After contact information is verified, sample members will be sent an email that contains a unique user name and password to access a secure Website through which they can complete the survey. If access to the Web is not available, a paper copy will be furnished upon request.

B.3 Methods to Maximize Response Rates

Compliance with the CEY evaluation is a requirement of the cooperative agreement awards to CEY grantees. Although some organizations may have major changes in key staff, in general we expect more limited challenges in locating FBCO respondents for the follow-up survey than is typical with individual respondents. Prior to the follow-up surveys, we will contact all sample members to ensure that we have correct contact information for key personnel. The OCS CEY Program Specialists will remind grantees of the importance of their participation in the study during their routine grant monitoring functions. Based on our experience with administration of the baseline survey and the experience in other CCF evaluations, we anticipate achieving a 70- 80% response rate.

Similar to procedures implemented to achieve a high response rate on the baseline survey (100% for lead organizations, 96% for FBCO partners), Abt Associates will establish a help-desk or “Solutions Desk,” comprised of a coordinator and a small team of liaisons who will be responsible for achieving the response rate target for a manageable “caseload” of grantees. The coordinator will be responsible for working with the team of liaisons to update contact information, follow up on nonrespondents, answer questions for prospective respondents, and if needed, administer a phone or hard-copy version of the survey.

B.4 Test of Procedures

The follow-up survey instrument is very similar to the baseline survey. The baseline survey was pilot-tested with three CEY grantees and feedback from respondents was obtained during baseline administration regarding the time required for survey completion and the clarity of the questions.

No pilot-test of the small number of new follow-up questions was conducted. The baseline data collection process, particularly the Solutions Desk liaisons, provided rich information about which survey questions were more difficult to answer. Using this information, we developed the follow-up questions, using questionnaire language and formats that were relatively easier to answer. For example, formats such as “Yes/No,” “Check all that apply,” and Likert scales were relatively easy to answer. The new follow-up questions were developed with these formats. The new follow-up questions were vetted by the Solutions Desk liaisons, as well as ACF and key staff on the CCF Evaluation, CEY Support evaluation, and the CEY Evaluation project.

B.5 Individuals Consulted on Statistical Aspects of Design

The plans for statistical analyses for this study were primarily developed by Abt Associates Inc. The team is led by JoAnn Jastrzab, Project Director; Howard Rolston and Jacob Klerman, co-Project Quality Advisors; and Ryoko Yamaguchi, Deputy Project Director. Contact information for these individuals is provided below.

JoAnn Jastrzab

Abt Associates Inc.

55 Wheeler St.

Cambridge, MA 02138

617-349-2372

Howard Rolston, Ph.D.

Abt Associates Inc.

4550 Montgomery Ave.

Bethesda, MD 20814

301-634-1820

Jacob Alex Klerman

Abt Associates Inc.

55 Wheeler St.

Cambridge, MA 02138

617-349-2372

Ryoko Yamaguchi, Ph.D.

Abt Associates Inc.

4550 Montgomery Avenue
Bethesda, MD 20814-3343

301-634-1778

1 During the kick-off meeting in December 2007, OCS stated an interest in urbanicity. Therefore, rural, urban, or mixed locales were included as part of the systematic probabilistic sampling.

Administration for Children and Families Part B: Collection of Information 6