memo 11-27-07

CEY ICR – ACF Response to OMB comments/questions November 27, 2007

• What % of CEY grantees also received grants under CCF’s other 2 grant programs (the demonstration and the targeted capacity building programs)? If this percentage is large, 1) is there a need to have them respond to multiple similar surveys and 2) how will ACF tease apart the effects of each program?

ACF Response: A total of 37 (28%) of the 2006 or 2007 CEY grantees have previously received a CCF grant (either a Demonstration program grant or a Targeted Capacity Building program grant). Sixteen (16) CEY grantees previously received Demonstration grants and 21 previously received a capacity-building grant. Seven of the 21 prior Targeted Capacity Building grantees received their grants in 2006 or 2007 and were not included in the study of the Targeted Capacity Building program because they were still involved in grant supported activities and could not be expected to report on final achievements. Therefore, a total of 30 (22%) CEY grantees may have responded to a prior survey administered as a part of another evaluation of CCF programs. In general, however, there is limited duplication of questions that these organizations would be asked to answer due to the different natures of the evaluations.

The survey previously sent to CCF Demonstration grantees (intermediaries) focused primarily on the approaches they took in providing intermediary services rather than on questions about the intermediaries’ organizational characteristics and capacity, as is asked in the CEY survey. There are only a few questions related to the organization’s characteristics (e.g., faith-based or secular) and prior experience (e.g., experience providing training, technical assistance, financial assistance) that are very similar to the CEY survey questions.

With regard to the Targeted Capacity Building grantee study, that survey focused primarily on activities and indicators of capacity building achievements rather than on organizational characteristics which are the focus of the CEY survey. There were a very limited number of questions which asked similar information such as organizational age and legal status.

Overall, although just over one-fifth of the CEY grantees may have responded to another CCF evaluation survey, the surveys have little duplication. We believe the small amount of duplication of burden is necessary for the integrity of the CEY Evaluation. For example, the characteristics of some organizations may have changed over time since completion of a prior survey (e.g., legal status). For the CEY evaluation, it is important to have current information, information in a consistent form across respondents, and the ability to attribute the response to the same period of time for all respondents.

With regard to the second part of your question about teasing apart the effects of each program, we will be able to consider the receipt of a prior award under another CCF component as a factor in the analyses for the CEY evaluation. We will attempt to assess whether there are differences in capacity achievements when this factor is considered. While we can compare outcomes for the different groups (those that received a prior award and those that did not), the data will not allow us to analytically assess the effects of the separate programs.

• It is not clear why some of the questions are being asked of only the 2006 grantees. Please provide a justification for each 2006-only question and why it is not appropriate to ask the 2007 cohort these questions.

ACF Response: Because the survey is being administered to the FY 2006 cohort after they have had some months to begin operations, we believe this initial survey presents an opportunity to capture information about some early activities/outcomes, in addition to organizational characteristics. We believe these questions are not appropriate for the FY 2007 grantees as they will have only had their grant for about four months at the time of the survey. We are concerned that asking the 2007 grantees the questions proposed only for the 2006 grantees would suggest to the 2007 grantees that ACF would have expected these kinds of actions/activities within the first few months of the award and may prompt respondents to feel compelled to report meeting these expectations.

Survey questions proposed to be asked only of FY 2006 grantees: 60, 60a, 61, 62, 68, 69, 71, most of 72, and 74 through 84. These questions are designed to capture information about a range of actions/activities that may have been undertaken within an active, operating partnership -- one that has had a number of months to potentially undertake such activities. While it is appropriate, we believe, to ask these questions of the 2006 cohort that has been operational for some months, they are not appropriate to ask of the 2007 grantees in the baseline survey because they are not expected to have yet participated in activities of this nature.

• It is not clear in some of the questions whether the 2006 grantees are being asked to discuss their organization “at or near the beginning of the grant period” (as the instructions on page 1 say), or whether they are being asked to talk about some later period. For example, beginning with question #21, there are a series of questions that ask “in the past 12 months…” Is this supposed to be the past 12 months starting now, or the first 12 months of the grant, or some other time period? Can this be clarified for the respondents in the 2006 cohort?

ACF Response: Upon further review, we agree that the instructions need to be clarified. The references to “at or near the beginning of the grant period” are unnecessary in the introductory text. We believe each survey question as written makes clear whether present circumstances or experience over a prior period is the focus of the question. We propose the changes noted via track changes below to the general overview on the first page.

The Administration for Children and Families, Office of Community Services (OCS) has sponsored a research study of the Compassion Capital Fund Communities Empowering Youth (CEY) program. As part of this study, a series of surveys will be used to gauge the level of organizational capacity growth and community partnership development that is stimulated through participation in the CEY program over the entire grant period. This first survey is designed to provide researchers with important information about your organization. The questions are designed to gather information on characteristics of your organization (e.g., when the organization was formed) and recent experience (e.g., in the past 12 months, whether the organization conducted performance reviews). Organizations initially approved for a CEY grant in Federal Fiscal Year 2006 have some additional questions related to partnership activities and interactions to date and a special “retrospective” section that asks about organizational characteristics at the time of their initial grant award. Your responses to this survey will provide researchers with information about your organization at or near the beginning of CEY grant activities. Please answer questions as honestly as possible. Sample questions and instructions on how to fill out the survey are provided on the following pages. to establish a baseline from which progress may be documented.

Completing this survey is part of the CEY grant requirements. Information obtained through this survey will be used to report about grantees as a group. That is, information about specific organizations and partnerships will not be reported. Federal CEY grant administrators will not view your organization’s survey responses. Other members of your partnership, including your designated lead organization, will not view your survey responses. Responses will be accessed only by staff at the research firm that is conducting the evaluation of the CEY program for OCS. All answers will be kept confidential to the best of the research contractor’s ability, as allowed by law.

As mentioned above, this study will utilize a series of surveys. Additional surveys will be sent to your organization to obtain updated information in the future. As mentioned above, this survey involves collecting baseline information at or near the outset of your CEY project implementation. Additional surveys will be sent to your organization to obtain updated information in the future. Your cooperation in completing this survey and future surveysreturning it by the date requested is appreciated.

• Why are the “how to fill out the survey” instructions provided at the end of the survey rather than at the beginning? Is there a way to make it “look different” so that respondents know that these are instructions and “mock questions” rather than actual questions?

ACF Response: We agree that it would be more effective and clearer to respondents to include the “how-to” instructions at the beginning of the survey. We expect that the majority of respondents will complete the survey via the website. We will ensure that the instructions and sample questions are in a module that is clearly identified as instructional with sample questions as examples. For the hard copy version, we will add a “sample” watermark over the questions to make clear that these are sample questions for illustration purposes, as shown in the attached.

• It might be worth reminding the 2006 cohort that they are being asked to talk about their organization at or near the beginning of the grant period. At the moment, this is stated only on page 1 of the survey: it might be worth repeating it throughout the survey.

ACF Response: Only the questions in the “Retrospective Glance” ask the 2006 cohort to base their responses on the period at or near the beginning of the grant period. The remaining questions in the survey are to be answered similarly by 2007 and 2006 grantees. The questions either ask about present time (e.g., do you keep records on program participants and services) or specify the time period (e.g., in the past 12 months, has your organization assessed its organizational needs/strengths). We have changed the introductory text (as shown in response to the third question, above) and removed reference to “at or near the beginning of the grant period.” We believe this change will allow each question to stand on its own in relation to the time frame covered by the question or set of questions.

• Is there some reason why this survey was not designed to be implemented earlier, so that the 2006 cohort could’ve filled out the baseline survey at or near the beginning of the grant period? This seems to be a repeating problem with all CCF surveys so far. It wouldn’t be such a problem if the first cohort were not critical to survey, but in this case, the 2006 cohort is the larger cohort and therefore more critical than the 2007 cohort.

ACF Response: The CEY grant program was created within a short time frame, following the announcement of the First Lady’s Helping America’s Youth (HAY) initiative. It is common to allow a new initiative to “settle in” a bit before an evaluation is conducted. The CCF CEY Program was launched in 2006. There were lessons learned through the initial grant award process that were incorporated into the 2007 process that ACF believes improved the grant program. Because, as you note, the 2006 cohort is large and represents a substantial Federal investment, we believed it was important to include a substantial number of the initial 2006 grantees in the evaluation, rather than limit it to the smaller 2007 cohort. The evaluation design, which includes a randomly selected sample of one half of the 2006 cohort in addition to all of the 2007 grantees, will provide a valid assessment of the accomplishments of both the 2006 and the 2007 cohorts.