Att_1875-NEW 4807 OMB SS B

B. Collection of Information Employing Statistical Methods

B.1. Respondent Universe and Sample Size

Through the use of internet research, examination of the federal 21^st CCLC Profile and Performance Information Collection System (PPICS) database, which annually collects data on the administration and outcomes of all 21^st CCLC programs, as well as interviews with federal officials and discussions with the Contracting Officer’s Representative, the research team will select sites in which to conduct in-depth case studies. A purposive sampling approach will be used to develop a matrix of case study options of up to 15 states and 15 corresponding

districts for ED to consider, along with additional data that illustrate the rationale for their selection and other important state and district characteristics. The sample will not be nationally representative; however, a cross section of states and districts will be selected that reflect the diversity of key state and grantee characteristics.

At the state level, the research team will select a distribution of candidate states that reflect differences with respect to each of the following: type of priorities set for grant awards; extent to which regular grant competitions are held; extent to which new programs/grantees are funded with each competition; and the selectivity of the grant competitions. In addition, selected sites will reflect the variation in states’ demographic characteristics, including size—as measured by both their 21^st CCLC funding allocation as well as their number of PK-12 students enrolled in schools. Further, the research team will include states that reflect the geographic diversity of the U.S. in order to capture variation in state competitions that may emerge as a result of the relative concentration—or spread—of state populations. In particular, to maximize the utility of the case studies, the research team will target states that appear to be managing their 21^st CCLC grant competitions using multiple processes and tools and that are operating within a variety of local program contexts.

At the grantee level, the research team will select a study sample that reflects variation in type and size, number of centers, number of students served, number of years receiving 21^st CCLC grant awards, and services provided. According to the PPICS database, 21^st CCLC grantees include school districts (66 percent), community-based organizations (16 percent), and nationally affiliated nonprofit agencies (4 percent), as well as other organization types (14 percent). Grantees receive awards of different durations and amounts, offer different types and amounts of services, serve different populations, and work with varying types of service providers; the grantee sample will reflect this diversity.

The research team will deliver a draft list of sites from this pool to ED for review. The list will be finalized after receiving input and approval from ED. The research team will then select a sample of up to nine states and nine districts for the study based on how well they meet the selection criteria and represent the current diversity of states and grantees. Once PPSS has approved the list of sites to be considered for the study and OMB has approved the case study data collection plan, the research team will send each 21^st CCLC State Director a letter inviting their participation in the study. These letters will explain the study and expectations for state participation. (Appendix A includes a draft of the recruitment letter.)

B.2. Procedures for Collection of Information

All PSA and AIR data collection staff will participate in training before conducting site visits to ensure familiarity with and understanding of the purpose of the study, content of the protocols, site visit procedures, and write-up procedures. The research team will be trained in what to look for on-site, and in how to motivate respondents to provide accurate and complete information during interviews, including strategies for providing effective follow-up questions, ensuring a natural conversation flow, and avoiding leading questions. The research team will also acquaint all site visitors with all the data collection forms and write-up responsibilities. Important procedural issues to be addressed during training will include all aspects of site recruitment, guidelines for ensuring privacy during interviews, guidelines for ensuring high-quality interview notes, and any necessary follow-up communications. Two researchers will visit each site; the senior researcher will conduct interviews and focus groups, while the junior researcher will help manage site visit logistics, including audio recordings and notes. Researchers will interview up to 153 individuals, as shown in Exhibit 2.

Exhibit 2

Expected Number of Respondents, by Role

Role	Number of Respondents
SEA 21st CCLC program staff (e.g., SEA liaison, other 21st CCLC staff, administrative consultant(s), budget/finance director)	27-54
SEA administrators of other federal discretionary grants programs (e.g., Improving Teacher Quality State Grants, School Improvement Grants, and Math and Science Partnership Grants)	18-27
Peer review team members	9-27
Local project directors	18-45

The specific number and role of respondents will vary, depending on the organizational structure of the departments within the SEA that administer 21^st CCLCs. In all cases, personnel who have specific responsibilities in administering the system and/or system operations will be included as respondents. We expect an 80-90% response rate per state.

Each individual interview will last approximately 45-60 minutes. (Appendix B contains interview protocols for state and local administrators and for peer review team members.) The research team will audio-record and take notes during each interview to ensure an accurate record.

Once final approval for the case study sites is received, the research team will mail recruitment letters to the appropriate state and local administrators in the study sample and begin scheduling the state- and local-level interviews. In advance of contacting state and local administrators, the research team will first ensure that they understand the purpose of the study and the research questions that the study will address, and that they are able to explain the study’s data collection needs.

1. Statistical Methodology

This study involves collection of qualitative data. A discussion of statistical methodology is not applicable to this study.

2. Analysis Methods

The use of qualitative methods will allow the study team to probe deeply into the issues surrounding implementation of the 21^st CCLC program. By synthesizing evidence from interviews and document reviews, the research team will have multiple sources and perspectives from which to gain an in-depth understanding of the context and circumstances surrounding the successes and challenges of implementing 21^st CCLC grants competitions. The study findings will provide rich descriptions of case studies that can help inform future guidance and support for 21^st CCLC program implementation, as well as for other Federal competitive grants programs.

Qualitative data collection allows respondents to describe actions, processes, and relationships in their own words. Qualitative analyses also rely, in part, on the analyst’s judgment and interpretation of responses. Thus, the study’s qualitative analytic procedures will establish and adhere to a set of standards to limit bias and ensure reliable findings. These include standards of evidence, triangulation of data, and procedures for measuring inter-rater agreement. For interview data, site visitors will begin compiling and synthesizing data immediately upon completion of each site visit, under the guidance of lead analysts. Data for each site will be compiled into a standardized write-up template, which will be used to capture verbatim quotes that address the evaluation questions and provide space for site visitors to summarize key themes. All interview data will be entered into a Microsoft Access database to facilitate analysis and consistency of entry.

Concise preliminary summaries of interviews will be completed—approximately two pages entered into a standard data capture template—with a bulleted list of key themes from each interview. Lead analysts will review these write-ups in detail, noting text that needs clarification or elaboration. A debriefing will take place when the site visits are completed and will begin the process of identifying findings. Following the debriefing, the lead analyst will review the site visit data and begin mapping verbatim quotes onto study themes. Analysts will identify, test, and refine tentative generalizations; at the same time, for every generalization that helps to characterize a central tendency among respondents, there will also be important exceptions that warrant explication. Finally, cross-site analyses will compare, contrast, and synthesize findings and propositions from the individual cases to triangulate data and make statements about the sample or segments of the sample (e.g., heavily or sparsely populated states).

3. Degree of Accuracy Needed

The research team will maximize the accuracy of the data collected for each of the case studies. First, prior to data collection, the research team will provide a thorough orientation to site visit team members to familiarize them with the overall study design and their responsibilities for data collection and case write-ups. This orientation will include special attention to the interview protocols and to site visitor responsibilities and strategies for tailoring and conducting individual interviews. Second, interviewers will take detailed notes during each interview, and all interviews will be recorded to ensure the accuracy of interview notes. Third, the research team will develop a telephone interview protocol, which will include the questions used for the site visit interviews, but will be scripted to simulate the tone and flow of an in-person conversation. Simulating an in-person conversation will be important for purposes of generating similar levels of candor and follow-up among telephone respondents. In addition, because the telephone interviews will be conducted after the site visits, the research team may want to use them as an opportunity to check assumptions about some of the information collected on site.

4. Use of Periodic Data Collection

Data collection will occur only one time.

B.3. Methods for Maximizing Response Rate and Dealing with Nonresponse

To solicit participation from selected 21^st CCLC state administrators and grantees, the research team will prepare notification letters and information packets about the study for the relevant state administrators, including the chief state school officer and the state director of 21^st CCLC programs, in all sampled states. In addition, notification letters for the district superintendents or project directors of all the local sub-grantee programs will be prepared. The letters will include (a) a study description with a discussion of its importance, purposes, and products; (b) information on the data collection schedule and plans; (c) provisions for maintaining anonymity of participants and data security; (d) the organizations and staff involved in the study; and (e) the benefits to be derived from the study. The letters will also explain that OMB clearance has been secured. Finally, the letters will include the names and phone numbers of ED staff and PSA employees who are available to answer questions about the study. (Appendix A includes draft notification letters). Letters to respondents will be mailed in advance of data collection once OMB clearance has been secured. Also, to maximize response rate, the research team will conduct interviews by telephone in cases where selected respondents are unavailable to schedule a meeting during the site visit or become unavailable on short notice.

B.4. Test of Procedures and Methods

The research team will pre-test each interview protocol with a few respondents appropriate to that protocol: SEA program directors, peer reviewers, and local project directors. No more than three individuals in roles similar to those included in the sample will be asked to pre-test the protocols. The research team will then use the pre-test results to refine the interview protocols.

B.5. Consultations on Statistical Aspects of the Design

Members of the research team who will be responsible for data collection and analysis are listed in Exhibit 3. As noted above, the evaluation will not require statistical analyses. The listed staff will conduct all planned qualitative analyses.

Exhibit 3

Staff Contact Information

Name	Organization	Title	Telephone
Leslie Anderson	PSA	Project Manager	202-939-5327
Derek Riley	PSA	Project Director II	202-939-5304
Monica Mielke	PSA	Project Director III	202-939-5320
Yvonne Woods	PSA	Researcher I	202-939-5335
Erikson Arcaira	PSA	Researcher I	202-939-5343
Carol McElvain	AIR	Senior Task Partner	312-288-7613

5