National Center on Early Head Start–Child Care Partnerships Evaluation
OMB Information Collection Request
New Collection
Supporting Statement
Part B
June 27, 2016
Submitted by:
Office of Planning, Research and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
7th Floor, West Aerospace Building
370 L’Enfant Promenade, SW
Washington, D.C. 20447
Project Officer:
Andrew Williams
CONTENTS
B1 Respondent universe and sampling methods 1
B2 Procedures for collection of information 2
A. Stakeholder survey 2
B. Stakeholder telephone interviews 2
B3 Methods to maximize response rates and deal with nonresponse 2
A. Expected response rates 2
B. Dealing with nonresponse 2
C. Maximizing response rates 3
B4 Tests of procedures or methods to be undertaken 3
B5 Individuals consulted on statistical aspects and individuals collecting or analyzing data 3
Proposed data collection activities for the study are (1) a web-based survey of 350 key stakeholders targeted by NCEHS-CCP, including regional T/TA specialists, implementation planners and fiscal consultants, Child Care and Development Fund (CCDF) administrators, and Head Start state and national collaboration office directors (“stakeholder survey”), and (2) telephone interviews with 75 purposively selected stakeholders who participated in T/TA offered by NCEHS-CCP (“stakeholder telephone interviews”).
The target population for the stakeholder surveys is the universe of T/TA providers targeted by NCEHS-CCP. NCEHS-CCP provides training, resources, and materials to OHS and OCC federal and national center staffs, regional T/TA specialists, implementation planners and fiscal consultants, CCDF administrators, and Head Start state and national collaboration office directors. We propose surveying regional T/TA specialists, implementation planners and fiscal consultants, CCDF administrators, and Head Start state and national collaboration office directors because we expect variation in the types of T/TA offered by NCEHS-CCP and accessed by stakeholders. Also, given variation in the roles of the stakeholders, we expect variation in how they use the information to inform the T/TA they provide to EHS-CCP grantees. The target population for the stakeholder telephone interviews is a purposively selected subset of the stakeholders who participated in T/TA offered by NCEHS-CCP.
The sampling frame for the stakeholder survey consists of the lists of regional T/TA specialists, implementation planners and fiscal consultants, CCDF administrators, and Head Start state and national collaboration office directors. Attempts will be made to survey all key stakeholders using a web-based survey.
The information from the surveys will be used to purposively select stakeholders who participated in T/TA offered by NCEHS-CCP for the stakeholder telephone interviews. Given that attempts will be made to survey all stakeholders, the sampling frame for the stakeholders selected for interviews corresponds to the lists of regional T/TA specialists, implementation planners and fiscal consultants, CCDF administrators, and Head Start state and national collaboration office directors who completed a survey.
We expect a response rate of 80 percent for the stakeholder survey (280 responses). The sample for the stakeholder telephone interviews will be a quota sample, meaning that we will recruit from stakeholder survey respondents until we obtain 75 respondents.
The instruments included in this OMB package will yield data that will be analyzed using quantitative and qualitative methods. These approaches will allow us to identify and learn about the type and quality of support participants received from NCEHS-CCP.
The stakeholder survey will be web-based. This survey will take, on average, 30 minutes to complete. Some advantages of the web-based survey include coverage (we expect that all stakeholders will have Internet access); convenience (for example, respondents may take the survey at a time suitable for them); and cost-effectiveness. The survey will be conducted in 2016 and repeated in 2018 with stakeholders in the same positions surveyed in 2016.
Stakeholders will be sent an advance email notification inviting them to take part in the study. Over the course of the data collection period, we will send one hard copy reminder letter and up to four email reminders to nonresponders at two-week intervals; we will also make one reminder call at the four-week mark. Supplemental materials for the stakeholder survey are included in Attachment E.
Estimation methods and software for quantitative analyses. The study team will summarize quantitative data from the stakeholder survey using basic descriptive methods. In addition, we will test differences in means or proportions between key groups of interest (for example, differences between regional T/TA specialists, CCDF administrators, and Head Start national and state collaboration office directors).
Data from the stakeholder survey will undergo a common set of steps involving data cleaning, variable construction, computing descriptive statistics, and conducting analyses. To conduct analyses, we will create variables to address the study’s research questions. Construction of these analytic variables will vary depending upon a variable’s purpose and the data source being used. Variables may combine several survey responses into a single composite or scale, may aggregate data from a set time period, or may compare responses to identify a level of agreement. We anticipate using statistical software packages (such as SAS and Stata) to conduct the quantitative analyses.
Sampling weights. Construction of sampling weights will not be necessary for the stakeholder survey because we plan to collect data from all stakeholders targeted for T/TA by NCEHS-CCP as described in Section B1 of the Supporting Statement, Part B.
Nonresponse weights. We will produce analysis weights for the stakeholder survey that account for selection probabilities and for differential nonresponse patterns. These weights will be constructed in such a way as to mitigate the risk for nonresponse bias. Should response rates fall below 80 percent, we will conduct a nonresponse bias analysis in accordance with OMB guidelines.
We plan to conduct 75 telephone interviews in year 3 and 75 in year 5 with purposively selected subsets of stakeholders who participated in T/TA offered by NCEHS-CCP and responded to the stakeholder surveys. Year 5 of data collection is not part of this OMB submission, but we will request an extension prior to summer 2019. The interviews will be semi-structured, one hour in length and will allow the study team to address the research questions in more depth than the web-based survey.
Qualitative analyses. Mathematica will use standard qualitative procedures to analyze and summarize information from the stakeholder telephone interviews which will be conducted by using a topic guide. Analysis will involve organization, coding, triangulation, and theme identification. For each interview, a standardized template will be used to organize and document the information and then code this documentation. Coded text will be searched to gauge consistency and triangulate across respondents. This process will reduce large volumes of qualitative data to a manageable number of topics, themes, and categories (Yin 1994; Coffey et al. 1996) that can then be analyzed to address the study’s research questions.
To code the qualitative data for key subtopics and themes, the evaluation team will first develop a coding scheme based on the interview questions. Senior members of the evaluation team will refine the coding scheme by reviewing codes and a preliminary set of data to make adjustments and ensure alignment with the topics that emerge from the data. For each round of coding, two team members will be trained to code the data using a qualitative analysis software package, such as Atlas.ti or NVivo. To ensure reliability across coders, all team members will code an initial document and compare codes to identify and resolve discrepancies. As coding proceeds, the lead team member will review a sample of coded documents from each coder to monitor reliability. Coded data will enable the team to compare responses across respondents by searching for specific codes. The software will also allow the team to retrieve data on particular codes by type of respondent (such as regional T/TA specialists).
As noted in section B.1, we expect a response rate of 80 percent for the web-based stakeholder survey in 2016 and 2018. This expected response rate is based on the response rates achieved in prior surveys with similar populations.
For the stakeholder telephone interview we will complete 75 interviews in 2017 and an additional 75 interviews in 2019. The interviews to be conducted in 2009 are not part of this OMB submission, but we will request an extension prior to data collection. Stakeholders who choose not to participate in the interview will be replaced with other respondents to meet the quota of 75 respondents.
The web-based stakeholder survey is programmed to skip items that ask questions about T/TA offerings from NCEHS-CCP that the stakeholders did not use. For example, if the respondent did not attend a training webinar, none of the follow-up questions will reference training webinars. Respondents may also indicate that they don’t know. Because the sample is a census, we will not be using sampling weights.
Because the stakeholder telephone interview is a quota sample we will continue recruiting respondents until we have 75 interviews.
To maximize response rates for this information collection, we will take the following steps:
Advance email notifications for the web-based surveys. Stakeholders will be sent an advance email notification inviting them to take part in the study. The advance email includes a brief overview of the purpose of the study, a description of the data collection activity in which we are asking them to participate, and an estimate of the amount of time required to complete the activity. The email will also include a link to the survey and log-in credentials. Respondents will also be provided with a number they can call should they have any questions about their participation in the study. Please see Attachment E.
Reminder notifications for the web-based surveys. Over the course of the data collection period, we will send one hard copy reminder letter and up to four email reminders to nonresponders at two-week intervals; we will also make one reminder call at the four-week mark. Please see Attachment E.
Trained and experienced data collection staff for the stakeholder telephone interviews. Interviews will be conducted by project staff who have extensive knowledge of the project, Early Head Start and child care, and NCEHS-CCP.
The items in the web-based survey have undergone several revisions by members of the NCEHS-CCP team. The survey was pre-tested with seven respondents, per OMB regulations. By pre-testing the survey with relevant stakeholders, we found the instrument is easily understandable and captures the range and depth of the T/TA provided by NCEHS-CCP and received by stakeholders. We also decided not to survey OHS and OCC federal and national center staff because cognitive interviews with these stakeholders revealed they do not feel as though they are the target audience for NCEHS-CCP. Once programmed, the survey will undergo extensive testing of all skip patterns and scenarios.
Angie Godfrey and Shireen Riley, OHS and OCC project officers, respectively; Rae Anderson, NCEHS-CCP project director; Patricia Del Grosso, project director for the NCEHS-CCP evaluation; and Cassandra Meagher, survey director for the NCEHS-CCP evaluation, lead the team. Additional staff contributed to the design of the study, including Scilla Albanese and Cheri Vogel.
Inquiries regarding the statistical aspects of the study’s planned analysis should be directed to the following individuals:
Patricia Del Grosso, project director for the NCEHS-CCP evaluation
Cassandra Meagher, survey director for the NCEHS-CCP evaluation
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Supporting Statement B |
| Author | JThomas |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |