Volume I:
Request for Clearance for the Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011) Spring First Grade Cognitive Interviews
OMB# 1850-0803 v.43
January 31, 2011
The Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011) is a survey that focuses on children’s early school experiences beginning with kindergarten and continuing through the fifth grade. It includes the collection of data from parents, teachers, school administrators, and nonparental care providers, as well as direct child assessments. Like its sister study, the Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K),1 the ECLS-K:2011 is exceptionally broad in its scope and coverage of child development, early learning, and school progress, drawing together information from multiple sources to provide rich data about the population of children who are kindergartners in the 2010-11 school year. As with the original ECLS-K, the ECLS-K:2011 is sponsored by the National Center for Education Statistics (NCES) within the Institute of Education Sciences (IES) of the U.S. Department of Education (ED). Fall and spring collections in the kindergarten year are being conducted for NCES by Westat, with the Educational Testing Service (ETS) as the subcontractor developing the child assessments. Clearances for studying the first ECLS-K cohort were granted in 1996 for the kindergarten data collection (OMB No. 1850-0719), in 1998 for the first grade to fifth grade data collections (OMB No. 1850-0750), and in 2005 for the spring 2006 field test data collection (conducted with eighth and tenth grade students and their teachers) and the spring 2007 national data collection (conducted with eighth graders) (OMB No. 1850-0750). Clearances for studying the ECLS-K:2011 cohort were granted for the fall 2009 field test data collection and fall 2010 and spring 2011 kindergarten national data collections (OMB No. 1850-0750).
ECLS-K:2011 is the third in an important series of longitudinal studies of young children sponsored by the U.S. Department of Education that examine child development, school readiness, and early school experiences. It shares many of the same goals as its predecessors, the ECLS-K and the Early Childhood Longitudinal Study, Birth Cohort (ECLS-B), but also advances research possibilities by providing updated information and addressing recent changes in education policy.
Like its predecessors, ECLS-K:2011 will provide a rich and comprehensive source of information on children’s early learning and development, transitions into kindergarten and beyond, and progress through school for a new cohort of children.
ECLS-K:2011 will provide data relevant to emerging policy-related domains not measured fully in previous studies.
Coming more than a decade after the inception of the ECLS-K, ECLS-K:2011 will allow cross-cohort comparisons of two nationally representative kindergarten classes experiencing different policy, educational, and demographic environments.
This is a request for clearance (under OMB# 1850-0803) to conduct cognitive interviews to inform the procedures and instruments planned for the ECLS-K:2011 spring first grade data collection. Specifically, the goals of these cognitive interviews are to identify and correct problems with ambiguity in or misunderstanding of new and revised items for the spring first-grade teacher and school administrator questionnaires, to examine the extent to which principals or other school administrators2 perceive that their burden would be reduced by using questionnaires prefilled with some of the information that was obtained in earlier rounds of data collection, and to determine whether some of the items could or should be asked on a biennial basis rather than annually.
NCES has contracted Westat to conduct the spring first grade data collection for ECLS-K:2011, including cognitive interviews.
Cognitive interviews will be conducted to evaluate new and revised items (appendix A) in the ECLS-K:2011 spring first-grade teacher and school administrator questionnaires. Most items from the questionnaires have been fielded over multiple rounds of the ECLS-K and, therefore, will not be tested in these cognitive interviews. However, respondents will be asked to briefly review the full questionnaire so they understand the general context in which the tested questions are asked. If time allows, at the end of the interview, they will also be asked if they have any other comments on the items that were not discussed or on the questionnaire in general. The new items in the questionnaires are intended to address issues that have recently been identified as important areas for research. Items that have been revised were found to have relatively low item response rates in the ECLS-K data collections or used wording that was not as clear as it could be, and revisions are intended to increase the clarity of the items and the rate at which respondents answer them. In general, participating teachers and principals will be asked to discuss their interpretation of items, to discuss the process they would use to obtain an accurate response, and to provide suggestions for any unclear or difficult to answer items.
School principals will also be asked about their perceptions regarding questionnaires that are prefilled with information about themselves and their schools that was collected in a prior round of data collection. Because the ECLS‑K:2011 is a longitudinal study, school principals respond to a series of similar school-level questions every spring. Feedback from school principals in the ECLS‑K indicated that some of the data do not change for their schools from year to year. As part of the cognitive interviews, school principals will be asked if prefilling some of the school information that was collected as part of the previous data collection round would make the current questionnaire easier for them to complete, as well as whether they would have any objections or concerns about this procedure.
The request to conduct the national spring first grade data collection will be submitted at a later date under OMB clearance for ECLS-K:2011 (OMB # 1850-0750).
The cognitive interviews will be conducted as one-on-one interviews between a responding teacher or school principal and an experienced qualitative interviewer. Cognitive interviews are intensive, one-on-one interviews in which the respondent is asked to “think aloud” as he or she answers survey questions and to answer a series of questions about the items he or she just answered. This approach includes asking probing questions, as necessary, to clarify points that are raised in the think-aloud comments.
Cognitive interviews will be conducted with 15 first grade teachers and 15 elementary school principals. To gather information about items related to Adequate Yearly Progress (AYP), we will attempt to recruit at least three principals from schools that were identified through information available through state department of education websites as “in need of improvement” in 2009-2010. Also, using the Common Core of Data (CCD), we will prioritize schools in which 10 percent or more of the students at the school were eligible for free or reduced price lunch in 2008-2009 in order to locate and recruit at least three principals from schools that are likely to currently participate in the school lunch program. This is a purposive sample, however we will recruit teachers and principals from among elementary schools with various characteristics, e.g., from public (including charter) and private schools, and rural and urban schools located throughout the country.
We will recruit among the 59 schools that participated in the 2009 ECLS-K:2011 Kindergarten Field Test (OMB # 1850-0750) and with whom Westat has a good working relationship because of this prior participation.3 The schools in the 2009 field test are in nine states in different regions of the country (i.e., northeast, southeast, central, and west), and include nine private schools, three charter schools, and 12 schools that were “in need of improvement” in the 2009-10 school year as defined by No Child Left Behind requirements. During recruitment, we will prioritize schools that have been identified as in need of improvement and as likely participating in the school lunch program. We will contact schools, beginning with the schools that have those characteristics, until we reach the target number of principals and teachers. To recruit respondents through these schools, we will contact the person who was designated as the school coordinator for the 2009 field test. The school coordinator will be asked to identify two first-grade teachers who might be willing to participate and to provide the contact information for those teachers and for the principal. We will only recruit one teacher per school, but we will obtain contact information for two teachers in the event that we are unable to reach one of the teachers or one of the teachers declines participation. In the event that the school coordinator from the 2009 field test is not available, we will contact the school principal to provide the contact information for two first-grade teachers.
In addition to recruiting from schools that participated in the 2009 field test, we will also recruit some elementary school principals and first-grade teachers through a network of contacts that Westat has with teachers and principals as well as through advertisements placed on listservs that teachers and principals may subscribe to (e.g., teacher association listservs, university education department listservs). Telephone interviews will be conducted with respondents recruited through both the 2009 field test schools and Westat’s network of contacts and listservs. Recruitment through the 2009 field test sample and through these other methods will occur simultaneously. Teachers and administrators will complete the interviews on their own time (i.e., outside of their normal work/school day).
We anticipate needing to make minor wording changes to the new and revised questions following the first round of cognitive interviews. A second round of cognitive interviews with 10 teachers and 10 principals who did not participate in the first round of cognitive interviews will be conducted via telephone to ensure that the revised questions are interpreted as intended and are not overly burdensome. The same procedures used for recruitment in the initial round of interviews will be used in the second round of recruitment.
Interviewers for the cognitive interviews for round 1 and round 2 will follow a prewritten protocol (appendix B) but will be free to deviate from the protocol in order to address specific issues or anomalies in the respondents’ verbal reports. After speaking with study staff and agreeing to participate in a cognitive interview, respondents will be mailed a letter explaining the purpose of the cognitive interviews (appendix D) and a copy of the questionnaire to review prior to the cognitive interview (appendices E and F). We will instruct respondents to take 5 minutes to briefly review the entire questionnaire prior to the cognitive interview. While the interviews will only focus on selected items, reviewing the entire questionnaire will provide some additional context for the respondent. The full teacher questionnaire covers a wide range of topics including classroom and teacher characteristics, class organization and resources, instructional activities and curricular focus, parental involvement, evaluation and grading practices, school and staff activities, school climate, and school environment and teacher background. The full school administrator questionnaire includes questions about school characteristics; school facilities and resources; school-family community connections; school policies and practices; school programs for particular populations; federal programs including Title I, Adequate Yearly Progress, and Title III; staffing and teacher characteristics; and finally school administrator characteristics.
In the cognitive interviews with teachers, focal items come from the sections on classroom characteristics, class organization and resources, and instructional activities and curricular focus. In the cognitive interview with principals, focal items come from each of the topic sections listed above. The interview will focus on questions about participation in the national school lunch program, questions about changes the school has seen or experienced as a result of current economic conditions, issues related to Adequate Yearly Progress, and staffing. We will also ask principals how stable their answers would be from year to year to determine if we could reduce burden by prefilling questions with answers provided in the previous data collection round. Topics for this line of questioning include items about school facilities and resources, school policies and practices, school programs for particular populations, and school administrator characteristics. The second round of interviews will focus on the items that have been modified based on round 1 results and we believe need retesting. The prewritten protocol for these items may change slightly in response to issues discovered during the first round of cognitive interviews.
Cognitive interviews with teachers are expected to last about an hour on average. Because there are more items for school administrators to review than for teachers, we anticipate that the school administrator cognitive interviews will take longer than the teacher cognitive interview, about an hour and 15 minutes for the full interview on average. Thus, the total burden per cognitive interview, including 5 minutes to review the questionnaire, is anticipated to be about 65 minutes for teachers and 80 minutes for school administrators.
Based on current school response rates from the fall 2010 kindergarten collection of the ECLS-K:2011, for the first and second rounds of cognitive interviewing, we anticipate a response rate of about 59 percent. Thus we anticipate contacting about 43 teachers to produce 25 responding teachers and about 43 principals to produce 25 responding principals. In total, an estimated 86 teachers and principals will be contacted. Information to contact teachers and principals recruited for telephone interviews will be gathered through a school coordinator. We anticipate contacting about 39 school coordinators. During recruitment, a recruiter will explain the purpose and length of the cognitive interview and ask the respondent if he or she would like to participate. The recruitment burden is anticipated to be about 5 minutes on average (see appendix C for recruitment script).
In preparation for the spring first and second grade data collections, a Technical Review Panel (TRP) and Content Review Panels (CRP) are being assembled for March 2011 to review and comment on issues related to the instruments. The members of these panels include experts in child development, teacher education and classroom activities, education policy, special education, bilingual education, and the learning environment. During the upcoming TRP and CRP meetings, experts may recommend adding new topics to the cognitive interviews. Should the panels suggest adding new content that needs to be included in these cognitive interviews, an amendment to this package will be filed.
To attract teachers and school principals to participate in the cognitive interviews and to thank them for their time and effort, participating teachers will receive $35 and administrators will receive $40.
Respondents who participate in cognitive interviews will largely be recruited through schools in which Westat has an established relationship. This includes schools that are local and those that are located throughout the country and thus will provide some diversity in respondents. Most of these schools participated in the 2009 ECLS-K:2011 Kindergarten Field Test.
In addition to recruiting volunteers through Westat’s network of contacts, advertisements will be placed on listservs geared toward teachers and school administrators.
Respondents will be informed that their participation is voluntary at the beginning of the cognitive interview. Oral consent will be obtained from all respondents at the time of the interview. The script for oral consent can be found in appendix G. No personally identifiable information will be maintained after the interview analyses are completed. With respondent permission, interviews will be audio recorded for later analysis. If the respondent indicates that he/she does not want to be audio recorded, only written notes will be taken. The recordings and notes will be destroyed at the conclusion of the ECLS-K:2011.
We anticipate contacting approximately 43 teachers and 43 principals during recruitment for cognitive interviews. As noted above, information to contact teachers and principals recruited for telephone interviews will be gathered through a school coordinator, and we anticipate contacting about 39 school coordinators. The expected burden for recruitment per person (including teachers, principals, and school coordinators) is 5 minutes on average for a total of about 10.4 burden hours. Cognitive interviews will be conducted with 25 teachers and 25 principals across two rounds of interviewing. Including the time to review the instrument prior to the interview, response burden for teachers is expected to be about 65 minutes and response burden for administrators is about 80 minutes for a total of about 60.4 burden hours. Total burden hours, including recruitment, time to review the questionnaire, and cognitive interviews is 70.8 hours.
There is no direct cost to the respondent.
The cost to the government to conduct the cognitive interviews is $95,556.
Recruitment is expected to begin in March, 2011, and the cognitive interviews will be completed by early May, 2011. The results of these cognitive interviews will be initially discussed via memorandum with NCES immediately following the completion of each round of the cognitive interviews, then a report summarizing the findings will be finalized in November, 2011.
1Throughout this package, reference is made to the Early Childhood Longitudinal Study, Kindergarten Class of 1998-99. For ease of presentation, it is referred to as the ECLS-K. The new study for which this submission requests approval is referred to as the ECLS-K:2011.
2 For ease of presentation, school administrators are referred to as principals throughout this package.
3 To avoid placing undue burden on schools or affecting response for either this cognitive laboratory work or other NCES studies, we will not contact 2009 field test schools that are part of a sample in other NCES studies that will be in the field at the same time the cognitive laboratory work is being conducted.
| File Type | application/msword |
| File Title | Volume I: |
| File Modified | 2011-03-09 |
| File Created | 2011-03-09 |