Part A IELS 2018 Main Study

The International Early Learning Study (IELS) main study, scheduled to be conducted in 2018, is a new study sponsored by the Organization for Economic Cooperation and Development (OECD), an intergovernmental organization of industrialized countries. The OECD also coordinates the Program for International Student Assessment (PISA), an assessment of 15-year-olds, and the Program for the International Assessment of Adult Competencies (PIAAC), which assesses adult skills. While PIAAC focuses on adults and PISA on students nearing the end of compulsory school, the IELS focuses on young children and their cognitive and non-cognitive skills and competencies as they transition to primary school. The IELS is designed to examine:

Children’s early learning and development in a broad range of domains, including social and cognitive skills;
The relationship between children’s early learning and children’s participation in early childhood education and care (ECEC);
The role of contextual factors, including children’s individual characteristics and their home backgrounds and experiences, in promoting young children’s growth and development; and
How early learning varies across and within countries prior to beginning, or in the early stages, of primary school.

In 2018, in the participating countries, including the United States, the IELS will assess nationally-representative samples of 5-year-old children enrolled in public and private schools that offer kindergarten, through direct and indirect measures, and will collect contextual data about the children’s home learning environments, ECEC histories, and demographic characteristics. The IELS will measure young children’s knowledge, skills, and competencies in both cognitive and non-cognitive domains, including language and literacy, mathematics and numeracy, executive function/self-regulation, and social emotional skills. This assessment will take place as children are transitioning to primary school and will provide data on how U.S. children entering kindergarten compare with their international peers on skills deemed important for later success¹.

In the United States, IELS is conducted by the National Center for Education Statistics (NCES), part of the U.S. Department of Education’s Institute of Education Sciences (IES). NCES, with the support of IELS 2018 U.S. data collection contractor, Westat, ensures proper implementation of the study and adoption of practices in adherence to the OECD‘s standards. The IELS is a collaboration among the participating countries, the OECD, and an international contractor which coordinates the study across countries.

To prepare for the main study, which will be conducted from October to December 2018, the IELS countries conducted a field test in the fall of 2017 to evaluate newly developed assessment instruments and questionnaires and also to test the study operations. The U.S. IELS field test data collection occurred from November 6 to December 20, 2017. In addition, main study recruitment activities with states and districts began in September 2017 to allow enough time to gain cooperation prior to contacting schools in early winter 2018. Because the IELS is a collaborative effort among many parties, the U.S. must adhere to the international schedule set forth by the OECD, including the availability of the final main study plans and questionnaires.

In September 2017, OMB approved the initial phase of this collection as part of request to conduct the 2017 IELS field test data collection and recruitment for the IELS 2018 main study (OMB# 1850-0936 v.3-4). With those submissions, NCES adequately justified the need for and overall practical utility of the main study as proposed, and an overarching plan for the phases of the data collection over the next 3 years, and provided as much detail on the measures to be used as was available at the time of the submission. Thus OMB approved all aspects of the initial phase of this collection, and now NCES published a notice in the Federal Register allowing a 30-day public comment period on the details of the subsequent phase of the IELS 2018 main study collection described in this submission. This request is to conduct the IELS 2018 main study. The IELS main study procedures and materials are based on those used in the field test and are provided in this submission. Appendices A-C and E provide all recruitment, consent, and administrative data collection materials, including the content of MyIELS portal. The final national versions of parent and teacher questionnaires are provided in Appendix D.

A. JUSTIFICATION

A.1 Importance of Information

As part of a continuing cycle of international studies, the United States, through NCES, participates in several international education assessments and surveys, including PISA, an assessment of 15-year-olds, and PIAAC, which assesses adult skills. IELS, sponsored by OECD, is a new addition to NCES’s international studies.

In light of the growing concerns related to international economic competitiveness, the changing face of our workplace, and the expanding international marketplace in which we trade, knowing how our students and adults compare with their peers around the world has become an even more prominent issue than ever before. Nationwide, interest in understanding what other nations are doing to further the educational achievement of their populations has increased beyond simple comparisons. Moreover, recent investments in early childhood education and care (ECEC) require a closer examination of what US children know as they enter school and how their knowledge compares to that of children in other nations with different ECEC experiences.

Data at critical points during the education career of U.S. students are used by policymakers in efforts to guide and examine the American education system. Consequently, generating comparative data about students in school and about adults in workplace and in community has become an important focus for NCES. The addition of the IELS to the slate of comparative assessments conducted by NCES will broaden what we know about U.S. students compared with those in other countries and provide comparative information about students at an earlier point in the educational process than has previously been possible.

The results from the IELS will allow national policymakers to compare the skills and competencies of children from different nations as they are readying to enter primary school and to evaluate whether changes might be needed in the U.S.’s ECEC system to promote additional skills and competencies seen in other countries. Without these types of data, U.S. policymakers are limited in their ability to gain insight into the educational performance and practices of other nations as they compare to the United States.

A.2 Purposes and Uses of Data

Overview of the IELS Study Design

The IELS 2018 will collect data on a nationally representative sample that will yield approximately 3,000 assessed five-year-old children^² enrolled in 200 schools through the United States. As in the 2017 field test, the data collection for the main study will include direct child assessments and online parent and teacher questionnaires. Instruments have been developed based on frameworks that define the constructs and provide specifications for distributions of items by framework dimensions. In order to develop the main study instruments, the field test cognitive and non-cognitive items were evaluated for bias and interpretation issues, following standard protocols. For the main study, the pool of items was reduced to only include those that demonstrate validity across the participating education systems, meet the goals of content coverage to adequately measure the frameworks, and provide the desired distribution of item types.

Cognitive Assessments. The IELS assessment will be comprised of four domains: emerging literacy, emerging numeracy, empathy and trust, and self-regulation. Each assessment will be monitored by a trained assessor, using a tablet. Each item will be read aloud to the child, using the microphone feature of the tablet. Headphones will be available for children to use in cases where there is a lot of distraction or noise in the assessment space. The child will be asked to respond to questions in a variety of methods, including tapping on a picture or dragging and dropping items. After a brief introductory session where the assessor demonstrates to the child how to use the tablet to respond to questions and the child answers a few practice items, the assessment will begin. The play-based assessment is untimed and self-administered, although the assessor will be available to respond to questions, demonstrate how to answer items if needed, and intervene if technical issues with the hardware or software arise. The assessment is expected to last approximately 60 minutes, although it will be broken into two sessions spread across two days. That is, each child will complete on average a 30-minute session on day 1 and another 30-minute session on day 2.

Parent questionnaire. One parent or guardian of each student will be asked to complete a brief online survey. If preferred, a paper-and-pencil version can be requested. The questionnaire asks demographic questions as well as questions about the child’s early learning environment and experiences, social skills, and participation in child care. The English questionnaire are also available in Spanish (see Appendix D).

Teacher questionnaire. Teachers of selected students will be asked to complete an online questionnaire, which includes questions about their professional background (about 3 minutes to complete) and the skills and abilities of the students selected for the study (about 5 minutes per child, with an average of 4 students per teacher). The survey does not need to be completed in a single session. Paper-and-pencil versions of the surveys will be available upon request (see Appendix D).

Governments and the general public want solid evidence of education outcomes. In the late 1990s, the OECD launched an extensive program for producing policy-oriented and internationally comparable indicators of student achievement on a regular basis and in a timely manner. The IELS is a new addition to this program.

Purposes of Data

The IELS will assess children as they are transitioning to primary school and, as such, will provide a view of how 5-year-old children in the U.S. compare with their international peers on the skills and competencies deemed important for success in school. The results will allow national policymakers to evaluate the readiness of U.S. children for formal schooling, and to analyze the relationship between constructs measured through the IELS adult questionnaires with child assessment results at national and international levels. Through the IELS, the OECD and NCES will produce two types of indicators:

Basic indicators that provide a baseline profile of the knowledge, skills, and competencies of young children; and
Contextual indicators that show how such skills relate to important demographic, social, economic, and education variables.

A.3 Improved Information Technology (Reduction of Burden)

Where feasible, available technology will be used to reduce burden and improve efficiency and accuracy. The burden of recruitment on districts and schools will be minimal, with most information gathered through E-filing (see below). Districts will primarily be asked to provide confirmation of data gathered from other sources, including school universe files and district and school websites.

A communication website, MyIELS.us, will be used during the main study in order to provide a simple, single source of information to engage sampled schools and maintain high levels of their involvement. This portal, used successfully during the field test, will be used again throughout the main study assessment cycle to inform schools, particularly school coordinators, of their tasks and to provide them with easy access to information tailored for their anticipated needs. We plan to gather student and teacher lists from participating schools electronically using an adaptation of a secure E-filing process through the web portal. E-filing is an electronic system for submitting lists of student and teacher information, including limited background information in school records. E-filing has been used successfully in the National Assessment of Educational Progress (NAEP) program for more than 10 years, and was used in the 2015 Trends in International Mathematics and Science Study (TIMSS 2015) and the PISA 2012 and 2015 assessments. The E-filing system provides advantageous features such as efficiency and data quality checks.

The IELS design and procedures are prescribed internationally. The data collected by direct, one-on-one child assessments will be captured electronically on tablet. Trained assessors will facilitate the assessment. Teacher and Parent Questionnaires will be made available as online questionnaires and in paper format.

A.4 Efforts to Identify Duplication

A number of international comparative studies exist to measure student achievement, including TIMSS, the Progress in International Reading Literacy Study (PIRLS), and PISA. In addition, the U.S. has been conducting its own national surveys of student achievement for more than 40 years through NAEP. The IELS is different from these comparative studies because it focuses on a younger population and collects information about early home learning experiences and early childhood education and care. None of the other studies provide comparative information about 5-year-old children. The IELS also looks at a broader set of domains than these other studies in that it includes not only literacy and numeracy but also social emotional skills.

NCES’s Early Childhood Longitudinal Studies (ECLS) program provides national data on children’s developmental status at birth and at various points thereafter; children’s transitions to early care and education programs, and school; and children’s home and school experiences, growth, and learning. The ECLS program also provides data that enable researchers to analyze how a wide range of child, family, school, classroom, early care and education provider, and community characteristics relate to children’s development and to their experiences and success in school. However, ECLS does not provide a view of U.S. children in an international context as the IELS will do. The IELS will afford an understanding of how U.S. children’s early experiences compare to those of children from other nations and thus allow a better evaluation of U.S. children’s preparedness for school.

The National Household Education Surveys Program (NHES) provides descriptive data on the educational activities of the U.S. population and offers a variety of statistics on the condition of education in the U.S. The NHES surveys have covered learning at all ages, from early childhood to school age through adulthood. However, to collect data on children, NHES surveys adults in households across the U.S., asking parents to report on their children’s competencies, but does not directly assess the children or collect data from their teachers.

Alternative sources of data comparable to IELS do not exist in the U.S. Additionally, IELS involves U.S. participation in an international study that includes other countries. In IELS, the U.S. must collect the same information, using the same instruments and procedures, at the same time as the other participating nations for purposes of making valid and meaningful international comparisons.

A.5 Minimizing Burden for Small Entities

No small entities are part of this sample. The school sample for the IELS will contain small-, medium-, and large-size schools from a wide range of school types, including private schools, and burden will be minimized wherever possible for all institutions participating in the data collection. Schools included in the field test will not be included in the main study sample. In addition, Westat staff will conduct all test administration and will assist with parental notification, sampling, and other study tasks as much as possible within each school.

A.6 Frequency of Data Collection

This request is to conduct the first administration of IELS main study data collection. Adherence to the proposed study schedule is necessary to establish consistency in survey operations among the participating countries. A second administration of IELS has not been scheduled at this time.

A.7 Special Circumstances

The special circumstances identified in the Instructions for Supporting Statement do not apply to this study.

A.8 Consultations outside NCES

Consultations outside NCES have been extensive and will continue throughout the life of the project. The OECD studies are developed as a cooperative enterprise involving participating countries. A steering group with representatives from participating countries guided the scoping phase and will guide reviews of study implementation. In addition, the OECD’s contractor convenes expert panels to seek input on the study design and instrument development. The majority of the consultations (outside NCES) have involved the OECD and the OECD’s international contractor, the Australian Council for Educational Research (ACER). Key to these ongoing consultations are: Ms. Rowena Phair (OECD), Drs. Sacha DeVelle and Wolfram Schultz (ACER), and Ms. Julianne Henke (IEA), all of whom have extensive experience in developing and operating international education surveys.

A.9 Payments or Gifts to Respondents

Currently, the minimum response rate targets required by the OECD are 50 percent of original schools (75 percent after replacement schools are included) and 75 percent of students, while the NCES statistical standards require a minimum response rate target of 85 percent at the student level. Historically, similarly high response rates have been difficult to achieve in school-based studies. For example, U.S. failed to reach the school response rate targets for all previous PISA administrations (2000, 2003, 2006, 2009, 2012, and 2015) and had to adjust incentives upwards in the middle of the recruitment and data collection period in order to meet minimum response rate requirements. Gaining sufficient student cooperation has also been challenging. U.S. has historically met the NCES target rate of 85 percent of responding students in PISA after a great deal of effort. IELS poses more burden on schools than PISA given that the assessments will be conducted one-on-one and over the course of two days for each child, and will include a parent questionnaire. NCES will use a multi-pronged approach to address the challenge of gaining school, student, and parent cooperation.

Staff with experience working on NAEP; international assessments such as PISA, PIRLS, and TIMSS; other large-scale data collections (e.g., Early Childhood Longitudinal Study Kindergarten Class of 1998–99 (ECLS-K) and Class of 2010–11 (ECLS-K:2011)); and with expertise in effective approaches to participant recruitment provided input to help identify strategies for achieving high response rates and are serving as an ongoing source of ideas and feedback. Literature reviews also provide information on methods to improve response rates. In addition, in a systematic review of articles published in major journals since 2002, Singer and Ye (2013) concluded that “incentives increase response rates to surveys in all modes, including the Web, and in cross-sectional and panel studies” and that “monetary incentives increase response rates more than gifts, and prepaid incentives increase them more than promised incentives….” More recently, Mercer et al. (2015) reviewed both published and unpublished research pertaining to incentives from the preceding 21 years and concluded that in mail and telephone surveys (web surveys were not included in the review): “consistent with prior research, the analysis found that prepaid incentives are more effective than promised incentives.”

Based on this information, pre-paid monetary incentives were used in the 2017 IELS field test. The incentive amounts were based on those used in the other international studies, ECLS-K:2011, and/or the Middle Grades Longitudinal Study of 2017-18 (MGLS:2017). Once the field test was concluded, the incentive structure and respondent participation in the IELS field test were analyzed, with the results contributing to slight changes in the proposed 2018 main study incentive plan, as detailed below.

As the terms of clearance of the IELS field test data collection package (1850-0936 v.4), OMB requested a summary describing the results of using the pre-paid incentive (“At conclusion of data collection, NCES will provide OMB with a report describing the results of using a pre-paid incentive for both teachers and parents. Include, at a minimum, a table showing if incentive checks were cashed by whether or not the response was successfully collected.”). The description of incentives for teachers and parents that follows in this section provides the information that fulfills these terms of clearance.

Schools. In order to meet the minimum school response rates mandated by the OECD, and to thank the school for accommodating the disruption of conducting the study, schools participating in IELS will receive a $200 check, included in the school administrator’s thank you letter after the conclusion of the data collection. The same amount was used in the field test and helped contribute to strong support of the study from the sampled schools.

However, due to differences in sampling requirements between the field test and the main study, we anticipate more difficulty in reaching the required school response rates in the latter. In the field test we did not need to produce nationally representative statistics and could thus omit schools in states where we have traditionally had difficulty gaining school cooperation. Furthermore, in the field test we only had to achieve sufficient school participation to obtain the required number of student responses for evaluation of the test items, and did not need to achieve a response rate threshold among the originally sampled schools. Thus after first approaching the original schools, we were able to move quickly to the replacement schools with very little, if any, conversion effort. In the main study, however, we have to collect nationally representative data and, to be included in the international comparisons, we will have to pursue the originally sampled, eligible schools until we obtain at least a 50 percent school response rate among them – the achievement of which has historically required additional efforts in the U.S. Thus to meet the more challenging school recruitment goals in the main study, we will utilize the second-tier school incentive used in other international K-12 studies (e.g., PISA 2012, 2015, and 2018; ICILS 2018), which allows us, when necessary, to offer schools an $800 participation incentive³ instead of the standard $200. This second tier incentive will be initiated only after we have approached all original schools and had tried different conversion efforts, such as addressing the specific concerns of refusing schools. If, by that time, we have not reached a participation rate of at least 52 percent⁴ of the original schools, we will implement the second tier incentive. In addition, if by October 2018, the un-weighted IELS school response rate with replacement schools is below 75%, a sufficient number of additional replacement schools will be selected to meet the minimum response rate requirements for U.S. data to be reported (i.e., an un-weighted school response rate with replacement of at least 75%, after at least 50% of original schools response rate). Given that these newly selected replacement schools will be approached late in the short data collection window, they will also be offered the second-tier $800 incentive. We will approach both original and replacement refusing schools with the second-tier incentive only if necessary and at the point of our last chance to convert them.

School coordinators. The role of the school coordinator is critical for the success of the study. The coordinator is expected to coordinate logistics with the data collection contractor; supply a list of eligible students and teachers for sampling to the data collection contractor; communicate with teachers, students, and parents about the study to encourage participation, collect parent consent; and assist the field staff in arranging for make-up sessions as needed. As was successfully implemented in the field test, to encourage their participation and extending their best effort for the study, each school coordinator will be offered $200, with a $100 check provided after they complete e-filing, and the remaining $100 being mailed with a thank you letter at the end of the data collection.

Teachers: We sampled 111 teachers in the field test, of which 91 percent completed part A and part B questionnaires. Teachers were given a prepaid incentive of $40 to thank them up-front for their time, and all of the teacher checks were cashed by the respondents. Prior to the field test, we estimated that each teacher would be asked to complete one part A teacher-focused background questionnaire plus part B student-focused questionnaires for approximately four sampled students. However, the field test results show that many teachers were asked to complete six part B questionnaires, imposing a greater variation in per teacher burden than originally estimated. Furthermore. because based on other NCES studies we expect that schools and teachers in the main study will be on average less cooperative than those in the field test, in the main study we will follow the incentive model successfully used in ECLS-K and ECLS-K:2011, where each sampled teacher received a check totaling $20 for completing part A of the teacher questionnaire plus an additional $7 for each part B questionnaire they were asked to complete about individual sampled students. Because in IELS we will “link” sampled students to teachers through the e-filing process, and thus will know the number of part B questionnaires each teacher will be asked to complete, we will calculate the incentive check amounts for each sampled teacher before recruiting them. Thus, for example, a teacher who completes one part A questionnaire plus: (a) six part B questionnaires versus (b) two part B questionnaires will receive a check for $62 versus $34, respectively. Given the variation in the number of part B questionnaires each teacher is asked to complete, this incentive model more fairly reflects the amount of time and effort each is being asked to spend. As in the field test, the teacher incentive check will accompany the study information materials that contain information on completing the questionnaires online. Cases where the actual number of part B questionnaires the teacher is asked to complete differs than the assumption made after sampling will be handled on a case-by-case basis.

Parents: To incentivize parents to participate in the IELS field test, the parent or guardian of each sampled student received a check for $20. When possible, this check was distributed at the beginning of the field period with the study informational materials and details on how to access the online questionnaire. However, in many cases the check could not be included because the school was unwilling to share parent names, preventing us from writing checks to the respondents. In these cases, parents received their incentive amounts at the conclusion of the study. In a few cases, schools were unwilling to share the parents’ names at all and thus no checks were distributed to these parents.

Table 1 summarizes the effect of the incentive payments on response rates. Upon analyzing the field test results, it does not appear that the receipt of a pre-payment positively influenced the parent response rates – 31 percent of parents who received a pre-payment and 64 percent of parents who received a post-payment completed their questionnaire. Furthermore, 74 percent of the parents who completed the questionnaire and 57 percent of parents who did not complete the questionnaire cashed their check.

Table 1. Parent questionnaire completion vs. checks cashed

	Checks mailed with advance package		Checks mailed after advance package		Total
	N	Percent of in-scope	N	Percent of in-scope	N	Percent
Total number of parents of in-scope students					523	100%
Checks mailed	189	36%	266	51%	455	87%
Checks cashed	119	22%	180	34%	299	57%

	Completed questionnaire		Did not complete questionnaire		Total
	N	Percent	N	Percent	N	Percent
Total number of parents of in-scope students	242	46%	281	54%	523	100%
Checks mailed in advance package	76	31%	113	40%	189	36%
Checks mailed after advance package	156	64%	110	39%	266	51%
No checks mailed due to lack of parent’s name	10	4%	58	21%	68	13%
Total checks written to parents	232	51%	223	49%	455	100%
Cashed check	172	74%	127	57%	299	66%
Did not cash check	60	26%	96	43%	156	34%

Note: parents were paid either a pre-paid or post-paid incentive depending on the school’s policy of sharing parent names with the study. These tables are not meant to reflect results from a random incentive experiment; the results should be interpreted based on the understanding that only 29 schools participated in the field test and the differences we see among parents in the pre-and post-pay groups may reflect school characteristics or other unknown factors.

Based on these results and the previous advice of experts, a $20 incentive for parents is again proposed for the main study. It is critical to continue providing an incentive to parents, given that parent survey response rates have been declining over the past decade. The ECLS-K:2011 baseline (fall 2010) parent survey response rate was more than 10 percentage points lower (74 percent) than the parent survey rate in the corresponding 1998 wave of the ECLS-K (85 percent), while the ninth-grade parent survey response rate for the HSLS:09 baseline was 68 percent. The IELS parent survey is a key component of the data being collected, especially given that this is the first time this type of respondent is included in an international study in the U.S.

In addition, we expect a large number of student cases to require active consent, given the age range of the sample. In the field test, 40 percent of schools required explicit consent, while 34 percent required implicit consent (the remaining 28 percent of schools required notification only). Thus the parent incentive is aimed to encourage parents to complete their questionnaire, complete the consent form for their sampled child’s participation in IELS, and minimize the need for costly follow-up.

However, instead of using checks as in the field test, we will use cash cards in the main study. Cash cards do not require parent name or other contact information, thereby allowing the IELS team to provide incentive cards to 100 percent of parents. Additionally, instead of offering a pre-paid incentive, we will use post-paid parent incentives in the main study. Parents will receive an empty cash card together with the study informational materials and MyIELS registration details distributed by the school coordinator or Westat, depending on the preference of the school. The cash card will be loaded with $20 upon completion of the parent questionnaire.

Students: IELS will not offer a monetary incentive to students. As has been done in other NCES studies, such as ECLS-K:2011, IELS will provide a nonmonetary incentive to all participating students, such as a sticker or a pencil.

A.10 Assurance of Confidentiality

Data security and confidentiality protection procedures have been put in place for IELS to ensure that Westat and its subcontractors comply with all privacy requirements, including:

The statement of work of this contract;
Privacy Act of 1974 (5 U.S.C. §552a);
Family Educational and Privacy Act (FERPA) of 1974 (20 U.S.C. §1232(g));
Privacy Act Regulations (34 CFR Part 5b);
Computer Security Act of 1987;
U.S.A. Patriot Act of 2001 (P.L. 107-56);
Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9573);
Confidential Information Protection and Statistical Efficiency Act of 2002;
E-Government Act of 2002, Title V, Subtitle A;
Cybersecurity Enhancement Act of 2015 (6 U.S.C. §151);
The U.S. Department of Education General Handbook for Information Technology Security General Support Systems and Major Applications Inventory Procedures (March 2005);
The U.S. Department of Education Incident Handling Procedures (February 2009);
The U.S. Department of Education, ACS Directive OM: 5-101, Contractor Employee Personnel Security Screenings;
NCES Statistical Standards; and
All new legislation that impacts the data collected through the inter-agency agreement for this study.

Furthermore, Westat will comply with the Department’s IT security policy requirements as set forth in the Handbook for Information Assurance Security Policy and related procedures and guidance, as well as IT security requirements in the Federal Information Security Management Act (FISMA), Federal Information Processing Standards (FIPS) publications, Office of Management and Budget (OMB) Circulars, and the National Institute of Standards and Technology (NIST) standards and guidance. All data products and publications will also adhere to the revised NCES Statistical Standards, as described at the website: http://nces.ed.gov/statprog/2012/.

Procedures for handling confidential aspects of the IELS study will mirror those used in other studies conducted by NCES, including signed confidentiality agreements and notarized nondisclosure affidavits obtained from all personnel who will have access to individual identifiers; personnel training regarding the meaning of confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses; protected access to computer files under the control of a single data base manager; built-in safeguards concerning status monitoring and receipt control systems; and a secured and operator-manned in-house computing facility.

Letters and other materials will be sent to parents and school administrators describing the voluntary nature of this survey. The materials sent will include a brochure that describes the study and conveys the extent to which respondents and their responses will be kept confidential (Appendices A and B):

The National Center for Education Statistics is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information you provide may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). Individuals are never identified in any reports. All reported statistics refer to the U.S. as a whole or to national subgroups.

The following statement will appear on the front cover and login page of the questionnaires:

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this voluntary survey is 1850-0936. The time required to complete this survey is estimated to average XX minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the survey. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this survey, or any comments or concerns regarding the status of your individual submission of this survey, please write to: International Early Learning Study (IELS), National Center for Education Statistics, Potomac Center Plaza, 550 12^th Street, SW, Washington, DC 20202.

OMB No. 1850-0936, Approval Expires 04/30/2020.

In addition, the following text appears on the MyNAEP log-in screen:

Notice: You are accessing a U.S. Government information system.

This warning banner provides privacy and security notices consistent with applicable federal laws, directives, and other federal guidance for accessing this Government system, which includes all devices/storage media attached to this system. This system is provided for Government-authorized use only. Unauthorized or improper use of this system is prohibited and may result in disciplinary action and/or civil and criminal penalties.

Data files, accompanying software, and documentation will be delivered to NCES at the end of the project. No school or individual names or addresses will be included on these files or documentation.

NCES understands the legal and ethical need to protect the privacy of the IELS respondents and has extensive experience in developing data files for release that meet the government’s requirements to protect individually identifiable data from disclosure. The contractor will conduct a thorough disclosure risk analysis of the IELS data when preparing the data files for use by researchers, in compliance with 20 U.S.C. §9573. Schools with high disclosure risk will be identified and, to ensure that individuals may not be identified from the data files, a variety of masking strategies will be used, including: swapping data and omitting key identification variables (e.g., school name and address) from both the public- and restricted-use files (though the restricted-use file will include an NCES school ID that can be linked to other NCES databases to identify a school); omitting key identification variables such as state or zip code from the public-use file; and collapsing or developing categories for continuous variables to retain information for analytic purposes while preserving confidentiality in public-use files. IES’s Disclosure Review Board (DRB) carefully reviews all datasets prior to release to ensure that disclosure risks have been properly addressed. The IELS data will be reviewed and approved by the DRB prior to any public release.

A. 11 Sensitive Questions

The IELS does not include questions usually considered to be of a sensitive nature, such as items concerning religion, substance abuse, or sexual activity. Several items in the background questionnaires may be considered sensitive by some of the respondents, such as parents’ education and occupation and family possessions. Research indicates that the constructs these items represent are strongly correlated to academic achievement, and they have been used in number of national and international studies, including ECLS, PISA, and TIMSS. These items are considered essential for the anticipated analyses and to retain consistency in planned comparisons with the international data.

A. 12 Estimates of Burden

This request is for main study data collection. Due to time overlap in activities, the approved main study recruitment activities are being carried over. Burden estimates are shown in table 2.

Recruitment activities include the time involved in a school deciding to participate, providing student listing forms, and arranging for assessment space. As shown in Part B, during the IELS main study, NCES will assess 3,000 students from 200 schools, with an expected 16 students per school (19 sampled per school).

Some districts are known as “special handling districts” which require completion of a research application before they will allow schools under their jurisdiction to participate in a study. The main study sample has 28 known special handling districts (shown in table 2), which we began to contact in fall 2017. Contacting special handling districts begins with updating district information based on what can be gleaned from online sources. Calls are then placed to verify the information about where to send the completed required research application forms, and, if necessary, to collect contact information for this process. During the call, inquiry is also made about the amount of time the districts spend reviewing similar research applications. The estimated number of such districts represents those with particularly detailed application forms and lengthy processes for approval.

The total response burden for districts and schools in the main study recruitment (approved in September 2017; OMB# 1850-0936 v.3-4) is based on the following:

There are 28 special handling districts in the main study sample – those known to require completion of a research application before they will allow schools under their jurisdiction to participate in a study. Estimated burden hours for special handling districts are included in table 1 with the provision of time for district application review. We estimate 1 hour for IRB staff of special districts to review materials and 30 minutes for IRB panels (on average made up of 6 people) to review materials.
It is estimated that it will take 10 minutes on average for school administrators to review the materials and either agree or decline to participate. It is estimated that an additional 4 hours will be needed for the provision of student rosters (including information about students for sampling, contact information for the sample students’ parents, and sample students' teachers; see Appendix A). That is, we estimate approximately 250 minutes total for each school.
For students’ parents, we estimate that it will take up to 10 minutes to review the recruitment materials and either consent or refuse to participate (on behalf of their student and themselves; see Appendix B). The provision of student rosters and the parents’ consent forms will serve as sources for parents’ contact information, which during the data collection period can be used for nonresponse follow-up.

The total response burden for the main study data collection, including pre-assessment activities, is based on the following:

We estimate that approximately 8 hours will be needed by school coordinators for the oversight of student rosters (including contact information for the sample students’ parents and teachers; see Appendix A-B); the collection and delivery of teacher and parent data; and the coordination of the student assessment.
It is estimated that it will take 30 minutes on average for parents to respond to the parent/guardian survey, including instructions.
For teachers, we estimate that it will take on average 33 minutes to complete all aspects of the teacher survey (on average 33 minutes total per teacher: approximately 3 minutes for the core portion and 5 minutes per sampled child, with an average of up to 6 sampled children per teacher).
For students, we expect it will take 60 minutes, including instructions, to complete all sections of a tablet-based assessment.

Based on the estimated hourly rates for principals/administrators/school coordinators of $46.85, kindergarten teachers (including special education teachers) of $28.62, and parents of $24.34⁵, and based on 4,461 total burden hours for IELS main study recruitment and data collection, the estimated total respondent burden time cost is $ 166,537.

Table 2. Burden estimates for IELS study

	Sample	Expected response rate	Number of respondents	Number of responses	Burden per respondent (minutes)	Total burden (hours)
Main Study
Recruitment
Special Handling Districts IRB staff	28	1.00	28	28	60	28
Special Handling Districts IRB Panel	168	1.00	168	168	30	84
Schools	235	1.00	235	235	250	979
Parent notification	3,600	1.00	3,600	3,600	10	600
Main Study Recruitment Burden			4,031	4,031		1,691
Pre-Assessment Activity
School Coordinator	235	0.85	200	200	240	800
Data Collection
Parent Questionnaire*	3,600	0.85	3,060*	3,060	30	1,530
Teacher Questionnaire	941	0.85	800	800	33	440
Students (direct assessment)	3,600	0.85	3,060	3,060	60	3,060
Total Burden Requested in this Submission			5,031	8,091		4,461

Note: Total respondent burden requested in this submission includes burden associated with the main study data collection and with the carried over approved burden for main study recruitment. Burden estimated for student cognitive assessment, which is not subject to PRA, is reflected in gray font, and is not included in the requested burden totals.

* A subset of the respondent group shown under recruitment, not double counted in the total number of respondents.

A.13 Total Annual Cost Burden

There are no additional costs to respond beyond the time to respond.

A.14 Annualized Cost to Federal Government

The cost to the federal government for conducting the upcoming phases of IELS, including remaining recruitment and data collection for the 2018 main study is estimated to be $4,587,968 (see table breakdown below). These figures include all direct and indirect costs (the grand total is the total value to the Government including other phases of the contract and including all optional tasks).

Components with breakdown	Estimated costs
FIELD TEST (2017)
Recruitment	736,675
Preparations (e.g., adapting instruments, sampling)	272,964
Data collection, scoring, and coding	663,598
MAIN STUDY (2017)
Recruitment	725,266
Preparations (e.g., adapting instruments, sampling)	61,993
MAIN STUDY (2018)
Recruitment	591,605
Preparations (e.g., adapting instruments, sampling)	365,057
Data collection, scoring, and coding	3,631,306
Current package components	4,587,968
Grand total	$7,048,464

A.15 Program Changes or Adjustments

The apparent increase in respondent burden is due to the fact that the last approval was for field test data collection and main study recruitment, while this request includes burden associated with the IELS 2018 main study recruitment and data collection.

A.16 Plans for Tabulation and Publication

The IELS field test was designed to provide a statistical review of the performance of items on the assessments and questionnaires in preparation for the main data collection. The international contractor provided the international instruments to be used in the field test, reported to the participating countries on the results of the field test and, based on the field test results, with input and agreement from the participating countries, made final revisions in the survey instruments, materials, and documents in preparation for the main study.

For the main study in 2018, an analysis of the U.S. and international data will be undertaken to provide an understanding of the U.S. national results in relation to the international results. Based on proposed analyses of the international data set by the OECD and the need for NCES to report results from the perspective of a U.S. constituency, a plan is being prepared for the statistical analysis of the U.S. national data set as compared to the international data set. Analysis of data will include examinations of student performance in relation to their international counterparts; and the relationships between student performance and contextual variables.

All reports and publications will be coordinated with the release of information from the international organizing body. Planned publications and reports for the IELS main study include the following:

General Audience Report. This report will present information on the skills and competencies of U.S. children in comparison to their international peers, written for a non-specialist, general U.S. audience. This report will present the results of analyses in a clear and non-technical way, conveying how U.S. children compare to their international peers, and what factors, if any, may be associated with the U.S. results.

Survey Operations/Technical Report. This document will detail the procedures used in the main study (e.g., sampling, recruitment, data collection, scoring, weighting, and imputation) and describe any problems encountered and the contractor’s response to them. The primary purpose of the main study survey operations/technical report is to document the steps undertaken by the U.S. in conducting and completing the study. This report will include an analysis of non-response bias, which will assess the presence and extent of bias due to nonresponse. Selected characteristics of respondent children and schools will be compared with those of non-respondent schools and students to provide information about whether and how they differ from respondents along dimensions for which we have data for the nonresponding units, as required by NCES standards.

Electronic versions of each publication are made available on the NCES website. Schedules for tabulation and publication of IELS results in the U.S. are dependent upon receiving data files from the international sponsoring organization. With this in mind, the expected data collection dates and a tentative reporting schedule are as follows:

August-November 2017	Recruitment for the field test
November-December 2017	Collect field test data
September 2017-October 2018	Recruitment for the main study
October-December 2018	Collect main study data
Early 2020	Release of main report on findings

A.17 Display OMB Expiration Date

The OMB expiration date will be displayed on all data collection materials.

A.18 Exceptions to Certification Statement

No exceptions are requested to the Certification for Paperwork Reduction Act Submissions of OMB Form 83-I.

References

Mercer, Andrew et al. 2015. “How Much Gets You How Much? Monetary Incentives and Response Rates in Household Surveys.” Public Opinion Quarterly 79:105-129.

Singer, Eleanor, and Cong Ye. 2013. “The Use and Effects of Incentives in Surveys.” ANNALS of the American Academy of Political and Social Science 645:112–41.

1 Although we expect that most children in this age range will attend kindergarten, a small percentage of children may be in pre-kindergarten or first grade. Because this is an age-based sample rather than a grade-sample, these off-grade students will be included in the study.

2 According to the IELS international technical standards, five-year-old children are defined as those who are five years old on the day in the middle of the administration period. Therefore, if the administration period for the main study is, for example, 15 October 2018 to 12 December 2018, then any child who was born between 1 November 2012 and 31 October 2013 are eligible for the study. The majority of these children will be in kindergarten, and a small number will be in first grade.

3 This amount was tested in the incentive experiment conducted as part of the PISA 2012 field test and then was first used in the PISA 2012 main study.

4 To meet the 50 percent of original schools minimum requirement, we need to recruit at least 52 percent of original schools factoring in a 3 percent attrition of schools before data collection begins in October 2018.

5 The average hourly wage of principals/education administrators/school coordinators derived from May 2017 Bureau of Labor Statistics (BLS) Occupation Employment Statistics is $46.85, of kindergarten teachers (including special education teachers) it is $28.62, and of parents it is $24.34. If mean hourly wage was not provided, it was computed assuming 2,080 hours per year. Source: BLS Occupation Employment Statistics, http://data.bls.gov/oes/ data type: Occupation codes: Education Administrators, Elementary and Secondary Schools (11-9032), Kindergarten Teachers, Except Special Education (25-2012), Special Education Teachers, Kindergarten and Elementary School (25-2052), and (for the parent estimate) All Occupations (00-0000); accessed on June 20, 2018.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	2021-01-20