Part A IELS 2018 MS Recruitment & Field Test

The International Early Learning Study (IELS), scheduled to be conducted in 2018, is a new study sponsored by the Organization for Economic Cooperation and Development (OECD), an intergovernmental organization of industrialized countries. The OECD also coordinates the Program for International Student Assessment (PISA), an assessment of 15-year-olds, and the Program for the International Assessment of Adult Competencies (PIAAC), which assesses adult skills. While PIAAC focuses on adults and PISA on students nearing the end of compulsory school, the IELS focuses on young children and their cognitive and non-cognitive skills and competencies as they transition to primary school. The IELS is designed to examine:

Children’s early learning and development in a broad range of domains, including social and cognitive skills;
The relationship between children’s early learning and children’s participation in early childhood education and care (ECEC);
The role of contextual factors, including children’s individual characteristics and their home backgrounds and experiences, in promoting young children’s growth and development; and
How early learning varies across and within countries prior to beginning, or in the early stages of primary school.

In 2018, in the participating countries, including the United States, the IELS will assess nationally-representative samples of 5-year-old children enrolled in public and private schools that offer kindergarten, through direct and indirect measures, and will collect contextual data about the children’s home learning environments, ECEC histories, and demographic characteristics. The IELS will measure young children’s knowledge, skills, and competencies in both cognitive and non-cognitive domains, including language and literacy, mathematics and numeracy, executive function/self-regulation, and social emotional skills. This assessment will take place as children are transitioning to primary school and will provide data on how U.S. children entering kindergarten compare with their international peers on skills deemed important for later success¹.

In the United States, IELS is conducted by the National Center for Education Statistics (NCES), part of the U.S. Department of Education’s Institute of Education Sciences (IES). NCES, with the support of IELS 2018 U.S. data collection contractor, Westat, ensures proper implementation of the study and adoption of practices in adherence to the OECD‘s standards. The IELS is a collaboration among the participating countries, the OECD, and an international contractor which coordinates the study across countries.

To prepare for the main study, which will be conducted from October to November 2018, the IELS countries will conduct a field test in the fall of 2017 to evaluate newly developed assessment instruments and questionnaires and also to test the study operations. The U.S. IELS field test data collection will occur from November to December 2017. In order to meet the international data collection schedule for the fall 2017, field test recruiting activities must begin by early September 2017. In addition, main study recruitment activities with states and districts must also begin in September 2017 to allow enough time to gain cooperation prior to contacting schools in early winter 2018. Because the IELS is a collaborative effort among many parties, the U.S. must adhere to the international schedule set forth by the OECD, including the availability of final field test and main study plans and questionnaires.

Recruitment activities for the 2017 field test were approved in April 2017 (OMB 1850-0936 v.1-2). This request is to conduct the field test data collection and recruitment for the main study. Because recruitment for the field test has not yet started and will continue beyond the beginning of field test data collection, previously cleared field test recruitment procedures and respondent burden and materials are being carried over in this submission, and have been updated in August 2017. IELS main study procedures and materials are based on those used in the field test and are included in Appendix A and B of this submission.

Final field test instruments, which have been approved by the international consortium, are included in Appendix C (revised August 2017). The appendix also includes a Spanish-language version of the parent interview, as well as content of the parent consent module fo the MyIELS communication website. The final details of the main study data collection will be submitted for OMB clearance in late 2017.²

A. JUSTIFICATION

A.1 Importance of Information

As part of a continuing cycle of international studies, the United States, through NCES, participates in several international education assessments and surveys, including PISA, an assessment of 15-year-olds, and PIAAC, which assesses adult skills. IELS, sponsored by OECD, is a new addition to NCES’s international studies.

In light of the growing concerns related to international economic competitiveness, the changing face of our workplace, and the expanding international marketplace in which we trade, knowing how our students and adults compare with their peers around the world has become an even more prominent issue than ever before. Nationwide, interest in understanding what other nations are doing to further the educational achievement of their populations has increased beyond simple comparisons. Moreover, recent investments in early childhood education and care (ECEC) require a closer examination of what US children know as they enter school and how their knowledge compares to that of children in other nations with different ECEC experiences.

Data at critical points during the education career of U.S. students are used by policymakers in efforts to guide and examine the American education system. Consequently, generating comparative data about students in school and about adults in workplace and in community has become an important focus for NCES. The addition of the IELS to the slate of comparative assessments conducted by NCES will broaden what we know about U.S. students compared with those in other countries and provide comparative information about students at an earlier point in the educational process than has previously been possible.

The results from the IELS will allow national policymakers to compare the skills and competencies of children from different nations as they are readying to enter primary school and to evaluate whether changes might be needed in the U.S.’s ECEC system to promote additional skills and competencies seen in other countries. Without these types of data, U.S. policymakers are limited in their ability to gain insight into the educational performance and practices of other nations as they compare to the United States.

The study’s success is dependent on the development of reliable, valid cross-cultural measures. The goal of the IELS field test is to evaluate the direct cognitive assessment, parent questionnaire, teacher questionnaire, and study operations. The IELS field test will provide the data needed to determine the psychometric properties of items and the predictive potential of assessment and survey items so that valid, reliable, and useful assessment and survey instruments can be composed for the main study, ensuring validity in multiple countries. As the focus of the IELS field test is the analyses of the psychometric properties of the survey items and assessments, the IELS field test requires a small, but diverse field test sample, though not a nationally representative one.

A.2 Purposes and Uses of Data

Overview of the IELS Study Design

The IELS 2018 will collect data on a nationally representative sample that will yield approximately 3,000 assessed five-year-old children³ enrolled in 200 schools through the United States. The fall 2017 field test sample is designed to yield a minimum of 450 children, enrolled in 30 randomly selected schools in five states chosen to reflect geographic diversity.

The data collection for both the field test and the main study will include direct child assessments and online parent and teacher questionnaires. Instruments have been developed based on frameworks that define the constructs and provide specifications for distributions of items by framework dimensions. Following the field test, cognitive and non-cognitive items will be evaluated for bias and interpretation issues, following standard protocols. For the main study, the pool of items will be reduced to only include those that demonstrate validity across the participating education systems, meet the goals of content coverage to adequately measure the frameworks, and provide the desired distribution of item types.

Cognitive Assessments. For the field test, the IELS assessment will be comprised of four domains: emerging literacy, emerging numeracy, empathy and trust, and self-regulation. Each assessment will be monitored by a trained assessor, using a tablet. Each item will be read aloud to the child, who listens through headphones. The child will be asked to respond to questions in a variety of methods, including tapping on a picture or dragging and dropping items. After a brief introductory session where the assessor demonstrates to the child how to use the tablet to respond to questions and the child answers a few practice items, the assessment will begin. The play-based assessment is untimed and self-administered, although the assessor will be available to respond to questions, demonstrate how to answer items if needed, and intervene if technical issues with the hardware or software arise. The assessment is expected to last approximately 60 minutes, although it will be broken into two sessions spread across two days. That is, each child will complete on average a 30-minute session on day 1 and another 30-minute session on day 2.

Parent questionnaire. One parent or guardian of each student will be asked to complete a brief online survey. If preferred, a paper-and-pencil version can be requested. The questionnaire asks demographic questions as well as questions about the child’s early learning environment and experiences, social skills, and participation in child care. The English questionnaire was translated into Spanish (see Appendix C for the field test parent questionnaire items).

Teacher questionnaire. Teachers of selected students will be asked to complete an online questionnaire, which includes questions about their professional background (about 3 minutes to complete) and the skills and abilities of the students selected for the study (about 5 minutes per child, with an average of 4 students per teacher). The survey does not need to be completed in a single session. Paper-and-pencil versions of the surveys will be available upon request (see Appendix C for the field test teacher questionnaire).

Governments and the general public want solid evidence of education outcomes. In the late 1990s, the OECD launched an extensive program for producing policy-oriented and internationally comparable indicators of student achievement on a regular basis and in a timely manner. The IELS is a new addition to this program.

Purposes of Data

The IELS will assess children as they are transitioning to primary school and, as such, will provide a view of how 5-year-old children in the U.S. compare with their international peers on the skills and competencies deemed important for success in school. The results will allow national policymakers to evaluate the readiness of U.S. children for formal schooling, and to analyze the relationship between constructs measured through the IELS adult questionnaires with child assessment results at national and international levels. Through the IELS, the OECD and NCES will produce two types of indicators:

Basic indicators that provide a baseline profile of the knowledge, skills, and competencies of young children; and
Contextual indicators that show how such skills relate to important demographic, social, economic, and education variables.

A.3 Improved Information Technology (Reduction of Burden)

Where feasible, available technology will be used to reduce burden and improve efficiency and accuracy. The burden of recruitment on districts and schools will be minimal, with most information gathered through E-filing (see below). Districts will primarily be asked to provide confirmation of data gathered from other sources, including school universe files and district and school websites.

A communication website, MyIELS.us, will be used during the field test and main study in order to provide a simple, single source of information to engage sampled schools and maintain high levels of their involvement. This portal will be used throughout the assessment cycle to inform schools, particularly school coordinators, of their tasks and to provide them with easy access to information tailored for their anticipated needs. We plan to gather student and teacher lists from participating schools electronically using an adaptation of a secure E-filing process through the web portal. E-filing is an electronic system for submitting lists of student and teacher information, including limited background information in school records. E-filing has been used successfully in NAEP for more than 10 years, and was used in TIMSS 2015 and the PISA 2012 and 2015 assessments. The E-filing system provides advantageous features such as efficiency and data quality checks.

The IELS design and procedures are prescribed internationally. The data collected by direct, one-on-one child assessments will be captured electronically on tablet. Trained assessors will facilitate the assessment. Teacher and Parent Questionnaires will be made available as online questionnaires and in paper format.

A.4 Efforts to Identify Duplication

A number of international comparative studies exist to measure student achievement, including the Trends in International Mathematics and Science Study (TIMSS), the Progress in International Reading Literacy Study (PIRLS) and PISA. In addition, the U.S. has been conducting its own national surveys of student achievement for more than 40 years through the National Assessment of Educational Progress (NAEP) program. The IELS is different from these comparative studies because it focuses on a younger population and collects information about early home learning experiences and early childhood education and care. None of the other studies provide comparative information about 5-year-old children. The IELS also looks at a broader set of domains than these other studies in that it includes not only literacy and numeracy but also social emotional skills.

NCES’s Early Childhood Longitudinal Studies (ECLS) program provides national data on children’s developmental status at birth and at various points thereafter; children’s transitions to early care and education programs, and school; and children’s home and school experiences, growth, and learning. The ECLS program also provides data that enable researchers to analyze how a wide range of child, family, school, classroom, early care and education provider, and community characteristics relate to children’s development and to their experiences and success in school. However, ECLS does not provide a view of U.S. children in an international context as the IELS will do. The IELS will afford an understanding of how U.S. children’s early experiences compare to those of children from other nations and thus allow a better evaluation of U.S. children’s preparedness for school.

The National Household Education Surveys Program (NHES) provides descriptive data on the educational activities of the U.S. population and offers a variety of statistics on the condition of education in the U.S. The NHES surveys cover learning at all ages, from early childhood to school age through adulthood. However, NHES conducts surveys of adults in households across the U.S., asking the parents to report on their children’s competencies, it does not assess nor collects other data on a nationally-representative sample of 5-year-old children enrolled in public and private U.S. schools.

Alternate sources for these data do not exist. This study represents the U.S. participation in an international study involving other countries. The United States must collect the same information, using the same instruments and procedures, at the same time as the other nations for purposes of making valid and meaningful international comparisons. No other study in the U.S. will be using the instruments developed by the OECD, and thus no alternative sources of comparable data are available.

A.5 Minimizing Burden for Small Entities

No small entities are part of this sample. The school sample for the IELS will contain small-, medium-, and large-size schools from a wide range of school types, including private schools, and burden will be minimized wherever possible for all institutions participating in the data collection. Schools included in the field test will not be included in the main study sample. In addition, Westat staff will conduct all test administration and will assist with parental notification, sampling, and other study tasks as much as possible within each school.

A.6 Frequency of Data Collection

This request is for the IELS field test data collection in fall 2017 and recruitment for the main study (for which recruitment will begin in September 2017 and data collection in the fall of 2018). A request for the main study data collection will be submitted for approval under a subsequent request. Adherence to this schedule is necessary to establish consistency in survey operations among the participating countries. A second administration of IELS has not been scheduled at this time.

A.7 Special Circumstances

The special circumstances identified in the Instructions for Supporting Statement do not apply to this study.

A.8 Consultations Outside NCES

Consultations outside NCES have been extensive and will continue throughout the life of the project. The OECD studies are developed as a cooperative enterprise involving participating countries. A steering group with representatives from participating countries guided the scoping phase and will guide reviews of study implementation. In addition, the OECD’s contractor convenes expert panels to seek input on the study design and instrument development. The majority of the consultations (outside NCES) have involved the OECD and the OECD’s international contractor, the Australian Council for Educational Research (ACER). Key to these ongoing consultations are: Ms. Rowena Phair (OECD), Drs. Sacha DeVelle and Wolfram Schultz (ACER), and Ms. Julianne Henke (IEA), all of whom have extensive experience in developing and operating international education surveys.

A.9 Payments or Gifts to Respondents

Currently, the minimum response rate targets required by the OECD are 75 percent of original schools and 75 percent of students, while the NCES statistical standards require a minimum response rate target of 85 percent at the student level. Historically, similarly high response rates have been difficult to achieve in school-based studies. For example, U.S. failed to reach the school response rate targets for all previous PISA administrations (2000, 2003, 2006, 2009, 2012, and 2015) and had to adjust incentives upwards in the middle of the recruitment and data collection period in order to meet minimum response rate requirements. Gaining sufficient student cooperation has also been challenging. U.S. has historically met the NCES target rate of 85 percent of responding students in PISA after a great deal of effort. IELS poses more burden on schools than PISA given that the assessments will be conducted one-on-one and over the course of two days for each child, and will include a parent questionnaire. NCES will use a multi-pronged approach to address the challenge of gaining school, student, and parent cooperation and learn as much as possible during the field test about how to achieve acceptable participation rates in the IELS main study.

Staff with experience working on the NAEP; international assessments such as PISA, PIRLS, and TIMSS; other large-scale data collections (e.g., ECLS-K); and with expertise in effective approaches to participant recruitment have provided input to help identify strategies for achieving high response rates and are serving as an ongoing source of ideas and feedback. Based on this input, one of the ideintified strategies are monetary incentives, which have been effective in other international studies. The proposed amounts are described below and are based on the amounts used in the other international, ECLS-K:2011, and the Middle Grades Longitudinal Study of 2017-18 (MGLS:2017) studies. For the MGLS:2017 study, a $20-30 parent incentive for a 40-minute parent interview was proposed and is currently under OMB review (OMB# 1850-0911), along with a teacher incentive of $20 plus $7 for each child-level questionnaire. Given the IELS assumption of an average of four sampled children linked to each teacher, this would translate to a $48 teacher incentive. IELS is proposing a $40 prepaid incentive for each teacher asked to participate.

Teacher and parent incentives will be prepaid – a check will accompany the study informational materials and details on how to complete the online survey that will be distributed to these respondents. A recent article updates previous reviews of research on the use of survey incentives (Singer and Ye 2013). The authors systematically reviewed articles appearing since 2002 in major journals, supplemented by searches of the Proceedings of the American Statistical Association’s Section on Survey Methodology. Among the conclusions they draw are that “Incentives increase response rates to surveys in all modes, including the Web, and in cross-sectional and panel studies” and “Monetary incentives increase response rates more than gifts, and prepaid incentives increase them more than promised incentives….” Another study which reviewed both published and unpublished research pertaining to incentives from the last 21 years resulted in similar conclusions: “Consistent with prior research, the analysis found that prepaid incentives are more effective than promised incentives.”(Mercer et al. 2015). This was especially true with mail and telephone surveys (web surveys were not included in the review). IELS is an unknown study that will compete with required testing and, in the case of teachers, instructional time over a 2-day period. Given our experience in ECLS-K:2011, we believe that providing a check up-front will encourage participation.

Schools. In order to meet the minimum school response rates mandated by the OECD, and to thank the school for accommodating the disruption of conducting the study, schools participating in IELS will receive a $200 check, included in the school administrator’s thank you letter after the conclusion of the data collection.

School coordinators. The role of the school coordinator is critical for the success of the study. The coordinator is expected to coordinate logistics with the data collection contractor; supply a list of eligible students and teachers for sampling to the data collection contractor; communicate with teachers, students, and parents about the study to encourage participation, collect parent consent; and assist the field staff in arranging for make-up sessions as needed. To encourage their participation and extending their best effort for the study, each school coordinator will be offered $200, with a $100 check provided after they complete e-filing, and the remaining $100 being mailed with a thank you letter at the end of the data collection.

Teachers: The teacher of each sampled student will be receive a $40 check with a request to complete the Teacher Questionnaire, which includes questions about the teacher and providing information about each sampled child. This amount is based on the amount offered to ECLS-K:2011 teachers for similar level of effort.

Parents: To incentivize them to complete the Parent Questionnaire, which includes providing information on the study child, the household learning environment, and the study child’s ECEC history , a parent or guardian of each sampled student will receive a $20 check included with the study informational materials and MyIELS registration details distributed by the school coordinator or Westat, depending on the preference of the school.

In IELS, we expect a large number of student cases to require active consent, given the age range of the sample. The cost of the parent pre-paid incentives is traded off against the cost of much more expensive additional field labor to follow-up on consent. Furthermore, we have found that parent survey response rates have declined over the past decade. The ECLS-K:2011 baseline (fall 2010) parent survey response rate was more than 10 percentage points lower (74 percent) than the parent survey rate in the corresponding 1998 wave of the ECLS-K (85 percent); the ninth-grade parent survey response rate for the HSLS:09 baseline was 68 percent. The IELS parent survey is a key component of the data being collected, especially as this is the first time this type of respondent is included in international studies’ data collections. The prepaid incentive is designed to encourage parent participation.

Students: There is no monetary incentive in the IELS for students, but prior NCES studies such as the ECLS have given children participating in assessments a sticker for participating. Therefore, the IELS plans for a nonmonetary incentive such as a sticker or a pencil.

A.10 Assurance of Confidentiality

Data security and confidentiality protection procedures have been put in place for IELS to ensure that Westat and its subcontractors comply with all privacy requirements, including:

The statement of work of this contract;
Privacy Act of 1974 (5 U.S.C. §552a);
Family Educational and Privacy Act (FERPA) of 1974 (20 U.S.C. §1232(g));
Privacy Act Regulations (34 CFR Part 5b);
Computer Security Act of 1987;
U.S.A. Patriot Act of 2001 (P.L. 107-56);
Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9573);
Confidential Information Protect and Statistical Efficiency Act of 2002;
E-Government Act of 2002, Title V, Subtitle A;
Cybersecurity Enhancement Act of 2015 (6 U.S.C. §151);
The U.S. Department of Education General Handbook for Information Technology Security General Support Systems and Major Applications Inventory Procedures (March 2005);
The U.S. Department of Education Incident Handling Procedures (February 2009);
The U.S. Department of Education, ACS Directive OM: 5-101, Contractor Employee Personnel Security Screenings;
NCES Statistical Standards; and
All new legislation that impacts the data collected through the inter-agency agreement for this study.

Furthermore, Westat will comply with the Department’s IT security policy requirements as set forth in the Handbook for Information Assurance Security Policy and related procedures and guidance, as well as IT security requirements in the Federal Information Security Management Act (FISMA), Federal Information Processing Standards (FIPS) publications, Office of Management and Budget (OMB) Circulars, and the National Institute of Standards and Technology (NIST) standards and guidance. All data products and publications will also adhere to the revised NCES Statistical Standards, as described at the website: http://nces.ed.gov/statprog/2012/.

Procedures for handling confidential aspects of the IELS study will mirror those used in other studies conducted by NCES, including signed confidentiality agreements and notarized nondisclosure affidavits obtained from all personnel who will have access to individual identifiers; personnel training regarding the meaning of confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses; protected access to computer files under the control of a single data base manager; built-in safeguards concerning status monitoring and receipt control systems; and a secured and operator-manned in-house computing facility.

Letters and other materials will be sent to parents and school administrators describing the voluntary nature of this survey. The materials sent will include a brochure that describes the study and conveys the extent to which respondents and their responses will be kept confidential (Appendices A and B):

The National Center for Education Statistics is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information you provide may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). Individuals are never identified in any reports. All reported statistics refer to the U.S. as a whole or to national subgroups.

The following statement will appear on the front cover and login page of the questionnaires:

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this voluntary survey is 1850-0936. The time required to complete this survey is estimated to average XX minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the survey. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this survey, or any comments or concerns regarding the status of your individual submission of this survey, please write to: International Early Learning Study (IELS), National Center for Education Statistics, Potomac Center Plaza, 550 12^th Street, SW, Washington, DC 20202.

OMB No. 1850-0936, Approval Expires 04/30/2020.

Data files, accompanying software, and documentation will be delivered to NCES at the end of the project. No school or individual names or addresses will be included on these files or documentation.

NCES understands the legal and ethical need to protect the privacy of the IELS respondents and has extensive experience in developing data files for release that meet the government’s requirements to protect individually identifiable data from disclosure. The contractor will conduct a thorough disclosure risk analysis of the IELS data when preparing the data files for use by researchers, in compliance with 20 U.S.C., § 9573. Schools with high disclosure risk will be identified and, to ensure that individuals may not be identified from the data files, a variety of masking strategies will be used, including: swapping data and omitting key identification variables (e.g., school name and address) from both the public- and restricted-use files (though the restricted-use file will include an NCES school ID that can be linked to other NCES databases to identify a school); omitting key identification variables such as state or zip code from the public-use file; and collapsing or developing categories for continuous variables to retain information for analytic purposes while preserving confidentiality in public-use files. IES’s Disclosure Review Board (DRB) carefully reviews all datasets prior to release to ensure that disclosure risks have been properly addressed. The IELS data will be reviewed and approved by the DRB prior to any public release.

A. 11 Sensitive Questions

The IELS does not include questions usually considered to be of a sensitive nature, such as items concerning religion, substance abuse, or sexual activity. Several items in the background questionnaires may be considered sensitive by some of the respondents, such as parents’ education and occupation and family possessions. Research indicates that the constructs these items represent are strongly correlated to academic achievement, and they have been used in number of national and international studies, including ECLS, PISA, and TIMSS. These items are considered essential for the anticipated analyses and to retain consistency in planned comparisons with the international data.

A. 12 Estimates of Burden

This request is for field test data collection and main study recruitment. Due to time overlap in activities, the approved field test recruitment activities are being carried over. Burden estimates are shown in table 1.

Recruitment activities include the time involved in a school deciding to participate, providing student listing forms, and arranging for assessment space. As shown in Part B, 30 schools will be recruited for the field test, and approximately 476 students from these 30 schools (about 16 per school) will participate in the field test (19 per school will be sampled). For the main study, NCES will assess 3,000 students from 200 schools, with an expected 16 students per school (19 sampled per school).

Some districts are known as “special handling districts” which require completion of a research application before they will allow schools under their jurisdiction to participate in a study. Based on an initial assessment of previous data collections, we have estimated the number of special handling districts in the field test (shown in table 1). Contacting special handling districts begins with updating district information based on what can be gleaned from online sources. Calls are then placed to verify the information about where to send the completed required research application forms, and, if necessary, to collect contact information for this process. During the call, inquiry is also made about the amount of time the districts spend reviewing similar research applications. The estimated number of such districts represents those with particularly detailed application forms and lengthy processes for approval. For the field test, this operation will begin in fall 2017 to allow sufficient time for special handling districts’ review processes. We will begin contacting these districts upon receiving OMB’s approval, and will continue to work with them until we receive a final response (approval or denial of request) from each school, up until October 31, 2017. For the main study, this process will also begin in fall 2017, upon approval of this clearance request.

The total response burden for districts and schools in the field test and main study recruitment is based on the following:

We estimate that there may be 4 special handling districts in the field test sample and 50 special handling districts in the main study sample – those known to require completion of a research application before they will allow schools under their jurisdiction to participate in a study. Estimated burden hours for special handling districts are included in table 1 with the provision of time for district application review. We estimate 1 hour for IRB staff of special districts to review materials and 30 minutes for IRB panels (on average made up of 6 people) to review materials.
It is estimated that it will take 10 minutes on average for school administrators to review the materials and either agree or decline to participate. It is estimated that an additional 4 hours will be needed for the provision of student rosters (including information about students for sampling, contact information for the sample students’ parents, and sample students' teachers; see Appendix B). That is, we estimate approximately 250 minutes total for each school.
For students’ parents, we estimate that it will take up to 10 minutes to review the recruitment materials and either consent or refuse to participate (on behalf of their student and themselves). The provision of student rosters and the parents’ consent forms will serve as sources for parents’ contact information, which during the data collection period can be used for nonresponse follow-up.

The total response burden for the field test data collection, including pre-assessment activities, is based on the following:

We estimate that approximately 8 hours will be needed by school coordinators for the oversight of student rosters (including contact information for the sample students’ parents and teachers; see Appendix B); the collection and delivery of teacher and parent data; and the coordination of the student assessment.
It is estimated that it will take 30 minutes on average for parents to respond to the parent/guardian survey, including instructions.
For teachers, we estimate that it will take on average 23 minutes to complete all aspects of the teacher survey (on average 23 minutes total per teacher: approximately 3 minutes for the core portion and 5 minutes per sampled child, with an average of 4 sampled children per teacher).
For students, we expect it will take 60 minutes, including instructions, to complete all sections of a tablet-based assessment.

Based on the estimated hourly rates for principals/administrators of $44.68, preschool and kindergarten teachers of $23.44, and parents of $22.71⁴, and based on 2,595 total burden hours for IELS main study recruitment and field test recruitment and data collection, the estimated total respondent burden time cost is $101,698.

Table 1. Burden estimates for IELS study

	Sample	Expected response rate	Number of respondents	Number of responses	Burden per respondent (minutes)	Total burden (hours)
Field Test
Recruitment
Special Handling Districts IRB staff	4	1.00	4	4	60	4
Special Handling Districts IRB Panel	24	1.00	24	24	30	12
Schools	35	1.00	35	35	250	146
Parent notification	560	1.00	560	560	10	93
Pre-Assessment Activity
School Coordinator	35	0.85	30	30	240	120
Data Collection
Parent Questionnaire*	560	0.85	476*	476	30	238
Teacher Questionnaire	141	0.85	120	120	23	46
Students (direct assessment)	560	0.85	476	476	60	476
Field Test Burden			773	1,249		659
Main Study
Recruitment
Special Handling Districts IRB staff	50	1.00	50	50	60	50
Special Handling Districts IRB Panel	300	1.00	300	300	30	150
Schools	235	1.00	235	235	250	979
Parent notification	3,600	1.00	3,600	3,600	10	600
Parent notification for additional sample (pending decision)	941	1.00	941	941	10	157
Main Study Recruitment Burden			5,126	5,126		1,936
Pre-Assessment Activity
School Coordinator	235	0.85	200	200	240	800
Data Collection
Parent Questionnaire*	3,600	0.85	3,060*	3,060	30	1,530
Teacher Questionnaire	941	0.85	800	800	23	307
Students (direct assessment)	3,600	0.85	3,060	3,060	60	3,060
Total Burden Requested in this Submission			5,899	6,375		2,595

Note: Total respondent burden requested in this submission includes burden associated with conducting the field test data collection, recruitment for the main study, and the carried over approved burden for field test recruitment. The previously approved field test recruitment burden is reflected in rows shaded in gray. Burden estimates for the main study data collection, which will be requested in a future submission, are reflected here in gray font. Burden estimated for student cognitive assessment, which is not subject to PRA, is reflected in italicized gray font. All burden reflected is gray font is presented here for information only purposes, and is not included in the requested burden totals.

* A subset of the respondent group shown under recruitment, not double counted in the total number of respondents.

A.13 Total Annual Cost Burden

There are no additional costs to respond beyond the time to respond.

A.14 Annualized Cost to Federal Government

The cost to the federal government for conducting initial phases of IELS, including the 2018 field test operations (preparations, data collection, and scoring) and recruitment for the 2018 main study is estimated to be $2,460,496 for a 1-year period (see table breakdown below). These figures include all direct and indirect costs (the grand total is the total value to the Government including other phases of the contract, including all optional tasks).

Components with breakdown	Estimated costs
FIELD TEST (2017)
Recruitment	736,675
Preparations (e.g., adapting instruments, sampling)	272,964
Data collection, scoring, and coding	663,598
MAIN STUDY (2017)
Recruitment	725,266
Preparations (e.g., adapting instruments, sampling)	61,993
Data collection, scoring, and coding	0
MAIN STUDY (2018)
Recruitment	591,605
Preparations (e.g., adapting instruments, sampling)	365,057
Data collection, scoring, and coding	3,631,305
Current package components	$2,460,496
Grand total	$7,048,464

A.15 Program Changes or Adjustments

The apparent increase in burden is due to the fact that the last approval was for field test recruitment only, while this request includes burden associated with field test recruitment, field test data collection, and main study recruitment.

A.16 Plans for Tabulation and Publication

The IELS field test is designed to provide a statistical review of the performance of items on the assessments and questionnaires in preparation for the main data collection. The international contractor will provide the international instruments to be used in the field test and will report to the participating countries on the results of the field test. Based on the field test results, the international contractor, with input and agreement from the participating countries, will make final revisions in the survey instruments, materials, and documents in preparation for the main study.

For the main study in 2018, an analysis of the U.S. and international data will be undertaken to provide an understanding of the U.S. national results in relation to the international results. Based on proposed analyses of the international data set by the OECD and the need for NCES to report results from the perspective of a U.S. constituency, a plan is being prepared for the statistical analysis of the U.S. national data set as compared to the international data set. Analysis of data will include examinations of student performance in relation to their international counterparts; and the relationships between student performance and contextual variables.

All reports and publications will be coordinated with the release of information from the international organizing body. Planned publications and reports for the IELS main study include the following:

General Audience Report. This report will present information on the skills and competencies of U.S. children in comparison to their international peers, written for a non-specialist, general U.S. audience. This report will present the results of analyses in a clear and non-technical way, conveying how U.S. children compare to their international peers, and what factors, if any, may be associated with the U.S. results.

Survey Operations/Technical Report. This document will detail the procedures used in the main study (e.g., sampling, recruitment, data collection, scoring, weighting, and imputation) and describe any problems encountered and the contractor’s response to them. The primary purpose of the main study survey operations/technical report is to document the steps undertaken by the U.S. in conducting and completing the study. This report will include an analysis of non-response bias, which will assess the presence and extent of bias due to nonresponse. Selected characteristics of respondent children and schools will be compared with those of non-respondent schools and students to provide information about whether and how they differ from respondents along dimensions for which we have data for the nonresponding units, as required by NCES standards.

Electronic versions of each publication are made available on the NCES website. Schedules for tabulation and publication of IELS results in the U.S. are dependent upon receiving data files from the international sponsoring organization. With this in mind, the expected data collection dates and a tentative reporting schedule are as follows:

August-November 2017	Recruitment for the field test
November-December 2017	Collect field test data
September 2017-October 2018	Recruitment for the main study
October-November 2018	Collect main study data
Early 2020	Release of main report on findings

A.17 Display OMB Expiration Date

The OMB expiration date will be displayed on all data collection materials.

A.18 Exceptions to Certification Statement

No exceptions are requested to the Certification for Paperwork Reduction Act Submissions of OMB Form 83-I.

References

Mercer, Andrew et al. 2015. “How Much Gets You How Much? Monetary Incentives and Response Rates in Household Surveys.” Public Opinion Quarterly 79:105-129.

Singer, Eleanor, and Cong Ye. 2013. “The Use and Effects of Incentives in Surveys.” ANNALS of the American Academy of Political and Social Science 645:112–41.

1 Although we expect that most children in this age range will attend kindergarten, a small percentage of children may be in pre-kindergarten or first grade. Because this is an age-based sample rather than a grade-sample, these off-grade students will be included in the study.

2 The materials that will be used in the 2018 main study will be based upon the field test materials included in this submission. This submission is designed to adequately justify the need for and overall practical utility of the full study and to present the overarching plan for all of the phases of the data collection, providing as much detail about the measures to be used as is available at the time of this submission. As part of this submission, NCES is publishing a notice in the Federal Register allowing first a 60- and then a 30-day public comment period. For the final proposal for the full study, after completion of the field test, NCES will publish a notice in the Federal Register allowing an additional 30-day public comment period on the final details of 2018 main study.

3 According to the IELS international technical standards, five-year-old children are defined as those who are five years old on the day in the middle of the administration period. Therefore, if the administration period for the main study is, for example, 8 October 2018 to 16 November 2018, then any child who was born between 1 November 2012 and 31 October 2013 are eligible for the study. The majority of these children will be in kindergarten, and a small number will be in first grade.

4 The average hourly earnings of principals/education administrators in the National Occupational and Employment Wage Estimates sponsored by the Bureau of Labor Statistics (BLS) is $44.68, of preschool and kindergarten teachers it is $23.44, and of parents it is $22.71. If mean hourly wage was not provided it was computed assuming 2,080 hours per year. Source: BLS Occupation Employment Statistics, http://data.bls.gov/oes/ data type: Occupation codes: Education Administrators, Elementary and Secondary Schools (11-9032), Preschool and Kindergarten Teachers (25-2011) and all employees (00-0000); accessed on April 11, 2017.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	2021-01-22