Part A IELS 2018 Field Test Recruitment

The International Early Learning Study (IELS), scheduled to be conducted in 2018, is a new study sponsored by the Organization for Economic Cooperation and Development (OECD), an intergovernmental organization of industrialized countries. The OECD also coordinates the Program for International Student Assessment (PISA), an assessment of 15-year-olds, and the Program for the International Assessment of Adult Competencies (PIAAC), which assesses adult skills. While PIAAC focuses on adults and PISA on students nearing the end of compulsory school, the IELS focuses on young children and their cognitive and non-cognitive skills and competencies as they transition to primary school. The IELS is designed to examine:

Children’s early learning and development in a broad range of domains, including social emotional skills as well as cognitive skills;
The relationship between children’s early learning and children’s participation in early childhood education and care (ECEC);
The role of contextual factors, including children’s individual characteristics and their home backgrounds and experiences, in promoting young children’s growth and development; and
How early learning varies across and within countries prior to beginning primary school.

In 2018, in the participating countries, including the United States, the IELS will assess nationally-representative samples of children ages 5.0-5.5 years enrolled in public and private school (enrolled in public and private kindergarten in the United States) through direct and indirect measures, and will collect contextual data about their home learning environments, ECEC histories, and demographic characteristics. The IELS will measure young children’s knowledge, skills, and competencies in both cognitive and non-cognitive domains, including language and literacy, mathematics and numeracy, executive function/self-regulation, and social emotional skills. This assessment will take place as children are transitioning to primary school and will provide data on how U.S. children entering kindergarten compare with their international peers on skills deemed important for later success.

In the United States, IELS is conducted by the National Center for Education Statistics (NCES) of the Institute of Education Sciences (IES), U.S. Department of Education. The IELS is a collaboration among the participating countries, the OECD, and an international contractor which coordinates the study across countries.

To prepare for the main study in 2018, which will be conducted from September to November 2018, the IELS countries will conduct a field test in the fall of 2017 with 300 children per nation, to evaluate newly developed assessment instruments and questionnaires and also to test the study operations. The U.S. IELS field test data collection will occur from September to October, 2017. In order to meet the international data collection schedule for the fall 2017 field test, recruiting activities must begin by May 2017. Because the IELS is a collaborative effort among many parties, U.S. must adhere to the international schedule set forth by the OECD, including the availability of final field test and main study plans and questionnaires.

This request is to conduct recruitment activities for the 2017 field test. Draft field test instruments will be added to this submission in January 2017. A request to administer the field test will begin clearance in March 2017, including the overarching plan for the 2018 main study sample¹.

Field test recruitment materials, including letters to state and district officials and school principals, and text for an IELS field test brochure and “Frequently Asked Questions” are included in Appendix A. Parental consent letters and related materials for the field test are included in Appendix B. Draft non-cognitive instruments will be provided in January 2017.

A. JUSTIFICATION

A.1 Importance of Information

As part of a continuing cycle of international studies, the United States, through NCES, participates in several international education assessments and surveys, including PISA, an assessment of 15-year-olds, and PIAAC, which assesses adult skills. IELS, sponsored by OECD, is a new addition to NCES’s international studies.

In light of the growing concerns related to international economic competitiveness, the changing face of our workplace, and the expanding international marketplace in which we trade, knowing how our students and adults compare with their peers around the world has become an even more prominent issue than ever before. Nationwide, interest in understanding what other nations are doing to further the educational achievement of their populations has increased beyond simple comparisons. Moreover, recent investments in early childhood education and care (ECEC) require a closer examination of what US children know as they enter school and how their knowledge compares to that of children in other nations with different ECEC experiences.

Data at critical points during the education career of U.S. students are used by policymakers in efforts to guide and examine the American education system. Consequently, generating comparative data about students in school and about adults in workplace and in community has become an important focus for NCES. The addition of the IELS to the slate of comparative assessments conducted by NCES will broaden what we know about U.S. students compared with those in other countries and provide comparative information about students at an earlier point in the educational process than has previously been possible.

The results from the IELS will allow national policymakers to compare the skills and competencies of children from different nations as they are readying to enter primary school and to evaluate whether changes might be needed in the U.S.’s ECEC system to promote additional skills and competencies seen in other countries. Without these types of data, U.S. policymakers are limited in their ability to gain insight into the educational performance and practices of other nations as they compare to the United States.

The study’s success is dependent on the development of reliable, valid cross-cultural measures. The goal of the IELS field test is to evaluate the direct cognitive assessment, parent questionnaire, teacher questionnaire, and study operations. The IELS field test will provide the data needed to determine the psychometric properties of items and the predictive potential of assessment and survey items so that valid, reliable, and useful assessment and survey instruments can be composed for the main study, ensuring validity in multiple countries. As the focus of the IELS field test is the analyses of the psychometric properties of the survey items and assessments, the IELS field test requires a small, but diverse field test sample, though not a nationally representative one.

A.2 Purposes and Uses of Data

Governments and the general public want solid evidence of education outcomes. In the late 1990s, the OECD launched an extensive program for producing policy-oriented and internationally comparable indicators of student achievement on a regular basis and in a timely manner. The IELS is a new addition to this program.

The IELS will assess children as they are transitioning to primary school and, as such, will provide a view of how U.S. children entering kindergarten compare with their international peers on the skills and competencies deemed important for success in school. The results will allow national policymakers to evaluate the readiness of U.S. children for formal schooling, and to analyze the relationship between constructs measured through the IELS adult questionnaires with child assessment results at national and international levels. Through the IELS, the OECD and NCES will produce two types of indicators:

Basic indicators that provide a baseline profile of the knowledge, skills, and competencies of young children; and
Contextual indicators that show how such skills relate to important demographic, social, economic, and education variables.

The instruments to be administered in IELS include the following:

Direct Child Assessment of oral language/emergent literacy, mathematics and numeracy, executive function/self-regulation, and social emotional skills (1 hour, adminsitered one-on-one)
Teacher Questionnaire, including rating forms to be completed regarding each assessed child (15 minutes); and
Parent Questionnaire, which will provide information about background characteristics, home learning environment, ECEC history, and social/emotional skills. (up to 30 minutes).

Instruments will be created using frameworks developed to define the constructs and provide specifications for distributions of items by framework dimensions. Following the field test, cognitive and non-cognitive items will be evaluated for bias and interpretation issues, following standard protocols. For the main study, the pool of items will be reduced to only include those that demonstrate validity across the participating education systems, meet the goals of content coverage to adequately measure the frameworks, and provide the desired distribution of item types.

A.3 Improved Information Technology (Reduction of Burden)

Where feasible, available technology will be used to reduce burden and improve efficiency and accuracy. The burden of recruitment on districts and schools will be minimal, with most information gathered through E-filing (see below). Districts will primarily be asked to provide confirmation of data gathered from other sources, including school universe files and district and school websites.

A communication website will be used during the field test and main study in order to provide a simple, single source of information to engage sampled schools and maintain high levels of their involvement. This portal will be used throughout the assessment cycle to inform schools, particularly school coordinators, of their tasks and to provide them with easy access to information tailored for their anticipated needs. We plan to gather student lists from participating schools electronically using an adaptation of a secure E-filing process through the web portal. E-filing is an electronic system for submitting lists of student and teacher information, including limited background information in school records. E-filing has been used successfully in NAEP for more than 10 years, and was used in TIMSS 2015 and the PISA 2012 and 2015 assessments. The E-filing system provides advantageous features such as efficiency and data quality checks.

The IELS design and procedures are prescribed internationally. The data collected by direct, one-on-one child assessments will be captured electronically by trained assessors. Teacher and Parent Questionnaires will be made available as online questionnaires and in paper format.

A.4 Efforts to Identify Duplication

A number of international comparative studies exist to measure student achievement, including the Trends in International Mathematics and Science Study (TIMSS), the Progress in International Reading Literacy Study (PIRLS) and PISA. In addition, the U.S. has been conducting its own national surveys of student achievement for more than 40 years through the National Assessment of Educational Progress (NAEP) program. The IELS is different from these comparative studies because it focuses on a younger population and collects information about early home learning experiences and early childhood education and care. None of the other studies provide comparative information about children entering primary school. The IELS also looks at a broader set of domains than these other studies in that it includes not only literacy and numeracy but also social emotional skills.

NCES’s Early Childhood Longitudinal Studies (ECLS) program provides national data on children’s developmental status at birth and at various points thereafter; children’s transitions to early care and education programs, and school; and children’s home and school experiences, growth, and learning. The ECLS program also provides data that enable researchers to analyze how a wide range of child, family, school, classroom, early care and education provider, and community characteristics relate to children’s development and to their experiences and success in school. However, ECLS does not provide a view of U.S. children in an international context as the IELS will do. The IELS will afford an understanding of how U.S. children’s early experiences compare to those of children from other nations and thus allow a better evaluation of U.S. children’s preparedness for school.

The National Household Education Surveys Program (NHES) provides descriptive data on the educational activities of the U.S. population and offers a variety of statistics on the condition of education in the U.S. The NHES surveys cover learning at all ages, from early childhood to school age through adulthood. However, NHES conducts surveys of adults in households across the U.S., asking the parents to report on their children’s competencies, it does not assess nor collects other data on a nationally-representative sample of children ages 5.0-5.5 years enrolled in public and private school (enrolled in public and private kindergarten in the U.S.).

Alternate sources for these data do not exist. This study represents the U.S. participation in an international study involving approximately 5-10 countries. The United States must collect the same information, using the same instruments and procedures, at the same time as the other nations for purposes of making valid and meaningful international comparisons. No other study in the U.S. will be using the instruments developed by the OECD, and thus no alternative sources of comparable data are available.

A.5 Minimizing Burden for Small Entities

No small entities are part of this sample. The school sample for the IELS will contain small-, medium-, and large-size schools from a wide range of school types, including private schools, and burden will be minimized wherever possible for all institutions participating in the data collection. Schools included in the field test will not be included in the main study sample. In addition, contractor staff will conduct all test administration and will assist with parental notification, sampling, and other study tasks as much as possible within each school.

A.6 Frequency of Data Collection

This request is for recruitment activities for the IELS field test, which will be conducted in fall 2017. The main study data collection will occur in the fall of 2018 and will be submitted for approval under a subsequent request. Adherence to this schedule is necessary to establish consistency in survey operations among the participating countries. A second administration of IELS has not been scheduled at this time.

A.7 Special Circumstances

The special circumstances identified in the Instructions for Supporting Statement do not apply to this study.

A.8 Consultations Outside NCES

Consultations outside NCES have been extensive and will continue throughout the life of the project. The OECD studies are developed as a cooperative enterprise involving participating countries. A steering group with representatives from participating countries guided the scoping phase and will guide reviews of study implementation. In addition, the OECD’s contractor convenes expert panels to seek input on the study design and instrument development. The majority of the consultations (outside NCES) have involved the OECD and the OECD’s international contractor. Key to these ongoing consultations are: Ms. Rowena Phair (OECD), Drs. Sacha DeVelle and Wolfram Schultz (Australian Council for Educational Research), and Ms. Julianne Henke (IEA), all of whom have extensive experience in developing and operating international education surveys.

A.9 Payments or Gifts to Respondents

Currently, the minimum response rate targets required by the OECD are 85 percent of original schools and 80 percent of students, while the NCES statistical standards require a minimum response rate target of 85 percent at the student level. Historically, these high response rates have been difficult to achieve in school-based studies. For example, U.S. failed to reach the school response rate targets for all previous PISA administrations (2000, 2003, 2006, 2009, 2012, and 2015) and had to adjust incentives upwards in the middle of the recruitment and data collection period in order to meet minimum response rate requirements. Gaining sufficient student cooperation has also been challenging. U.S. has historically met the NCES target rate of 85 percent of responding students in PISA after a great deal of effort. IELS poses more burden on schools than PISA given that the assessments will be conducted one-on-one and will include a parent questionnaire. NCES will use a multi-pronged approach to address the challenge of gaining school, student, and parent cooperation and learn as much as possible during the field test about how to achieve acceptable participation rates in the IELS main study.

Staff with experience working on the NAEP; international assessments such as PISA, PIRLS, and TIMSS; other large-scale data collections (e.g., ECLS); and with expertise in effective approaches to participant recruitment will provide input so that strategies can be identified for achieving high response rates and serve as an ongoing source of ideas and feedback. NCES will also offer incentives, a strategy which has been effective in other international studies. The proposed amounts are described below and are based on the amounts used in the other international and/or the ECLS studies.

Schools. In order to meet the minimum school response rates mandated by the OECD, and to thank the school for accommodating the disruption, schools participating in IELS will receive $200.

School coordinators. The role of the school coordinator is critical for the success of the study. The coordinator is expected to coordinate logistics with the data collection contractor; supply a list of eligible students for sampling to the data collection contractor; communicate with teachers, students, and parents about the study to encourage participation; and assist the test administrator in arranging for make-up sessions as needed. To encourage their participation and extending their best effort for the study, each school coordinator will be offered $200.

Teachers: The teacher of each assessed student will be offered approximately $40 to complete the Teacher Questionnaire, which includes providing information about each assessed child. This amount is based on the amount offered to ECLS teachers ($7 for each student) with the assumption that each IELS teacher will provide information on an average of six students.

Parents: One parent or guardian of each participating student will be offered $20 to complete the Parent Questionnaire, which includes providing information on the study child, the household learning environment, and the study child’s ECEC history.

Students: There is no monetary incentive in the IELS for students, but prior NCES studies such as the ECLS have given children participating in assessments a sticker for participating. Therefore, the IELS plans for a nonmonetary incentive such as a sticker or a pencil.

A.10 Assurance of Confidentiality

Procedures for handling confidential aspects of the IELS study will mirror those used in other studies conducted by NCES. The plan for maintaining confidentiality will include signed confidentiality agreements and notarized nondisclosure affidavits obtained from all personnel who will have access to individual identifiers. Also included in the plan is personnel training regarding the meaning of confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses; protected access to computer files under the control of a single data base manager; built-in safeguards concerning status monitoring and receipt control systems; and a secured and operator-manned in-house computing facility.

Letters and other materials will be sent to parents and school administrators describing the voluntary nature of this survey. The materials sent will include a brochure that describes the study and conveys the extent to which respondents and their responses will be kept confidential (Appendices A and B). The following statement will appear on the front cover and login page of the questionnaires:

The National Center for Education Statistics (NCES), within the U.S. Department of Education, conducts the International Early Learning Study (IELS) in the United States as authorized by the Education Sciences Reform Act of 2002 (20 U.S.C., § 9543). Your responses are protected by federal statute (20 U.S.C., § 9573) and may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law.

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this voluntary survey is 1850-xxxx. The time required to complete this survey is estimated to average XX minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the survey. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this survey, or any comments or concerns regarding the status of your individual submission of this survey, please write to: International Early Learning Study (IELS), National Center for Education Statistics, Potomac Center Plaza, 550 12^th Street, SW, Washington, DC 20202.

OMB No. 1850-XXXX, Approval Expires xx/xx/2020.

Data files, accompanying software, and documentation will be delivered to NCES at the end of the project. No school or individual names or addresses will be included on these files or documentation.

NCES understands the legal and ethical need to protect the privacy of the IELS respondents and has extensive experience in developing data files for release that meet the government’s requirements to protect individually identifiable data from disclosure. The contractor will conduct a thorough disclosure risk analysis of the IELS data when preparing the data files for use by researchers, in compliance with 20 U.S.C., § 9573. Schools with high disclosure risk will be identified and, to ensure that individuals may not be identified from the data files, a variety of masking strategies will be used, including: swapping data and omitting key identification variables (e.g., school name and address) from both the public- and restricted-use files (though the restricted-use file will include an NCES school ID that can be linked to other NCES databases to identify a school); omitting key identification variables such as state or zip code from the public-use file; and collapsing or developing categories for continuous variables to retain information for analytic purposes while preserving confidentiality in public-use files. IES’s Disclosure Review Board (DRB) carefully reviews all datasets prior to release to ensure that disclosure risks have been properly addressed. The IELS data will be reviewed and approved by the DRB prior to any public release.

A. 11 Sensitive Questions

The IELS does not include questions usually considered to be of a sensitive nature, such as items concerning religion, substance abuse, or sexual activity. Several items in the background questionnaires may be considered sensitive by some of the respondents, such as parents’ education and occupation and family possessions. Research indicates that the constructs these items represent are strongly correlated to academic achievement, and they have been used in number of national and international studies, including ECLS, PISA, and TIMSS. These items are considered essential for the anticipated analyses and to retain consistency in planned comparisons with the international data.

A. 12 Estimates of Burden

This request is for field test recruitment. Burden estimates are shown in table 1. Recruitment activities include the time involved in a school deciding to participate, providing student listing forms, and arranging for assessment space.

As shown in Part B, 30 schools will be recruited for the field test, and approximately 476 students from these 30 schools (16 per school) will participate in the field test (19 per school will be sampled).

Some districts are known as “special handling districts” which require completion of a research application before they will allow schools under their jurisdiction to participate in a study. Based on an initial assessment of previous data collections, we have estimated the number of special handling districts in the field test (shown in table 1). Contacting special handling districts begins with updating district information based on what can be gleaned from online sources. Calls are then placed to verify the information about where to send the completed required research application forms, and, if necessary, to collect contact information for this process. During the call, inquiry is also made about the amount of time the districts spend reviewing similar research applications. The estimated number of such districts represents those with particularly detailed application forms and lengthy processes for approval. This operation should begin in spring 2017 to allow sufficient time for special handling districts’ review processes. We will begin contacting these districts upon receiving OMB’s approval, and continue to work with them until we receive a final response (approval or denial of request) up until September 30, 2017.

The total response burden for districts and schools in the field test is based on the following:

We estimate that there may be 4 special handling districts in the field test sample – those known to require completion of a research application before they will allow schools under their jurisdiction to participate in a study. Estimated burden hours for special handling districts are included in table 1 with the provision of time for district application review. We estimate 1 hour for IRB staff of special districts to review materials and 30 minutes for IRB panels (on average made up of 6 people) to review materials.
It is estimated that it will take 10 minutes on average for school administrators to review the materials and either agree or decline to participate. It is estimated that an additional 4 hours will be needed for the provision of student rosters (including information about students for sampling, contact information for the sample students’ parents, and sample students' teachers; see Appendix B). That is, we estimate approximately 250 minutes total for each school.
For students’ parents, we estimate that it will take up to 10 minutes to review the recruitment materials and either consent or refuse to participate (on behalf of their student and themselves). The provision of student rosters and the parents’ consent forms will serve as sources for parents’ contact information, which during the data collection period can be used for nonresponse follow-up.

Based on the estimated hourly rates for principals/administrators and parents of $44.68 and $22.71, respectively², and based on 234 total burden hours for IELS recruitment, the estimated total burden time cost is $8,412.

Table 1. Burden estimates for IELS field test

	Sample	Expected response rate	Number of respondents	Number of responses	Burden per respondent (minutes)	Total burden (hours)
Recruitment
Special Handling Districts IRB staff	4	1.00	4	4	60	4.0
Special Handling Districts IRB Panel	24	1.00	24	24	30	12.0
Schools	30	1.00	30	30	250	125
Parent notification	560	1.00	560	560	10	93
Pre-Assessment Activity
School Coordinator	30	1.00	30	30	240	120
Field Test Data Collection
Parent Questionnaire	560	0.85	476	476	30	238
Teacher Questionnaire	63	0.85	54	54	15	14
Students (direct assessment)	560	0.85	476	476	60	476
Total Burden Requested in this Submission			618	618	0	234

Note: Total burden requested in this submission includes burden associated with recruiting schools and contacting districts and parents for the IELS field test. Burden that will be requested in a future submission is presented here for information only purposes (in gray font). It is estimated that the teacher questionnaire will be completed for about 6 students per teacher.

A.13 Total Annual Cost Burden

There are no additional costs to respond beyond the time to respond.

A.14 Annualized Cost to Federal Government

The cost to the Federal Government for conducting IELS is still to be determined. The contract for the national data collection has not been awarded. This information will be provided in January 2017.

A.15 Program Changes or Adjustments

This is a new data collection effort by the U.S., and as such shows all burden as new.

A.16 Plans for Tabulation and Publication

The IELS field test is designed to provide a statistical review of the performance of items on the assessments and questionnaires in preparation for the main data collection. The international contractor will provide the international instruments to be used in the field test and will report to the participating countries on the results of the field test. Based on the field test results, the international contractor, with input and agreement from the participating countries, will make final revisions in the survey instruments, materials, and documents in preparation for the main study.

For the main study in 2018, an analysis of the U.S. and international data will be undertaken to provide an understanding of the U.S. national results in relation to the international results. Based on proposed analyses of the international data set by the OECD and the need for NCES to report results from the perspective of a U.S. constituency, a plan is being prepared for the statistical analysis of the U.S. national data set as compared to the international data set. Analysis of data will include examinations of student performance in relation to their international counterparts; and the relationships between student performance and contextual variables.

All reports and publications will be coordinated with the release of information from the international organizing body. Planned publications and reports for the IELS main study include the following:

General Audience Report. This report will present information on the skills and competencies of U.S. children in comparison to their international peers, written for a non-specialist, general U.S. audience. This report will present the results of analyses in a clear and non-technical way, conveying how U.S. children compare to their international peers, and what factors, if any, may be associated with the U.S. results.

Survey Operations/Technical Report. This document will detail the procedures used in the main study (e.g., sampling, recruitment, data collection, scoring, weighting, and imputation) and describe any problems encountered and the contractor’s response to them. The primary purpose of the main study survey operations/technical report is to document the steps undertaken by the U.S. in conducting and completing the study. This report will include an analysis of non-response bias, which will assess the presence and extent of bias due to nonresponse. Selected characteristics of respondent children and schools will be compared with those of non-respondent schools and students to provide information about whether and how they differ from respondents along dimensions for which we have data for the nonresponding units, as required by NCES standards.

Electronic versions of each publication are made available on the NCES website. Schedules for tabulation and publication of IELS results in the U.S. are dependent upon receiving data files from the international sponsoring organization. With this in mind, the expected data collection dates and a tentative reporting schedule are as follows:

September-October 2017	Collect field test data
September-November 2018	Collect main study data
Early 2020	Release of main report on findings

A.17 Display OMB Expiration Date

The OMB expiration date will be displayed on all data collection materials.

A.18 Exceptions to Certification Statement

No exceptions are requested to the "Certification for Paperwork Reduction Act Submissions" of OMB Form 83-I.

1 The materials that will be used in the 2018 main study will be based upon the field test materials included in this and the subsequent IELS field test submission. Additionally, this and the subsequent submission are designed to adequately justify the need for and overall practical utility of the full study and to present the overarching plan for all of the phases of the data collection, providing as much detail about the measures to be used as is available at the time of this submission. As part of this submission, NCES is publishing a notice in the Federal Register allowing first a 60- and then a 30-day public comment period. For the final proposal for the full study, after the field test, NCES will publish a notice in the Federal Register allowing an additional 30-day public comment period on the final details of 2018 main study.

2 The average hourly earnings of principals/education administrators in the May 2015 National Occupational and Employment Wage Estimates sponsored by the Bureau of Labor Statistics (BLS) is $44.68, and of parents is $22.71. If mean hourly wage was not provided it was computed assuming 2,080 hours per year. Source: BLS Occupation Employment Statistics, http://data.bls.gov/oes/ data type: Occupation codes: Education Administrators, Elementary and Secondary Schools (11-9032) and all employees (00-0000); accessed on Nov. 20, 2016.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	2021-01-23