Revision Memo

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: July 7, 2022

TO: Robert Sivinski, OMB

THROUGH: Carrie Clarady, NCES

FROM: Linda Hamilton, NCES

SUBJECT: International Computer and Information Literacy Study (ICILS) 2023 Main Study Questionnaire Revision (OMB# 1850-0929 v.10)

The International Computer and Information Literacy Study (ICILS) is a computer-based international assessment of eighth-grade students’ computer and information literacy (CIL) skills and computational thinking (CT). ICILS was first administered internationally in 2013 in 21 education systems and again in 2018 in 14 education systems. The third cycle will be administered in 2023. The United States participated for the first time in the 2018 administration. U.S. participation in the 2023 study will allow for trend comparisons of data on students’ skills and experience using technology to investigate, create, and communicate, and will provide a comparison of U.S. student performance and technology access and use with those of the international peers. The data collected through ICILS will provide valuable information with which to understand the nature and extent of the “digital divide” and has the potential to inform understanding of the factors that may influence student computer and information literacy skills and the relationship between technology skills and experience and student performance in other core subject areas. The global coronavirus pandemic made the study even more pertinent and policy-relevant because the pandemic forced so many U.S. schools to rely much more on the technologies that are at the heart of what ICILS measures and brought to the fore how much American students’ ability to navigate those technologies is critical to their success.

ICILS is organized by the International Association for the Evaluation of Educational Achievement (IEA), an international collective of research organizations and government agencies that create the assessment framework, assessment, and background questionnaires. The IEA decides and agrees upon a common set of standards and procedures for collecting and reporting ICILS data, and defines the study timeline, all of which must be followed by all participating countries. As a result, ICILS is able to provide a reliable and comparable measure of student skills in participating countries. In the U.S., the National Center for Education Statistics (NCES) conducts this study and works with the IEA and RTI International to ensure proper implementation of the 2023 study and adoption of practices in adherence to the IEA’s standards. Participation in ICILS will allow NCES to meet its mandate of acquiring and disseminating data on educational activities and student achievement in the United States compared with foreign nations [The Educational Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C. § 9543].

ICILS 2023 will be a computer-based student assessment, administered using a customized assessment platform on Chromebooks, a tablet-like computer with an attached keyboard. In preparation for the ICILS 2023 main study, all countries were asked to implement a field test between March 1 and April 15, 2022.¹ The purpose of the ICILS field test is to evaluate new assessment items and background questions, to ensure practices that promote low exclusion rates, and to ensure that classroom and student sampling procedures proposed for the main study are successful. The U.S. ICILS main study will be conducted from March through May 2023 and will involve a nationally-representative sample of at least 3,000 eighth-grade students from a minimum of 150 schools.

Because ICILS is a collaborative effort among many parties, the United States must adhere to the international schedule set forth by the IEA, including the availability of final field test and main study plans as well as draft and final questionnaires. In order to meet the international data collection schedule and to align with recruitment for other NCES studies (e.g., TIMSS), approval for the main study sampling, recruitment, and data collection activities were approved in April 2022 (OMB# 1850-0929 v9).

This request is for approval for the (1) revised study revision timeline; (2) updated screen shots of the study portal; (3) changes to the main study contact materials; and (4) addition of COVID-related items in the questionnaires. A final change memo will be submitted in December 2022 with the final adapted main study questionnaire. Changes are described below and are included in the supporting documentation.

New text is provided in red below and text removed since the prior approved package is shown with strikethrough text.

1. Revised study revision timeline in Part A

The request to This request is to conduct the ICILS 2023 main study data recruitment and collection (OMB# 1850-0929 v.9) was approved in April 2022. The materials to be used in the main study are based upon those that were proposed most recently in October 2021. This That submission describesd the overarching plan for all phases of the data collection for the 2023 main study, including the Non-Response Bias Analysis Plan in Appendix C.

Because ICILS is a collaborative effort among many parties, the United States must adhere to the international schedule set forth by the IEA, including the availability of draft and final questionnaires. For this the previous submission, the content of Appendix B washas been updated with the latest approved drafts of the 2023 ICILS field study questionnaires. After feedback from interested parties, we chose to add If there are new items related to the effects of the COVID pandemic added to the main study questionnaires; this submission , we will submitseeks approval for a questionnaire revision along with 30 days of public commentin August 2022. We expect the final U.S. versions of the main study questionnaires to be approved by the IEA in late December 2022, at which point we will submit them to OMB for approval as a change request. We expect the final versions to be very similar to the drafts included in this submission. Lastly, the Non-Response Bias Analysis Plan has been added in Appendix C.

2. Updated screen shots of the study portal

The prior submission used the TIMSS screen shots of the portal and some materials as a placeholder for ICILS. Updated screen shots were provided for appendices A13 (p. 23), A15 (p. 28), A17 (p. 31), A18a (p. 35), A20 (p. 40), A31 (p. 53), A32 (p. 54), A33 (p. 56), A34 (p. 57), and A35 (p. 58). The content of the ICILS pages were approved in prior packages and only the graphics were updated, with the exception of one new screenshot in Appendix A17 which provides fields to record the contact information for an ICT coordinator, as follows:

3. Changes to the main study contact materials

Throughout Appendix A, updates were made to replace placeholders with the study’s toll-free number, email address, and website URL.

Changes were made to the state, district, and school contacting letters, as follows:

Appendix A1. ICILS 223 State Recruitment Letter:

The United States is participating in an important international study: the International Computer and Information Literacy Study 2023 (ICILS). The ICILS study is developed internationally as a response to the increasing use of information and communication (ICT) in modern society and the need for citizens to fevelop relevant skills to participate effectively in the digiital age. administered once every 5 years and provides crucial information on U.S. students’ skills and experience using technology compared with that of our peer countries around the world. Schools in your state have been selected to represent the nation in ICILS 2023.

The continuation of U.S. participation allows for the study of past and current education policies that have shaped the role of technologies in teaching and learning. In addition to an assessment of students’ computer and information skills, ICILS provides information on factors such as teachers’ experiences and school resources in support of computer and information literacy education. This high quality, internationally comparative trend data study provides key information to inform education policy discussions while identifying existing educational inequalities across socioeconomic, racial/ethnic, and gender lines.

Schools in your state have been selected to participate in the ICILS 2023. We realize these are unprecedented times, and we are very aware of the strain the coronavirus pandemic has put on schools. Results from the ICILS indicated that only about 50% of 8th grade teachers used information and communication technologies (ICT) in teaching In 2018 U.S. 8^th-graders scored above the international average for computer and information literacy (CIL), and on par with the international average in computational thinking (CT). With the dramatic shift to virtual learning in 2020, the results of the 2023 ICILS are more important than ever in understanding the role of technology in education within the U.S. and globally. We are reaching out to you now with the expectation ICILS will take place in on campus classroom settings in spring 2023.

Appendix A4. ICILS 2023 District Recruitment Letter

The United States is participating in an important international study: International Computer and Information Literacy Study 2023 (ICILS). The ICILS study is administered once every 5 years and provides crucial information on U.S. students’ skills and experience using technology compared with that of our peer countries around the world. In addition to an assessment of students’ computer and information skills, ICILS provides information on factors such as teachers’ experiences and school resources in support of computer and information literacy education. ICILS is developed internationally as a response to the increasing use of information and communication technology (ICT) in modern society and the need for citizens to develop relevant skills to participate effectively in the digital age. The continuation of U.S. participation allows for the study of past and current education. This high quality, internationally comparative trend data study provides key information to inform education policy discussions while identifying existing educational inequalities across socioeconomic, racial/ethnic, and gender lines.

One or more schools in your <district/diocese> has been selected to participate in the ICILS 2023 to occur in spring 2023. ICILS is the only international student assessment that measures students’ digital skills at grade 8 in countries around the world, including the United States, since 2013. Results from the 2018 ICILS indicated that only about 505 of 8th grade teachers used information and communication technologies (ICT) in teaching. In 2018 U.S. 8th-graders scored above the international average for computer and information literacy (CIL), and on par with the international average in computational thinking (CT). With the dramatic shift to virtual learning in 2020, the results of the 2023 ICILS are more important than ever in understanding the role of technology in education within the U.S. and globally. We are reaching out to you now with the expectation that ICILS will take place in on-campus classroom settings in spring 2023.

. . . .

Within a few days, study Study materials will be sent to selected schools in your district.

. . . .

Within the next few days, a A representative of RTI International will contact the following school or schools in your <district/diocese> that have been selected for ICILS 2023 to discuss study logistics and answer any questions you may have.

Appendix A5. ICILS 2023 Main Study School Letter

The United States is participating in an important international study: International Computer and Information Literacy Study 2023 (ICILS). The ICILS study is administered once every 5 years and provides crucial information on U.S. students’ skills and experience using technology compared with that of our peer countries around the world. In addition to an assessment of students’ computer and information skills, ICILS provides information on factors such as teachers’ experiences and school resources in support of computer and information literacy education.

I would like to invite your school to participate in the ICILS 2023 Main Study to occur in spring 2023. Please put this assessment window date on your 2022-2023 school calendar. The purpose of the Main Study is to measure international differences in students’ computer and information literacy (CIL) and computational thinking (CT). Students will take the assessment digitally on computers supplied by the ICILS 2023 Team.

Within the next few weeks, our contractor, RTI International, will reach out to your school to discuss your school’s participation and answer any questions. Your district has been notified of your selection. A representative from ICILS will contact you to discuss your participation. For any participating schools, we would like to obtain the name of a school coordinator (SC) and work with you to identify an assessment date coordinate the study logistics at your school. As a token of appreciation for participating in ICILS, your school will receive $200. Students who participate in ICILS will receive a small gift as a token of appreciation. Your school’s ICILS School Coordinator (the staff person you designate to liaise with ICILS project staff) will receive $100 for his or her time and effort coordinating ICILS activities. Sampled teachers and the ICT coordinator will receive $25 for completing a brief questionnaire their questionnaires.

4. Addition of COVID-related items in the questionnaires

Student questionnaire

These additional items appear in Appendix B on pages 31-32.

1.7 Your school experience with COVID-19

Q32. In the last three school years (including this year), did you ever take part in remote learning for more than a week because of the following reasons?

Remote learning is any time you you attended school from home or somewhere else outside of school.

Do not count the time that your school was scheduled to be closed for school holiday or vacations.

If you changed schools during the past three years, please count the time across all schools you attended.

If took part in remote learning multiple different times, please count all times together.

(Please mark one choice in each row)

	No	Yes, up to 1 month	Yes, more than 1 month and up to 3 months	Yes, more than 3 months and up to 6 months	Yes, more than 6 months and up to 12 months	Yes, more than 12 months	I don’t know
Because of COVID-19
For another reason (such as a natural disaster, strikes or demonstrations, or air pollution)

Q33. How confident do you feel about doing the following things if your school had to close for more than a week due to COVID-19 or another reason (e.g., a natural disaster, strikes or demonstrations, air pollution)?

(Please mark one choice in each row)

	Not at all confident	Not very confident	Confident	Very confident
Finding learning resources on my own
Planning when to do school work on my own
Motivating myself to do school work
Focusing on school work without reminders
Completing school work independently
Assessing my progress with learning

Teacher questionnaire

These additional items appear in Appendix B on pages 65-66.

Q29. So far this school year, how confident do you feel about addressing gaps between students’ knowledge/skills and achievement standards that may have occurred due to the COVID-19 outbreak school closures?

(Please mark only one choice)

• Not at all confident

• Not confident

• Somewhat confident

• Quite confident

• Extremely confident

Q30. So far this school year, how often have you taken the following measures to address gaps in learning that may have occurred due to the COVID-19 outbreak school closures?

(Please mark one circle in each row)

	Not applicable	Never	About once or twice a month	About once or twice a week	Every day or almost every day
Diagnostic assessments to evaluate gaps between students’ knowledge/skills and achievement standards
Remedial measures to reduce gaps between students’ knowledge/skills and achievement standards
Remedial measures with a special focus on students with disabilities
Remedial measures with a special focus on English language learners

Q31. Do you think that you would be able to do the following things while teaching in a remote learning format?

Remote learning refers to students learning only from home or a location other than the school building. Please only consider remote learning that occurs for more than a week.

(Please mark one choice in each row)

	I definitely can’t	I probably can’t	Maybe	I probably can	I definitely can
Create materials to engage students in remote learning (e.g., prepare daily or weekly instructional packets, record videos or screencasts)
Conduct a remote learning lesson with students in real-time (e.g., via phone, video conferencing, online chat, online learning platforms)
Provide feedback to students in a remote learning format (e.g., via phone, email, virtual office hours)
Develop strategies to help students who are having difficulties mastering content in their remote learning

Principal questionnaire

This additional item appears in Appendix B on page 122.

Q29. For this school year, were the following actions taken to prepare for teaching in a remote learning format?

Remote learning refers to students learning only from home or a location other than the school building. Please only consider remote learning that occurs for more than a week.

(Please mark one choice in each row)

	Yes, before the school year	Yes, during this school year	Yes, before and during this school year	No
Training of teachers and staff on the use of digital applications for conducting remote learning (e.g., Google Classroom, Zoom)
Training of students and/or their parents/guardians on the use of digital devices and applications for remote learning (e.g., Google Classroom, Zoom)
Preparing digital resources for students’ remote learning (e.g., curating existing resources, designing new resources)
Preparing paper-based resources for students’ remote learning (e.g., curating existing resources, designing new resources)
Adapting existing curriculum plans and resources for remote learning
Planning additional instructional support for students with disabilities (SD)
Planning additional instructional support for English language learners (ELLs)

1 In October 2021, NCES submitted, and OMB approved a separate package for the ICILS 2023 Pilot Field Test (OMB# 1850-0803 v.304), which can be inspected at https://www.reginfo.gov/public/do/PRAViewIC?ref_nbr=201903-1850-001&icID=249841. Although this submission refers to research activities associated with the field test, in this submission NCES is not requesting review or approval of the procedures, burden, budget, or instruments for the ICILS 2023 Pilot Field Test.