Vol 1 ICILS 2018 Pretest Rev.

International Computer and Information Literacy Study (ICILS 2018) Pretest

OMB# 1850-0803 v.207

Volume 1

Submitted by:

National Center for Education Statistics (NCES)

Institute of Education Sciences (IES)

U.S. Department of Education

Washington, DC

August 2017

revised September 2017

Table of Contents

1) Submittal-Related Information 3

2) Background and Study Rationale 3

3) Recruitment and Data Collection 3

4) Consultations outside the Agency 5

5) Justification for Sensitive Questions 5

6) Paying Respondents 5

7) Assurance of Confidentiality 6

8) Estimate of Hourly Burden 6

9) Cost to Federal Government 6

10) Project Schedule 7

1. Submittal-Related Information

This material is being submitted under the generic National Center for Education Statistics (NCES) clearance agreement (OMB# 1850-0803), which provides for NCES to conduct various procedures (such as pilot tests, cognitive interviews, and usability studies) to test new methodologies, question types, or delivery methods to improve survey and assessment instruments and procedures.

2. Background and Study Rationale

The International Computer and Information Literacy Study (ICILS) is a computer-based international assessment of eighth-grade students’ computer and information literacy (CIL) skills. ICILS was first administered internationally in 2013 in 21 education systems and will be administered again in 2018. The United States will participate for the first time in the 2018 administration. U.S. participation in this study will provide data on students’ skills and experience using technology to investigate, create, and communicate, and will provide a comparison of U.S. student performance and technology access and use with those of the international peers. This study will also allow the U.S. to begin monitoring the progress of its students compared to that of other nations and to provide data on factors that may influence student computer and information literacy skills. The data collected through ICILS will provide valuable information with which to understand the nature and extent of the “digital divide” and has the potential to inform understanding of the relationship between technology skills and experience and student performance in other core subject areas.

ICILS is coordinated by the International Association for the Evaluation of Educational Achievement (IEA), an international collective of research organizations and government agencies that create the assessment framework, assessment, and background questionnaires. The IEA decides and agrees upon a common set of standards and procedures for collecting and reporting ICILS data, and defines the study timeline, all of which must be followed by all participating countries. As a result, ICILS is able to provide a reliable and comparable measure of student skills in participating countries. In the U.S., the National Center for Education Statistics (NCES) conducts this study and works with the IEA and Westat to ensure proper implementation of the study and adoption of practices in adherence to the IEA’s standards. Participation in ICILS will allow NCES to meet its mandate of acquiring and disseminating data on educational activities and student achievement in the United States compared with foreign nations [The Educational Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C. §9543].

This request is to conduct a real-world test of the IEA ICILS assessment systems with students, allowing the systems to be tested in the manner that will be used in the main study to help identify system issues during the software development process. The rationale for this study is based on lessons learned and issues encountered by students during the 2017 ICILS field test. In addition, the IEA will be using a different assessment system in the ICILS main study than in the 2017 field test, and it needs to be evaluated. It is believed that students use and interact with the systems differently than adult QC testers. Conducting pretesting and QC process with students should allow us to identify any issues and address them prior to the operational use of the system in the main study in spring 2018.

The pretesting will be conducted with students from grade 8 and will be held in a simulated classroom after two preliminary/draft versions of the IEA ICILS assessment systems are produced. Two pretesting events will take place during fall 2017, with event sessions lasting approximately 175 minutes each.

3. Recruitment and Data Collection

Recruitment and Sample Characteristics

EurekaFacts, under contract with Westat, will recruit up to 30 students for each session of ICILS pretesting. Each ICILS session will be exclusively composed of 8^th graders, and there will be a maximum of two sessions. Each session will be held at the EurekaFacts facility in Rockville, Maryland, most likely on a Saturday.

EurekaFacts will recruit participants for the pretesting study from the District of Columbia, Maryland, Virginia, and West Virginia. Although the sample will include a mix of student characteristics, the results will not explicitly measure differences by those characteristics. Students will be recruited to obtain the following criteria:

• A mix of race/ethnicity (Black, Asian, White, Hispanic);

• A mix of socioeconomic background; and

• A mix of urban/suburban/rural areas.

While EurekaFacts will use various outreach methods (see Appendices) to recruit students to participate, the bulk of the recruitment will be conducted by telephone and be based on their acquisition of targeted mailing lists containing residential addresses and landline telephone listings. EurekaFacts will also use a participant recruitment strategy that integrates multiple outreach methods and resources such as newspaper and internet ads, community organizations (e.g., Boys and Girls Clubs, Parent-Teacher Associations), and mass media recruitment (e.g., postings on the EurekaFacts website).

Interested students will be screened (see Appendices E-F) to ensure that they meet the criteria for participation in the pretesting study (i.e., the students are from the targeted demographic groups outlined above and their parents/legal guardians have given consent). When recruiting participants, EurekaFacts staff will speak to the parent/legal guardian of the interested minor before starting the screening process. During this communication, the parent/legal guardian will be informed about the objectives, purpose, and participation requirements of the data collection effort as well as the activities that it entails. After confirming that a participant is qualified, willing, and available to participate in this study, he or she will receive a confirmation email/letter and phone call. Written informed parental consent (see Appendix H) will be obtained for all respondents who are interested in participating in the data collection efforts. Shortly after each of the two preliminary/draft IEA ICILS assessment versions are released, a pretesting event with students will be held in a simulated classroom.

All events will have one session lasting approximately 175 minutes each, and it will be structured as follows:

• During each session, each student will be asked to take the assessment under standard ICILS assessment conditions (approximately 165 minutes, which includes the directions/tutorial, the assessment and the student questionnaire). Westat will administer the session using standard procedures. Students will take the full assessment, including the cognitive items and the student questionnaire.¹

• A group debrief (up to 10 minutes) will be conducted to solicit feedback from the students. See Volume 2 for the debriefing script.

As part of the assessment administration in all event sessions, students will take a set of survey questions. The maximum time for the student questionnaire component is 30 minutes (included in the 175-minute time estimation for each session). Volume 2 includes the student survey items that were administered in the field test.² As the items for the 2018 ICILS are finalized, the final questions will be included in the ICILS assessment system for pretesting. As such, session one will use the field test versions of the questionnaires and session two is likely to use the final questionnaires. If, as anticipated, questionnaire items for session two will differ from those that will be used in session one (as included here), we will submit to OMB for review a new clearance request under OMB# 1850-0803 (please note that the estimated required response burden for session two is included in this request; if additional burden will be needed at the time of session two request submission, we will explicitly request the additional burden, otherwise we will not ask for new burden amount).

Data Collection Process

Normal data collection will be enabled by the ICILS systems, and any errors generated will be collected automatically by the system. Note that student responses will not be scored. In addition to the ICILS systems recording information, administrators and observers from NCES, Westat, and/or EurekaFacts will monitor the assessments and record notes detailing any issues encountered by the students, as well as what the students were doing at the time each issue occurred. In addition, observers may ask individual students for clarification of the actions he or she took prior to an issue or error occurring. For example, observers may ask questions such as, “What is the error?”; “What was the last thing you saw before the error?”; “What were you expecting to happen?”; or “What did you do right before the error happened?”. Understanding and documenting what caused the system error is necessary in order to have enough information for staff to replicate the error and develop a fix for it.

The sessions will be audio and/or video recorded to capture information regarding any student actions that resulted in system errors or issues.

4. Consultations outside the agency

Westat is the contractor for ICILS. Westat will provide the laptops and tablets for the students’ use and carry out the pretesting study.

The majority of the consultations involve the Australian Council for Educational Research (ACER), the international study center for ICILS. ACER staff are responsible for designing and implementing the study in close cooperation with the IEA Secretariat, the IEA Data Processing and Research Center, and the national centers of participating countries. Key staff from ACER include: Dr. John Ainley (project coordinator), Mr. Julian Fraillon (research director); and Dr. Wolfram Schulz (assessment coordinator), all of whom have extensive experience in developing and operating international education surveys (especially related to ICILS).

EurekaFacts is located in Rockville, Maryland. It is an established for-profit research and consulting firm, offering facilities, tools, and staff to collect and analyze both qualitative and quantitative data. EurekaFacts is working as a subcontractor for ETS to recruit participants and provide the facilities to be used for the study. In addition, EurekaFacts staff may assist in administering and/or observing some sessions.

5. Justification for Sensitive Questions

Throughout the item and debriefing question development processes, effort has been made to avoid asking for information that might be considered sensitive or offensive.

6. Paying Respondents

To encourage participation and thank students for their time and effort, a $50 gift card from a major credit card will be offered to each participating student. If a parent or legal guardian brings their student to and from the testing site, they will also receive a $50 gift card to thank them for their time and effort in transporting their child. Similar NAEP studies (e.g. OMB# 1850-0803 v. 199) have offered $25 to the student participant and $25 to the parent/guardian for sessions that last approximately 1.5 hours. Given that this research study requires about twice the amount of time (approximately 3 hours per session), $50 is being offered to both the student and parent/guardian in order to aid in recruitment and gain cooperation in the study.

7. Assurance of Confidentiality

The study will not retain any personally identifiable information. Prior to the start of the study, students will be notified that their participation is voluntary. As part of the study, students will be notified that the information they provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Before each pretesting session is administered, for each participant a written consent will be obtained from their parent/legal guardian. Participants will be assigned a unique student identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form. The consent forms, which include the participant name, will be separated from the participant interview files, secured for the duration of the study, and will be destroyed after the final report is released. Pretesting activities may be recorded using audio and/or screen capture technology. The only identification included on the files will be the participant ID. The recorded files will be secured for the duration of the study and will be destroyed after the final report is completed.

8. Estimate of Hourly Burden

The estimated burden for recruitment assumes attrition throughout the process.³ In all events, each student will participate in one session for a total of 175 minutes. Table 1 details the estimated burden.

Table 1. Estimate of Hourly Burden

Respondent	Number of respondents	Number of responses	Hours per respondent	Total hours
Parent or Legal Guardian for Student Recruitment
Initial contact	268	268	0.05	14
Follow-up via phone	134*	134	0.15	20
Consent & confirmation	67*	67	0.15	10
Recruitment Totals	268	469	-	44
Participation (Pretesting)
Students in ICILS Events	60	60	2.91	175
Total	328	529	-	219

* Subset of initial contact group

Note: numbers have been rounded and therefore may affect totals

9. Cost to federal government

The total cost of the study is $90,235 as detailed in Table 2.

Table 2: Estimate of Costs

Activity	Provider	Estimated Cost
Recruiting students and providing facilities for the study	EurekaFacts	$54,353
Administering the study	Westat	$35,882
Total		$90,235

10. Project Schedule

The schedule for this study, including all activities, will begin in September 2017 through December 2017 as detailed in Table 3.

Table 3: Project Schedule

Activity	Dates
Recruitment, pretesting, data collection, analysis and final report	September–December 2017

1 Draft content may be used in the earlier versions.

2 The final items will consist of those selected for ICILS 2018 administration (to be submitted for OMB review in fall 2017: OMB #1850-0929 v.6-7). Volume 2 provides the questionnaires approved as part of the 1850-0929 v.1-5 field test submittal.

3 Assumptions for approximate attrition rates are 50 percent from initial contact to follow-up, 50 percent from follow-up to confirmation, and 90 percent from confirmation to participation.