Justification

Volume I:

Fast Response Survey System (FRSS) 109: Public School Teachers’ Use of Technology for School and Homework Assignments – Pretest

OMB# 1850-0803 v.226

March 2018

National Center for Education Statistics (NCES)

U.S. Department of Education

Justification

The National Center for Education Statistics (NCES), within the Institute of Education Sciences (IES) of the U.S. Department of Education (ED), requests OMB approval under the NCES system clearance for Cognitive, Pilot and Field Test studies (OMB #1850-0803) to conduct two rounds of pretest calls for the Fast Response Survey System (FRSS) survey #109 on public school teachers’ use of technology for school and homework assignments. NCES is conducting this FRSS survey as part of the IES response to the request in the Every Student Succeeds Act of 2015 (ESSA 2015, 20 U.S.C. §6301 et seq.) to provide information about the educational impact of access to digital learning resources (DLR) such as computers and the Internet outside of the classroom.

The expanding use of technology affects the lives of students both inside and outside the classroom. For this reason, the role of technology in education is an increasingly important area of research. While access to technology can provide valuable learning opportunities to students, technology by itself does not guarantee successful outcomes. Schools and teachers play an important role in successfully integrating technology into teaching and learning.

ESSA provides guidance to state governments on how to receive supplemental federal funding for public education. As part of the ESSA legislation, IES is required to produce a report on the educational impact of access to digital learning resources (DLR) outside of the classroom. Specifically, ESSA requests that IES conduct research in the five areas listed below.

1. An analysis of student habits related to digital learning resources outside of the classroom, including the location and types of devices and technologies that students use for educational purposes.

2. An identification of the barriers students face in accessing DLR outside of the classroom.

3. A description of the challenges that students who lack home internet access face, including challenges related to student participation and engagement in the classroom and homework completion.

4. An analysis of how the barriers and challenges such students face impact the instructional practice of educators.

5. A description of the ways in which state education agencies, local education agencies, schools, and other entities, including partnerships of such entities, have developed effective means to address the barriers and challenges students face in accessing DLR outside of the classroom.

ESSA refers to the term “digital learning” as “any instructional practice that effectively uses technology to strengthen a student’s learning experience and encompasses a wide spectrum of tools and practices” (20 U.S.C. §7112 Definitions). However, for this survey, the main focus of digital learning resources will be computers (laptops, desktops, tablets), smartphones, and Internet access.

The purpose of this FRSS 109 survey is to collect nationally representative data from public school teachers about their use of computer and the Internet for school and homework assignments, and how their knowledge and beliefs about their students’ access to computers and the Internet outside the classroom affect the assignments they give. The survey will focus on information that can best be provided by teachers from their unique perspective and direct interaction with students. Findings from the FRSS 109 study will provide insights on the types and availability of DLRs outside of the classroom, and will contribute to IES reports on the educational impact of access to DLRs outside the classroom.

The purpose of the FRSS 109 pretest is to identify and correct any potential issues with the content and format of the survey before conducting full-scale implementation, and to ensure that the survey captures the intended meaning of the questions and minimizes the burden imposed on respondents. The pretest involves asking respondents to complete the draft survey and participate in a telephone debriefing. The request to conduct FRSS 109 full-scale preliminary activities, including securing research approval from special contact school districts beginning in April 2018 and obtaining teacher lists from sampled schools beginning in August 2018, was approved by OMB under the OMB clearance for FRSS teacher technology surveys (OMB#1850-0857 v.2-3). The request to conduct the full-scale FRSS 109 data collection will be submitted to OMB in the Spring of 2018 (OMB#1850-0857). NCES is authorized to conduct FRSS by the Education Sciences Reform Act of 2002 (ERSA 2002, 20 U.S.C. §9543).

Design

Overview of Survey Development

NCES has contracted Westat to prepare for and administer FRSS 109, including development of the survey instrument. FRSS has established procedures for developing short surveys on a wide variety of topics. The techniques used to shape the survey design on FRSS 109 include input from the NCES Quality Review Board (QRB), several rounds of feasibility calls, and up to two pretests.

The current survey reflects lessons learned from topics and issues developed and refined during three rounds of feasibility calls with public school teachers during fall and winter 2017–18. Because this is a new survey topic, the initial feasibility calls used an open-ended interview guide rather than a questionnaire. As rounds of feasibility calls progressed, respondents were asked to review but not complete draft questionnaire items and ultimately a draft questionnaire. Conducting multiple rounds of feasibility calls informed us about public school teachers’ use of computers and the Internet for school and homework assignments, and what they know about their students’ access to computers and the Internet outside of school. During the pretest calls, we will test all questions on the questionnaire and obtain estimates of the respondent time required to complete the survey.

NCES Review and Consultations Outside of Agency

The NCES QRB members reviewed a draft list of questionnaire and discussion topics prior to our request for OMB clearance of the feasibility calls (OMB# 1850-0803 v.202). Revisions were made to the list of topics based on input from the reviewers, and the list was used to develop an interview guide for the feasibility calls. As rounds of feasibility calls progressed, draft questionnaire items and then a draft questionnaire were developed. Following the last round of feasibility calls, the QRB members reviewed the draft questionnaire, and revisions were made based on their input. The revised version will be used for the pretest and is provided in this submission. In addition to staff from NCES’s Statistical Standards group, the Annual Reports group, and each of the three Divisions, the QRB also included staff from ED’s Office of Educational Technology (OET) and the Policies and Programs Studies Service of the Office of Planning, Evaluation, and Policy Development (OPEPD); the U.S. Commerce Department’s National Telecommunication and Information Administration; and the IBM Center for The Business of Government. The QRB members for this survey are listed below:

Rafi Goldberg, National Telecommunications and Information Administration, Commerce

Bernadette Adams, Office of Educational Technology

Andrew Abrams, OPEPD (Policy and Program Studies Service)

Dan Chenok, the IBM Center for The Business of Government

Halima Adenegan, NCES (Assessment Division)

Jamie Deaton, NCES (Assessment Division, NAEP)

John Ralph, NCES (Annual Reports and Information)

Tom Snyder, NCES (Annual Reports and Information)

Mark Glander, NCES (Administrative Records Division, CCD)

Chris Chapman, NCES (Sample Surveys Division, Longitudinal Branch)

Maura Spiegelman, NCES (Sample Surveys Division, Cross-sectional Surveys Branch)

Marilyn Seastrom, NCES (Statistical Standards and Data Confidentiality)

Kashka Kubzdela, NCES (Statistical Standards and Data Confidentiality)

Sample, Burden, and Cost

In this submission, we are requesting approval to conduct up to two rounds of pretest calls to test the revised questionnaire, with 15 or fewer public school teachers around the nation in each round. Teachers will be recruited to participate in feasibility calls based on various school characteristics including level (elementary or secondary), size, urbanicity (locale), and geographic region. Respondents will be recruited by email and telephone and will be identified as a regular self-contained classroom teacher at the elementary level or a teacher of a core academic subject at the secondary level.

Telephone interviewers will recruit participants for the pretest calls using the recruitment script in Attachment 1. Interviewers will schedule an appointment to complete the pretest calls with cooperating teachers. Following telephone recruitment, interviewers will email a cover letter and draft questionnaire to the participating teachers (as discussed below in the Data Collection Instrument section). Participants will be asked to review, complete, and fax back the paper and pencil questionnaire, and will be scheduled to participate in a telephone debriefing.

In order to recruit 15 respondents per round, we anticipate contacting 45 public schools per round (Table 1). On average, recruitment calls with respondents who agree to participate in the pretest calls are expected to take about 10 minutes to explain the purpose of the call and set up an appointment to discuss the survey; all other recruitment calls are expected to take about 3 minutes. The questionnaire and the pretest debriefing interview are each expected to take respondents approximately 30 minutes to complete, for a total of one hour per respondent. The total estimated burden is approximately 40 hours for two rounds of pretest calls. We anticipate that the estimated cost to the federal government will be approximately $6,000 for each round of pretest calls.

Table 1. Maximum burden time for each of up to two rounds of pretest calls for FRSS 109

Respondents	Number of Respondents	Number of Responses1	Burden Hours per Respondent	Total Burden Hours
Recruitment – Teachers not participating in the pretest	30	30	0.05	2
Recruitment – Teachers participating in the pretest	15	15	0.17	3
Pretest questionnaire completion and debriefing	15	15	1.00	15
Total per round	45	60	-	20
Total for two rounds	90	120	-	40

¹ Counts each response (e.g., recruitment and debriefing interview are counted separately even when they are with the same respondents).

Data Collection Instrument

For each round of pretest calls, a cover letter and draft questionnaire will be emailed to each participating teacher. The cover letter and questionnaire appear in attachments 2 and 3. The cover letter thanks the respondent for agreeing to participate in the pretest, introduces the purpose and content of the survey, indicates that participation is voluntary, indicates that respondents should complete the questionnaire and fax it back to Westat, includes questions for respondents to consider while completing the questionnaire, and provides contact information should any questions arise before the scheduled discussion with the survey manager. On the cover letter and on the cover of the survey, respondents are assured that their participation is voluntary and their answers may not be disclosed or used in identifiable form for any other purpose except as required by law. The public law is cited on the cover letter and the front page of the survey. The materials for the second pretest round (if it takes place) will be similar, except the survey instrument will include the modifications that will result from the first round. The current draft of the questionnaire is discussed below.

Timeline

Pretest activities are expected to begin as soon as approval is received from OMB. It is anticipated that recruitment, debriefing, write up of the memorandum summarizing the results, and survey revisions will take approximately 4-6 weeks for each pretest round.

Questionnaire

This survey is designed to collect information to help NCES respond to the request in the Every Student Succeeds Act of 2015 to provide information about the educational impact of access to digital learning resources such as computers and the Internet outside of the classroom. The survey focuses on information that can best be provided by teachers from their perspective and based on their direct interaction with students. The questions have been developed and refined through three sets of feasibility calls and two reviews by the NCES QRB for this survey.

Instructions and Definitions

The cover of the questionnaire provides instructions indicating that teachers should respond based on the students they are teaching during the current school year. The cover also includes a definition of computers to be used by respondents throughout the questionnaire. To help ensure that respondents read this important definition, it is repeated in a box above question 1.

District- or School-Provided Computers Or Hotspots Available to Students (Questions 1 through 6)

Question 1 asks whether the teacher’s students have a district- or school-provided computer that the student takes home on a long-term basis. Because the Congressional request was for information about student access to technology at home, the key part of this question is whether the student takes the computer home. Some districts and schools have 1:1 programs where each student is provided a computer, but the program does not allow students to take the computers home. Such programs do not provide students with district- or school-provided technology access outside of the school.

If a teacher answers yes to question 1, question 2 asks whether there are some students who are not able to take their district- or school-provided computer home. During the feasibility calls, teachers reported that some students were not able to take their computers home because parents did not give permission for the student to bring the computer home, the student had previously damaged or lost a computer, or the computer would not be safe at home or during the trip home.

If a teacher answers no to question 1, question 3 asks whether the school allows students to borrow computers to take home on a short-term basis. Loaner programs provide at least some student access to technology at home using district- or school-provided computers. If a teacher answers no to question 1, question 4 asks whether students can access school computers outside of class time. Teachers reported that schools tried to provide as much access as possible to school computers so that students who did not have computers at home would be able to use school resources to complete technology-based assignments.

Question 5 asks whether the school has an additional academic period for all students during the school day when students can use computers and the Internet to work on school assignments from other classes. Teachers reported that this was another way in which schools tried to provide as much access as possible to school computers and the Internet so that students who did not have it at home would be able to use school resources to complete technology-based assignments.

Question 6 asks whether the district or school provides mobile hotspots for students to take home. Providing these hotspots is another way that some schools provide Internet access to students who do not have that access at home.

Teacher Knowledge about Student Access Outside of School to Computers and the Internet for Doing School Assignments (Questions 7 through 10)

Question 7 asks how knowledgeable teachers are about their students’ access to computers and the Internet for doing school assignments at home. Knowledge about computers and Internet access differed, with teachers often reporting they were more knowledgeable about computers than Internet access. Question 8 asks teachers how they find out information about their students’ access to computers and the Internet at home. The response items are based on teacher reports across the feasibility calls.

Question 9 asks teachers about the extent to which their students use various locations for computer and/or Internet access to work on school assignments. The response items are based on teacher reports across the feasibility calls.

Access and Availability of Computers at Home (Questions 10 through 12)

Question 10 asks teachers to estimate the percentage of their students with access to a computer at home. Responses from teachers during the feasibility calls indicated substantial variability in the reported access to computers at home. Even teachers whose students had district- or school-provided computers that they could take home (Q1=yes) often reported that less than 100 percent of their students had access to a computer at home because some students were not able to take their computers home.

Question 11 asks teachers to estimate how available those home computers are for students to use for school assignments. Discussions with teachers indicated that home computers were often shared computers, and that there was variability in how available those home computers were for students to use for school assignments. Teachers extensively used the text after the availability labels, which was developed over the course of the feasibility calls, to guide their responses.

Question 12 asks teachers to estimate how likely it is that the home computer has reliable Internet access from home. Discussions with teachers indicated variability in reliable Internet access for computers, particularly for students who lived in remote rural or mountainous areas, or whose families could not afford Internet service for a computer (although they may have it for a phone).

Access, Usefulness, and Availability of a Smartphone at Home (Questions 13 through 16)

Question 13 asks teachers to estimate the percentage of their students with access to a smartphone at home. Early discussions with teachers indicated that the two devices to which students would be likely to have access are computers and smartphones. Responses from teachers during the feasibility calls indicated variability in the reported access to a smartphone at home, particularly for younger students.

Question 14 asks teachers how useful those smartphones are for completing the assignments that they give their students. Discussions with teachers indicated that some teachers designed some of their assignments in ways that could be easily done on a smartphone (e.g., vocabulary review or practice quizzes), while other teachers indicated that smartphones would not be useful for the types of assignments they give to students.

Question 15 asks teachers to estimate how available those smartphones are for students to use for school assignments. Discussions with teachers indicated that smartphone availability varied, and that particularly for younger students, the smartphone may belong to a parent.

Question 16 asks teachers to estimate how likely it is that the smartphone has reliable Internet access from home.

Technology and Homework (Questions 17 through 21)

The box above question 17 provides definitions for teachers to use while responding to this set of questions.

Question 17 asks teachers how much influence their students’ access to technology and the Internet outside of school has on the homework assignments they give them. As indicated in the box above the question, technology is defined to include devices such as computers and smartphones, software such as computer programs and digital apps, and the Internet. Some teachers mentioned that students’ home computers or devices did not have the same software or digital apps available that were used on the school devices, such as tablets. This influenced the homework assignments given by these teachers.

Question 18 asks how often teachers assign technology-based homework to their students, and question 19 asks how often teachers assign homework of any type to their students. The question about assigning homework of any type (Q19) is asked to provide context for assigning technology-based homework. Both of these questions showed considerable variability during the feasibility calls, ranging from “never” to “often” for both categories of homework. Early feasibility calls identified age of the students and subject matter taught as influences on these questions. During early feasibility calls, questions 18 and 19 were in the opposite order, with the question about all types of homework first. We found that some (but not all) teachers were then responding to technology-based homework as a subset of all homework. For example, a teacher might say that they rarely assigned homework, and then say that they sometimes assigned technology-based homework. Discussion indicated that these teachers meant that, on the rare occasions when they did assign homework, it was sometimes technology-based homework. However, this is not the base that we would like teachers to use for technology-based homework, and we made two changes to address this situation. The main change was to ask about technology-based homework first. The second change was to the wording of question 19, to ask about “any type” of homework, and to add the parenthetical instruction to include both technology-based and non-technology-based homework.

Question 20 asks teachers about the extent to which their students have difficulty completing technology-based homework because they are not familiar with how to use technology. Feasibility calls showed variability in responses to this question. Teachers from schools with a large number of migrant students or sizeable refugee populations especially indicated that familiarity with technology had a large effect.

Question 21 asks teachers about the extent to which they provide various types of assistance for doing technology-based homework to their students who have limited access to technology and the Internet outside of school. The types of assistance are those reported by teachers during several rounds of feasibility calls.

Student Preparedness for Online or Computerized Assessments

Question 22 asks teachers how prepared their students are to use the technology required for online or computerized assessments given by their district or school. This question was added during the feasibility calls because some teachers mentioned that their students had trouble with these types of assessments, particularly those that included extended written responses, because they did not have much experience using a keyboard and mouse, although they might have experience using touch screens.

Teacher Instructional Setting

Questions 23 through 25 ask about the teacher’s instructional setting – self-contained classroom or departmentalized instruction, subjects taught (for departmentalized instruction), and grades currently taught at the school. These variables will be used for analysis.

Attachment 1: FRSS 109 Pretest Call Recruitment Script

FRSS 109: Teachers’ Use of Technology For School and Homework Assignments

Hello, my name is __________________.

I am calling from Westat on behalf of the National Center for Education Statistics, within the U.S. Department of Education, regarding a survey on teachers’ use of technology and the Internet. We are developing this survey, and would like help from one of your teachers to review our draft questionnaire to ensure that it is clear and easy to complete. This input will help us develop a survey that makes sense to teachers and which they can easily answer.

Can you give me the name and school email address of a few of your teachers who might be interested in helping us with this important survey? We would like to talk to [a regular self-contained classroom teacher {FOR ELEMENTARY SCHOOLS} / a teacher who teaches English, history or social studies, math, or science {FOR SECONDARY SCHOOLS}].

Is this phone number the best number on which to reach these teachers? When would be the best time to call?

SPEAKING TO A TEACHER

Hello, my name is __________________.

I’m calling from Westat on behalf of the National Center for Education Statistics, within the U.S. Department of Education, regarding a survey on teachers’ use of technology and the Internet. We would like your help in reviewing our draft questionnaire to ensure that it is clear and easy to complete. Specifically, we would like you to complete the questionnaire and fax it back to us, and then obtain your comments about the survey by telephone. This is a short questionnaire that should take about 30 minutes to complete. [The call will take about 30 minutes.]

Your input, while voluntary, will be essential in developing a questionnaire that is relevant. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

1. May I have your email address to send you the survey materials?

2. We ask that you complete the questionnaire and fax it back to us before you talk to the survey manager. When would be a good time for the survey manager, Cindy Gray, to call you to discuss the survey and obtain your comments? How about [SUGGEST A TIME]. [Just to be sure, you are in the [Eastern, Central, Mountain, Pacific] time zone?]

3. What is the best telephone number at which the survey manager can reach you?

Thank you. Your insights will be very helpful.

Attachment 2: FRSS 109 Cover Letter

U.S. Department of Education  Institute of Education Sciences  National Center for Education Statistics

[Date] 2018

Dear Pretest Participant,

Thank you for agreeing to participate in the pretest of the survey on teachers’ use of technology for school and homework assignments. Westat, a research company located in Rockville, Maryland, is administering this survey for the National Center for Education Statistics (NCES), within the U. S. Department of Education. The survey is part of the Fast Response Survey System (FRSS), which is charged with collecting information on important and emerging issues related to education. This survey is being conducted as part of the NCES response to the request in the Every Student Succeeds Act of 2015 to provide information about the educational impact of access to digital learning resources such as computers and the Internet outside of the classroom. The survey focuses on information that can best be provided by teachers from their perspective and direct interaction with students.

We are currently conducting the pretest of the survey. Your input, while voluntary, will be essential in developing a survey that is relevant, clear, and not overly burdensome to respondents. Your participation is very important because your comments will improve the survey before the actual data collection begins. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

We ask that you (1) complete the questionnaire and fax it to Westat; (2) keep track of the time you spend filling out the questionnaire; (3) write down any comments about the questionnaire; and (4) discuss your comments with me by telephone at the time scheduled. Please keep the following questions in mind as you complete the questionnaire:

1. Are the instructions and definitions clear and helpful to you?

2. Are the survey questions clear and easy to interpret?

3. Would you have access to the information necessary for answering these questions?

4. How long did it take you to answer the survey?

Please fax the completed questionnaire to me. My toll-free fax number is 1-800-254-0984. My colleague and I will call you at the scheduled time to get your feedback on the questionnaire and discuss any comments or suggestion you may have about the study. In the meantime, feel free to call me at Westat’s toll-free number,
800-937-8281, ext. 4336, if you have any questions. You may also reach me by email at [email protected].

Thank you for your much-needed assistance!

Sincerely,

Cindy Gray

Westat Survey Manager