Justification - Volume 1 FRSS 110 Ed Technology in Public Schools Pretest

Volume I:

Fast Response Survey System (FRSS) 110: Use of Educational Technology for Instruction in Public Schools – Pretest

OMB# 1850-0803 v. 254

August 2019

National Center for Education Statistics (NCES)

U.S. Department of Education

Justification

The National Center for Education Statistics (NCES), within the Institute of Education Sciences (IES) of the U.S. Department of Education (ED), requests OMB approval under the NCES system clearance for Cognitive, Pilot and Field Test studies (OMB #1850-0803) to conduct two rounds of pretest calls for Fast Response Survey System (FRSS) survey #110 on use of educational technology for instruction in public schools. The Office of Educational Technology, ED, requested that NCES conduct this FRSS survey.

The expanding use of technology affects the lives of students both inside and outside the classroom. For this reason, the role of technology in education is an increasingly important area of research. While access to technology can provide valuable learning opportunities to students, technology by itself does not guarantee successful outcomes. Schools and teachers play an important role in successfully integrating technology into teaching and learning. The purpose of this FRSS 110 survey is to collect nationally representative data from public schools about their use of educational technology for instruction.

The purpose of the FRSS 110 pretest is to identify and correct any potential issues with the content and format of the survey before conducting full-scale implementation, and to ensure that the survey captures the intended meaning of the questions and minimizes the burden imposed on respondents. The pretest involves asking respondents to complete the draft survey and participate in a telephone debriefing. The request to conduct FRSS 110 preliminary activities, including securing research approval from special contact school districts beginning in May 2019, was approved in May 2019 (OMB#1850-0733 v.35). The request to conduct the full-scale survey will be submitted in fall 2019 under the OMB generic clearance for quick response surveys (OMB#1850-0733 v.36). NCES is authorized to conduct FRSS by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543).

Design

Overview of Survey Development

NCES has contracted Westat to prepare and administer FRSS 110, including development of the survey instrument. FRSS has established procedures for developing short surveys on a wide variety of topics. The techniques used to shape the survey design on FRSS 110 include input from the NCES Quality Review Board (QRB), several rounds of feasibility calls, and up to two pretests.

The current survey reflects lessons learned from topics and issues developed and refined during three rounds of feasibility calls with public school principals or their designees during winter through summer 2019. Because this is a new survey topic, the initial feasibility calls used an open-ended interview guide rather than a questionnaire. As rounds of feasibility calls progressed, respondents were asked to review but not complete draft questionnaire items and ultimately a draft questionnaire. Conducting multiple rounds of feasibility calls informed us about the use of educational technology for instruction in public schools. During the pretest calls, we will test all questions on the questionnaire and obtain estimates of the respondent time required to complete the survey.

NCES Review and Consultations Outside of Agency

The NCES QRB members reviewed a draft list of questionnaire and discussion topics prior to our request for the feasibility calls (OMB#1850-0803 v.244). Revisions were made to the list of topics based on input from the reviewers, and the list was used to develop an interview guide for the feasibility calls. As rounds of feasibility calls progressed, draft questionnaire items and then a draft questionnaire were developed. Following the last round of feasibility calls, the QRB members reviewed the draft questionnaire, and revisions were made based on their input. The revised version will be used for the pretest and is provided in this submission. In addition to staff from NCES’s Statistical Standards group, the Annual Reports group, and the Administrative Records and Sample Surveys Divisions, the QRB also includes staff from ED’s Office of Educational Technology (OET) and four education technology organizations. The QRB members for this survey are listed below:

Bernadette Adams, Office of Educational Technology

Jake Steele, Office of Educational Technology

Tom Snyder, NCES (Annual Reports and Information)

Ross Santy, NCES (Administrative Records Division)

Chris Chapman, NCES (Sample Surveys Division,

Kashka Kubzdela, NCES (Statistical Standards and Data Confidentiality)

Maria Worthen, iNACOL

Christina Luke, Digital Promise

Susan Bearden, CoSN

Ji Soo Song, ISTE

Sample, Burden, and Cost

In this submission, we are requesting approval to conduct up to two rounds of pretest calls with members of the target population to test the draft questionnaire, with 15 or fewer respondents per round. School personnel will be recruited to participate in pretest calls based on various school characteristics including level (elementary, middle, or high school), size, urbanicity (locale), and geographic region. Respondents will be recruited by email and telephone and will be identified as the person at the school most knowledgeable about the use of educational technology for instruction.

Telephone interviewers will recruit participants for the pretest calls using the recruitment script in Attachment 1. Interviewers will schedule an appointment to complete the pretest calls with cooperating school personnel. Following telephone recruitment, interviewers will email a cover letter and draft questionnaire to the participating school staff member (as discussed below in the Data Collection Approach section). In order to recruit 15 respondents per round, we anticipate contacting 45 public schools per round (Table 1). On average, recruitment calls with respondents who agree to participate in the feasibility calls are expected to take about 10 minutes to explain the purpose of the call and set up an appointment to discuss the survey; all other recruitment calls are expected to take about 3 minutes. The questionnaire and the pretest debriefing interview are each expected to take respondents approximately 30 minutes to complete, for a total of one hour per respondent. The total estimated burden is approximately 40 hours for two rounds of pretest calls.

Table 1. Estimated maximum burden time for up to three rounds of feasibility calls for FRSS 110

Respondents	Number of Respondents	Number of Responses1	Burden Hours per Respondent	Total Burden Hours
Recruitment: Schools not participating in the pretest	30	30	0.05	2
Recruitment: Schools participating in the pretest	15	15	0.17	3
Pretest questionnaire completion and debriefing	15	15	1.0	15
Total per round	45	60	--	20
Total for two rounds	90	120	--	40

¹ Counts each response (e.g., recruitment and debriefing interview are counted separately even when they are with the same respondents).

Cost to the Federal Government

The total estimated cost to the federal government for FRSS 110 is $793,665 (see Table 2). We anticipate that the cost to the federal government will be approximately $6,000 for each round of pretest calls, putting the total estimated cost to the federal government for the two rounds of pretest calls requested in this submission at $12,000.

Table 2. Cost to the federal government for all FRSS 110 activities

FRSS 110 Activity	Cost
Feasibility calls (3 rounds x $6,000 per round)	$18,000
Pretest calls (2 rounds x $6,000 per round)	$12,000
Total survey development work (2 clearances under 1850-0803)	$30,000
Preliminary activities/research applications for national data collection	$200,000
Remaining activities national data collection (data collection, analysis and reporting, data files, project management)	$563,665
Total national data collection (2 clearances under $1850-0733)	$763,665
Total cost for FRSS 110	$793,665

Timeline

Pretest activities are expected to begin in September 2019, as soon as approval is received from OMB. It is anticipated that recruitment, debriefing, write up of the memorandum summarizing the results, and survey revisions will take approximately 4-6 weeks for each pretest round.

Data Collection Approach

For each round of pretest calls, a cover letter and draft questionnaire will be emailed to each participant. The cover letter and questionnaire appear in attachments 2 and 3. The cover letter thanks the respondent for agreeing to participate in the pretest, introduces the purpose and content of the survey, indicates that participation is voluntary, indicates that respondents should complete the questionnaire and fax it back to Westat, includes questions for respondents to consider while completing the questionnaire, and provides contact information should any questions arise before the scheduled discussion with the survey manager. On the cover letter and on the cover of the survey, respondents are assured that their participation is voluntary and their answers may not be disclosed or used in identifiable form for any other purpose except as required by law. The public law is cited on the cover letter and the front page of the survey. The materials for the second pretest round (if it takes place) will be similar, except the survey instrument will include the modifications that will result from the first round. The survey questions have been developed and refined through three sets of feasibility calls and two reviews by the NCES QRB for this survey.

The cover of the questionnaire indicates that the survey is designed to be completed by the principal or the person most knowledgeable about the use of educational technology for instruction at the school indicated on the front of the survey. The cover also includes a definition of computers to be used by respondents while completing the questionnaire. To help ensure that respondents read this important definition, it is repeated in a box above question 1, and called out in questions 1 and 4.

Research Questions and Questionnaire

The purpose of the FRSS survey is to collect information from public schools about their use of educational technology for instruction, with a focus on equity and access. The questionnaire collects information about the following research questions.

• What is the availability and location of computers for student instructional use?

• Are there school-provided computers assigned to individual students, and are students allowed to take these computers home?

• What is the quality of the instructional computers and software at the school, and to what extent do the computers meet the school’s instructional needs?

• How easy is it for teachers at the school to find enough computers to use with their students?

• How reliable is the Internet connection in the instructional areas of the school?

• How much flexibility do school-level leaders have in determining which types and how much educational technology is purchased for the school?

• How much flexibility do school-level leaders have in determining which types and how much professional development in educational technology is provided for the school?

• Does the school allow students to borrow computers to take home on a short-term basis?

• Does the school provide mobile hotspots or web-enabled devices with paid data plans for students to take home for Internet access?

• To what extent are various types of online resources used for instruction at the school?

• For what types of classroom activities do teachers at the school use educational technology?

• What type of professional development in educational technology is provided to teachers at the school?

• What type of staff work with the teachers at the school to integrate educational technology into instruction?

• How is student learning affected by the use of educational technology in the instructional program at the school?

• What challenges are faced by teachers in the school in using educational technology for instruction?

Attachment 1: FRSS 110 Pretest Call Recruitment Script

FRSS 110: Use of Educational Technology for Instruction in Public Schools

Pretest Call Recruitment Script

Hello, my name is __________________.

I am calling from Westat on behalf of the National Center for Education Statistics, within the U.S. Department of Education, regarding a survey on school use of educational technology for instruction. I would like to ask the school-level person who is most familiar with education technology use at the school to provide us with information about how your school uses technology for instruction.

Who is the person at your school who is most knowledgeable about how your school uses educational technology for instruction?

(This is often the school principal or a school-level education technology coordinator.)

May I please speak to that person?

CONNECTED TO SCHOOL–LEVEL PERSON MOST FAMILIAR WITH EDUCATIONAL TECHNOLOGY USE FOR INSTRUCTION AT THE SCHOOL.

Hello, my name is __________________.

I’m calling from Westat on behalf of the National Center for Education Statistics, within the U.S. Department of Education, regarding a survey on use of educational technology for instruction in public schools. We would like your help in reviewing our draft questionnaire to ensure that it is clear and easy to complete. Specifically, we would like you to complete the questionnaire and fax or email it back to us, and then participate in a brief telephone call with us to provide your comments about the survey. This is a short questionnaire that should take about 30 minutes to complete. (The call will take about 30 minutes.)

Your input, while voluntary, will be essential in developing a questionnaire that is relevant. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

1. May I have your email address to send you the survey materials?

2. We ask that you complete the questionnaire and fax or email it back to use before you talk to the survey managers. When would be a good time for the survey managers, Laurie Lewis and Cindy Gray, to call you to discuss the survey and obtain your comments? How about [SUGGEST A TIME]. [Just to be sure, you are in the [Eastern, Central, Mountain, Pacific] time zone?]

3. What is the best telephone number at which the survey managers can reach you?

Thank you. Your insights will be very helpful.

Attachment 2: FRSS 110 Cover Letter

U.S. Department of Education  Institute of Education Sciences  National Center for Education Statistics (NCES)

[Date] 2019

Dear Pretest Participant,

Thank you for agreeing to participate in the pretest of the survey on use of educational technology for instruction. The purpose of the survey is to collect information from public schools about their use of educational technology for instruction, with a focus on equity and access. The study is being conducted by the National Center for Education Statistics (NCES) in partnership with the Office of Educational Technology, U. S. Department of Education. NCES is authorized to conduct this survey by the Education Sciences Reform Act of 2002 (ERSA 2002, 20 U.S.C. §9543). Westat, a research company located in Rockville, Maryland, is administering this survey on behalf of NCES, using the Fast Response Survey System (FRSS).

We are currently conducting the pretest of the survey. Your input, while voluntary, will be essential in developing a survey that is relevant, clear, and not overly burdensome to respondents. Your participation is very important because your comments will improve the survey before the actual data collection begins. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

We ask that you (1) complete the questionnaire and fax it to Westat; (2) keep track of the time you spend filling out the questionnaire; (3) write down any comments about the questionnaire; and (4) discuss your comments with me by telephone at the time scheduled. Please keep the following questions in mind as you complete the questionnaire:

1. Are the instructions and definitions clear and helpful to you?

2. Are the survey questions clear and easy to interpret?

3. Would you have access to the information necessary for answering these questions?

4. How long did it take you to answer the survey?

Please fax the completed questionnaire to me. My toll-free fax number is 1-800-254-0984. My colleague and I will call you at the scheduled time to get your feedback on the questionnaire and discuss any comments or suggestions you may have about the study. In the meantime, feel free to call me at Westat’s toll-free number,
800-937-8281, ext. 8284, if you have any questions. You may also reach me by email at [email protected].

Thank you for your much-needed assistance!

Sincerely,

Laurie Lewis

Westat Survey Manager