NAEP Teacher and School Online Questionnaires Feedback 2014

National Assessment of Educational Progress

Supporting Statement

&

Appendices

2014 Teacher and School Feedback

Regarding Online Questionnaire Process

OMB# 1880-0542

(Generic Clearance of Customer Service Satisfaction Collections)

2/14/14

Table of Contents

1) Submittal-Related Information 2

2) Study Purpose, Design, Content, and Data Collection Process 2

3) Consultants outside the Agency 3

4) Assurance of Confidentiality 3

5) Estimate of Hourly Burden 3

6) Cost to Federal Government 4

7) Project Schedule 4

Appendix A: Notification Email to Respondents 5

Appendix B: Survey of Respondents that Did Not Finish Online Questionnaire 6

Appendix C: Survey of Respondents that Did Not Start Online Questionnaire 8

Appendix D: Survey of Respondents that Completed Hardcopy Questionnaire 10

1. Submittal-Related Information

This material is being submitted under the generic U.S. Department of Education clearance agreement (OMB # 1880-0542), which provides for the Department to carry out collections of routine customer feedback.

The National Assessment of Educational Progress (NAEP) is a federally authorized survey of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, civics, geography, economics, and the arts. NAEP is administered by the National Center for Education Statistics (NCES), part of the Institute for Education Sciences, in the U.S. Department of Education.

NCES proposes to conduct a brief interview of a selection of teachers and principals regarding their experiences with the NAEP 2014 online contextual questionnaires, with a focus on partial or non-completion.

2. Study Purpose, Design, Content, and Data Collection Process

NAEP is designed to provide achievement and education contextual information about the nation, geographic regions, states, and select large districts and groups of students. In addition to assessing subject-area achievement, NAEP collects information that serves to fulfill reporting requirements of federal legislation and to provide a context for reporting student performance.

The school questionnaire, completed by the principal or another school official, is used to gather information concerning school administration, staffing patterns, curriculum, and student services. The teacher questionnaires, completed by the subject-specific teachers of assessed students, are used to gather information concerning years of teaching experience, frequency of assignments, and use of teaching materials.

Prior to 2014, NAEP teacher and school questionnaires were offered both online and in hard copy format, but the latter was used more frequently. In 2014, in an effort to reduce operational costs and improve data quality, NAEP provided school and teacher questionnaires online only. Given concerns with technology access by all schools, an alternate option was made available in which the respondent could print out the questionnaires and complete a hardcopy of the form in order to maintain high response rates.

The interviews proposed in this study will determine why some teachers and principals chose to respond using a hardcopy questionnaire, as well as why some chose not to start the online questionnaire and others started the online questionnaire but did not complete it. This information will help NCES evaluate how to proceed with online teacher and schools questionnaires in future NAEP assessments.

The interviews have been designed with open-ended questions that will probe why respondents ultimately chose not to complete the 2014 online questionnaire. Data collection will specifically focus on why the respondents did not use or complete the online questionnaire, determine any technical issues with accessing the online questionnaire, and some of the reasons why completing a hardcopy of the questionnaire was their preference. The interviews will also provide respondents with an opportunity to suggest ways in which NAEP might make the online questionnaire easier to complete.

Three categories of both school and teacher questionnaire respondents will be identified to participate in the interviews, those who:

• Did not start the online questionnaire;

• Started but did not complete the online questionnaire; and

• Completed the questionnaire in hardcopy format.

Westat (see Section 3 for more information) will identify participants based on the NAEP survey/data collection system. Westat will interview up to 10 respondents in each cell, as shown below in Table 1.

Table 1

Type of Respondent	Did Not Start Online Questionnaire	Did Not Complete Online Questionnaire	Completed Hardcopy Questionnaire	Total
School	10	10	10	30
Teacher	10	10	10	30
Total	20	20	20	60

Westat will contact respondents by phone to conduct the short interviews (see Appendices B, C, and D for the interview scripts). If the respondent is not available, the interviewer will leave a message asking him or her to call back. After three unsuccessful attempts to contact a respondent, the interviewer will send an email (see Appendix A) to the respondent asking for a good date/time to conduct the interview. After six unsuccessful attempts to contact a respondent, interviewers will select a replacement respondent from the same response category, if available.

3. Consultants outside the Agency

Westat is the Sampling and Data collection contractor for NAEP. Westat will make initial contact and follow-up calls to conduct the interviews with teachers and schools.

4. Assurance of Confidentiality

Participants are notified that their participation is voluntary and that their answers may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (20 U.S.C. §9573)].

No personally identifiable information will be maintained for the participants. However, the comments from teachers and principals might be associated with the characteristics of their school to determine any possible patterns nationwide.

5. Estimate of Hourly Burden

This study includes email and phone contact with principals and teachers to initially set up phone interviews. Westat will complete interviews with up to 10 respondents in each of the categories. An initial call to set up time for the phone interview will take two minutes and the brief phone interview will be comprised of five minutes on average.

Table 2

Type of Respondent	Maximum Number of Respondents	Burden Hours Per Respondent	Total Hours
Initial contact	120	.033 (2 minutes)	4
Phone interview	60*	.083 (5 minutes)	5
Total	120		9

* Subset of the previous line (this study estimates a total of 120 respondents and 180 responses).

6. Cost to Federal Government

The total estimated cost to the federal government for all recruitment, data collection, and analysis activities for this study is $5,000.

No incentives or stipend will be provided to the respondents.

7. Project Schedule

The following table depicts the schedule for efforts related to the implementation of the focus groups.

Table 3

Activity	Date
Begin Principal and Teacher recruitment process upon approval from OMB	February 2014
Conduct Principal and Teacher phone interviews	March-April 2014
Draft and submit to NCES a summary report on the interviews	May 2014

Appendix A: Notification Email to Respondents

To: <RESPONDENT EMAIL ADDRESS>

From: <NAEP FIELD STAFF>

Subject: NAEP Questionnaire Follow Up Interview

Dear <RESPONDENT NAME>

Hello, my name is <YOUR NAME> and I work for the National Assessment of Educational Progress or NAEP. NAEP was recently conducted in your school with grade [4/8/12] students and you were asked to complete a questionnaire online. To improve the user experience completing the questionnaires online, I have just a few questions about your experience.

When would be a good time to reach you for a brief, five-minute phone interview?

I would be happy to contact you at your convenience or you can contact me at the phone number listed below.

Thank you in advance for your time and cooperation.

We appreciate your support of the National Assessment of Educational Progress.

Regards,

<FIELD STAFF NAME>

<FIELD STAFF PHONE NUMBER>

<FIELD STAFF EMAIL ADDRESS>

Appendix B: Survey of Respondents that Did Not Finish Online Questionnaire

Respondent Type:  School Questionnaire  Teacher Questionnaire

Teacher Grade/Subject  Grade 4  Grade 8 Science  Grade 8 Social Studies

School ID: _____________

Hello, may I speak to <INSERT RESPONDENT NAME>?

IF NOT AVAILABLE Does he/she have a voicemail where I could leave a message?

• IF NO May I leave my name and number so he/she can call me back?

WHEN CONNECTED TO RESPONDENT

Hello, my name is <YOUR NAME> and I’m calling on behalf of the National Assessment of Educational Progress or NAEP. NAEP was recently conducted in your school with grade [4/8/12] students and you were asked to complete a questionnaire. The questionnaire was available online. Although you started the questionnaire online; you did not complete the questionnaire.

To improve the user experience completing the questionnaires online, I have just a few questions about your experience. Your responses will be used only for research purposes and will not be disclosed or used, in identifiable form, for any other purpose except as required by the Education Sciences Reform Act of 2002 [(ESRA 2002) 20 U.S.C., § 9573].

Are you willing to answer a few questions about your experience?

• IF NO: Thank you for your time. Good-bye.

CONTINUE WITH THE INTERVIEW

1. Why didn’t you complete the questionnaire online? DO NOT READ RESPONSES. PROBE FOR ONLY ONE RESPONSE.

• Had trouble completing the questionnaire (Go to Q1a)

• Didn’t want to complete it online (Go to Q1b)

• Forgot to complete the questionnaire (Go to Q2)

• Other (specify; Go to Q1a, Q1b, or Q2 as appropriate)

1a. I’m sorry you had trouble completing the questionnaire online. Can you describe the trouble you had?

SKIP TO Q2

1b. Please tell me why you did not want to complete the questionnaire online. PROBE FOR ANY TECHNOLOGICAL CONCERNS OR CHALLENGES.

2. How could we have made it easier for you to complete the questionnaire online?

That’s all the questions I have for you today. Thank you for your time and for your support of the National Assessment of Educational Progress. Good-bye.

Appendix C: Survey of Respondents that Did Not Start Online Questionnaire

Respondent Type:  School Questionnaire  Teacher Questionnaire

Teacher Grade/Subject  Grade 4  Grade 8 Science  Grade 8 Social Studies

School ID:

Hello, may I speak to <INSERT RESPONDENT NAME>?

IF NOT AVAILABLE Does he/she have a voicemail where I could leave a message?

• IF NO May I leave my name and number so he/she can call me back?

WHEN CONNECTED TO RESPONDENT

Hello, my name is <YOUR NAME> and I’m calling on behalf of the National Assessment of Educational Progress or NAEP. NAEP was recently conducted in your school with grade [4/8/12] students and you were asked to complete a questionnaire. The questionnaire was available online and you did not complete the questionnaire.

To improve the user experience completing the questionnaires online, I have just a few questions about your experience. Your responses will be used only for research purposes and will not be disclosed or used, in identifiable form, for any other purpose except as required by the Education Sciences Reform Act of 2002 [(ESRA 2002) 20 U.S.C., § 9573].

Are you willing to answer a few questions about your experience?

• IF NO: Thank you for your time. Good-bye.

CONTINUE WITH THE INTERVIEW

1. Why didn’t you complete the questionnaire online? DO NOT READ RESPONSES. PROBE FOR ONLY ONE RESPONSE.

• Didn’t know how to access the questionnaire (Go to Q1a)

• Didn’t want to complete it online (Go to Q1b)

• Didn’t have time to complete (Go to Q2)

• Other (specify; Go to Q1a, Q1b, or Q2 as appropriate)

1a. I’m sorry you didn’t know how to complete the questionnaire online. Did you receive the emails with the links and your password?

• Yes PROBE FOR DETAILS ON WHETHER THE USER TRIED TO ACCESS THE QUESTIONNAIRE AND WHAT TROUBLE WAS EXPERIENCED

• No

• Don’t know

SKIP TO Q2

1b. Please tell me why you did not want to complete the questionnaire online. PROBE FOR ANY TECHNOLOGICAL CONCERNS OR CHALLENGES.

2. How could we have made it easier for you to complete the questionnaire online?

That’s all the questions I have for you today. Thank you for your time and for your support of the National Assessment of Educational Progress. Good-bye.

Appendix D: Survey of Respondents that Completed Hardcopy Questionnaire

Respondent Type:  School Questionnaire  Teacher Questionnaire

Teacher Grade/Subject  Grade 4  Grade 8 Science  Grade 8 Social Studies

School ID:

Hello, may I speak to <INSERT RESPONDENT NAME>?

IF NOT AVAILABLE Does he/she have a voicemail where I could leave a message?

• IF NO May I leave my name and number so he/she can call me back?

WHEN CONNECTED TO RESPONDENT

Hello, my name is <YOUR NAME> and I’m calling on behalf of the National Assessment of Educational Progress or NAEP. NAEP was recently conducted in your school with grade [4/8/12] students and you were asked to complete a questionnaire. Although the questionnaire was available online, you chose to complete a hardcopy.

To improve the user experience completing the questionnaires online, I have just a few questions about your experience. Your responses will be used only for research purposes and will not be disclosed or used, in identifiable form, for any other purpose except as required by the Education Sciences Reform Act of 2002 [(ESRA 2002) 20 U.S.C., § 9573].

Are you willing to answer a few questions about your experience?

• IF NO: Thank you for your time. Good-bye.

CONTINUE WITH THE INTERVIEW

1. Why didn’t you complete the questionnaire online? DO NOT READ RESPONSES. PROBE FOR ONLY ONE RESPONSE.

• Had trouble completing the questionnaire (Go to Q1a)

• Didn’t want to complete it online (Go to Q1b)

• Other (specify; Go to Q1a, Q1b, or Q2 as appropriate)

1a. I’m sorry you had trouble completing the questionnaire online. Can you describe the trouble you had? PROBE FOR ANY TECHNOLOGICAL CONCERNS OR CHALLENGES.

SKIP TO Q2

1b. Please tell me why you did not want to complete the questionnaire online. PROBE FOR ANY TECHNOLOGICAL CONCERNS OR CHALLENGES.

2. How could we have made it easier for you to complete the questionnaire online?

That’s all the questions I have for you today. Thank you for your time and for your support of the National Assessment of Educational Progress. Good-bye.