National Assessment of Educational Progress (NAEP) NAEPQ Giving Back Website 2018 Focus Groups

National Center for Education Statistics

National Assessment of Educational Progress

Volume II

Interview Protocol

National Assessment of Educational Progress (NAEP)

NAEPQ Giving Back Website 2018 Focus Groups

OMB# 1880-0542

October 2018

Table of Contents

Paperwork Burden Statement 2

Moderator’s Interview Script 3

Figure 1 7

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be also included in the materials used in the study.

Paperwork Burden Statement

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1880-0542. The time required to complete this information collection is estimated to average 90 minutes, including the time to review instructions and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or any comments or concerns regarding the status of your participation, please write to: National Assessment of Educational Progress (NAEP), National Center for Education Statistics (NCES), Potomac Center Plaza, 550 12th St., SW, 4th floor, Washington, DC 20024.

Authorization and Confidentiality Assurance

National Center for Education Statistics (NCES) is authorized to conduct NAEP by the National Assessment of Educational Progress Authorization Act (20 U.S.C. §9622).

All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C.§151).

OMB No. 1880-0542 Approval Expires 07/31/2020

Moderator’s Interview Script

Interviewer Script for NAEPQ Giving Back Website 2018 Focus Groups

Materials needed:

• Colored paper to be passed out at the end of session for comments

• Laptop and headphones for each participant

• Internet connection

• Screen, projector, and laptop with speakers for moderator (to project website and demonstrate creation of a report)

• Username and password information cards for each participant

• Hand-outs with links if participants are interested in learning more about NAEP

• Copies of the actual opt-in questions in the NAEPQ for 2015, 2017, 2018, 2019

Room set-up:

• Tables and chairs positioned so that each participant can access a laptop and see the screen

Time:

• 90 minutes total

  • Facilitator intro, ground rules, questions, participant intros: 15 minutes

  • Discussion questions/responses: 20 minutes

  • Demo of NAEPQ Giving Back Website 10 minutes

  • Participant access/exploration of website: 10 minutes

  • Discussion: 30 minutes

  • Wrap-up: 5 minutes

NOTE: Text placed in brackets and highlighted provides notes and cues for the moderator.

Facilitator intro, ground rules, questions, participant intros: 15 minutes

My name is [NAME] and I will be facilitating our discussion today.

I want to thank you for participating in today’s focus group. You are all here to provide feedback on the NAEP Questionnaire (or NAEPQ) Giving Back Website, which provides participating schools preliminary data from other teachers and schools, as a way of “giving back”. As part of NAEP, students, teachers, and school administrators complete contextual questionnaires about learning environments and educational opportunities depending on the subject and grade being assessed. Although NAEP collects student questionnaire data, they are not included in this website. Our hopes are to figure out how we can help this program provide more useful data to teachers and school administrators like yourselves. I will discuss this in more detail later.

First, let me begin with a brief background of both the National Assessment of Educational Progress, or NAEP, and the National Center for Education Statistics, or NCES. NAEP was first administered in 1969 and is the largest continuing and nationally representative assessment of what our nation’s students know and can do in subjects such as mathematics, reading, science, and writing. NAEP is a standardized test, meaning that the test is administered under the same conditions in each participating school, resulting in a common measure of student achievement. Teachers, principals, parents, policymakers, and researchers all use NAEP results to assess progress and develop ways to improve education in the United States. The results of NAEP are released as The Nation’s Report Card, and are available for the nation and, in some cases, states and urban districts – known as the Trial Urban District Assessment. This program is run by the National Center for Education Statistics, or NCES, which is part of the U.S. Department of Education.

NAEP and State Assessments:

While NAEP and state assessments may differ in scope and content, both can be used to assess progress and develop ways to improve education in our nation. Let’s quickly discuss some of those similarities and differences.

NAEP	State Assessments
Measures what the nation’s students know and can do in various subjects	Measure students’ progress toward required knowledge and skills as set and defined by each state’s content standards
A nationally representative sample of students in grades 4, 8, and 12 is asked to participate	Participation may be required for all students in public schools within the state
Evaluates trends in student achievement over time by demographic groups for the nation, states, and some urban districts	Report state, district, school, and individual student results

As I mentioned earlier, I will be facilitating a discussion about NAEP data and the NAEPQ Giving Back Website, which is based on the teacher and school questionnaires. This site allows educators to enter a web portal to explore preliminary teacher and school questionnaire results that interest them and see the results within the context of their own state, school type, selected demographics, and more. I will be giving a demonstration of the website in a few minutes.

Before we get started today, I’d like to offer some reminders and set a few ground rules for our discussion.

This interview is being audio recorded so that researchers can review the recordings later. This recording is only meant to help us summarize what we learn today. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (if needed: 20 U.S.C. §9573 and 6 U.S.C. §151). Please note that there are some interested colleagues listening in on the discussion from the other room.

1. What is said here today will be aggregated based on the themes and topics we will be addressing. No names or any identifiable information will be associated with responses or appear on any presentation or report on this focus group. I will be using a recording device to ensure that we preserve a complete rendering of what is shared in this group, and it will be destroyed when the project is completed. Only NCES and its NAEP affiliated contractors will be privy to the outcomes of this session.

2. I want to point out that I am not involved in the development of this site, so please feel free to be open and honest in your comments.

3. We want to have an open discussion. It is important to hear what everyone thinks, so please speak up and please let others do the same.

4. You don’t have to wait for me to call on you, but please speak one at a time.

5. You can respond to each other as well as to me.

6. You might disagree about certain topics, but please be respectful of each other.

7. There are no right or wrong answers. All comments, positive and negative, what you know and don’t know, are important.

8. If I ask a question you’re not comfortable answering, feel free not to answer.

9. We are recording the discussion so I can write a report. However, no names will be included in the report, and all comments mentioned in the report will be anonymous.

10. My job is to keep us on the right track and moving along so that we’re done in an hour and a half. From time to time, I may have to jump in to keep our conversation moving.

11. If you need to go to the restroom or need a break, feel free to step away, but please come back as soon as possible because all opinions are important.

12. Please turn off or silence all of your electronic devices to avoid distractions to yourself and those around you.

Do you have any questions before we get started? [Pause for questions]

Let’s get started with introductions – Please tell us your name and what grade and subject(s) you teach. For administrators, tell us what school level you lead. [Participant introductions]

Discussion questions/responses: 20 minutes

Now, let me ask you some general questions about your use of education data:

• What are your current data use practices?

• What would you say are your data needs?

• What types of data do you find useful?

• Do you use external data sources? If so, what types of data do you use and where do you retrieve them from?

• Do you have an interest in, or do you explore data for your state?

• Do you explore or have an interest in national data around teacher/principal practices?

Ok, so [none/one/few/most/all] of you use data in some way.

Moving on, let’s talk about your familiarity with NAEP:

• Have students in your school participated in NAEP? Raise your hand if they have.

• If students in your school have not participated in NAEP, or if you don’t know, are you familiar with this national assessment or the Nation’s Report Card?

• Have you ever used NAEP data? If yes, how? [If no one responds yes, skip the next question]

• Have NAEP data helped you with your work?

• Are you familiar with the NAEP Questions Tool (NQT) and have you assessed your students using any NAEP sample questions?

[The moderator will record the number of hands raised and the participants’ responses to open ended questions.]

Ok, so [none/one/few/most/all] of you are familiar with NAEP and the data resources that NAEP provides. We are focusing today on NAEP data that could be valuable to you and sharing the NAEPQ Giving Back Website with you. Access to this website is provided to those who “opted in” when they completed the NAEP teacher or school survey. Beside your tablet, there is a copy of the opt-in question on the questionnaires, which you can review later when you are exploring the website. Once the website is launched, teachers and administrators are emailed a user name and password. The website was created to serve a number of purposes, but the most important one for this conversation is to provide informative feedback from the Teacher and School Questionnaires to educators.

Demo of NAEPQ Giving Back Website: 10 minutes

[Teachers and principals have laptops. User name and password will be provided] I am going to get us started by opening the portal on my computer and projecting onto the screen. Can everyone see the screen alright? If not, please relocate so you can all get a good view. After I demonstrate how to log in, I’ll give you a quick tour of the website with the overview tutorial. Then you’ll log in on the laptops provided [password/access will be arranged for the loaner laptops prior to the focus group; headphones will be provided for each participant so they can complete the tutorial, if desired]. The tutorial will pop up on your screen, which we just viewed, so you can click on the “x” in the upper right corner of the tutorial box. There are other tutorials that you can view, so plug in your headphones, put them on, and let the tutorials guide you if you would like—this doesn’t take long.

Participant access/exploration of website: 10 minutes

Now, try creating and running a report on your own, exploring what interests you. You’ll have about 10 minutes for exploring and then we’ll talk more about your experiences with the website.

Discussion: 30 minutes

[After 10 minutes] Okay, let’s start with some preliminary questions.

• What did you think?

• Is this information valuable to you?

• What would you do with these data?

• What were your expectations? Were they met? If not, why?

• Was it too much information?

• On a scale of 1 to 5, 1 being not useful, and 5 being extremely useful, how useful are these data for the work that you do?

• What additional information would be helpful for the work that you do?

• What type of data do you need that overlap with the data that NAEP provides?

Now, some additional questions.

[Prompts - pause for response to each question]

• Could someone tell me one thing you liked about the website? [pause for responses].

• Why do you like that feature?

• Great! What’s something you didn’t like? [pause for responses]

• What’s something you would like to be able to do with this website or the data it provides?

• Let’s talk about the reports you generated with the website. What did you learn from the reports?

• Where did you spend most of your time and where did you spend the least amount of time?

• Now, for those of you who aren’t familiar with NAEP and haven’t accessed the website, what would make you want to access these data? It turns out that few teachers and principals opt in. Why do you think that is? Also, it seems that many of the teachers and administrators who opted in never visit the site. Why might that be?

• If you opted-in to receive access to these data, but never accessed the site or created a report, what are your thoughts about NAEP contacting you to find out why?

• For those of you that opted-in to receive these data, did you have any difficulties accessing the data from school, such as connectivity issues? [Moderators will not ask this question if no one opted in to receive data]

• What would make accessing the website easier?

• When is the best time of year to provide access to this website?

Wrap up: 5 minutes

Final thoughts…this is your opportunity to let the people who designed the website know what you want and need in order to make it truly useful to you in your work. I would love to hear your suggestions [pause for responses]. Thanks for your responses. If you didn’t get a chance to respond, feel free to write your suggestions on a piece of paper [hand out half-sheets of colored paper] and give them to me on your way out.

Thank you again for your time and your willingness to share your experiences and thoughts. They will be invaluable to the developers as they consider how to provide the information you need, in a format that is helpful to you in your work. Have a great [day/evening]!

END

Figure 1

NAEPQ Opt In Questions

2015

2017

2018

2019