National Assessment of Educational Progress (NAEP) Assessment Literacy Focus Groups

National Center for Education Statistics

National Assessment of Educational Progress

Volume II

Screener Calls & Focus Group Protocols

Assessment Literacy Focus Group Studies

OMB# 1880-0542

May 4, 2015

Contents

1) Paperwork Burden Statement 1

2) Phone Script and Screening Questionnaire for Parent/Guardian Focus Group 1

3) Phone Script and Screening Questionnaire for Teacher Focus Group 1

4) Focus Group Protocols 2

Part I. Interviewer Welcome Scripts and Assent/Consent 9

Part II. Focus Group Questions 11

Attachment A: Handout Materials for Parent/Guardian Focus Group 17

Attachment B: Handout Materials for Teacher Focus Group 18

1. Paperwork Burden Statement

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement, OMB Information

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1880-0542. The time required to complete this information collection is estimated to average 90 minutes including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or any comments or concerns regarding the status of your individual submission of this information collection, please write to: National Assessment of Educational Progress, National Center for Education Statistics, 1990 K Street, N.W., Washington, DC 20006.

This is a project of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences, within the U.S. Department of Education.

Your feedback may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].

2. Phone Script and Screening Questionnaire for Parent/Guardian Focus Group

CALL FROM DATABASE OR AT RANDOM

RECRUIT 10 PER GROUP FOR 8-9 TO SHOW

Hello, I’m __________ with __________________, an independent consumer research firm. We are not selling or promoting any product or service. May I please speak to [RECRUIT’S NAME]?

We are conducting a focus group on behalf of the U.S. Department of Education’s National Center of Education Statistics and would like to include your views. This research is about educational testing in schools. Would you be interested in participating in a 90 minute focus group, for which you would receive a $75 VISA gift card as a thank you for your time and participation?

If no: Thank you for your time. Have a wonderful <DAY/EVENING>.

If yes: Great, thank you very much. To ensure that we have the correct mix of participants, we would like to ask some questions. My questions will only take a few minutes. You don’t have to answer any question you do not wish to respond to. May we proceed with the questions?

Yes (PROCEED TO PROTOCOL SCRIPT BELOW)

No (THANK YOU, HAVE A GOOD DAY/EVENING)

1. First, do you, or does any member of your household or immediate family, work for or is retired from:

A market research company _____

A direct mail company or direct marketing company _____

An advertising agency or public relations firm _____

The media (TV/radio/newspapers/magazines) _____

[IF YES TO ANY >> TERMINATE AND THANK¹]

2. Please tell me your occupation and the name of the company or organization you work for (if retired get former occupation and company worked for).

____________________________________________________________

____________________________________________________________

[TERMINATE IF TEACHER, TEACHER’S ASSISTANT, SCHOOL GUIDANCE COUNSELOR, PRINCIPAL, OR OTHER SCHOOL ADMINISTRATOR]

3. Does any member of your household or immediate family currently work in or is retired from any of the following positions in a K-12 school: teacher, teacher’s assistant, school guidance counselor, principal, or other school administrator?

[TERMINATE AND THANK] Yes_____

No _____

4. Are you the parent of a child who attends a public school in any grade 3 through 12 in Maryland?

Yes_____

[TERMINATE AND THANK] No _____

5. Does this child live in your household?

Yes_____

[TERMINATE AND THANK] No _____

6. Which of the following categories best describes your total annual household income? [MIX]

Under $20,000 _____

$20,000 - $39,999 _____

$40,000 - $59,999 _____

$60,000 - $79,999 _____

$80,000 - $100,000 _____

Over $100,000 _____

7. Please tell me your age. [RECORD AGE] __________________

8. What is your current marital status? Are you ….. ?

Married or living with a partner_____

Single _____

Divorced _____

Widowed _____

Separated _____

9. What is the highest level of education you have completed? [GOOD MIX]

Less than high school graduate _____

High school graduate _____

Some college _____

College graduate _____

Post graduate studies or degree _____

10. Do you or your children speak more than one language at home?

YES_____

NO______

11. So that we can be sure that all backgrounds are represented in our study, please tell me your ethnic or race background. Are you …?

[RECRUIT MIX]

Ethnicity

Hispanic or Latino_____

Not Hispanic or Latino_____

Race (can select more than one)

American-Indian or Alaska Native _____

Asian _____

Black or African American _____

Native Hawaiian-Indian or Other Pacific Islander _____

White _____

12. [RECORD GENDER] [MIX]

Male _____

Female _____

(INTERVIEWER: Determine recruitment needs in advance. Record all responses. End screener at whatever point the parent or legal guardian gives a response that confirms that their group has already been sampled adequately. End screener script: Based on the requirements of this study, we are not able to include you in the interviews at this time.)

13. Have you ever attended a focus group discussion or individual interview? By that we mean an informal, round-table discussion, conducted by a professional moderator in which you were asked your opinions regarding a product, a service, or advertising?

>> ASK A-C >> Yes _____

INVITE TO GROUP >> No _____

1. How many of these groups have you attended?

______________________________ [MAX. 5 EVER]

2. What was/were the topics discussed?

______________________________ [IF EDUCATIONAL TESTING, TERMINATE]

3. How long ago was the last one of these groups you attended?

______________________________ [MUST BE AT LEAST 6 MONTHS AGO]

[INVITE TO GROUP]

Are you willing to participate in this focus group?

If NO: Thank you for your time. Have a wonderful <day/evening>.

If YES: Great, thank you for answering all of my questions. As I mentioned earlier, we are conducting a focus group regarding educational testing in schools. The focus group will be audio and video recorded, but all information will be reported only in a combined form, so individuals will not be identified. Participation is voluntary and your answers may be used only for research purposes and will not be disclosed or used, in identifiable form, for any other purpose except as required by the Education Sciences Reform Act of 2002 [(ESRA 2002) 20 U.S.C., § 9573].

The discussion will last about 1.5 hours and will be both fun and informative. The discussion will be held at <LOCATION> on <DATE> at <TIME>. Prior to the start of the focus groups, all participants will be asked to read and sign a consent form. No one will attempt to sell you anything and no one will call you as a result of your participation. As a token of our appreciation for your help in our research effort, you will receive a $75 VISA gift card after the session is completed. This is an important research effort and we hope that you will be part of it. We can only invite about 10 people to take part.

Can we schedule your attendance?

[If NO:] Thank you for your time. Have a wonderful <day/evening>

[If YES, read … ]

Do you have any questions at this time? If you do have any questions, please call [PHONE NUMBER].

[STAFF NAME] will follow-up with you and send a [LETTER/EMAIL] indicating the time and date of the focus group.

If you need glasses for reading or for using the computer, please be sure to have them available for this session.

ID Number _________________________________________________

NAME: _________________________________________________

ADDRESS: _________________________________________________

CITY: _________________________________________________

ZIP CODE: _________________________________________________

PHONE: (DAY) _____________________________________

(EVE) _____________________________________

(CELL) _____________________________________

(EMAIL) _____________________________________

Interviewer: ____________________

Supervisor Confirm: ____________________

3. Phone Script and Screening Questionnaire for Teacher Focus Group

CALL FROM DATABASE OR AT RANDOM

RECRUIT 10 PER GROUP FOR 8-9 TO SHOW

Hello, I’m __________ with __________________, an independent consumer research firm. We are not selling or promoting any product or service. May I please speak to [RECRUIT’S NAME]?

We are conducting a focus group on behalf of the U.S. Department of Education’s National Center of Education Statistics and would like to include your views. This research is about educational testing in schools. Would you be interested in participating in a 90 minute focus group, for which you would receive a $75 VISA gift card as a thank you for your time and participation?

If no: Thank you for your time. Have a wonderful <DAY/EVENING>.

If yes: Great, thank you very much. To ensure that we have the correct mix of participants, we would like to ask some questions. My questions will only take a few minutes. You don’t have to answer any question you do not wish to respond to. May we proceed with the questions?

Yes (PROCEED TO PROTOCOL SCRIPT BELOW)

No (THANK YOU, HAVE A GOOD DAY/EVENING)

1. Are you currently an active classroom teacher?

[ ] Yes – [CONTINUE WITH QUESTION 3]

[ ] No – [THANK AND TERMINATE]

2. Are you currently in a public school, a charter school, or at a private school?

[ ] Public ---- |

| -- [CONTINUE]

[ ] Charter --- |

[ ] Private ---- | -- [THANK AND TERMINATE]

3. What grades do you teach/are you involved with? RECRUIT ONLY THOSE WHO TEACH OR ARE INVOLVED WITH THIRD THROUGH TWELFTH GRADES.

[ ] Kindergarten through Second grade [THANK AND TERMINATE]

[ ] Third through Eighth grade [CONTINUE WITH QUESTION 4]

[ ] Ninth through TWELFTH grade [CONTINUE WITH QUESTION 4]

4. How long have you been teaching or involved in education? [RECRUIT A MIX]

[ ] 3 years or less |

[ ] 4-6 years |

[ ] 7-10 years |

[ ] 10 years or more |

5. Are you of either Hispanic or Latino origin? [RECRUIT A MIX]

[ ] Yes

[ ] No

6. What is your race? (YOU MAY SELECT ONE OR MORE) [RECRUIT A MIX]

[ ] White

[ ] Black or African American

[ ] Asian

[ ] American Indian or Alaska Native

[ ] Native Hawaiian or Other Pacific Islander

7. RECORD GENDER. DO NOT ASK [RECRUIT A MIX]

[ ] Female

[ ] Male

8. Do you speak more than one language?

[ ] Yes

[ ] No

(INTERVIEWER: Determine recruitment needs in advance. Record all responses. End screener at whatever point the teacher gives a response that confirms that their group has already been sampled adequately. End screener script: Based on the requirements of this study, we are not able to include you in the interviews at this time.)

9. Do you or anyone in your family work [READ LIST]:

[ ] In market research ------------------------------------------------| THANK

[ ] In advertising or public relations | AND

[ ] In journalism or the news media ----------------------------|TERMINATE

10. In the past 6 months, have you participated in a market research group discussion?

[ ] Yes – [THANK AND TERMINATE]

[ ] No – [CONTINUE]

Thank you for answering all of my questions. As I mentioned earlier, we are conducting a focus group regarding educational testing in schools. The discussion will last approximately 90 minutes and will be held via WebEx on <DATE> at <TIME>. The focus group will be audio recorded, but all information is reported in combined form so individuals will not be identified. Participation is voluntary and your answers may be used only for research purposes and will not be disclosed or used, in identifiable form, for any other purpose except as required by the Education Sciences Reform Act of 2002 [(ESRA 2002) 20 U.S.C., § 9573].

The discussion will last about an hour and a half and will be both fun and informative. Prior to the start of the focus groups, all participants will be asked to read and sign a consent form. Your opinions are very important to us and to thank you for your time, we will give you $75 VISA gift card as a token of our appreciation. No one will attempt to sell you anything and no one will call you as a result of your participation. This is an important research effort and we hope that you will be part of it. We can only invite about 10 people to take part. Can we schedule your attendance?

[If no, thank and end call]

[If yes, read …..]

Do you have any questions at this time? If you do have any questions, please call [PHONE NUMBER].

[STAFF NAME] will follow-up with you and send a [LETTER/EMAIL] indicating the time and date of the focus group.

If you need glasses for reading or for using the computer, please be sure to have them available for this session. We will follow up within the next two or three weeks with details about how to go online to participate in this session.

RESPONDENT NAME: ___________________________________

TELEPHONE/EMAIL: ____________________________________

4. Focus Group Protocols

Part I. Interviewer Welcome Scripts and Assent/Consent

Parent/Guardian Participant Welcome Script

NOTE: Text placed in brackets provides notes and cues for the moderator.

[Welcome participants and thank them for participating in this focus group.]

My name is <NAME> and I have been hired to facilitate our discussion today. I am an independent moderator – I’m not an expert on the topic we are going to discuss today. I am primarily interested in your views on this topic – my views don’t matter. During our discussion, you may have questions that I can’t answer, but at the end of our discussion I will refer you to resources for more information.

[Sponsor and purpose of the group]

We’re here to discuss educational testing in schools on behalf of the National Center for Education Statistics (NCES), which is part of the U.S. Department of Education. NCES collects, analyzes, and makes available data related to education in the U.S. and other nations. You might be familiar with an educational test NCES manages – it’s called the National Assessment of Educational Progress (NAEP). Many know it as “The Nation’s Report Card.” NAEP is the largest national assessment of what America's students know and can do in various subject areas, including math, reading, science, writing, the arts, civics, economics, geography, and U.S. history.

Tonight, we’ll talk about various kinds of tests that your child will experience in his or her school years. We are also interested in hearing what questions you may have about the tests that your child takes, and how you might look for information about those tests.

We want to remind you that everything you tell us will be used only for research purposes and will not be disclosed or used, in identifiable form, for any other purpose except as required by law [if needed: Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573]. What is said here today will be aggregated based on the themes and topics we will be addressing. No names or identifying information will be associated with responses or appear on any presentation or report on this focus group. We will be using a recording device to ensure that we preserve a complete rendering of what is shared in this group. Are there any questions? [Respond to any questions.]

[Focus group process and general rules]

Let’s review our general focus group process and some general rules.

1. We will be looking at some statements and materials throughout the group today; please make sure you have reading glasses if you need them.

2. I did not create any of the materials we are going to talk about today, so please feel free to be open and honest in your comments.

3. We want to have an open discussion. It is important to hear what everyone thinks, so please speak up and please let others do the same.

4. You don’t have to wait for me to call on you, but please speak one at a time.

5. You can respond to each other as well as to me.

6. You might disagree about certain topics, but please be respectful of each other.

7. There are no right or wrong answers. All comments, positive and negative, what you know and don’t know, are important.

8. If I ask a question you’re not comfortable answering, feel free not to answer.

9. We are recording the discussion so I can write a report. However, no names will be included in the report and all comments mentioned in the report will be anonymous.

10. There are some interested colleagues listening to our discussion behind the glass.

11. My job is to keep us on the right track and moving along so that we’re done in an hour and a half. From time to time, I may have to jump in to keep our conversation moving.

12. If you need to go to the restroom or need a break, feel free to step away, but please come back as soon as possible because all opinions are important.

13. Please turn off or silence all of your electronic devices to avoid disturbing others.

Do you have any questions before we get started?

[Participant Introductions]

Let’s get started with introductions - Please tell us your first name, and what grade your child attends at his or her school.

Teacher Participant Welcome Script

NOTE: Text placed in brackets provides notes and cues for the moderator.

[Welcome participants and thank them for participating in this focus group.]

My name is <NAME> and I have been hired to facilitate our discussion today. I am an independent moderator – I’m not an expert on the topic we are going to discuss today. I am primarily interested in your views on this topic – my views don’t matter. During our discussion, you may have questions that I can’t answer, but at the end of our discussion I will refer you to resources for more information.

[Respond to questions about the WebEx. Information about joining the session will be distributed prior to the meeting.]

We’re here to discuss educational testing in schools on behalf of the National Center for Education Statistics (NCES), which is part of the U.S. Department of Education. NCES collects, analyzes and makes available data related to education in the U.S. and other nations. You might be familiar with an educational test NCES manages – it’s called the National Assessment of Educational Progress (NAEP). Many know it as “The Nation’s Report Card.” NAEP is the largest national assessment of what America's students know and can do in various subject areas, including math, reading, science, writing, the arts, civics, economics, geography, and U.S. history.

We’ll talk about various kinds of tests that the children you teach will experience during their school years. We are also interested in hearing what questions you may have heard from parents about tests that their children take, and where they typically look for information on those tests.

We want to remind you that everything you tell us will be used only for research purposes and will not be disclosed or used, in identifiable form, for any other purpose except as required by law [if needed: Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573]. What is said here today will be aggregated based on the themes and topics we will be addressing. No names or identifying information will be associated with responses or appear on any presentation or report on this focus group. We will be using a recording device to ensure that we preserve a complete rendering of what is shared in this group. Are there any questions? [Respond to any questions.]

Before we get started today, I’d like to offer some reminders and set a few ground rules for our discussion.

1. We will be looking at some statements and materials throughout the group today; please make sure you have reading glasses if you need them.

2. I did not create any of the materials we are going to talk about today, so please feel free to be open and honest in your comments.

3. We want to have an open discussion. It is important to hear what everyone thinks, so please speak up and please let others do the same.

4. You don’t have to wait for me to call on you, but please speak one at a time.

5. You can respond to each other as well as to me.

6. You might disagree about certain topics, but please be respectful of each other.

7. There are no right or wrong answers. All comments, positive and negative, what you know and don’t know, are important.

8. If I ask a question you’re not comfortable answering, feel free not to answer.

9. We are recording the discussion so I can write a report. However, no names will be included in the report and all comments mentioned in the report will be anonymous.

10. My job is to keep us on the right track and moving along so that we’re done in an hour and a half. From time to time, I may have to jump in to keep our conversation moving.

11. If you need to go to the restroom or need a break, feel free to step away, but please come back as soon as possible because all opinions are important.

12. Please turn off or silence all of your electronic devices to avoid disturbing others.

Do you have any questions before we get started?

[Participant Introductions]

Let’s get started with introductions - Please tell us your name and what grade and subject(s) you teach. .

Part II. Focus Group Questions

NOTE: Text placed in brackets provides notes and cues for the moderator.

The purpose of the focus groups is to inform the development of materials to increase assessment literacy among parents/guardians. Below, we present questions and probes that parents/guardians will be asked during the focus group sessions. Please note that some probes may not be addressed if time does not permit.

Parent/Guardian (90 minute sessions)

1. GENERAL AWARENESS / KNOWLEDGE / ATTITUDES REGARDING EDUCATIONAL TESTING (10 MINUTES)

1. Let’s get started by talking about tests your child has taken in school – what are some examples? [Probe, if necessary: Classroom tests, the Maryland School Assessment, the High School Assessment, the SAT, the AP test, NAEP, etc.]

2. Do these tests have similar or different purposes? What are some of the purposes of various tests? [Probe, if necessary: Classrooms tests, the Maryland School Assessment, the High School Assessment, the SAT, the AP test, NAEP, etc.]

3. We’ve mentioned quite a few tests – how easy or difficult is it to understand what each of these tests is meant to measure? How easy or difficult is it to understand how the tests differ from one another?

4. In general, what questions do you have about the tests your child takes at school? What are some of the things you want to know about them?

   1. Probe, if necessary:

      1. What are different TYPES of tests (that is, categories of existing tests)?

      2. What are common FUNCTIONS of tests (that is, what do they do)?

      3. What is each test intended to measure (e.g., outcomes in student learning and teacher performance)?

      4. How should one interpret the results of various tests? (e.g., what the tests mean and don’t mean)

      5. How do various tests align with what students learn in the classroom?

Now I’d like us to look at a couple of “storyboards” that explain some of the issues we’ve discussed. A storyboard is a series of drawings and dialogue showing planned shots for a video. The two storyboards we will talk about show two different ways to explain these issues through a video.

2. STORYBOARD #1: Assessments: Perspectives from Teachers and Parents (30 MINUTES)

Let’s look at the first storyboard – follow along with me as I read aloud through the first page and a half.

Assessments: Perspectives from Teachers and Parents

[Read first 1.5 pages on Formative Assessments]

1. Conceptual understanding of formative assessment

1. Based on this part of the script, how would you explain the ways that classroom assessments inform teaching and learning?

2. How does this storyboard describe classroom assessments? (That is, what are they?)

3. In what ways, if at all, do you think classroom assessments can help students?

Now let’s continue to the bottom of the second page. Please follow along as I read through the rest of this script.

[Read final 1.5 pages on Summative Assessments]

2. Conceptual understanding of summative assessment

1. Based on this part of the script, how would you explain the purpose of unit tests and final exams?

2. How does this storyboard describe unit tests and final exams?

3. In what ways, if at all, does it explain how unit tests and final exams can help students?

3. Overall understanding of and reaction to Storyboard #1: Perspectives from Teachers and Parents

1. What do you think about the idea of using perspectives from parents and teachers to explain testing?

   1. -Does this way of explaining classroom assessments and unit tests and final exams make sense to you?

   2. -What do you like about this way of explaining these tests? What do you not like about it?

2. This video would feature comments from teachers and parents. It would also include words on the screen, and there would be voiceover explanations at certain points.

   1. To what extent do you feel that parents like you would be interested in this video? Why do you feel this way?

   2. To what extent do you feel that parents like you would relate to this video, or feel that it is relevant to the lives of their children? Why do you say that?

3. Now that you’ve read through the entire storyboard script, if another parent asked you to explain the difference between ongoing classroom assessments and unit tests and final exams, what would you say?

4. If a friend or another parent asked you to describe this video, what would you say?

5. Is there anything that you would recommend changing about this storyboard before it is produced as a video? What would you change?

3. STORYBOARD #2: Assessments: Correcting Course on the Road to Academic Success (30 MINUTES)

[Distribute handouts with shots from second storyboard- “Correcting Course on the Road to Academic Success.”].

Let’s look at the second storyboard – please follow along with me as I read aloud through the first page.

Read first page on Formative Assessments

Assessments: Correcting Course on the Road to Academic Success

[Read first page on Formative Assessments]

1. Conceptual understanding of formative assessment

1. Based on this part of the script, how would you explain the ways that classroom assessments inform teaching and learning?

2. How does this storyboard describe classroom assessments? (That is, what are they?)

3. In what ways, if at all, do you think classroom assessments can help students?

Now let’s continue to the second page. Please follow along with me as I read through the rest of this script.

[Read second page on Summative Assessments]

2. Conceptual understanding of summative assessment

1. Based on this part of the script, how would you explain the purpose of unit tests and final exams?

2. How does this storyboard describe unit tests and final exams?

3. In what ways, if at all, does it explain how unit tests and final exams can help students?

3. Overall understanding of and reaction to Storyboard #2: Correcting Course on the Road

1. What do you think about the idea of “education as a journey”?

   1. Does this way of explaining classroom assessments and unit tests and final exams make sense to you?

   2. What do you like about this way of explaining these tests? What do you not like about it?

2. The drawings in this storyboard represent frames in an animated video that would have music playing throughout, and a voiceover reading through the script.

   1. To what extent do you feel that parents like you would be interested in this video? Why do you feel this way?

   2. To what extent do you feel that parents like you would relate to this video, or feel that it is relevant to the lives of their children? Why do you say that?

3. Now that you’ve read through the entire storyboard script, if another parent asked you to explain the difference between classroom assessments and unit tests and final exams what would you say?

4. If a friend or another parent asked you to describe this video, what would you say?

5. Is there anything that you would recommend changing about this storyboard before it is produced as a video? What would you change?

4. STORYBOARD PREFERENCES (10 MINUTES)

Now that you have reviewed the two storyboards—one looking at assessments from the perspectives of teachers and parents, and a second describing “the education journey,” --which video do you believe parents like you would find to be more helpful in explaining different types of assessments?

1. Let’s have a show of hands—how many prefer the first storyboard? And how many prefer the second storyboard?

2. For those who chose the first storyboard, why do you think this video would be more helpful for parents like you?

   1. How, if at all, do you think it should be changed to be more helpful to parents like you?

   2. How well does it answer your questions about types of tests that your child takes in school?

3. For those who chose the second storyboard, why do you think this video would be more helpful for parents like you?

   1. How, if at all, do you think it should be changed to be more helpful to parents like you?

   2. How well do you think it answers your questions about types of tests that your child takes in school?

5. INFORMATION-SEEKING PREFERENCES, AND CONCLUSION (10 MINUTES)

1. We’ve now spent ninety minutes or so talking about educational tests. Have your perceptions or feelings on this topic changed from before we began this discussion?

• If yes, how have they changed?

2. Is there more information you would like to know on this topic?

• What would be of interest to you?

3. How would you like to receive more information about educational testing?

4. Suppose there could be a website that would help parents like you to learn basic information about tests that may be administered to your child over the course of his or her education. Which of the following Taglines for the site would you find to be more compelling?

   1. Testing Matters

   2. Assessment Matters

   3. Is there another name website name that you would find to be more compelling?

5. Those are all of my questions. Do you have any final comments on anything we’ve talked about?

6. I’d like to share several web sites that provide more information about the tests and concepts we discussed this evening [Distribute one page with URLs]:

• A Parent’s Guide to the MSA: http://marylandpublicschools.org/NR/rdonlyres/D880A550-54C3-4621-AE7E-1AC0CABC022C/34521/Parents_Guide_MSA_2012_r012012_.pdf

• district, or a state: High School Assessment Program: http://mdk12.org/assessments/high_school/index.html

• NAEP: http://nces.ed.gov/nationsreportcard/

• Thank you so much; your help has been invaluable. [Provide a gift card, if applicable].

Teacher (90 minute sessions)

A. GENERAL AWARENESS / KNOWLEDGE / ATTITUDES REGARDING EDUCATIONAL TESTING (10 MINUTES)

1. Let’s get started by talking about the tests that are administered in the schools where you teach. What are some examples of these? [High School Assessment, the SAT, the AP test, NAEP, etc.]

2. Do these tests have similar or different purposes? What are some of the purposes of various tests? [Probe, if necessary: the High School Assessment, the SAT, the AP test, NAEP, etc.]

3. Have you faced any challenges in communicating about these tests with parents, students, policymakers, or the general public?

   1. What challenges have you faced?

   2. Do you feel you need more tools or resources that you can use to explain educational tests to parents, students, and the general public?

   3. What tools and resources have you used in the past?

4. In general, what questions have parents asked you about the tests? What are some of the things that they are interested in knowing about them?

   1. Probe, if necessary:

      1. What are the different types of tests?

      2. What are common functions of tests?

      3. What is each test intended to measure?

      4. How to interpret the results of tests (e.g., what they mean and don’t mean)

      5. How various tests align with what students learn in the classroom

5. What resources do you use to help parents to understand tests that are administered to their children?

B. EXPLORATION OF KEY CONCEPTS (25 MINUTES TOTAL)

NCES is developing educational materials to help parents take an active role in their children’s education by talking with teachers about tests that their children may take. These materials will cover issues such as: the purpose of various tests, how the results can be used to improve teaching and learning, and how parents should interpret the results of various tests. I’d like to get your opinion on some language that may be used for these materials.

Let’s take look at a draft guide that could be provided to parents as a resource to help them talk with teachers about tests. [Hand out Testing Matters Discussion Guide]

This is a dense document, so I’d like to break up our discussion into sections before we talk about it as a whole.

Please note that near the end of this session, we’ll give you an opportunity to make written edits and/ or comments on this document. In the interest of time, we’d like our discussion to focus on concepts, rather than phrasing.

Let’s start by taking a close look at the introduction, which begins with the phrase, “As a parent…” Please follow along with me as I read through the introduction. [Read introduction, from “As a parent” to “teacher conferences.”]

1. The introduction makes several assertions before it explains the purpose of the guide. These include:
   (15 minutes)

   • Parents play a central role in guiding their children’s success

   • Parents need tools to open doors to educational opportunities for their children

   • Parents need to understand tests and how they can be used to improve students’ learning

   • Getting the facts and asking questions can help parents to advance their child’s education

1. To what extent do you believe that the introduction will engage parents’ interest? Why?

2. To what extent do you feel that the statements in the introduction are accurate? Please explain.

3. Do any of the statements in the introduction seem inaccurate or inappropriate? Please explain.

4. Do you believe that parents would be convinced by the arguments presented here? Please explain.

5. Is the final sentence of the introduction, explaining the purpose of the guide, clear and straightforward?

2. The Discussion Guide makes use of four topics to organize questions that parents might have four teachers about tests. These topics include: (10 minutes)

   • Test purposes and uses.

   • Tests to improve teaching and learning.

   • Test scores as estimates of a student’s knowledge and skills.

   • Test fairness.

1. Are these topics important for parents who want to learn about tests that their children are taking? Please explain.

2. Would any of these topics would be of limited interest or usefulness to parents? Why?

3. When it comes to learning about tests, are there other topics—not listed in the Discussion Guide—that may be more important to parents than the ones included here? What are they?
   
   
   

C. CONTENT OF EACH SECTION (30 MINUTES)

1. Now I’d like us to look at the questions for each section. Instead of going through them one at a time, we’ll review them by section, starting with the first. I’ll read the section aloud and then ask for feedback.

[Ask the following questions for each of the four sections]

1. Are the questions listed in this section pretty straightforward and clear?

2. Are any of them confusing?

3. How important are these questions in helping parents think about their child’s education?

4. Are there any important questions for this topic that are not listed here?

5. Are there any questions listed for this topic that will not be useful for parents?

D. WRITTEN FEEDBACK ON THE GLOSSARY (10 MINUTES)

1. Please turn to the second page of the Discussion Guide. I’m going to give you a couple of minutes to read through the list of terms, and then I’d like you to take 5 minutes to make any revisions or edits that you feel would make the glossary more understandable or useful for parents. Please feel free to write any comments you may have in the “NOTES” section at the bottom of the page. I’ll call time in a few minutes, and then I’d like you to attach your revised version of the document to an email and send it to me at: [email protected].

2. Thank you for providing that feedback. Without commenting on specific phrasing or choice of words, do you have any general comments about the Glossary of Assessment Terms that you’d like to share?

1. Is the glossary pretty straightforward and clear?

2. Do you think this will be helpful for parents?

E. FINAL FEEDBACK ON THE GUIDE AS A WHOLE (5 MINUTES)

1. Finally, I’d like to ask your opinions about the Testing Matters Discussion Guide as a whole.

   • Do you think this will be a useful resource for parents? Please explain.

   • Do you think this resource would help you to explain testing to parents?

   • If this resource was available, how likely do you think it is that parents would use it?

CONCLUSION (10 MINUTES)

• We’ve now spent about ninety minutes talking about resources for explaining educational tests to parents. Have your perceptions or feelings on this topic changed from before we began this discussion?

• If yes, how have they changed?

• Suppose there could be a website that would help parents to learn basic information about tests that may be administered to their children as a part of their education. Which of the following names for the site would you find to be more compelling?

  1. Testing Matters

  2. Assessment Matters

  3. Is there another website name that you would find to be more compelling?

• Those are all of my questions. Do you have any final comments on anything we’ve talked about?

• I will now post on WebEx an informational handout on educational assessments. This is intended to provide you with additional information for your own use. Please feel free to download the document when it appears. If you have any trouble downloading it, you can send me a note through WebEx with your email address and I will be happy to send you the document after the end of the session.

• Thank you so much; your help has been invaluable. For your participation today, you will receive a $75 VISA gift card. Your gift card will be mailed to you within two to three weeks.

Attachment A: Handout Materials for Parent/Guardian Focus Group

Handout Materials for Parent/Guardian Focus Group

The Purpose of Educational Tests

• Tests aren’t one size fits all. They are not all designed to measure the same thing.

• Like tools, various tests do different jobs. It is important to use the right tool for the job.

• Some quizzes and tests measure learning along the way and help guide next steps in the classroom. Others tests, like final exams and end-of-the year state tests, assess what’s been learned over a long period of time.

• Some tests measure performance of a school, a district, or a state – but they aren’t designed to evaluate your child’s performance.

Interpreting Results of Tests

• Test scores are more than a single number. They offer several ways to interpret a child’s performance.

• If you know what’s on the test and how it’s being scored, you can better understand the performance of your child and schools.

• It’s important to understand how test results will be used to help your child or school.

Validity or Appropriateness of Tests

• The best tests have clearly stated purposes and are designed to achieve them.

• Some tests, such as the SAT, can be used to predict whether a student will succeed in the future. Others have different purposes, such as measuring what students have learned during a school year.

• If a test is designed to evaluate a student’s progress in school, it needs to focus on what the child is being taught. It shouldn’t include anything unrelated to the subject or unfamiliar to the child.

How Tests Can Help the Learning Process

If classroom assessments are routinely employed as part of the ongoing instructional process to measure what students are learning, and thus permit both teachers and students to decide whether to adjust what they are doing, students learn substantially better.

Resources for More Information:

• A Parent’s Guide to the MSA: http://marylandpublicschools.org/NR/rdonlyres/D880A550-54C3-4621-AE7E-1AC0CABC022C/34521/Parents_Guide_MSA_2012_r012012_.pdf

• HSA: High School Assessment Program: http://mdk12.org/assessments/high_school/index.html

• NAEP: http://nces.ed.gov/nationsreportcard/

Attachment B: Handout Materials for Teacher Focus Group

Handout Materials for Teacher Focus Group

The Purpose of Educational Tests

• Tests aren’t one size fits all. They are not all designed to measure the same thing.

• Like tools, various tests do different jobs. It’s important to use the right tool for the job.

• Some quizzes and tests measure learning along the way and help guide next steps in the classroom. Others tests, like final exams and end-of-the year state tests, assess what’s been learned over a long period of time.

• Some tests measure performance of a school, a district, or a state – but they aren’t designed to evaluate a student’s individual performance.

Interpreting Results of Tests

• Test scores are more than a single number. They offer several ways to interpret a child’s performance.

• If you know what’s on the test and how it’s being scored, you can better understand the performance of your child and schools.

• It’s important to understand how test results will be used to help your child or school.

Validity or Appropriateness of Tests

• The best tests have clearly stated purposes and are designed to achieve them.

• Some tests, such as the SAT, can be used to predict whether a student will succeed in the future. Others have different purposes, such as measuring what students have learned during a school year.

• If a test is designed to evaluate a student’s progress in school, it needs to focus on what the child is being taught. It shouldn’t include anything unrelated to the subject or unfamiliar to the child.

How Tests Can Help the Learning Process

• If classroom assessments are routinely employed as part of the ongoing instructional process to measure what students are learning, and thus permit both teachers and students to decide whether to adjust what they are doing, students learn substantially better.

1 Thank and Terminate script: “Thank you for answering my questions. Based on the requirements of this study, we are not able to include you in the interviews at this time.”

2