NAEP Assessment Literacy Concepts Focus Groups 2014

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement, OMB Information

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1800-0011. The time required to complete this information collection is estimated to average < 90, 120 > minutes including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, D.C. 20202-4537. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: NAEP/NCES, U.S. Department of Education, 1990 K Street, N.W., 9^th floor, Washington, DC 20006.

This is a project of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences, within the U.S. Department of Education.

Your feedback may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].

Focus Group Protocol

Part I. Interviewer Welcome Scripts and Assent/Consent

A. Parent/Guardian Participant Welcome Script

NOTE: Text placed in brackets provides notes and cues for the moderator.

[Welcome participants and thank them for participating in this focus group.]

My name is <NAME> and I have been hired to facilitate our discussion today. I am an independent moderator – I’m not an expert on the topic we are going to discuss today. I am primarily interested in your views on this topic – my views don’t matter. During our discussion, you may have questions that I can’t answer, but at the end of our discussion I will refer you to resources for more information.

[Sponsor and purpose of the group]

We’re here to discuss educational testing in schools on behalf of the National Center for Education Statistics (NCES), which is part of the U.S. Department of Education. NCES collects, analyzes, and makes available data related to education in the U.S. and other nations. You might be familiar with an educational test NCES manages – it’s called the National Assessment of Educational Progress (NAEP). Many know it as “The Nation’s Report Card.” NAEP is the largest national assessment of what America's students know and can do in various subject areas, including math, reading, science, writing, the arts, civics, economics, geography, and U.S. history.

We’ll talk just a little bit about NAEP this evening, but we’re really interested in a variety of different tests your child will experience in his or her school years. We are also interested in talking about educational testing in general, including the questions you might have about it.

[Focus group process and general rules]

We will be looking at some statements and materials throughout the group today; please make sure you have reading glasses if you need them.
I did not create any of the materials we are going to talk about today, so please feel free to be open and honest in your comments.
We want to have an open discussion. It is important to hear what everyone thinks, so please speak up and please let others do the same.
You don’t have to wait for me to call on you, but please speak one at a time.
You can respond to each other as well as to me.
You might disagree about certain topics, but please be respectful of each other.
There are no right or wrong answers. All comments, positive and negative, what you know and don’t know, are important.
If I ask a question you’re not comfortable answering, feel free not to answer.
We are recording the discussion so I can write a report. However, no names will be included in the report and all comments mentioned in the report will be anonymous.
There are some interested colleagues listening to our discussion behind the glass.
My job is to keep us on the right track and moving along so that we’re done in two hours. From time to time, I may have to jump in to keep our conversation moving.
If you need to go to the restroom or need a break, feel free to step away, but please come back as soon as possible because all opinions are important.
Please turn off or silence all of your electronic devices to avoid disturbing others.

Do you have any questions before we get started?

[Participant Introductions]

Let’s get started with introductions - Please tell us your first name, and what grade your child is in at his or her school.

B. Student Participant Welcome Script

NOTE: Text placed in brackets provides notes and cues for the moderator.

[Welcome participants and thank them for participating in this focus group.]

[Sponsor and purpose of the group]

We’re here to discuss educational testing in schools on behalf of the National Center for Education Statistics (NCES), which is part of the U.S. Department of Education. NCES collects, analyzes and makes available data related to education in the U.S. and other nations. You might be familiar with an educational test NCES manages – it’s called the National Assessment of Educational Progress (NAEP). Many know it as “The Nation’s Report Card.” NAEP is the largest national assessment of what America's students know and can do in various subject areas, including math, reading, science, writing, the arts, civics, economics, geography, and U.S. history.

We’ll talk just a little bit about NAEP this evening, but we’re really interested in a variety of different tests you will experience in school. We are also interested in talking about educational testing in general, including the questions you might have about it.

[Focus group process and general rules]

I did not create any of the materials we are going to talk about today, so please feel free to be open and honest in your comments.
We want to have an open discussion. It is important to hear what everyone thinks, so please speak up and please let others do the same.
You don’t have to wait for me to call on you, but please speak one at a time.
You can respond to each other as well as to me.
You might disagree about certain topics, but please be respectful of each other.
There are no right or wrong answers. All comments, positive and negative, what you know and don’t know, are important.
If I ask a question you’re not comfortable answering, feel free not to answer.
We are recording the discussion so I can write a report. However, no names will be included in the report and all comments mentioned in the report will be anonymous.
There are some interested colleagues listening to our discussion behind the glass.
My job is to keep us on the right track and moving along so that we’re done in an hour and a half. From time to time, I may have to jump in to keep our conversation moving.
If you need to go to the restroom or need a break, feel free to step away, but please come back as soon as possible because all opinions are important.
Please turn off or silence all of your electronic devices to avoid disturbing others.

Do you have any questions before we get started?

[Participant Introductions]

Let’s get started with introductions - Please tell us your first name, what school you attend, and what grade you are in.

C. Policymaker Participant Welcome Script

Text written in italics is to be read aloud by the staff facilitator.

NOTE: Text placed in brackets provides notes and cues for the moderator.

[Welcome participants and thank them for participating in this focus group.]

[Sponsor and purpose of the group]

[Explain WebEx information]

We’re here to discuss educational testing in schools on behalf of the National Center for Education Statistics (NCES), which is part of the U.S. Department of Education. NCES collects, analyzes and makes available data related to education in the U.S. and other nations. You might be familiar with an educational test NCES manages – it’s called the National Assessment of Educational Progress (NAEP). Many know it as “The Nation’s Report Card.” NAEP is the largest national assessment of what America's students know and can do in various subject areas, including math, reading, science, writing, the arts, civics, economics, geography, and U.S. history.

We’ll talk just a little bit about NAEP this evening, but we’re really interested in a variety of different tests administered in the schools you oversee. We are also interested in talking about educational testing in general, including the questions you might have about it.

[Focus group process and general rules]

We will be looking at some statements and materials throughout the group today; please make sure you have reading glasses if you need them.
I did not create any of the materials we are going to talk about today, so please feel free to be open and honest in your comments.
We want to have an open discussion. It is important to hear what everyone thinks, so please speak up and please let others do the same.
You don’t have to wait for me to call on you, but please speak one at a time.
You can respond to each other as well as to me.
You might disagree about certain topics, but please be respectful of each other.
There are no right or wrong answers. All comments, positive and negative, what you know and don’t know, are important.
If I ask a question you’re not comfortable answering, feel free not to answer.
We are recording the discussion so I can write a report. However, no names will be included in the report and all comments mentioned in the report will be anonymous.
There are some interested colleagues listening to our discussion behind the glass.
My job is to keep us on the right track and moving along so that we’re done in an hour and a half. From time to time, I may have to jump in to keep our conversation moving.
If you need to go to the restroom or need a break, feel free to step away, but please come back as soon as possible because all opinions are important.
Please turn off or silence all of your electronic devices to avoid disturbing others.

Do you have any questions before we get started?

[Participant Introductions]

Let’s get started with introductions - Please tell us your name and a description of the schools you oversee.

D. Teacher Participant Welcome Script

NOTE: Text placed in brackets provides notes and cues for the moderator.

[Welcome participants and thank them for participating in this focus group.]

[Sponsor and purpose of the group]

[Explain WebEx information]

We’re here to discuss educational testing in schools on behalf of the National Center for Education Statistics (NCES), which is part of the U.S. Department of Education. NCES collects, analyzes and makes available data related to education in the U.S. and other nations. You might be familiar with an educational test NCES manages – it’s called the National Assessment of Educational Progress (NAEP). Many know it as “The Nation’s Report Card.” NAEP is the largest national assessment of what America's students know and can do in various subject areas, including math, reading, science, writing, the arts, civics, economics, geography, and U.S. history.

[Focus group process and general rules]

We will be looking at some statements and materials throughout the group today; please make sure you have reading glasses if you need them.
I did not create any of the materials we are going to talk about today, so please feel free to be open and honest in your comments.
We want to have an open discussion. It is important to hear what everyone thinks, so please speak up and please let others do the same.
You don’t have to wait for me to call on you, but please speak one at a time.
You can respond to each other as well as to me.
You might disagree about certain topics, but please be respectful of each other.
There are no right or wrong answers. All comments, positive and negative, what you know and don’t know, are important.
If I ask a question you’re not comfortable answering, feel free not to answer.
We are recording the discussion so I can write a report. However, no names will be included in the report and all comments mentioned in the report will be anonymous.
There are some interested colleagues listening to our discussion behind the glass.
My job is to keep us on the right track and moving along so that we’re done in an hour and a half. From time to time, I may have to jump in to keep our conversation moving.
If you need to go to the restroom or need a break, feel free to step away, but please come back as soon as possible because all opinions are important.
Please turn off or silence all of your electronic devices to avoid disturbing others.

Do you have any questions before we get started?

[Participant Introductions]

Let’s get started with introductions - Please tell us your name and a description of the schools you oversee.

Part II. Focus Group Questions

NOTE: Text placed in brackets provides notes and cues for the moderator.

The purpose of the focus groups is to inform the development of assessment literacy among parents, students, and policymakers. Below are the probes that students, parents/guardians, and policymakers will be asked during the focus group session. Please note that some probes may not be addressed if time does not permit.

Parent/Guardian (120 minute sessions)

GENERAL AWARENESS / KNOWLEDGE / ATTITUDES REGARDING EDUCATIONAL TESTING

So, let’s get started by talking about the tests your child has taken in school – what are some examples? [Probe, if necessary: Classroom tests, the Maryland School Assessment, the High School Assessment, the SAT, the AP test, NAEP, etc.]
Do these tests have similar or different purposes? What are some of the purposes of various tests? [Probe, if necessary: Classrooms tests, the Maryland School Assessment, the High School Assessment, the SAT, the AP test, NAEP, etc.]
We’ve mentioned quite a few tests – how easy or difficult is it to keep track of which test does what?
What questions in general do you have about the tests your child takes at school? What are some of the things you want to know about them?
1. Probe, if necessary:
  1. What are different types of tests?
  2. What are common functions of tests?
  3. What is each test intended to measure?
  4. What is a particular test supposed to be telling me?
  5. How should I interpret the results of various tests? (e.g., what they mean and don’t mean)
  6. How do various tests align with what students learn in the classroom?

EXPLORATION OF KEY CONCEPTS

NCES plans to develop educational materials to inform parents and others about the various types of tests students take in their school years. These materials will cover several things – the purpose of various tests, how they are scored, how the results are used, and how parents should interpret the results of various tests. I’d like to get your opinion on some language that may be used for these materials.

First let’s start with some statements that talk about the purpose of various tests.

[Hand out one page with the following statements]

Let’s look at a few statements – read along with me as I read aloud.

The Purpose of Educational Tests

Tests aren’t one size fits all. They are not all designed to measure the same thing.
Like tools, various tests do different jobs. It’s important to use the right tool for the job.
Some quizzes and tests measure learning along the way and help guide next steps in the classroom. Others tests, like final exams and end-of-the year state tests, assess what’s been learned over a long period of time.
Some tests measure performance of a school, a district, or a state – but they aren’t designed to evaluate your child’s performance.

Let’s talk through these statements one by one, starting with the first….

[Ask the following questions for each of the statements]

Is this statement pretty straightforward and clear?
Is there anything confusing about it?
How important is this statement to you as you think about your child’s education?
What do you like about this statement? What do you dislike?
Do you have any questions after reading this statement?
What questions do you have?
If you had to tell a friend or another parent about this topic, what would you say? What are the top three things?

I have another handout for you – I have a chart that shows you examples of various tests your child may take in the state of Maryland. [Hand out chart on assessments]

I’ll hand it out now, and I would like for you to take a few moments to look at the information.

Does the chart give you enough information about the purpose of the various tests?
1. Do you feel there’s anything missing?
How well do you think it answers the following questions?
1. What are different types of tests?
2. What are common functions of tests?
3. What is each test intended to measure?

Let’s set aside the chart for a moment because I want to talk about some statements that might help us understand how to interpret the results of various educational tests. [Hand out one page with the following statements].

Let’s look at a few more statements – read along with me as I read aloud.

Interpreting Results of Tests

Test scores are more than a single number. They offer several ways to interpret a child’s performance.
If you know what’s on the test and how it’s being scored, you can better understand the performance of your child and schools.
It’s important to understand how test results will be used to help your child or school.

Let’s talk through these statements one by one, starting with the first….

[Ask the following questions for each of the statements]

Is this statement pretty straightforward and clear?
Is there anything confusing about it?
How important is this statement to you as you think about your child’s education?
What do you like about this statement? What do you dislike?
Do you have any questions after reading this statement?
1. What questions do you have?

If you had to tell a friend or another parent about this topic, what would you say? What are the top three things?

Now, let’s look at our chart again - When you look at the chart, do you think it gives you enough information about how to interpret the results of each of the tests?

Do you feel there’s anything missing?
How well do you think it answers the following questions?
1. What is a particular test supposed to be telling me?
2. How should I interpret the results of various tests? (e.g., what they mean and don’t mean)
3. How do various tests align with what students learn in the classroom?

So, we’ve talked about the purpose of various tests, and we’ve talked about how to interpret results from various tests. Now I want to talk about how you can tell if a test is being used appropriately to measure the results it was designed to measure.

Has anyone ever had questions about how a test is being used or how it was designed? What are some questions you’ve had?

Probe, if necessary:

What makes a test appropriate for a certain function?
How do I know if a test is being used appropriately?
How do I know if a test is a “fair” test – e.g.,

Is the length appropriate?
Does it measure what a student had an opportunity to learn?
Was it administered appropriately?
Is it at an appropriate difficulty level?
Is it age/grade level appropriate?
Is it biased in terms of culture, gender, socioeconomic, or linguistic factors?

When we are talking through these questions, we are talking about a test’s validity – you can also think of it as a test’s appropriateness for the job it is being used to do.

These statements explore the validity of tests. [Hand out one page with the following statements].

Let’s look at a few statements – read along with me as I read aloud.

Validity or Appropriateness of Tests

The best tests have clearly stated purposes and are designed to achieve them.
Some tests, such as the SAT, can be used to predict whether a student will succeed in the future. Others have different purposes, such as measuring what students have learned during a school year.
If a test is designed to evaluate a student’s progress in school, it needs to focus on what the child is being taught. It shouldn’t include anything unrelated to the subject or unfamiliar to the child.

Let’s talk through these statements one by one, starting with the first….

[Ask the following questions for each of the statements]

Is this statement pretty straightforward and clear?
Is there anything confusing about it?
How important is this statement to you as you think about your child’s education?
What do you like about this statement? What do you dislike?
Do you have any questions after reading this statement?
1. What questions do you have?
Do you have any other questions overall about this topic?

I have one last statement for you about how tests can help the learning process. [Hand out one page with the following statement].

Read along with me as I read aloud.

How Tests Can Help the Learning Process

If classroom assessments are routinely employed as part of the ongoing instructional process to measure what students are learning, and thus permit both teachers and students to decide whether to adjust what they are doing, students learn substantially better.

Let’s talk through this statement.

Is this statement pretty straightforward and clear?
Is there anything confusing about it?
How important is this statement to you as you think about your child’s education?
What do you like about this statement? What do you dislike?
Do you have any questions after reading this statement?

What questions do you have?

Do you have any other questions overall about this topic?

If you had to tell a friend or another parent about this topic, what would you say? What are the top three things?

CONCLUSION

We’ve now spent two hours or so talking about educational tests. Have your perceptions or feelings about this topic changed from before this discussion?

If yes, how have they changed?

Is there more information you would like to know on this topic?

What would be of interest to you?
Are you more likely now to take a look at sample test questions or take a sample test, for example?

How would you like to receive more information about educational testing?

Through your child’s teacher or school principal?
Through media? Are there particular channels, magazines, web sites, or other media outlets you pay attention to? What are these?

Are there other sources you would trust for this information? What or who are these?
That’s all of my questions. Do you have any final comments on anything we’ve talked about?
I’d like to share several web sites that provide more information about the tests and concepts we discussed this evening [Distribute one page with URLs]:

A Parent’s Guide to the MSA: http://marylandpublicschools.org/NR/rdonlyres/D880A550-54C3-4621-AE7E-1AC0CABC022C/34521/Parents_Guide_MSA_2012_r012012_.pdf
district, or a state: High School Assessment Program: http://mdk12.org/assessments/high_school/index.html
NAEP: http://nces.ed.gov/nationsreportcard/
Thank you so much; your help has been invaluable. [Provide a gift card, if applicable].

B. Student (90 minute sessions)

GENERAL AWARENESS / KNOWLEDGE / ATTITUDES REGARDING EDUCATIONAL TESTING

So, let’s get started by talking about the tests you have taken in school – what are some examples? [Probe, if necessary: Classroom tests, the Standards of Learning (SOL) Tests, the SAT, the AP test, NAEP, etc.]
Do these tests have similar or different purposes? What are some of the purposes of various tests? [Probe, if necessary: Classrooms tests, the Standards of Learning (SOL) Tests, the High School Assessment, the SAT, the AP test, NAEP, etc.]
We’ve mentioned quite a few tests – how easy or difficult is it to keep track of which test does what?
What questions in general do you have about the tests you take at school? What are some of the things you want to know about them?
1. Probe, if necessary:
  1. What are different types of tests?
  2. What is the purpose of each test?
  3. What is each test intended to measure?
  4. What are the results of a particular test supposed to tell me?
  5. How should I interpret the results of various tests? (e.g., what they mean and don’t mean)
  6. How do various tests relate to what I learn in the classroom?
  7. How can I do better on tests?
Let’s talk a bit about your personal experiences with standardized tests, like the Standards of Learning (SOL) Tests, the SAT, the AP test, or the NAEP.
1. Who has taken one of these tests?
  1. Which one?
2. In general, how did it go?
  1. Did the content of the test reflect what you have been learning in your classroom?
    1. Did the test seem easier or harder than what you’ve been learning in the classroom?
  2. Did you feel prepared?
    1. Why or why not?
  3. Did your teacher or school do anything special to help you prepare for the test?
  4. Can you think of anything that might have helped you feel more prepared?

EXPLORATION OF KEY CONCEPTS

NCES plans to develop educational materials to help students better understand the various types of tests they take in their school years. I’d like to get your opinion on some language that may be used for these materials.

First let’s start with some statements about tests and test taking. [Hand out one page with the following statements]

Let’s look at a few statements – read along with me as I read aloud.

The Importance of Effort

Although your performance on an educational test can sometimes predict how well you will do later on tests and with grades, the level of effort you expend in trying to master the skills and knowledge being assessed is much more important in determining your later success.

Let’s talk through this statement….

Is this statement pretty straightforward and clear?
Is there anything confusing about it?
How important is this statement to you as you think about your education?
What do you like about this statement? What do you dislike?
Do you have any questions after reading this statement?
What questions do you have?
If you had to tell a friend about this, what would you say?

The Purpose of Educational Tests

Tests aren’t one size fits all. They are not all designed to measure the same thing.
Like tools, various tests do different jobs. It’s important to use the right tool for the job.
Some quizzes and tests measure learning along the way and help guide next steps in the classroom. Others tests, like final exams and end-of-the year state tests, assess what’s been learned over a long period of time.
Some tests measure performance of a school, a district, or a state – but they aren’t designed to evaluate your individual performance.

Let’s talk through these statements one by one, starting with the first….

[Ask the following questions for each of the statements]

Is this statement pretty straightforward and clear?
Is there anything confusing about it?
How important is this statement to you as you think about your education?
What do you like about this statement? What do you dislike?
Do you have any questions after reading this statement?
What questions do you have?
If you had to tell a friend about this, what would you say?

The Accuracy of Test Scores

Assessments provide an estimate of student performance and learning in a particular area.
Assessments are often believed to be precise indicators of student learning, but this is not always true. The accuracy of a test depends on what the test was designed to measure.
Because assessments provide only an estimate, multiple assessments can provide a clearer idea of student performance than a single assessment.

Let’s talk through these statements one by one, starting with the first….

[Ask the following questions for each of the three statements]

Is this statement pretty straightforward and clear?
Is there anything confusing about it?
How important is this statement to you as you think about your education?
What do you like about this statement? What do you dislike?
Do you have any questions after reading this statement?
What questions do you have?
If you had to tell a friend about this, what would you say?

How Tests Can Help the Learning Process

Although some educational tests are intended to evaluate what a student has learned, other tests are intended to help students decide whether they need to alter the way they are trying to learn something. This distinction between the two different uses of tests can often permit students to monitor their own learning as part of the test-informed learning process.

Let’s talk through this statement...

[Ask the following questions for each of the four statements]

Is this statement pretty straightforward and clear?
Is there anything confusing about it?
How important is this statement to you as you think about your education?
What do you like about this statement? What do you dislike?
Do you have any questions after reading this statement?
What questions do you have?
If you had to tell a friend about this, what would you say?

The Importance of Preparing for Tests

Research shows that students who use effective test-taking procedures tend to perform well on educational tests. Therefore, it is important for all students to understand the most common test-taking tactics that have been proven effective.

Let’s talk through this statement...

[Ask the following questions for each of the four statements]

Is this statement pretty straightforward and clear?
Is there anything confusing about it?
How important is this statement to you as you think about your education?
What do you like about this statement? What do you dislike?
Do you have any questions after reading this statement?
What questions do you have?
If you had to tell a friend about this, what would you say?

CONCLUSION

We’ve now spent one and a half hours or so talking about educational tests. Have your perceptions or feelings about this topic changed from before this discussion?

If yes, how have they changed?

Is there more information you would like to know on this topic?

What would be of interest to you?

How would you like to receive more information about educational testing?

In school?

That’s all of my questions. Do you have any final comments on anything we’ve talked about?

Thank you so much; your help has been invaluable. [Provide a gift card, if applicable].

Policymaker (90 minute sessions)

GENERAL AWARENESS / KNOWLEDGE / ATTITUDES REGARDING EDUCATIONAL TESTING

So, let’s get started by talking about the tests that are administered in the schools you oversee. What are examples of these? [High School Assessment, the SAT, the AP test, NAEP, etc.]

Do these tests have similar or different purposes? What are some of the purposes of various tests? [Probe, if necessary: the High School Assessment, the SAT, the AP test, NAEP, etc.]
Have you had any challenges in communicating about these tests with parents, students, other policymakers, or the general public?
1. What challenges have you had?
2. Do you feel you need more tools or resources to explain educational tests to parents, students, and the general public?
3. What tools and resources have you used in the past?
What questions in general do you have about the tests? What are some of the things you want to know about them?
1. Probe, if necessary:
  1. The quality of the test
  2. The validity of the test
  3. How it was designed
  4. What each test is intended to measure
  5. How to interpret the results (e.g., what they mean and don’t mean)
  6. How various tests align with what students learn in the classroom
Each educational test has technical documentation that provides details about the purpose of the test, what it was designed to measure, how items were developed, how precise of estimate of student learning it provides, and other aspects of the test. Have you ever reviewed this type of technical documentation for a test?
1. Was it helpful or not helpful?
2. Is there a format for this information that would be more useful for you?
3. What are the key points you would want a new format to cover?

I want to talk about how you can tell if a test is being used appropriately to measure the results it was designed to measure. Has anyone ever had questions about how a test is being used or how it was designed? What are some questions you’ve had?

Probe, if necessary:

What makes a test appropriate for a certain function?
How do you know if a test is being used appropriately?
How do you know if a test is a “fair” test – e.g.,
- Is the length appropriate?
- Does it measure what a student had an opportunity to learn?
- Was it administered appropriately?
- Is it at an appropriate difficulty level?
- Is it age/grade level appropriate
- Is it biased in terms of culture, gender, socioeconomic, or linguistic factors?

What resources do you currently use to make decisions about the tests that are administered in your schools?

EXPLORATION OF KEY CONCEPTS

NCES plans to develop educational materials to guide educational policymakers about certain aspects of educational tests, such as the purpose of various tests, how they are scored, how the results are to be used, and how results should be interpreted. I’d like to get your opinion on some language that may be used for these materials.

First let’s start with a statement about the purpose and quality of various tests.

[Hand out one page with the following statements]

Let’s look at a few statements…

The Purpose of Educational Tests

Because educational tests are employed for many important purposes, it is imperative that persuasive evidence of an educational test’s appropriateness be available for each function that the test’s results are asked to fulfill.

Let’s talk through this statement….

[Ask the following questions for each of the four statements]

Is this statement pretty straightforward and clear?
Is there anything confusing about it?
How important is this statement to you as an administrator?
What do you like about this statement? What do you dislike?
Do you have any questions after reading this statement?
What questions do you have?
How would you describe this to a colleague?

The Quality of Tests

Because educational assessments, whether created by teachers for use in their classrooms or by large test-development organizations differ—sometimes dramatically—in their quality without persuasive evidence of high quality, it should not be assumed that an educational test is good enough to be used.

Let’s talk through this statement….

[Ask the following questions for each of the four statements]

Is this statement pretty straightforward and clear?
Is there anything confusing about it?
How important is this statement to you as an administrator?
What do you like about this statement? What do you dislike?
Do you have any questions after reading this statement?
What questions do you have?
How would you describe this to a colleague?

Accuracy of Test Scores

Although many people, educators as well as non-educators, believe that educational tests provide a remarkably accurate fix regarding test-takers’ status, the reality is that educational tests are considerably less accurate than widely thought.

Let’s talk through this statement….

Is this statement pretty straightforward and clear?
Is there anything confusing about it?
How important is this statement to you as an administrator?
What do you like about this statement? What do you dislike?
Do you have any questions after reading this statement?
What questions do you have?
How would you describe this to a colleague?

The Need for Interpretable Analyses

Although many reports regarding such topics as the function and quality of educational tests are written in technical and abstruse language, if assessment specialists are directed by educational policymakers to provide readily comprehensible analyses, such reports can be produced.

Let’s talk through this statement….

Is this statement pretty straightforward and clear?
Is there anything confusing about it?
How important is this statement to you as an administrator?
What do you like about this statement? What do you dislike?
Do you have any questions after reading this statement?
What questions do you have?
How would you describe this to a colleague?

CONCLUSION

We’ve now spent one and a half hours or so talking about educational tests. Have your perceptions or feelings about this topic changed from before this discussion?

If yes, how have they changed?

Is there more information you would like to know on this topic?

What would be of interest to you?

How would you like to receive more information about educational testing?

That’s all of my questions. Do you have any final comments on anything we’ve talked about?

Thank you so much; your help has been invaluable. For your participation today, you will receive a $40 Visa gift card. Your gift card will be mailed to you within two to three weeks.

D. Teacher (90 minute sessions)

GENERAL AWARENESS / KNOWLEDGE / ATTITUDES REGARDING EDUCATIONAL TESTING

So, let’s get started by talking about the tests that are administered in the schools in which you teach. What are examples of these? [High School Assessment, the SAT, the AP test, NAEP, etc.]
Do these tests have similar or different purposes? What are some of the purposes of various tests? [Probe, if necessary: the High School Assessment, the SAT, the AP test, NAEP, etc.]
Have you had any challenges in communicating about these tests with parents, students, policymakers, or the general public?
1. What challenges have you had?
2. Do you feel you need more tools or resources to explain educational tests to parents, students, and the general public?
3. What tools and resources have you used in the past?
What questions in general do you have about the tests? What are some of the things you want to know about them?
1. Probe, if necessary:
  1. The quality of the test
  2. The validity of the test
  3. How it was designed
  4. What each test is intended to measure
  5. How to interpret the results (e.g., what they mean and don’t mean)
  6. How various tests align with what students learn in the classroom
Each educational test has technical documentation that provides details about the purpose of the test, what it was designed to measure, how items were developed, how precise of estimate of student learning it provides, and other aspects of the test. Have you ever reviewed this type of technical documentation for a test?
1. Was it helpful or not helpful?
2. Is there a format for this information that would be more useful for you?
3. What are the key points you would want a new format to cover?

Probe, if necessary:

What makes a test appropriate for a certain function?
How do you know if a test is being used appropriately?
How do you know if a test is a “fair” test – e.g.,

Is the length appropriate?
Does it measure what a student had an opportunity to learn?
Was it administered appropriately?
Is it at an appropriate difficulty level?
Is it age/grade level appropriate?
Is it biased in terms of culture, gender, socioeconomic, or linguistic factors?

What resources do you currently use to make decisions about the tests that are administered in your schools?

EXPLORATION OF KEY CONCEPTS

NCES plans to develop educational materials to guide parents, students, and policymakers about certain aspects of educational tests, such as the purpose of various tests, how they are scored, how the results are to be used, and how results should be interpreted. I’d like to get your opinion on some language that may be used for these materials, particularly among parents and students.

First let’s start with a statement about the purpose and quality of various tests.

[Hand out one page with the following statements]

Let’s look at a few statements…

The Purpose of Educational Tests

Tests aren’t one size fits all. They are not all designed to measure the same thing.
Like tools, various tests do different jobs. It’s important to use the right tool for the job.
Some quizzes and tests measure learning along the way and help guide next steps in the classroom. Others tests, like final exams and end-of-the year state tests, assess what’s been learned over a long period of time.
Some tests measure performance of a school, a district, or a state – but they aren’t designed to evaluate a student’s individual performance.

Let’s talk through these statements one by one, starting with the first….

[Ask the following questions for each of the statements]

Is this statement pretty straightforward and clear?
Is there anything confusing about it?
How important is this statement in helping parents think about their child’s education?
What do you like about this statement? What do you dislike?
Do you have any questions after reading this statement?
What questions do you have?
Do you think this will be helpful for parents? For students?

I have another handout for you – I have a chart that shows you examples of various tests students may take in the state of Maryland. [Hand out chart on assessments]

I’ll hand it out now, and I would like for you to take a few moments to look at the information.

1. Does the chart give parents enough information about the purpose of the various tests?

a. Do you feel there’s anything missing?

2. How well do you think it answers the following questions?

a. What are different types of tests?

b. What are common functions of tests?

c. What is each test intended to measure?

Let’s look at a few more statements – read along with me as I read aloud.

Interpreting Results of Tests

Test scores are more than a single number. They offer several ways to interpret a child’s performance.

If you know what’s on the test and how it’s being scored, you can better understand the performance of your child and schools.

It’s important to understand how test results will be used to help your child or school.

Let’s talk through these statements one by one, starting with the first….

[Ask the following questions for each of the statements]

Is this statement pretty straightforward and clear?
Is there anything confusing about it?
How important is this statement in helping parents think about their child’s education?
What do you like about this statement? What do you dislike?
Do you have any questions after reading this statement?
What questions do you have?
Do you think this will be helpful for parents? For students?

Now, let’s look at our chart again - When you look at the chart, do you think it gives you enough information about how to interpret the results of each of the tests?

Do you feel there’s anything missing?
How well do you think it answers the following questions for parents?
1. What is a particular test supposed to be telling me?
2. How should I interpret the results of various tests? (e.g., what they mean and don’t mean)
3. How do various tests align with what students learn in the classroom?

Has anyone ever had questions about how a test is being used or how it was designed? What are some questions you’ve had?

Probe, if necessary:

What makes a test appropriate for a certain function?
How do I know if a test is being used appropriately?
How do I know if a test is a “fair” test – e.g.,

Is the length appropriate?
Does it measure what a student had an opportunity to learn?
Was it administered appropriately?
Is it at an appropriate difficulty level?
Is it age/grade level appropriate?
Is it biased in terms of culture, gender, socioeconomic or linguistic factors?

When we are talking through these questions, we are talking about a test’s validity – you can also think of it as a test’s appropriateness for the job it is being used to do.

These statements explore the validity of tests. [Hand out one page with the following statements].

Let’s look at a few statements – read along with me as I read aloud.

Validity or Appropriateness of Tests

The best tests have clearly stated purposes and are designed to achieve them.
Some tests, such as the SAT, can be used to predict whether a student will succeed in the future. Others have different purposes, such as measuring what students have learned during a school year.
If a test is designed to evaluate a student’s progress in school, it needs to focus on what the child is being taught. It shouldn’t include anything unrelated to the subject or unfamiliar to the child.

Let’s talk through these statements one by one, starting with the first….

[Ask the following questions for each of the statements]

Is this statement pretty straightforward and clear?
Is there anything confusing about it?
How important is this statement in helping parents think about their child’s education?
What do you like about this statement? What do you dislike?
Do you have any questions after reading this statement?
What questions do you have?
Do you think this will be helpful for parents? For students?

I have one last statement for you about how tests can help the learning process. [Hand out one page with the following statement].

Read along with me as I read aloud.

How Tests Can Help the Learning Process

Let’s talk through this statement.

Is this statement pretty straightforward and clear?
Is there anything confusing about it?
How important is this statement in helping parents think about their child’s education?
What do you like about this statement? What do you dislike?
Do you have any questions after reading this statement?
What questions do you have?
Do you think this will be helpful for parents? For students?

CONCLUSION

We’ve now spent one and a half hours or so talking about educational tests. Have your perceptions or feelings about this topic changed from before this discussion?

If yes, how have they changed?

Is there more information you would like to know on this topic?

What would be of interest to you?

How would you like to receive more information about educational testing?

That’s all of my questions. Do you have any final comments on anything we’ve talked about?

Thank you so much; your help has been invaluable. For your participation today, you will receive a $40 VISA gift card. Your gift card will be mailed to you within two to three weeks.

Attachment A: Handout Materials for Parent/Guardian Focus Group

Handout Materials for Parent/Guardian Focus Group

The Purpose of Educational Tests

Tests aren’t one size fits all. They are not all designed to measure the same thing.
Like tools, various tests do different jobs. It is important to use the right tool for the job.
Some quizzes and tests measure learning along the way and help guide next steps in the classroom. Others tests, like final exams and end-of-the year state tests, assess what’s been learned over a long period of time.
Some tests measure performance of a school, a district, or a state – but they aren’t designed to evaluate your child’s performance.

Interpreting Results of Tests

Test scores are more than a single number. They offer several ways to interpret a child’s performance.
If you know what’s on the test and how it’s being scored, you can better understand the performance of your child and schools.
It’s important to understand how test results will be used to help your child or school.

Validity or Appropriateness of Tests

The best tests have clearly stated purposes and are designed to achieve them.
Some tests, such as the SAT, can be used to predict whether a student will succeed in the future. Others have different purposes, such as measuring what students have learned during a school year.
If a test is designed to evaluate a student’s progress in school, it needs to focus on what the child is being taught. It shouldn’t include anything unrelated to the subject or unfamiliar to the child.

How Tests Can Help the Learning Process

Resources for More Information:

A Parent’s Guide to the MSA: http://marylandpublicschools.org/NR/rdonlyres/D880A550-54C3-4621-AE7E-1AC0CABC022C/34521/Parents_Guide_MSA_2012_r012012_.pdf
HSA: High School Assessment Program: http://mdk12.org/assessments/high_school/index.html
NAEP: http://nces.ed.gov/nationsreportcard/

Attachment C: Handout Materials for Student Focus Group

Handout Materials for Student Focus Group

The Importance of Effort

The Purpose of Educational Tests

Tests aren’t one size fits all. They are not all designed to measure the same thing.
Like tools, various tests do different jobs. It’s important to use the right tool for the job.
Some quizzes and tests measure learning along the way and help guide next steps in the classroom. Others tests, like final exams and end-of-the year state tests, assess what’s been learned over a long period of time.
Some tests measure performance of a school, a district, or a state – but they aren’t designed to evaluate your individual performance.

The Accuracy of Test Scores

Assessments provide an estimate of student performance and learning in a particular area.
Assessments are often believed to be precise indicators of student learning, but this is not always true. The accuracy of a test depends on what the test was designed to measure.
Because assessments provide only an estimate, multiple assessments can provide a clearer idea of student performance than a single assessment.

How Tests Can Help the Learning Process

The Importance of Preparing for Tests

Attachment D: Handout Materials for Policymaker Focus Group

Handout Materials for Policymaker Group

The Purpose of Educational Tests

The Quality of Tests

Because educational assessments, whether created by teachers for use in their classrooms or by large test-development organizations, differ—sometimes dramatically—in their quality, without persuasive evidence of high quality, it should not be assumed that an educational test is good enough to be used.

Accuracy of Test Scores

The Need for Interpretable Analyses

Attachment E: Handout Materials for Teacher Focus Group

Handout Materials for Teacher Focus Group

The Purpose of Educational Tests

Tests aren’t one size fits all. They are not all designed to measure the same thing.
Like tools, various tests do different jobs. It’s important to use the right tool for the job.
Some quizzes and tests measure learning along the way and help guide next steps in the classroom. Others tests, like final exams and end-of-the year state tests, assess what’s been learned over a long period of time.
Some tests measure performance of a school, a district, or a state – but they aren’t designed to evaluate a student’s individual performance.

Interpreting Results of Tests

Test scores are more than a single number. They offer several ways to interpret a child’s performance.
If you know what’s on the test and how it’s being scored, you can better understand the performance of your child and schools.
It’s important to understand how test results will be used to help your child or school.

Validity or Appropriateness of Tests

The best tests have clearly stated purposes and are designed to achieve them.
Some tests, such as the SAT, can be used to predict whether a student will succeed in the future. Others have different purposes, such as measuring what students have learned during a school year.
If a test is designed to evaluate a student’s progress in school, it needs to focus on what the child is being taught. It shouldn’t include anything unrelated to the subject or unfamiliar to the child.

How Tests Can Help the Learning Process

AutoShape 1

File Type	application/msword
Author	Debby E. Almonte
Last Modified By	Kubzdela,Kashka
File Modified	2014-10-01
File Created	2014-09-11

NAEP Assessment Literacy Concepts Focus Groups 2014

Generic Clearance for the Collection of Qualitative Feedback on Agency Service Delivery

Volume II NAEP Assessment Literacy Focus Groups 2014 - Protocol