1) NPSAS:12 Cognitive Testing 2010; and 2) Program for International Student Assessment (PISA) 2012 Participation Improvement Focus Groups

Program for International Student Assessment (PISA)

Principal Focus Group Recruitment Script

(will be either email, letter, or phone call)

Dear _____________,

My name is ______________ and I am writing on behalf of NCES, the National Center for Education Statistics. NCES supports international efforts to collect and report data on education. In 2009, your school was selected for PISA or the Program for International Student Assessment. As you may recall, PISA is a system of international assessments that focus on 15-year-olds' performance in reading, math and science literacy every three years. At present, we are preparing for PISA 2012 and would like to better understand the factors that motivated you and your students to participate or not participate in PISA 2009.

To that end, we would like to meet in Washington, DC from ___ to ___ (2.5-3 hrs) on September ___ with 8 to 10 principals from high schools in MD, DC, and VA. During the meeting, we will discuss your experiences with PISA and garner advice to guide the development of communication materials for PISA 2012. Based on your recommendations, we will create or adapt new materials for PISA 2012. Toward the end of October, we would like to meet again by phone for a one hour group call to elicit your feedback about the newly created or revised PISA materials.  

In addition to principals, in mid-October, we would also like to talk with students. As with principals, our goal is to learn about students’ attitudes towards testing and to gather advice about how to improve PISA-related messages and communications in the future. We will ask for your help in identifying 10-15 students from your school that we might approach about participating in a 90-minute focus group discussion about PISA. It is important that your list well represents the cross-section of the 15-year-old students in your school (e.g., with respect to gender, race/ethnicity, academic achievement, school attendance), thus we ask you to select with care a diverse group of students. Please bring your list (along with student addresses to which we can mail recruitment materials) with you to the Principal meeting in Washington, DC. We will send a letter home to students’ parents to describe the study and seek consent. We will select up to 2 students to participate in a focus group with other students.

So what’s in it for you? If you agree to participate in both the initial in-person meeting in September AND the follow up call in October you will receive a $150 stipend.

What’s in it for your students? Students will receive a $35 stipend for their participation in the focus group discussion and their parent will be offered $20 to offset the travel/transportation costs to bring the participating student to the focus group discussion site.

NCES is authorized to conduct this study under Section 9543, 20 US Code. Your participation is voluntary. Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Section 9573, 20 US Code).

While we recognize that this is a busy time for schools, we would very much appreciate the opportunity to hear your insights, as well as those of your students, in order to more effectively reach out to schools during our PISA 2012 recruitment efforts. Please let us know at your earliest convenience if you are able to participate.  

Sincerely, ….

Program for International Student Assessment (PISA)

Principal Focus Group Consent Form

CONSENT FORM

PARTICIPATION IN A QUALITATIVE RESEARCH PROJECT

Focus Group Purpose

A series of focus groups will be held with principals in the Washington, DC area about participation in voluntary educational assessments and communication and motivation strategies for student recruitment in those assessments. The information from these groups will guide school and student recruitment in the effort to improve participation rates and collect valid data on educational achievement across the U.S.

Sponsoring Agency

This project is the National Center for Education Statistics, which is part of U.S. Department of Education. Representatives from Shattuck & Associates, Inc., an independent research consulting contractor, will be conducting the focus groups.

Confidentiality

Participation in this study is voluntary. Although notes will be taken during the conversation, no information will be used to identify participants and participant names will not be used in any report.

Questions

If you have any questions about the discussion group, please call Teresa Shattuck of Shattuck & Associates at 301-829-5737.

Consent

I, __________________________________, agree to participate in this focus group.

SIGNATURE DATE

Program for International Student Assessment (PISA)

Principal Focus Group Facilitator’s Guide for In Person Working Group

Focus Group Prep Sheet

Focus Group Objectives

By the end of today’s discussion, stakeholders will have an increased understanding of:

1. How principals perceive standardized testing.

2. How principals might make decisions about whether or not to participate in PISA.

3. Factors that might motivate principals to agree to participate in voluntary assessments such as PISA.

4. Factors that might motivate 15-year-old students to participate and put forth their best effort on an assessment that has no personal consequences.

5. Recommendations for improving existing messages and materials related to PISA.

Date of Event: Start Time: End Time:

Location:

Facilitator: Note Taker:

Total # of participants: # of Men: ___ # of Women: ______

Materials:
Facilitator’s Guide/Script Note Taker’s Guide/Pad/Laptop Markers Name Tags Index Cards Sticky Dots Pens/Pencils Audio Recorder/extra batteries	Registration Form Handout: Participant Survey Handout: Flip Charts Food/Drink Paper Goods Other: Other:

Room Set Up: Registration will be set up by host organization outside of the room. Inside the room, a welcome table will be set up with participant surveys, name tags, pens/pencils, markers, and sticky dots. Also set up will be flip chart(s) with markers and the note taker’s computer station.

Flip Chart Posters:

Flip Chart for Brainstorm/Discussion Notes:

Principal Focus Group Facilitator’s Guide

Program for International Student Assessment or PISA

I. Welcome Table/Participant Survey (5 minutes)

Set up welcome table name tags, markers, and surveys. Include a welcome sign with the following instructions:

WELCOME!

After making yourself comfortable and enjoying some food/drink, please:

• Fill out a name tag

• Complete a participant survey

II. Introduction

1. Welcome, Thank You, Logistical Info

• Welcome the group.

• Thank participants for coming and for completing the participant survey.

• Point out where restrooms are located and discuss other necessary logistics.

2. Introductions

• Facilitator & Note-taker Introductions- First name and tell group a little about yourselves

• Participant Introductions – First name and single favorite thing about your job

3. Background/Purpose/Sponsorship

• The purpose of today’s conversation is to discuss strategies for securing participation of principals and students in PISA 2012.

• More specifically we will seek to better understand:

1. How principals perceive standardized testing.

2. How principals might make decisions about whether or not to participate in PISA.

3. Factors that might motivate principals to agree to participate in voluntary assessments such as PISA.

4. Factors that might motivate 15-year-old students to participate and put forth their best effort on an assessment that has no personal consequences.

5. Recommendations for improving existing messages and materials related to PISA.

• These discussion groups are sponsored by the National Center for Education Statistics (NCES) which supports international efforts to collect and report data on education. NCES is housed in the Institute of Education Sciences at the US Department of Education.

• The information you share today is very important. Your opinions will guide NCES in its efforts to collect data that enables comparisons of US students to students around the world.

4. Confidentiality

The National Center for Education Statistics is authorized to conduct this study under Section 9543, 20 US Code. Your participation is voluntary. Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Section 9573, 20 US Code). The things you say may be put in a summary of this discussion, but your names will not be included in a summary report.

5. Note taking and Recording Session

We will be taking notes and audio recording the discussion so that we can accurately report the important information that you will be sharing. Again, this information will be kept confidential and will be destroyed once the final report is created. Is this OK with everyone?

Because we are using an audio recorder, please speak one at a time so that all opinions can be clearly heard.

6. Ground Rules

I would like to review some basic guidelines that will be used during our discussion:

• There are no right or wrong answers to the questions being asked today; all ideas are good ideas.

• We also welcome and respect different points of view. Please answer what you truly think, regardless of the opinions of the other group members.

• If you are uncomfortable with a question, feel free to pass. You are under no obligation to answer any question that makes you feel uncomfortable in any way.

• We ask that you share your honest feedback so that the greater community can benefit from your opinions.

• Although you may tell colleagues that you have participated in a focus group around this topic, we ask that you please not share any specific information you hear in this group with anyone outside the group. In this way individual’s perspectives can remain confidential.

• Finally, during today’s discussion, there are quite a few things to talk about; at times we may need to stop the discussion in order to move on. I apologize in advance for that.

7. PISA Overview – post information on wall and leave up throughout the discussion; (the information below was pulled from the PISA website)

   • PISA - Program for International Student Assessment

   • PISA is a system of international assessments that focus on 15-year-olds' performance in reading literacy, mathematics literacy, and science literacy every three years. PISA emphasizes functional skills that students have acquired as they near the end of mandatory schooling.

   • PISA is organized by the Organization for Economic Cooperation and Development (OECD), an intergovernmental organization of industrialized countries, and was administered for the first time in 2000.

   • PISA 2012 will focus primarily on math with minor emphasis on reading and science (2006 focused primarily on science; 2003 was math).

   • Administered every 3^rd year since 2000; 66 countries participated in 2009.

   • Schools and students are randomly selected to participate.

   • The assessment takes approximately 2.5 hours to complete. Some students will take an additional second shorter assessment.

3. Focus Group Questions

Objective 1: Perceptions of Standardized Testing

1. Standardized Testing -

• What are your views about standardized tests? (e.g., likes, dislikes)

• How much standardized testing is already going on in your school? (e.g., too little, reasonable amount, too much)

• To what extent are standardized tests valuable to your school? (to administrators, teachers, students, parents)

B. Voluntary & International Assessments

• Are you familiar with any ‘voluntary assessments’? If so, what do you know about them?

• What comes to mind when you hear the term ‘voluntary assessment’?

• Has your school ever participated in a ‘voluntary assessment’?

  • If yes, please tell us about the test and your experiences (Probes: What was the test? Was it a valuable experience? If so, why? Are you glad your school participated? Would you like to see your school participate in future ‘voluntary assessments’?

  • If no, do you think you would be willing to have students sit for a ‘voluntary assessment’? Why or why not?

• Are you familiar with international assessments?  If so, what do you know about them?

• What comes to mind when you hear the term ‘international assessment’?

• Has your school ever participated in an ‘international assessment’?

  • If yes, please tell us about the test and your experiences (Probes: What was the test? Was it a valuable experience? If so, why? Are you glad your school participated? Would you like to see your school participate in future ‘international assessments’?

  • If no, do you think you would be willing to have students sit for a ‘international assessment’? Why or why not?

C. Perceptions of PISA – refer to posted info about PISA

• What are your initial reactions to PISA? (e.g., likes, dislikes)

• What value do you see in PISA? (to participating students, schools, districts, states, US)

• What challenges do you see associated with PISA? (e.g., for schools, students)

• Do you think it is important for the US to know how its students compare internationally?

• With regard to international competitiveness, do you think PISA outcomes have implications for business or workforce development?

• Would it be valuable to understand information about international school policies and characteristics, students’ attitudes and experiences in school?

Objective 2: How principals might make decisions about whether or not to participate in PISA

1. Let’s suppose your school was selected for PISA. How would feel about that?

2. Knowing what you know now, as a first instinct, would you be inclined to say ‘yes’ or ‘no’ to participation? Why?

3. With more time to think about participating in PISA, how would you go about making a decision?

• Who would you talk to?

• What would you be curious about? What more would you want to learn?

• Would the length of the test (e.g., 2 vs. 3 hours) influence your decision to participate?

• Would the type of test administration (paper/pencil vs. computer based assessment or both) influence your decision to participate?

• Would you put someone on your staff in charge of learning more about or coordinating PISA-related efforts? If yes, who would that be (job title)?

Objective 3: Incentives for School Participation

1. What would you see as the primary obstacles to participation in PISA at your school? (list & prioritize)

2. What incentives might motivate you to participate in PISA? (list & prioritize)

3. What data would you like to receive in return for your participation in PISA?

4. On newsprint share PISA incentives that were for schools and School Coordinators in 2009:

• School: $200 goes to school; school receives PISA report comparing school to US and OECD averages and National Report with final results

• School Coordinators: $100; receives National Report with final results

5. What are your thoughts about these incentives? (Schools? School coordinators?)

Objective 4: Incentives for Student Participation

1. What do you think students would see as the primary obstacle to participation in PISA? (list & prioritize)

2. What incentives might motivate students to participate in PISA? (list & prioritize)

3. On newsprint share PISA incentives that were used for students in 2009:

• For Students: $20 cash, Certificate of Volunteer Service for 2.5 hours of community Service, and school specific incentives such as student recognition, volunteer hours, tickets to sporting events

2. What are your thoughts about these incentives?

Objective 5: Recommendations for improving existing messages and materials related to PISA

Note: The items below will be selected and adapted for the various existing PISA materials under review (e.g., study brochure, guide, draft 2009 school report, video); not all questions will be asked of all PISA materials.

1. In terms of format for the materials, what are your preferences? (Probe using items below.)

• Size - What size should material be? Small/pocket-sized? Medium? Large?

• Colors - What colors are most appealing to you? Bold/bright? Muted/pastel?

• Graphics - What type of graphics are most appealing to you? Simple or elaborate? Black and white or colorful? Real photos?

• Balance – Does the guide strike a good balance between text/writing and pictures/graphics? Please explain

• Length - About how long should the materials be? Is there a maximum length that would appeal to you? How long should the chapters be? What’s a reasonable range?

• Organization - How should the materials be organized? Can NCES improve the way the existing materials are organized? If yes, how?

• Final comments about format:

• Is there one thing (in terms of format) that really stood out to you / that you really liked?

• Is there anything (in terms of format) that bothered you / that you really didn’t like?

2. How would you describe the quality of the information in the materials? Probes:

• What information did you find helpful?

• What did you learn? Please describe.

• What surprised you?

• Was the information clear? Easy/hard to read? Easy/hard to understand?

• Do the materials present the right amount of information (too much/too little)?

2. Capacity of PISA Materials to Motivate

• What, if anything, about the materials made you WANT to read them?

• What, if anything, about the materials made you NOT want to read them?

• How would you describe how relevant the materials were for you? (Probe: Did you feel the materials were created for you?)

• To what extent does the design of the materials motivate you to continue reading or learning about PISA?

• To what extent does the design of the materials motivate you to participate in PISA?

IV. Closure

1. Summarize/review the main objectives/points that arose during the group discussion.

2. Ask for comments or questions.

3. THANK the group very much for their time and for sharing their opinions and suggestions; the information they shared today was very VALUABLE.

4. Distribute incentives and get signatures for ‘Receipt Sheet.’

Program for International Student Assessment (PISA)

Principal Focus Group Survey

Target Audience: High School Principals from the Washington Metro Area

[Perceptions about PISA]

1. After learning about PISA today, how inclined would you be to participate?

 Not at all inclined  Somewhat inclined  Very inclined

Please explain your answer:

2. What is the #1 thing that could be done to enhance the likelihood that you would participate?

3. After learning about PISA today, how inclined do you think your 15-year-old students would be to participate? Explain

 Not at all inclined  Somewhat inclined  Very inclined

Please explain your answer:

4. What is the #1 thing that could be done to enhance the likelihood that your 15-year-old students would participate? Explain.

5. In your view, are some schools more or less likely to participate in PISA based on socio-demographics (e.g., urban vs. rural) and academic performance (e.g., high vs. low performing schools)? Explain.

6. Any closing thoughts?

[Experiences with and Attitudes about Standardized Tests]

1. In our school we make regular use of standardized test data to better serve students.

 Not at all true  Somewhat true  True

2. I am a big fan of standardized tests.

 Not at all true  Somewhat true  True

3. Standardized tests are not worth the effort.

 Not at all true  Somewhat true  True

4. In our school, standardized tests are invaluable to decision making.

 Not at all true  Somewhat true  True

5. Standardized tests do not accurately reflect students’ understanding of the material.

 Not at all true  Somewhat true  True

Participant Demographics

1. How long have you been a principal?

• 1-2 years  3-4 years  5-6 years  7 or more years

2. I am:  Male  Female

3. I am a principal in a(n) ___________ setting.

 Urban  Rural  Suburban

4. In what jurisdiction do you work?

• Maryland  Virginia  Washington, DC

Program for International Student Assessment (PISA)

Principal Focus Group Facilitator’s Guide for Webinar Follow-Up

Focus Group Prep Sheet

Focus Group Objectives

By the end of today’s discussion, stakeholders will have an increased understanding of:

6. Reactions to and recommendations for improving existing messages and materials related to PISA.

Date of Event: Start Time: End Time:

Location:

Facilitator: Note Taker:

Total # of participants: # of Men: ___ # of Women: ______

Materials:
Facilitator’s Guide/Script Note Taker’s Guide/Pad/Laptop Markers Name Tags Index Cards Sticky Dots Pens/Pencils Audio Recorder/extra batteries	Registration Form Handout: Participant Survey Handout: Flip Charts Food/Drink Paper Goods Other: Other:

Principal Focus Group Facilitator’s Guide – Webinar Follow-Up

Program for International Student Assessment or PISA

I. Introduction

1. Welcome, Thank You, Logistical Info

• Welcome the group.

• Thank participants for coming and for completing the participant survey.

2. Introductions

• Facilitator & Note-taker Introductions

• Participant Introductions

3. Background/Purpose/Sponsorship

• The purpose of today’s conversation is to discuss strategies for securing participation of principals and students in PISA 2012.

• More specifically we will seek to better understand:

6. Your reactions to and recommendations for improving NEWLY revised messages and materials related to PISA.

• These discussion groups are sponsored by the National Center for Education Statistics (NCES) which supports international efforts to collect and report data on education. NCES is housed in the Institute of Education Sciences at the US Department of Education.

• The information you share today is very important. Your opinions will guide NCES in its efforts to collect data that enables comparisons of US students to students around the world.

4. Confidentiality

The National Center for Education Statistics is authorized to conduct this study under Section 9543, 20 US Code. Your participation is voluntary. Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Section 9573, 20 US Code). The things you say may be put in a summary of this discussion, but your names will not be included in a summary report.

5. Note taking and Recording Session

We will be taking notes and audio recording the discussion so that we can accurately report the important information that you will be sharing. Again, this information will be kept confidential and will be destroyed once the final report is created. Is this OK with everyone?

Because we are using an audio recorder, please speak one at a time so that all opinions can be clearly heard.

6. Ground Rules

I would like to review some basic guidelines that will be used during our discussion:

• There are no right or wrong answers to the questions being asked today; all ideas are good ideas.

• We also welcome and respect different points of view. Please answer what you truly think, regardless of the opinions of the other group members.

• If you are uncomfortable with a question, feel free to pass. You are under no obligation to answer any question that makes you feel uncomfortable in any way.

• We ask that you share your honest feedback so that the greater community can benefit from your opinions.

• Although you may tell colleagues that you have participated in a focus group around this topic, we ask that you please not share any specific information you hear in this group with anyone outside the group. In this way individual’s perspectives can remain confidential.

• Finally, during today’s discussion, there are quite a few things to talk about; at times we may need to stop the discussion in order to move on. I apologize in advance for that.

7. PISA Review – (the information below was pulled from the PISA website)

   • PISA - Program for International Assessment

   • PISA is a system of international assessments that focus on 15-year-olds' performance in reading literacy, mathematics literacy, and science literacy every three years. PISA emphasizes functional skills that students have acquired as they near the end of mandatory schooling.

   • PISA is organized by the Organization for Economic Cooperation and Development (OECD), an intergovernmental organization of industrialized countries, and was administered for the first time in 2000.

   • PISA 2012 will focus primarily on math with minor emphasis on reading and science (2006 focused primarily on science; 2003 was math).

   • Administered every 3^rd year since 2000; 66 countries participated in 2009.

   • Schools and students are randomly selected to participate.

   • The assessment takes approximately 2.5 hours to complete.

3. Focus Group Questions

Objective 1: Recommendations for improving existing messages and materials related to PISA

Note: The items below will be selected and adapted for the various existing PISA materials under review (e.g., study brochure, guide, video); not all questions will be asked of all PISA materials.

2. In terms of format for the materials, what are your preferences? (Probe using items below.)

• Size - What size should material be? Small/pocket-sized? Medium? Large?

• Colors - What colors are most appealing to you? Bold/bright? Muted/pastel?

• Graphics - What type of graphics are most appealing to you? Simple or elaborate? Black and white or colorful? Real photos?

• Balance – Does the guide strike a good balance between text/writing and pictures/graphics? Please explain

• Length - About how long should the materials be? Is there a maximum length that would appeal to you? How long should the chapters be? What’s a reasonable range?

• Organization - How should the materials be organized? Can NCES improve the way the existing materials are organized? If yes, how?

• Final comments about format:

• Is there one thing (in terms of format) that really stood out to you / that you really liked?

• Is there anything (in terms of format) that bothered you / that you really didn’t like?

2. How would you describe the quality of the information in the materials? Probes:

• What information did you find helpful?

• What did you learn? Please describe.

• What surprised you?

• Was the information clear? Easy/hard to read? Easy/hard to understand?

• Do the materials present the right amount of information (too much/too little)?

2. Capacity of PISA Materials to Motivate

• What, if anything, about the materials made you WANT to read them?

• What, if anything, about the materials made you NOT want to read them?

• How would you describe how relevant the materials were for you? (Probe: Did you feel the materials were created for you?)

• To what extent does the design of the materials motivate you to continue reading or learning about PISA?

• To what extent does the design of the materials motivate you to participate in PISA?

IV. Closure

1. Summarize/review the main objectives/points that arose during the group discussion.

2. Ask for comments or questions.

3. THANK the group very much for their time and for sharing their opinions and suggestions; the information they shared today was very VALUABLE.

4. Distribute incentives and get signatures for ‘Receipt Sheet.’

24