PISA Justification rev.

PROGRAM FOR INTERNATIONAL STUDENT ASSESSMENT
(PISA 2012)

REQUEST FOR OMB CLEARANCE

To Conduct Focus Groups to Improve Participation

in PISA 2012

OMB# 1850-0803 v.36

September 1, 2010

Prepared by:

National Center for Education Statistics

U.S. Department of Education

Institute of Education Sciences

Washington, DC

CONTENTS

Background and Study Rationale 1

Design 2

Estimates of Burden 6

Cost to the Government 6

Assurances of Confidentiality 6

Appendices

A. Principal Focus Group Materials

B. Student Focus Group Materials

C. Materials for Principal Review

PISA Summary of Activities

PISA Timeline

PISA 2009 Presentation for Teachers

Locating Info about PISA

NAEP 2010 Facts for Principals

NAEP 2010 In Your School – School Coordinator Responsibilities

Draft PISA 2009 School Report

Screenshots from the Draft PISA 2009 School Website

Online data tool

D. Materials for Student Review

PISA 2009 Student Invite

NAEP 2010 Save-the-Date Letter

NAEP 2010 Customizable Presentation for Students

NAEP 2010 Sample School Newspaper Article

PISA Certificate of Voluntary Service

The NAEP student video at

http://www.nces.ed.gov/nationsreportcard/videos/naepstudent.asp  

E. Affidavit of Nondisclosure

Background and Study Rationale

The Program for International Student Assessment (PISA) is an international assessment, conducted every three years, that focuses on 15-year-olds' capabilities in reading, mathematics, and science literacy. In 2012, PISA will assess these subjects as well as problem-solving and financial literacy. PISA emphasizes functional skills that students have acquired as they near the end of mandatory schooling. PISA is organized by the Organization for Economic Cooperation and Development (OECD), an intergovernmental organization of industrialized countries. A request for clearance for the PISA 2012 field test and school recruitment has been submitted to OMB on August 12, 2010. The purpose of this request is to seek approval to conduct focus groups with school principals and students in order to improve school and student participation rates.

In the United States, participation in PISA is voluntary for states, districts, schools, and students. In order to provide valid and reliable information about student performance, the participation of sampled schools and students is critical. In the United States, there has been a steady decline in the willingness to participate in voluntary assessments like PISA, especially on the part of schools but also students. In each of the past PISA administrations, the United States has barely achieved the minimum school response rate.¹ Student response rates have also been dropping. The data of countries that do not attain the minimum response rates are not included in international PISA results or in the international PISA database.

There will be more burden than usual for the schools sampled for PISA in 2012. In addition to paper-and-pencil assessments of reading, mathematics, and science, PISA 2012 will include an assessment of financial literacy, which means more students to be sampled per schools, and the inclusion of computer-based assessments, to be conducted in a second session, which means additional testing time and time away from the classroom. Moreover, the focus on mandatory testing across the United States continues to grow, which means schools feel overburdened and even less apt to agree to participate in voluntary assessments such as PISA. All of these factors are likely to make it more difficult to recruit schools for PISA 2012, yet even a minor drop in response rates from what the United States has been able, with great effort, to obtain in the past, could mean results for the U.S. would not be reported.

Given the importance of achieving high response rates, the historical challenges in doing so in PISA, and the increased burden in PISA 2012, NCES is seeking additional information to optimize school and student recruitment for PISA 2012. NCES’s PISA 2012 national data collection contractor, Westat, and its subcontractors, Hagar Sharp and Shattuck & Associates, will conduct focus groups with (a) principals and (b) students to collect information that will guide school and student recruitment with the goal of improving participation rates for PISA 2012. The results of the focus groups will be shared across NCES program areas to inform school recruitment efforts.

Design and Recruitment

The purpose of the principal focus groups is to collect information about their decision-making regarding school participation in the assessment and their communication and motivation strategies for student recruitment. We will focus on the following research questions:

• What value do international assessments and comparisons have in education?

• How might principals make decisions about whether or not to participate in PISA?

• What factors might motivate principals to agree to participate in voluntary assessments such as PISA?

• What factors might motivate 15-year-old students to participate and put forth their best effort on an assessment that has no personal consequences?

• What recommendations do principals have for improving existing messages and materials related to PISA?

The purpose of the student focus groups is to collect information about a) how students make the decision of whether or not to participate in voluntary assessments and b) what might motivate them to put in their best effort on those assessments. We will focus on the following research questions:

• What value do students see in making comparisons to students internationally?

• How might students make decisions about whether or not to participate in PISA?

• What factors might motivate 15-year-old students to participate and put forth their best effort on an assessment that has no personal consequences?

• What recommendations do 15-year-old students have for improving existing messages and materials related to PISA?

Using the suggestions and information garnered from these conversations, we will incorporate the principal and student feedback into the development of materials for PISA 2012 recruitment and promotion. The following materials used or developed for the PISA 2009 assessment or for the National Assessment of Educational Progress (NAEP) 2010 will be presented to principals and students during focus groups to elicit feedback and suggestions (see Appendix B and C for copies of all materials).

Materials for principal review (to be reviewed during the focus groups)

• PISA Summary of Activities (from 2009)

• PISA Timeline (from 2009)

• PISA 2009 Presentation for Teachers

• Locating Info about PISA (from 2009)

• NAEP 2010 Facts for Principals

• NAEP 2010 In Your School – School Coordinator Responsibilities

• PISA School Report (from 2009, draft in development)

• PISA online school data tool (from 2009, draft in development)

Materials for student review

• PISA 2009 Student Invite

• NAEP 2010 Save-the-Date Letter

• The NAEP student video at http://www.nces.ed.gov/nationsreportcard/videos/naepstudent.asp 

Principal Focus Group

We will convene a focus group of public school principals, who will be asked to meet twice, in order to best collect the information we are seeking. During the first meeting, principals will be asked to provide their feedback on topics that include decision-making regarding the assessment, internal communications at their respective schools, school incentives, and student motivation strategies. To hear the principals’ thoughts on potential communication plans for PISA 2012, materials from the 2009 administration of PISA and NAEP 2010 will be shared and discussed. During the second focus group, the principals will be asked to respond to draft PISA 2012 promotion and recruitment materials and strategies that were developed based on the first meeting.

The focus groups will consist of approximately eight high school principals – a size that allows for in-depth collection of information and buy-in from this audience. We will seek members who are diverse, representing both rural and inner city schools, to ensure that different perspectives are represented at the table.

Group participants will be recruited by Westat, Hager Sharp, and Shattuck & Associates. Potential participants will be from city, suburban, and rural areas in the surrounding Washington, DC area representing diverse backgrounds and experiences. Westat will identify 20 principals who have had their schools selected for participation in previous PISA administrations. Hager Sharp or Shattuck & Associates will contact the principals to recruit them for the effort, with the intention of having participants who have agreed to participate in PISA and those who have not (see recruitment script in appendix A).

The first meeting of the principal working group will be an in-person discussion for three hours on a weekend. Hager Sharp’s office in Washington, DC can be used to facilitate the focus group. Due the challenges facing principals at the start of the school year, meeting on a weekend day will reduce the distractions of being on campus and in front of a computer for a long duration of time. We believe we will gather more meaningful feedback and buy-in if the group meets in person to discuss PISA and review related materials. The second meeting of the principal working group will be convened via webinar, for approximately one hour.

The first meeting should happen in September, with the second meeting happening four to six weeks later. This schedule will allow for materials to be updated prior to recruitment for the field trial, which will begin in November.

Schedule for Principal Groups

Activity	Tasks	Date Ranges
OMB clearance	OMB submission	September, 2010
Data collection	Recruit participants	Early-Mid September, 2010
	First Discussion	Mid-Late September, 2010
	Second Discussion	Early-Mid October, 2010

Student Focus Groups

We will organize two student focus groups at two different locations, local to the groups. Each focus group will consist of approximately eight high school students – a size large enough that we may hear various opinions, but small enough for participants to feel comfortable sharing their suggestions in an informal setting. While these focus groups will seek a degree of diversity/variation among participants, these participants are not intended to be a representative group of students across the United States. The students selected, however, will all be 15 years old.

Group participants will be recruited from the Washington, DC and surrounding area schools overseen by principals in the principal working group. We will ask each principal to identify 10-15 possible students. All recruitment will be handled by Hager Sharp or Shattuck & Associates. Identifying 10-15 students will help attain a representative cross-section with respect to gender, race/ethnicity, academic achievement, and school attendance, with the goal of enrolling two participating students from each school. It is necessary to identify 10-15 possible students from each school as some students may opt not to participate as well as to help attain a representative cross-section.

Because the students will be from the schools overseen by principals in the working group, they will represent students in urban, suburban, and rural areas. We will ask the principals to identify students who represent a cross-section of their student population (e.g., with respect to gender, race/ethnicity, academic achievement, school attendance). A letter from NCES will be sent home to those students and their parents to inform them of the study and receive their consent to participate. As those students volunteer to participate in the study, a screening questionnaire (included in appendix B) will be administered to each potential participant to ensure two students are selected from each school and an appropriate amount of diversity is achieved among the students.

We anticipate holding one session in Montgomery County, MD and another session in Arlington County, VA on a weekday evening. Each will last 90 minutes.

The student focus groups should take place in mid-October so that recruitment materials and strategies can be updated, if necessary, prior to recruitment for the field trial beginning in November.

Schedule for Student Groups

Activity	Tasks	Date Ranges
OMB clearance	OMB submission	September, 2010
Data collection	Recruit participants	Late September, 2010
	Focus Groups	Mid-October, 2010

Session Activities

During all sessions, Teresa Shattuck will lead participants through a discussion, using the appropriate Moderator’s Guide (attached for each group) that she and Hager Sharp have produced. Participants will also be asked to complete a short survey at the close of their session (attached). In addition, one Hager Sharp team member will observe and take notes of the participants’ comments and suggestions.

Following the conclusion of each focus group, Hager Sharp will compile the minutes and share summaries with Westat and NCES. The end product will be a final report that presents the answers to the research questions outlined above, the methodology, the recommendations heard, and the suggestions for next steps. This report can serve as the foundation for making changes to PISA materials in order to more effectively recruit schools and student for participation in the assessment.

Estimates of Burden

The total time required of each principal will not exceed five hours during the course of recruitment and participating in the two discussions. The total time required of each student will not exceed two hours during the course of recruitment and participating in a single discussion. There is no cost to participants.

Respondent	Hours Per Respondent	Number of Respondents	Number of Responses	Total Burden Hours
Principal – Recruitment	.5	20	20	10
Principal - Discussion	4	8	8	32
Total Burden/Principal	--	20	28	42
Student – Recruitment	.5	80	80	40
Student - Discussion	1.5	16	16	24
Total Burden /Student	--	80	96	64
Total Combined Burden		100	124	106

Cost to the Government

As part of recruitment efforts, participants in the principal and student focus groups will be given a monetary payment to compensate for the time spent participating and traveling to and from the focus group locations. Principals will each receive $75 for each session. Students will each receive $35 and their parents will be offered $20 to offset travel/transportation costs to bring the participating student to the focus group discussion site. A monetary payment is deemed necessary both to compensate individuals for their time and trouble in attending, but also to ensure a representative sample – the motivation to attend without an incentive can lead to greater bias based on attitudes or lifestyle (e.g., only those people who are more curious or those who are less busy may be more likely to attend; a financial incentive encourages a broader range of individuals to participate). The estimated cost is approximately $21,500 to conduct the four focus group meetings and report the results.

Assurances of Confidentiality

All contractor and sub-contractor staff working on PISA 2012 focus groups will sign Affidavits of Non-disclosure.  Once these affidavits have been signed and received by NCES, Westat will share the names of a subset of schools sampled for PISA in the past with Hager Sharp for the purpose of forming the focus groups. Westat and its contractors, Shattuck & Associates and Hager Sharp, will collect personal information (e.g., name, address) for recruitment purposes, but on the data file used for analyses (and presented to NCES), respondents will be identified only by a unique Study ID number assigned to each participant. This unique Study ID will appear only on participant survey data. Within 48 hours of respondents’ participation in the focus group, the discussion notes will be edited, organized, and cleaned and all identifiers will be stripped from the data set. At no point will the files containing the participants’ personal information (e.g., name, address) be linked to focus group or survey data files. All computer files will be password protected and hard copies will be locked in secure locations (e.g., data will be in locked file cabinets within locked offices). Only Westat and contract staff working directly on the data analysis portion of the project will have access to the data files. All personally identifiable information will be destroyed once the final report is created. All presentations of data in reports will be in aggregate form, with no links to individuals.

1 OECD requires a minimum of 65 percent of originally sampled schools if replacement schools are used to supplement the sample. If replacement schools are not used, the minimum response rate is 85 percent of originally sampled schools. The United States includes replacement schools and so our minimum allowable rate is 65 percent of originally sampled schools, provided we can also provide evidence that the sample is not biased.