PISA 2015 Principal Interviews and Focus Group Volumn

	Program for International Study Assessment (PISA 2015) Interviews and Focus Groups with School Principals Request for OMB Clearance OMB # 1850-0803 v. 81
	June 19, 2013
		Submitted by: National Center for Education Statistics U.S. Department of Education Institute of Education Sciences Washington, D.C.

Table of Contents

Section Page

1

Background and Study Rationale 1

2 Design and Recruitment 2

2.1 Research Questions 2

2.2 Procedures 4

2.3 Session Activities 5

3 Estimates of Burden 5

4 Cost to the Government 6

5 Assurances of Confidentiality 6

Appendixes

A Principal Focus Group Materials A-1

Principal In-depth Interview and Focus Groups Recruitment Scripts

Principal Focus Group Screener

Principal In-depth Interview and Focus Group Consent Form

Facilitator’s Guide for In-Depth Interviews

Facilitator’s Guide for In-Person and Webinar Focus Groups

B PISA 2012 Materials for Principal Review B-1

PISA 2012 Summary of Activities

PISA 2012 Brochure

PISA 2012 School FAQ

PISA 2012 Student FAQ

PISA 2012 Presentation for Students

Screenshots from the PISA School Website

C PISA 2015 Materials for Principal Review C-1

PISA 2015 Draft Summary of Activities

PISA 2015 Draft Brochure

PISA 2015 Draft School FAQ

PISA 2015 Draft Student FAQ

NAEP WCBA Brochure

PISA 2012 Presentation for Students

Screenshots from the PISA School Website

PISA School Report

Tables

1 Schedule for In-Depth Interviews and Principal Focus Groups 4

2 Burden calculations for principal in-depth interviews and focus groups 5

1. Background and Study Rationale

The Program for International Student Assessment (PISA) is an international assessment conducted every 3 years that focuses on 15-year-olds’ capabilities in reading, mathematics, and science literacy. In 2015, PISA will assess these subjects as well as collaborative problem-solving and financial literacy. PISA emphasizes functional skills that students have acquired as they near the end of mandatory schooling. PISA is coordinated by the Organization for Economic Cooperation and Development (OECD), an intergovernmental organization of industrialized countries, and is conducted in the United States by the National Center for Education Statistics (NCES). The 60-day public comment period for the PISA 2015 field test and school recruitment ended on June 18, 2013. The purpose of this request is to seek approval to conduct in-depth interviews and focus groups with school principals to explore ways to improve school and student participation rates.

In the United States, participation in PISA is voluntary for states, districts, schools, and students. To provide valid and reliable information about student performance, the participation of sampled schools and students is critical. In the United States, there has been a steady decline in the willingness to participate in voluntary assessments like PISA among both schools and students. In each of the past PISA administrations, the United States has barely achieved the minimum school response rate.¹ In PISA 2012, the last PISA administration, the original school response rate was 67.1 percent. The data of countries that do not attain the minimum response rates are not included in international PISA results or in the international PISA database.

PISA 2015 will be entirely computer based, and the inclusion of financial literacy means that some students will take both the main assessment and financial literacy assessment in each school. Additionally, the assessments will be administered on equipment carried in by data collection staff rather than on school equipment. While this form of assessment administration lessens the burden on schools, it limits the number of computers that can be transported and set up in schools and will require 2 days of assessment sessions for some students. Moreover, as mandatory testing across the United States continues to grow, schools feel overburdened and even less apt to agree to participate in voluntary assessments such as PISA.

All these factors are likely to make it more difficult to recruit schools for PISA 2015―a troubling development since even a minor drop in response rates from what the United States has been able to obtain in the past, with great effort, could mean that results for the United States would not be reported.

Given the importance of achieving high response rates, the historical challenges in doing so in PISA, and the increased burden in PISA 2015, NCES is seeking additional information to optimize school and student recruitment for PISA 2015. NCES’s PISA 2015 national data collection contractor, Westat, and its subcontractor, Hager Sharp, will conduct in-depth interviews and focus groups with school principals during the summer 2013. The information collected from these meetings will guide school and student recruitment strategies for the main study with the goal of improving participation rates for PISA 2015. The results of the focus groups will be shared across NCES program areas to inform school recruitment efforts.

2. Design and Recruitment

The purpose of the principal interviews and focus groups is to better understand both the barriers and benefits schools associated with participation in PISA and to identify communication strategies that will help overcome barriers to participation. This information would guide recruitment strategies and materials for PISA 2015. We propose speaking with three sets of principal groups in four steps to examine these issues.

1. We will hold in-depth telephone interviews with school principals who participated in PISA 2012.

2. We will meet with a group of principals who refused to participate in PISA 2012 to better understand their objections and barriers to participation.

3. We will arrange an in-person focus group with principals who have never been asked to participate in PISA to understand their perceptions of the study, international assessments, and how they would respond if selected for PISA.

4. Finally, if we find that there is a topic that should be probed further and we need more time, we may go back to one of these groups for an additional focus group examining a specific topic(s).

2.1 Research Questions

During in-depth interviews, we will focus on the following questions with principals who participated in PISA 2012:

• Having participated in PISA 2012, what were the benefits of participation?

• What convinced principals to participate in PISA 2012

• What materials were most useful?

• What materials were not useful or informative?

• PISA 2015 will require 2 days for administration. How might we minimize the burden of PISA participation on schools?

• Could we, or should we, have done anything differently in presenting information or establishing contacts beyond the principal, such as at the district level?

In a focus group, we will ask principals who refused to participate on PISA 2012 about the following:

• What value do international assessments and comparisons have in education?

• What barriers did the principals associate with participation in PISA 2012?

• What are their reactions to the benefits articulated by principals who did participate?

• What are their reactions to the current PISA 2015 materials?

• What are their suggestions for recruitment strategies and materials?

• How was the decision to refuse participation made for their school? Were they the sole decisionmaker, or were there others within the school (teachers, administrators) or at the state or district level who influenced the decision not to participate?

• What suggestions do they have for how we might have been successful in gaining the cooperation of their school (e.g., other communication with districts)?

In a focus group, we will ask principals who have never been asked to participate in PISA about the following:

• What value do international assessments and comparisons have in education?

• Given the structure of PISA 2015 (computer-based assessments and the associated assessment time), what factors might motivate principals to agree to participate in voluntary assessments such as PISA?

• What would minimize the burden in their school?

• What are the perceived benefits of participation? How do the principals value PISA?

• What are their reactions to the current PISA 2015 materials? What materials convey useful information? What materials are not useful or informative?

• If they were approached to participate, what would be the most effective ways to communicate with them?

• What other groups (district, other administrators) would be central to the decision making process and what would be the best way to reach them?

Using the suggestions and information garnered from these conversations, we will incorporate the principals’ feedback into the development of materials and strategies for engaging schools in PISA 2015 recruitment. The following materials, which were used for PISA 2012 or are being developed for the PISA 2015 assessment, will be presented to principals during focus groups and in-depth interviews to elicit feedback and suggestions (see appendix B for copies of all materials).

Materials for principal review (to be reviewed during the interviews)

• PISA 2012 Summary of Activities

• PISA 2012 brochure

• PISA 2012 Presentation for Students

• PISA 2012 School Coordinator Responsibilities

• PISA 2012 Presentation for Students

• Screenshots from the PISA School Website

Materials for principal review (to be reviewed during the focus groups)

• PISA 2015 Summary of Activities (Draft)

• PISA 2015 Brochure (Draft)

• NAEP brochure about computer-based assessment

• PISA Presentation for Students (from PISA 2012)

• Screenshots from the PISA School Website

• PISA School Report mock-up

2.2 Procedures

Participants in the interviews will be sent a package of PISA 2012 materials in advance of the discussion (appendix B), while participants in the two focus groups will be sent a package of PISA 2015 materials (appendix C). All discussions will be conducted by a trained researcher working from a facilitator guide. Four groups are identified as follows. Table 1 provides a schedule of activities.

1. In-depth telephone interviews with principals whose schools have participated in PISA. We would start with an initial set of five interviews with five principals, adding up to three more if opinions were very diverse in the first set of five. Each telephone interview would last approximately 30-45 minutes and would focus on identifying the benefits principals associate with PISA participation—both when they initially agreed to participate and in hindsight as they reflect on the experience. In each interview we would also explore which PISA recruitment strategies and marketing materials they believe would be most and least useful as well as their suggestions for recruitment strategies or materials.

2. Focus group with principals who refused in PISA 2012 to explore barriers and benefits and get reactions to current materials. This focus group would include seven to eight principals and would be conducted via WebEx or comparable technology so that principals across the country could participate. Topics would include barriers they associate with participation, their reactions to the benefits articulated by principals who did participate, their reactions to the current PISA 2015 materials, and their suggestions for recruitment strategies and materials. The discussions would last for up to 2 hours.

3. Focus group with principals who have never been asked to participate in PISA. To the extent possible, participants would be from schools with characteristics of the schools that tend to decline participation: high- or low-performing schools and large suburban schools. This group would take place in person at Westat’s Rockville focus group facility with seven to eight principals. Topics would be essentially the same as those in the first focus group. Should local recruitment prove challenging, we could convene the group via WebEx or comparable technology. The focus group would last up to 2 hours.

4. Subsequent focus group(s) with two types of principals. We may hold an additional focus group using the same methodology described in working with groups 2 and 3 above. It would cover most of the same topics and would include review of any new materials developed as a result of the research. We would convene this focus group if we heard reaction or ideas that warranted a significant change in the recruitment materials and would elicit feedback to the changes.

Table 1. Outline for In-Depth Interviews and Principal Focus Groups

Activity	Tasks
Step 1. Interviews with PISA 2012 participating principals	Recruit participants for in-depth interviews (Step 1)
	Hold interviews
Step 2. Focus group with PISA 2012 principal refusals	Recruit participants for focus group (principal refusals in PISA 2012)
	Hold focus group
Step 3. Focus group with principals never contacted for PISA	Recruit participants for focus group (principal refusals in PISA 2012)
	Hold focus group
Step 4. Subsequent focus group with either focus group 2 or 3	Hold focus group

2.3 Session Activities

Hager Sharp will conduct in-depth interviews with principals who participated in PISA 2012, using the attached Facilitator’s Guide (appendix A). Hager Sharp will take notes during the interviews and compile feedback into a report.

During all focus group sessions, Maria Ivancin of Market Research Bureau LLC will lead participants through a discussion, using the Facilitator’s Guide (appendix A). In addition, one Hager Sharp team member will observe and take notes on the participants’ comments and suggestions.

Following the conclusion of the in-depth interviews and each focus group, Hager Sharp will compile the minutes and share summaries with Westat and NCES. The end product will be a final report that presents the answers to the research questions outlined above, the methodology, the recommendations heard, and the suggestions for next steps. This report can serve as the foundation for making changes to PISA materials and recruitment strategies to more effectively recruit schools and students for participation in the assessment.

3. Estimates of Burden

The total time required of each principal will not exceed 3 hours during the course of recruitment and participating in the discussions. There is no cost to participants. Table 2 provides the burden estimates for each respondent group.

Table 2. Burden calculations for principal in-depth interviews and focus groups

Respondent group	Hours per respondent	Number of respondents	Number of responses	Total burden hours
1. Interviews with participating principals
Principal – Recruitment	0.5	20	20	10
Principal ‒ Discussion	1	8	8	8
2. WebEx with PISA 2012 principal refusals
Principal – Recruitment	0.5	20	20	10
Principal ‒ Discussion	2	8	8	16
3. Focus group with principals not contacted for PISA
Principal – Recruitment	0.5	20	20	10
Principal ‒ Discussion	2	8	8	16
4. Follow-up focus group	2	8	8	16
Total Burden		92	92	86

As part of the recruitment efforts, participants in the principal interviews and focus groups will be given a monetary payment to thank them for their time and effort, and for to compensate those traveling to and from the focus group locations. Principals participating in interviews or a distance focus group via WebEx will each receive $50. Principals participating in the in-person focus group will each receive $75. A monetary payment is deemed necessary both to compensate individuals for their time and trouble in attending, and to ensure a representative sample. The motivation to attend without an incentive can lead to greater bias based on attitudes or lifestyle (e.g., only those people who are more curious or those who are less busy may be more likely to attend, while a financial incentive encourages a broader range of individuals to participate). The dollar amounts are based on similar dollar amounts proposed for NAEP focus groups with principals held in 2011 that examined school-level reports. These focus groups were of similar duration to those we plan to hold. The focus groups were conducted both in-person and via WebEx. The principals attending the WebEx sessions received $50, while the principals attending the in-person focus group received $75.

4. Cost to the Government

The estimated cost to the federal government to conduct in-depth interviews and focus groups and to report the results is approximately $21,500. These costs include salaried labor for Hager Sharp staff and other direct costs associated with organization of the meeting.

5. Assurances of Confidentiality

All contractor and sub-contractor staff working on PISA 2015 focus groups will sign the NCES Affidavits of Nondisclosure. Once these affidavits have been signed and received by NCES, Westat will share the names of a subset of schools sampled for PISA in the past with Hager Sharp for the purpose of forming the focus groups. Westat and its contractor, Hager Sharp, will collect personal information (e.g., name, address) for recruitment purposes, but on the data file used for analyses (and presented to NCES) respondents will be identified only by a unique study ID number assigned to each participant. This unique study ID will appear only on participant survey data. Within 48 hours of respondents’ participation in the focus group, the discussion notes will be edited, organized, and cleaned, and all identifiers will be stripped from the data set. At no point will the files containing the participants’ personal information (e.g., name, address) be linked to focus group or survey data files. All computer files will be password protected and hard copies will be locked in secure locations (e.g., data will be in locked file cabinets within locked offices). Only Westat and contract staff working directly on the data analysis portion of the project will have access to the data files. All personally identifiable information will be destroyed once the final report is created. All presentations of data in reports will be in aggregate form, with no links to individuals.

The following voluntary and confidentiality statement will be included in respondent recruitment materials: “NCES is authorized to conduct this study under Section 9543, 20 US Code. Your participation is voluntary. Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Section 9573, 20 US Code).”

1 OECD requires a minimum of 65 percent of originally sampled schools if replacement schools are used to supplement the sample. If replacement schools are not used, the minimum response rate is 85 percent of originally sampled schools. The United States includes replacement schools, so our minimum allowable rate is 65 percent of originally sampled schools, provided we can also provide evidence that the sample is not biased.