OMB83 C Memo

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: February 22, 2011

TO: Shelly Martinez, OMB

FROM: Dana Kelly, NCES

THROUGH: Kashka Kubzdela, NCES

RE: PISA 2012 Field Test and Recruitment Change Request (OMB# 1850-0755 v.11)

The PISA 2012 Field Test and Recruitment OMB clearance request was approved on November 22, 2010 (1850-0755 v.10). The PISA 2012 field test design and costs have not changed since the approval, but some of the burden estimates and recruitment materials have, and the field test questionnaires have been finalized with approval from the international consortium. We therefore seek OMB’s approval for these changes:

• The international consortium that manages PISA is estimating the field-test version of the School Questionnaire to take 40 minutes rather than the originally estimated 30 minutes. We fully expect the main study version of the school questionnaire to take 30 minutes, but the increased length of the field test version means that our burden estimate for school administrators completing the questionnaire increased by 13 hours total (from 41 to 54 hours). An updated burden table is provided in attachment 1 (changes are shown in red).

• The final field test student and school questionnaire items have by now been approved by the international consortium and are provided in attachment 2. There will be one school questionnaire and four forms of the student questionnaire so that alternative item formats and topics can be field-tested. Attachment 3 provides a summary of how the final field test questionnaire items compare to the versions included in the original submission.

• Final recruitment materials are provided in attachment 4. Some revisions were made as a result of focus groups with principals and students (conducted under OMB# 1850-0803).

• The PISA School Coordinator Manual explaining the internationally-prescribed procedures that the school staff person designated as the school coordinator for PISA must follow has been provided by the international consortium that manages PISA and adapted by NCES to the U.S. context (see attachment 5).

This memo first describes the status of the questionnaires and then describes changes to materials.

Questionnaires

The school and student questionnaire included in the initial clearance request were the questionnaires administered in PISA 2003. The 2003 instruments were included because (a) the international versions of the PISA 2012 Field Test questionnaires were not available at the time of the original submission and (b) it was expected that the school and student questionnaires for the 2012 field test would be very much like questionnaires administered in PISA 2003 because like PISA 2003, PISA 2012 focuses on mathematics. The original clearance request indicated expected changes to the 2003 questionnaires, such as which items we expected would be eliminated, the kinds of new item types we expected would be developed by the international consortium, and new topics we expected to see in the final questionnaires..

In September 2010, NCES shared draft PISA 2012 Field Test questionnaires (international versions) with OMB. The draft questionnaires included many new items not administered in 2003; some new items reflect new topics to be addressed by PISA 2012 and others are new item formats. The draft questionnaires were reviewed by the international Questionnaire Expert Group and participating countries in October and November 2010, and final international versions were prepared by the international consortium based on that feedback. In December 2010, the international consortium released the final international versions of the questionnaires. NCES has adapted them to the U.S. context and the international consortium has approved the U.S. versions.

To aid OMB’s review of the final U.S. versions of the PISA 2012 Field Test questionnaire items (provided in attachment 2), in attachment 3 we show how the final versions compare with the PISA 2003 questionnaire items. We show, for each instrument, which items are the same as those that were administered in 2003, which items used in 2003 have been revised (including a side-by-side with the 2003 version), which items were not administered in 2003 (but may have been administered in 2000, 2006 or 2009), and which 2003 items have been eliminated.

Recruitment and Other Materials for Participants

Recruitment materials included in the original submission have been refined based on a review of the materials by our contractor’s (Westat) subcontractor, Hager Sharp (the organization that also worked with NAEP to develop improved recruitment materials for grade 12 NAEP), and the focus group with school principals held in November 2010 (conducted under OMB# 1850-0803). Hager Sharp reviewed the materials to make them more consistent in terminology, make it clear that they referred to the PISA field test, and to be better able to be used together or as a stand-alone. The principals gave their reactions to the materials, some as representatives of schools that had participated in previous rounds of PISA. During the focus group, principals were asked to review and comment on PISA 2012 recruitment materials, including a letter to principals, frequently asked questions (FAQ), and the PISA 2012 brochure. Based on comments received, the following documents were revised to simplify them, make them more appealing to potential respondents, and to highlight the international nature of the study and the importance of participation.

• School Letter. The school letter was reduced to a single page based on the advice of the focus groups principals. The text was edited to better engage the interest of school principals. For example, the opening sentence of the letter was changed from “I am writing to inform you about the upcoming Program for International Student Assessment (PISA 2012)” to “I am pleased to inform you that your school has the opportunity to be part of the upcoming Program for International Student Assessment (PISA 2012).” Other text was simplified to both engage principals and clearly communicate the importance of the study. The confidentiality language was placed in italics at the bottom of the letter because principals said they found the legal language interrupted the flow of the letter and distracted them. School letters were tailored to provide specific information to schools depending on whether they were a computer and paper/pencil assessment school or a computer-based assessment only school. School letters also reflected the financial incentives a school would receive based on whether they were a control or experiment school in the field trial’s incentive experiment; the versions used for schools participating in each of the modes or the treatment condition are attached to this memorandum. The state and district letters updated with contractor contact information are also provided.

• Frequently Asked Questions (FAQ) for schools. A brief description of PISA was added to the beginning of the FAQ. Language in the FAQ was revised to clarify the amount of time required of students and to explain school selection. The FAQ also now explains that students are participating in a field test to help prepare for PISA 2012.

• PISA 2012 Brochure. The list of participating countries was updated to accurately reflect PISA 2012 participants. The list of endorsing organizations was deleted.

• Summary of Activities for School Coordinators. This was reoriented and reformatted as a “timeline of activities” to show the content of what was previously presented in a Word document without dates. Several versions were created so that the timeline would accurately describe the process in schools being assessed under different assessment and incentive schemes. There were 4 timelines created to account for the incentive experiment ($800 for schools and $200 for school coordinators versus $200 for schools and $100 for school coordinators), and the mode of data collection in the schools (paper and pencil assessment only versus paper and pencil and computer-based assessment or computer based only). The version used for schools participating in both modes and the treatment condition is attached to this memorandum.

• Certificate of Volunteer Service for students: This certificate will be provided to students to document that they will receive credit for volunteer service for having participated. It is an important recruiting device and well received by students.

• Student Invitation: This will be used to inform students about their selection for the assessment.

A focus group with students, also conducted under OMB# 1850-0803, suggested the usefulness of other materials in the recruiting phase.

• A Powerpoint presentation to orient sampled students to PISA. Feedback from the student focus group suggested ways that a similar Powerpoint presentation used by NAEP (provided in the focus group clearance request) could be adapted to PISA. The Powerpoint presentation developed for PISA aims to help students understand what PISA is and encouraging them to participate. It will be made available to schools if they wish to use it with their students. The Powerpoint includes an embedded video clip that is attached as a separate file; when provided to schools it will be possible to simply click on the video clip.

• FAQs for students. Feedback from the students in the focus group suggested that students are keen to understand more about PISA, particularly how schools and students are sampled, and that they would appreciate getting a tangible document that describes PISA that would help them make a decision about participation. A short PISA “FAQ” sheet was developed specifically for students, by adapting information taken largely from other materials (what is PISA, how long does it take, etc.).

Attachment 1:

Updated Burden Table

Table A-3. Burden estimates for PISA 2012 field trial and main study (revised)

	Sample	Expected response rate	Number of respondents	Number of responses	Per respondent (minutes)	Total burden (hours)
FIELD TRIAL
Student
Directions paper-and-pencil	1,512	0.80	1,210	1,210	10	202
Paper-and-pencil test booklet	1,512	0.80	1,210	1,210	120	2,420
Directions (computer-based assessment)	2,406	0.80	1,925	1,925	10	321
Computer-based assessment in addition to paper-and-pencil	756	0.80	605	605	40	404
Computer-based assessment only	1,650	0.80	1,320	1,320	40	880
Financial Literacy background items	302	0.80	242	242	5	21
Core Questionnaire	1,512	0.80	1,210	1,210	30	605
Total Student Burden Field Trial			1,210	1,452		626
School Administrator
Questionnaire	124	0.65	81	81	40	54
Recruitment and Pre-Assessment Activity
School Administrator	124	1.00	124	124	90	186
School Coordinator	124	0.65	81	81	240	323
Total School Burden Field Trial			205	286		563
MAIN STUDY—Based on core + international options
Student
Directions	8,000	0.85	6,800	6,800	10	1,134
Paper-and-pencil test booklet	8,000	0.85	6,800	6,800	120	13,600
Core Questionnaire	8,000	0.85	6,800	6,800	30	3,400
Financial Literacy background items	1,412	0.85	1,200	1,200	5	100
Directions (computer-based assessment)	2,353	0.85	2,000	2,000	10	334
Computer-based assessment	2,353	0.85	2,000	2,000	40	1,334
Total Student Burden Main Study			6,800	8,000		3,500
School Administrator
Questionnaire	194	0.85	165	165	30	83
Recruitment and Pre-Assessment Activity
School Administrator	194	1 .00	194	194	90	291
School Coordinator	194	0.85	165	165	240	660
Total School Burden Main Study			359	8,524		4,534

NOTES: Total student burden does not include time for cognitive assessment and its associated instructions.

6