memo

Date: October 13, 2006

Subject: Change request for PISA 2006 incentives

We would like discuss changing the incentive for the Saturday administrations of PISA 2006. This memo provides some background about the study, details the problem, and ends with a request to increase the student incentive for Saturday testing from $50 to $75 for the final two Saturdays in our field period (November 11 and November 18). We are in jeopardy of not meeting the student response rate required by the international sponsors and feel that an adjustment of the incentive for the Saturday test administrations will bring the response rates for those administrations to an adequate level.

Background of PISA 2006

For this study, there are two response rate requirements:  65% of original sample schools must participate, and countries must achieve an 80% student response rate.  If a country fails to achieve these minimums, their data will not be included in international reports.

The school target has been especially difficult for the US.  The US did not make the target in 2000 or 2003 but data were included.  However, in advance of this round, the US was told there would be no exceptions.  One of the major reasons schools refuse participation is concern over the loss of instructional time.  To address this concern, in the field trial we decided to experiment and offer schools that refused to administer PISA during the school day the option to administer PISA on a Saturday or after school.  For these sessions, the student incentive was increased from $15.00 (in school) to $75.00.  In the field test, in spring 2005, we had a 92% student participation rate in our Saturday sessions.  Based on the strong response rate, we decided to offer Saturday sessions for the main study.

Main study data collection began 2 weeks ago on September 25 and must end on November 22, 2006.  Nearly half the 166 schools in our sample have opted for Saturday test dates.  The OMB-approved incentive for these sessions is $50, which most of us thought would be adequate.

The Problem - Student Participation on Saturdays

On the first Saturday, our student response rate was 71.19%.  (By comparison, our in-school response rate is about 88%).  Two schools had response rates under 50% (49% and 24%); schools must have at least 50% student response rate to be counted as a participating school so we must have good makeup days in these schools.  We tried to put some new procedures in place for this past Saturday but did not improve much – a 73.04% response rate with all schools equal to or above 50%. An additional worry is that over one-quarter of our schools require explicit parental consent, mostly for Saturday sessions.  The table below presents response rates by type of consent and type of administration.

Response Rate by Type of Administration and Type of Consent    

	In-school	Out-of-school
Passive (34 schools)	87.25%  (18)	74.61% (16)
Active (7 schools)	95.45%%  (1)	68.86% (6)
Total (41 schools)	88% (19)	72% (22)

Notice that only a small percent of school thus far have been active consent.  If we project the results of this table out to the remaining schools, we estimate an overall student response rate of 77%, not including make-up sessions.  At this point, we have not conducted many make up sessions—we have been attempting to get school coordinators to allow us to conduct makeup sessions in school.

Remedies

We have established several new procedures to try to improve our response rates.

To monitor progress at each school and brainstorm ways to get students to sessions, we have calls with test administrators 10 days and 4 days before Saturday administrations. We have also asked our test administrators to go to the school a day in advance and meet with students for 5-10 minutes, if possible, to make a connection with the students and convey the importance of their participation.

The major problem for us is communicating with the students and parents to get them to come to the Saturday sessions. The incentive alone is not motivating their participation. We ask the school coordinator to give us home phone numbers so that we can remind students and parents about the Saturday sessions. Unfortunately, only 25% of schools can give us student or parent phone numbers. Since these reminder calls make a major difference for turnout, we have asked school coordinators to make those calls for Saturday or after-school sessions and we will reimburse them for their time. Note that in our original OMB submission, we stated that we would pay schools for any reasonable expenses, and we believe that these calls, while not anticipated when setting the coordinator incentive, are essential and represent real time and effort. We expect that additional time of school coordinators for phone calls (42 5-minute calls for test day), arranging advance sessions (1 hour), and phone calls for make-up sessions (18 5-minute calls) will be about $200 per school, for Saturday and active consent schools. This is based on an average teacher salary of $266 per day ($48000 annually (NCES 2004) divided by 180 days). We would also like to reimburse them for preparing additional mailings to parents and students.

Finally, we are concerned that we will not have sufficient time for makeup sessions following our last two Saturday administrations (November 11 and November 18). We would like to increase the incentives for those sessions to $75. In the field trial, we found that students and parents made time for the Saturday sessions, without prompting, and we believe that this was due largely to the high incentive. Without this incentive, we are worried that the response rates for these sessions will pull down our overall rate below the required 80%. There are ten administrations currently scheduled for those dates, and we expect another six to be scheduled from among the nine schools that have not yet set test dates.

We realize that this is a large incentive.  We also know that we are achieving our response rates by a very narrow margin and do not want to lose the battle in the final week of data collection.