OMB Passback (Q&A)

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: September 23, 2010

TO: Shelly Martinez, OMB

FROM: Dana Kelly, NCES

THROUGH: Kashka Kubzdela, NCES

RE: Response to 09-16-2010 through 09-28-2010OMB Passback for PISA 2012 Focus Groups

(OMB# 1850-0803 v. 36)

The revisions referred to in the responses below have been made in the memo and appendices A and B. In addition, as part of our response, two documents have been added to appendix C (screen shots of draft PISA online school reporting tool and draft PISA school report); and three documents have been added to appendix D (NAEP 2010 Customizable PowerPoint for Students, NAEP 2010 Sample School Newspaper Article, and PISA 2009 Certificate of Voluntary Service).

Q1. It would be nice to use these focus groups to inform the largest set of NCES studies possible, so increase of only about the decision to participation in PISA, try to work in more general feelings about participating in international studies and even national studies.

Response: We agree that we want the information from these focus groups to inform other studies conducted by NCES. Because our primary task is to collect information specific to recruitment efforts for PISA 2012, with its specific target population and increased burden due to the computer-based assessment, we will focus on PISA in these focus groups, but the focus group materials (appendix A, p. 6, for principals and appendix B, p. 7, for students) and memo (p. 2) have been revised to incorporate additional questions about principal and student perceptions of international and voluntary assessments in general. The feedback on these items, the more specific feedback on materials designed for PISA, and the voluntary components of NAEP are expected to provide helpful information for school recruitment efforts across NCES data collection programs.

Q2. Why not show principals the school level statistic that HSLS was able to provide since that seemed to be a useful recruiting tool for them?

Response: HSLS is sharing school-level information similar to, and based on, what NCES developed to report back to PISA 2009 schools (to be shared with PISA 2009 schools in December 2010). The PISA 2009 sample report will be presented to principals in the focus group as part of the materials on which we are seeking their feedback. PISA 2009 school reporting includes a secure interactive web-based tool through which principals can see how their school compares with: the United States and Organization of Economic Cooperation and Development (OECD) countries on average, similar schools in the United States and OECD countries, and all PISA-participating countries. The site will also include a print report showing whether the school’s average scores (reading, three reading subscales, math, and science) were above, below, or not different from the U.S. averages, averages of the OECD countries, and similar schools in the United States, and which countries were above, below or not different from the school in reading. When recruiting PISA 2009 schools, we told the schools that they would receive reports showing their position relative to the United States and OECD averages. Since that time, we were able to further develop PISA school reporting to include more information and we developed the web-based tool, both of which we feel the schools will view very positively.

We have added to appendix C screen shots from the online tool and a draft final report. Both are in development and final approved versions will be shared with PISA 2009 schools in December 2010 after the PISA 2009 data are released. We will submit the final copy of the final version of the report and screenshots of the online tool to amend the PISA 2009 OMB record.

Q3. Has Laura LoGerfo looked at this?  It would be good to hear if she has other thoughts about recruiting strategies from either HSLS or from chatting with the larger set of PIs.

Response: We have shared our focus group plans and materials with Laura LoGerfo, who had no comments, and will also share the focus group results with her. More generally, for over a year, the PISA staff at NCES have been in dialogue with Laura and others about improving school recruitment outcomes (including participation in a meeting convened in 2009 to discuss how different IES programs were addressing recruitment and incentivizing schools and students).

Q4. Since it seems that NAEP has been largely unsuccessful at motivating 12^th graders, why not show the students something else (e.g., maybe even the materials designed for the adults)?

Response: NAEP developed a Best Practices Guide for improving participation and motivation of 12th grade students in NAEP 2007. The guide included the materials for students that were listed in the memo as well as talking points and materials for the principals to “sell” the importance of NAEP to teachers and students. Student participation increased 13 percentage points from 66% in 2005 to 79% in 2007. Given this, the fact that NAAL 2003 achieved the final weighted response rate for the household sample of only 62% and ALL (international literacy assessment) 2003 of 68% overall, plus the effort that was put into developing NAEP and PISA recruitment materials specifically targeting teenagers, we recommend using the examples of NAEP student materials (included in Appendix D) in the student focus groups rather than the NAAL or ALL materials. Prompted by the OMB passback, we have reviewed again the available materials and decided to include three additional materials to obtain feedback on during the student focus groups: NAEP 2010 Customizable PowerPoint for Students, NAEP 2010 Sample School Newspaper Article, and PISA 2009 Certificate of Voluntary Service (Appendix D).

Q5. We want to be very clear that we find this exercise to be potentially very useful but that the questions to principals and students are somewhat analogous to hypothesis generation.  Such opinions will not constitute sufficient justification for any incentive proposals.  Instead, they would be useful input for an incentive experiment.

Response: We agree that findings from the focus groups cannot justify incentive proposals alone. In the request for OMB approval clearance package for PISA 2012 Recruitment and Field Test (OMB# 1850-0755 v.10) that is currently awaiting OMB’s review, we proposed to conduct an incentive experiment during the PISA 2012 field test to inform incentive proposals for the main study. Ideally we would have completed the focus groups before submitting the PISA 2012 Recruitment and Field Test clearance package, but we had to await the award of the PISA 2012 contract (awarded on August 6, 2010) before we could design and propose the focus groups on recruitment strategies. Depending on what we learn during the focus groups, we may need to submit a change request proposing revisions to the incentive experiment.

Q6. Without much more explicit guidance than students who “represent a cross-section of their student population” we would be concerned that principals will give names only of students likely to represent the school well, i.e., more motivated students.  How will NCES ensure that less motivated or even just “typical” students are included?

Response: The memo (p. 5) and principal focus group materials (p. 1) have been revised to provide further guidance to principals about which students to recruit for the study.

Q7. Can NCES provide other relevant examples where we have approved $50 for teens to participate in a focus group? 

Response: NAEP has offered $50 to 12^th grade students for focus groups and $30 to students and $25 to parents for cognitive laboratories (see excerpts below). Because these are focus groups (with a higher pre-approved incentive level of $75 than cognitive labs of $40 to make up for the lesser flexibility in scheduling a focus group) and because some students may come without parents’ help, we propose $50 per participating student. We assume that if parents will drive the student to the focus group, the parents will be aware of the incentive amount and able to negotiate with the student a reimbursement for their time and gas, if needed.

In NAEP Focus Group Study of High School Student Motivation (OMB# 1850-0803 v.10), NAEP provided $50 for participation of 12^th grade students. OMB was concerned about the originally proposed $75 and the proposal was revised to state the following:

“Participants in the student focus groups will be given a monetary payment to compensate for the time spent participating, as well as traveling to and from the market research facility. The “standard” rate currently is $75 (there may be some variation from city to city and by the type of respondent, but $75 is on the lower end of what is currently being paid). A monetary payment is deemed necessary both to compensate individuals for their time and trouble in attending, but also to ensure that we draw a better “sample” – the motivation to attend without an incentive can lead to some bias based on attitudes or lifestyle (e.g., only those people who are more curious or those who are less busy may be more likely to attend; a financial incentive encourages a broader range of individuals to participate). Given the plan to recruit high school seniors through a market research company, it is essential to provide compensation at a rate that is competitive with similar efforts, to ensure that an adequate number of participants actually participate. There appears to be no precedent, however, for paying $75 per participant for studies of this nature sponsored by the Department of Education. We therefore propose an incentive payment of $50 per student.”

Below is information regarding student incentives for $30 and a parent incentive of $25 for the for the In the NAEP 2011 Writing Assessment Audiovisual Stimuli Cognitive Interviews (OMB# 1850-0803 v.27), the total incentive per student amounted to $55: students were provided $30 and parents $25 to offset the travel/transportation costs to bring the participating student to the cognitive laboratory site. The total burden for each student/parent was estimated at 1 hour and 50 minutes (we estimate for the PISA 2012 student focus group at 2 hours).

“Each participating student will receive a $30 gift card in compensation for time and effort. Because the study will take place outside of regular academic school hours, the monetary incentive is aimed at ensuring participation and motivation on behalf of the students. In addition, we are offering a check of $25 per parent to help offset the travel/transportation costs to bring the participating student to the cognitive laboratory site.”

Q8: Is NCES assuming that no parental assent/consent is needed? We do not think calling a student "cold" without involving the parents is a good idea. Please consider making the initial contact with the parent.

Response: The (principal (Appendix A, p. 2) and student (Appendix B, p. 1 and 3) focus groups consent forms are already included in the package. The memo (p. 6) has been revised to emphasize that parental consent will be sought and the parent/students consent form has been revised to correct the spelling of “guardian”.

Q9:  On the PISA piece, we would prefer not to give to youth of this age $50.  We prefer the "split" between the child and parent, or, if NCES really thinks that this group will not be accompanied by their parents, then will ok $40 as a maximum amount to give to the youth.

Response: We will provide $35 to each student that participates and offer $20 to the students’ parent to offset the travel/transportation costs to bring the participating student to the focus group discussion location.