System Clearance for Cognitive, Pilot and Field Test Studies -- NAEP Focus Group Study of High School Student Motivation

Background

As part of the continuing effort to increase participation and motivation in high school NAEP, NCES has requested that NAEP Outreach and Dissemination contractor Hager Sharp conduct focus groups with high school seniors. These focus groups will serve as a continuation of the work that was done with the NCES Principals’ Working Group, Teachers’ Working Panel and Superintendents’ Working Panel.

The purpose of the student focus groups is to obtain feedback from students on their perception of their own and other seniors’ engagement in voluntary assessments. We will focus on four primary research questions:

1. What factors serve to motivate high school seniors in general?

2. What are the attitudes of high school seniors towards testing and what motivates them to do well?

3. Will high school seniors put forth their best effort on tests that have no personal consequences?

4. What factors can be used to motivate high school seniors to put forth their best effort on tests that have no personal consequences?

Specifically, it will be important to explore what strategies and tactics will encourage students to try their best. Using the suggestions and information that we have learned in our past working groups and panels, as well as what we expect to learn in these focus groups, we will incorporate student feedback into the development of future field and outreach materials. NCES and Hager Sharp will work together to produce a final report that summarizes the focus groups’ findings.

Methodology

We recommend holding two focus groups in five different sites that represent regions across the country. Usually, focus groups are held during the evening in back-to-back sessions (i.e., from 6:00-7:30 p.m. and from 8:00.-9:30 p.m.) We will coordinate with market research facilities to carry out the logistical details outlined below.

Maria Ivancin, a professional facilitator with Market Research Bureau LLC, will serve as the moderator for each of the focus groups. Maria has experience facilitating prior working groups and panels for NCES and Hager Sharp, including our efforts with superintendents, principals and teachers. (Please see the attached resume for further information). Maria is also an assistant professor at American University and is familiar with working with students at this age level.

Each focus group will consist of approximately 10-12 public high school students – a size large enough that we may hear various opinions, but small enough that participants will feel comfortable sharing their suggestions in an informal setting. Altogether, we will meet with approximately 100-120 students – a sample size large enough to obtain important qualitative information and feedback. While these focus groups will seek a degree of diversity/variation among participants, these participants are not intended to be a representative group of students across the United States. The students selected, however, will all be high school seniors and will vary in their post-graduation plans (i.e., attending college, working or joining the military). The participants will also differ in their class rank and will represent various racial/ethnic backgrounds. About half of the students in each site will be male and the other half female.

At this time, we do not recommend conducting more than ten focus groups total, as we feel that additional sessions will not be necessary. In focus groups, the first few meetings often begin with the observers hearing new perspectives and recommendations. As time goes on and more focus groups on similar topics take place, we often hear similar suggestions and recommendations during each session.

Group participants will be recruited by the contracted focus group facilities. As mentioned previously, participants will be high school seniors from public schools, and we will coordinate with the market research facilities to include students that represent the appropriate amount of diversity and backgrounds. We have already produced a screening questionnaire for potential candidates (see attached document). Market research facilities will use their database of potential participants and contact individuals by telephone. The screening questionnaire will be administered to each potential participant to ensure that he or she has the characteristics desired for this study – i.e., to be a high school senior and to represent diversity in future plans, academic achievement, ethnicity and gender.

NCES/Hager Sharp will provide the facilities with lists of schools that will be represented in the groups. In most cases, these facilities will work with their own databases of potential respondents. These databases of households typically include basic demographic information (including the number and ages of children and zip codes). Potential households will be selected based on their proximity to the schools that will be represented and the likelihood that there is a high school senior in the home. If there is a high school senior in the household, that student will be asked the questions in the screening questionnaire.

If there is a need to supplement these databases, two possible methods will be used – (1) households will be drawn from local telephone directories and screened as to whether there is a high school senior; or (2) the focus group facilities will work with the schools and parent-teacher-student organizations to recruit the respondents. In all cases, the initial contact will likely be with an adult in the household (to simply determine whether a high school senior is present), but the screening questionnaire will be administered to the high school student. It is anticipated that as many as 8 to 10 households will be contacted to recruit a single participant. The use of the facilities’ databases will minimize the burden on the public. The information in the databases will allow the recruiters to contact only those households where there is a high likelihood that there is a high school senior.

In addition to the characteristics identified through the screening questionnaire, each group will be representative of the demographics of that regional location and participants will be recruited from small and large schools in urban, inner-city and rural environments. At least two of the focus groups will take place in rural areas. In some cases, students may only represent one of these particular characteristics. These characteristics will be achieved by selecting appropriate sites, specific locations of the research in each site and identification of the schools that should be represented in the research.

In the first session of the night, we may meet with students from larger, urban or inner-city schools. The following session may consist of students from smaller, more suburban and/or rural schools. In all cases, students will be from public high schools in the area and no more than two participants within a group will come from the same high school. In each focus group, we will encourage the students to recognize that every opinion is valuable and ensure that all student names remain anonymous. We will observe the room and participants from behind glass mirrors.

Proposed Dates/Locations

We recommend conducting focus groups throughout the students’ senior years in high school.  Doing so will allow us to hear student reactions and opinions at various important points during a student’s high school career, including before college applications are due/when students are considering plans after high school and at times when students are preparing for high school exit exams or other state assessments.

Participants in the student focus groups will be given a monetary payment to compensate for the time spent participating, as well as traveling to and from the market research facility. The “standard” rate currently is $75 (there may be some variation from city to city and by the type of respondent, but $75 is on the lower end of what is currently being paid). A monetary payment is deemed necessary both to compensate individuals for their time and trouble in attending, but also to ensure that we draw a better “sample” – the motivation to attend without an incentive can lead to some bias based on attitudes or lifestyle (e.g., only those people who are more curious or those who are less busy may be more likely to attend; a financial incentive encourages a broader range of individuals to participate). Given the plan to recruit high school seniors through a market research company, it is essential to provide compensation at a rate that is competitive with similar efforts, to ensure that an adequate number of participants actually participate. There appears to be no precedent, however, for paying $75 per participant for studies of this nature sponsored by the Department of Education. We therefore propose an incentive payment of $50 per student.

To maximize the diversity of opinions, we recommend conducting focus groups in four regions of the United States: Northeast, South, Midwest and West.  As previously mentioned, we will include students from urban, suburban and rural high schools and will recruit participants from the attached screening questionnaire.  We will work with Maria Ivancin and market research facilities to select the appropriate sites.

Pre-Planning Support

Hager Sharp will coordinate with Maria Ivancin to arrange the logistical details where the focus groups will take place. We will work in conjunction to recruit the appropriate number of students, provide snacks/sandwiches for participants, distribute compensation and organize the practical arrangements.

On-site Logistics

We will arrive at each facility with sufficient time to review the logistical arrangements and ensure that the setting is adequately prepared for the focus group session that evening. During the meeting, Maria Ivancin will lead participants through a discussion, using the Moderator’s Guide (attached) that she and Hager Sharp have produced. In addition, two Hager Sharp team members will observe and take notes of the participants’ comments and suggestions. One of these team members will also serve as the liaison, as necessary, with the focus group facility for any on-site issues.

Post-Meeting Activities

Following the conclusion of each focus group, Hager Sharp will compile the minutes and include updates on the meetings in our monthly reports. The end product will be a final report that presents the answers to the research questions outlined above, the methodology, the recommendations heard and the suggestions for next steps. This report can serve as the foundation for making changes to NAEP processes, as the focus will be on enhancing student engagement in and motivation to try hard on NAEP. Additionally, the recommendations may be used for future briefings to the National Assessment Governing Board. The suggestions and ideas that we expect to hear will also help to guide our work in future field and outreach materials to students who will eventually participate in NAEP.

Estimated Budget

The following budget breaks down the costs associated with conducting two focus groups in each site:

Facilities fee/recruiting participants: $4,500

Room/equipment/supplies: $3,500

Moderation/facilitation: $3,000

Reimbursements: $1,200

Travel (flights, lodging and meals for three people): $2,000

Labor: $2,000

Other indirects and out of pockets: $1,000

Produce report (including graphic design): $800

Therefore, we project a budget of approximately $18,000 per site/series of two focus groups. Since we are proposing to hold five sets of two groups, we envision a total budget of around $90,000 altogether.

Next Steps

Pending approval of this proposal, we will begin preparing logistical arrangements for conducting the focus groups. We know that this work has an important impact on the future of NAEP participation at the high school level. Meeting with seniors will help to determine feedback on student engagement in voluntary assessments, as well as the methods and incentives that might enhance participation in future assessments. We will incorporate summaries of the student focus groups into our regular monthly reports and prepare a final report that outlines the findings. We look forward to continuing the efforts related to high school participation and motivation with an important additional audience – high school seniors in public schools.

5