Background
As part of the continuing effort to increase participation and motivation in high school NAEP, NCES has requested that NAEP Outreach and Dissemination contractor Hager Sharp conduct focus groups with high school seniors. These focus groups will serve as a continuation of the work that was done with the NCES Principals’ Working Group, Teachers’ Working Panel and Superintendents’ Working Panel.
The purpose of the student focus groups is to obtain feedback from students on their perception of their own and other seniors’ engagement in voluntary assessments. We will focus on four primary research questions:
1. What factors serve to motivate high school seniors in general?
2. What are the attitudes of high school seniors towards testing and what motivates them to do well?
3. Will high school seniors put forth their best effort on tests that have no personal consequences?
4. What factors can be used to motivate high school seniors to put forth their best effort on tests that have no personal consequences?
Specifically, it will be important to explore what strategies and tactics will encourage students to try their best. Using the suggestions and information that we have learned in our past working groups and panels, as well as what we expect to learn in these focus groups, we will incorporate student feedback into the development of future field and outreach materials. NCES and Hager Sharp will work together to produce a final report that summarizes the focus groups’ findings.
Methodology
We recommend holding two focus groups in five different sites that represent regions across the country. Usually, focus groups are held during the evening in back-to-back sessions (i.e., from 6:00-7:30 p.m. and from 8:00.-9:30 p.m.) We will coordinate with market research facilities to carry out the logistical details outlined below.
Maria Ivancin will
serve as the moderator for each of the focus groups. Maria has
experience facilitating prior working groups and panels for NCES and
Hager Sharp, including our efforts with superintendents, principals
and teachers. Maria is also an assistant professor at American
University and is familiar with working with students at this age
level.
Each focus group will consist of approximately 10-12 public high school students – a size large enough that we may hear various opinions, but small enough that participants will feel comfortable sharing their suggestions in an informal setting. Altogether, we will meet with approximately 100-120 students – a sample size large enough to obtain important qualitative information and feedback. We plan to screen approximately 800 to 1200 people to achieve the sample of 100-120 students. But we won't necessarily be administering the entire screening questionnaire to each person. Some people will fall out because they don't qualify (e.g. if we ask the first question and they aren't a senior in high school we wouldn't continue).
Group participants will be recruited by the contracted focus group facilities. NCES/Hager Sharp will provide the facilities with lists of schools that will be represented in the groups. In most cases, these facilities will work with their own databases of potential respondents. These databases of households typically include basic demographic information (including the number and ages of children and zip codes). Potential households will be selected based on their proximity to the schools that will be represented and the likelihood that there is a high school senior in the home. If there is a high school senior in the household that student will be asked the questions in the screening questionnaire.
If there is a need to supplement these databases, two possible methods will be used – (1) households will be drawn from local telephone directories and screened as to whether there is a high school senior; or (2) the focus group facilities will work with the schools and parent-teacher-student organizations to recruit the respondents. In all cases, the initial contact will likely be with an adult in the household (to simply determine whether a high school senior is present), but the screening questionnaire will be administered to the high school student. It is anticipated that as many as 8 to 10 households will be contacted to recruit a single participant. The use of the facilities’ databases will minimize the burden on the public. The information in the databases will allow the recruiters to contact only those households where there is a high likelihood that there is a high school senior.
Monetary incentives will be provided to the participants to encourage participation. The “standard” rate currently is $75 (there may be some variation from city to city and by the type of respondent, but $75 is on the lower end of what is currently being paid). This amount is deemed necessary both to compensate individuals for their time and trouble in attending, but also to ensure that we draw a better “sample” – the motivation to attend without an incentive can lead to some bias based on attitudes or lifestyle (e.g. only those people who are more curious, or those who are less busy may be more likely to attend; a financial incentive encourages a broader range of individuals to participate).
While these focus groups will seek a degree of diversity/variation among participants, these participants are not intended to be a representative group of students across the United States. The students selected, however, will all be high school seniors and will vary in their post-graduation plans (i.e., attending college, working or joining the military). The participants will also differ in their class rank and will represent various racial/ethnic backgrounds. About half of the students in each site will be male and the other half female.
At this time, we do not recommend conducting more than ten focus groups total, as we feel that additional sessions will not be necessary. In focus groups, the first few meetings often begin with the observers hearing new perspectives and recommendations. As time goes on and more focus groups on similar topics take place, we often hear similar suggestions and recommendations during each session.
As mentioned previously, participants will be high school seniors from public schools, and we will coordinate with the market research facilities to include students that represent the appropriate amount of diversity and backgrounds. We have already produced a screening questionnaire for potential candidates (see attached document). Market research facilities will use their database of potential participants and contact individuals by telephone. The screening questionnaire will be administered to each potential participant to ensure that he or she has the characteristics desired for this study – i.e., to be a high school senior and to represent diversity in future plans, academic achievement, ethnicity and gender. In addition to the characteristics identified through the screening questionnaire, each group will be representative of the demographics of that regional location and participants will be recruited from small and large schools in urban, inner-city and rural environments. At least two of the focus groups will take place in rural areas. In some cases, students may only represent one of these particular characteristics. These characteristics will be achieved by selecting appropriate sites, specific locations of the research in each site and identification of the schools that should be represented in the research.
In the first session of the night, we may meet with students from larger, urban or inner-city schools. The following session may consist of students from smaller, more suburban and/or rural schools. In all cases, students will be from public high schools in the area and no more than two participants within a group will come from the same high school. In each focus group, we will encourage the students to recognize that every opinion is valuable and ensure that all student names remain anonymous.
We will also videotape each focus group. Doing so will allow us, as needed, to refer to the students’ comments as we produce a final report, as well as have footage available for future presentations. We will observe the room and participants from behind glass mirrors.
Proposed Dates/Locations
We recommend conducting focus groups throughout the students’ senior years in high school. Doing so will allow us to hear student reactions and opinions at various important points during a student’s high school career, including before college applications are due/when students are considering plans after high school and at times when students are preparing for high school exit exams or other state assessments.
Participants in the student focus groups will be given a monetary payment to compensate for the time spent participating, as well as traveling to and from the market research facility. Participants will be paid a standard compensation rate of $75 per person for their participation in the focus group. Given the plan to recruit high school seniors through a market research company, it is essential to provide compensation at a rate that is competitive with similar efforts, to ensure that an adequate number of participants actually participate. The students will either take a car or bus to the focus group facility where the sessions will take place, and will be reimbursed for their transportation costs. Although the actual incentive payments for these respondents would be $75, we propose providing them with a separate travel reimbursement, particularly for students in suburban and rural areas. We believe that this additional travel compensation is necessary to ensure that these important populations be included in the focus groups. Without covering their direct travel expenses, it would be extremely difficult to draw these populations to the market research facility.
To maximize the diversity of opinions, we recommend conducting focus groups in four regions of the United States: Northeast, South, Midwest and West. As previously mentioned, we will include students from urban, suburban and rural high schools and will recruit participants from the attached screening questionnaire. We will work with Maria Ivancin and market research facilities to select the appropriate sites.
Pre-Planning Support
Hager Sharp will coordinate with Maria Ivancin to arrange the logistical details where the focus groups will take place. We will work in conjunction to recruit the appropriate number of students, provide snacks/sandwiches for participants, distribute compensation and secure the audiovisual arrangements.
On-site Logistics
We will arrive at each facility with sufficient time to review the logistical arrangements and ensure that the setting is adequately prepared for the focus group session that evening. During the meeting, Maria Ivancin will lead participants through a discussion, using the Moderator’s Guide (attached) that she and Hager Sharp have produced. In addition, two Hager Sharp team members will observe and take notes of the participants’ comments and suggestions. One of these team members will also serve as the liaison, as necessary, with the focus group facility for any on-site issues.
Post-Meeting Activities
Following the conclusion of each focus group, Hager Sharp will compile the minutes and include updates on the meetings in our monthly reports. The end product will be a final report that presents the answers to the research questions outlined above, the methodology, the recommendations heard and the suggestions for next steps. This report can serve as the foundation for making changes to NAEP processes, as the focus will be on enhancing student engagement in and motivation to try hard on NAEP. Additionally, the recommendations may be used for future briefings to the National Assessment Governing Board. The suggestions and ideas that we expect to hear will also help to guide our work in future field and outreach materials to students who will eventually participate in NAEP.
Estimated Budget
The following budget breaks down the costs associated with conducting two focus groups in each site:
Facilities fee/recruiting participants: $4,500
Room/equipment/supplies: $3,500
Moderation/facilitation: $3,000
Reimbursements: $1,200
Travel (flights, lodging and meals for three people): $2,000
Labor: $2,000
Other indirects and out of pockets: $1,000
Produce report (including graphic design): $800
Therefore, we project a budget of approximately $18,000 per site/series of two focus groups. Since we are proposing to hold five sets of two groups, we envision a total budget of around $90,000 altogether.
Next Steps
Pending approval of this proposal, we will begin preparing logistical arrangements for conducting the focus groups. We know that this work has an important impact on the future of NAEP participation at the high school level. Meeting with seniors will help to determine feedback on student engagement in voluntary assessments, as well as the methods and incentives that might enhance participation in future assessments. We will incorporate summaries of the student focus groups into our regular monthly report and prepare a final report that outlines the findings. We look forward to continuing the efforts related to high school participation and motivation with an important additional audience – high school seniors in public schools.
| File Type | application/msword |
| File Title | To: |
| Author | Hager Sharp |
| Last Modified By | DoED User |
| File Modified | 2007-12-11 |
| File Created | 2007-12-11 |