Vol 1 Justification

National Center for Education Statistics

National Assessment of Educational Progress

Volume I

Supporting Statement

Key Concepts for Assessment Literacy

Focus Groups

OMB# 1880-0542

September 30, 2014

Contents

1) Submittal-Related Information 1

2) Background and Study Rationale 1

3) Sampling and Recruitment Plans 1

4) Data Collection Process 3

5) Consultations Outside the Agency 4

6) Assurance of Confidentiality 4

7) Justification for Sensitive Questions 4

8) Estimate of Hourly Burden 4

9) Recruitment Costs 5

10) Costs to Federal Government 5

11) Schedule 5

1. Submittal-Related Information

This material is being submitted under the generic U.S. Department of Education clearance agreement (OMB #1880-0542), which provides for the Department to conduct customer feedback surveys and similar studies.

2. Background and Study Rationale

The National Assessment of Educational Progress (NAEP) is a federally authorized assessment of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, civics, geography, economics, and the arts. NAEP is administered by the National Center for Education Statistics (NCES), part of the Institute for Education Sciences, within the U.S. Department of Education.

In 2013, a major assessment-literacy initiative was undertaken by the National Assessment Governing Board (NAGB) to explore current perception and knowledge of educational assessments. The purpose of this initiative is to promote greater understanding of assessments and the role of NAEP in assessments for three key target audiences: students, parents/guardians, and policymakers.

To help inform efforts to educate audiences about assessment literacy, focus groups with students, parents, teachers, and educational policymakers are planned. Hager Sharp, a communications firm experienced in conducting focus groups has been selected by NCES to conduct the NAEP Assessment Literacy focus groups designed to discern the knowledge, attitudes, and beliefs of the participants concerning educational assessments. Topics include the function of formative and summative assessments, interpreting the results of educational assessments, considerations of their quality and accuracy, and questions audiences may have about educational assessments.

The results of this research will be used to develop communication materials to encourage assessment literacy and engagement. To that end, the focus groups will also provide needed feedback on the content, presentation, and usability of materials to be developed in support of the educational literacy initiative. It is important to note that the focus groups will not inform policy – they are intended to inform education and communication efforts only. Findings from these focus group will be summarized in a report.

Volume I of this submittal contains descriptions of the design and sampling, estimated respondent burden, cost, and schedule information for the study. Volume II contains the protocols for the various focus groups. The appendices contain consent forms, screening checklists, and phone scripts.

3. Sampling and Recruitment Plans

Hager Sharp will conduct four in-person focus groups with parents/guardians and students in Baltimore, MD, and Richmond, VA respectively; plus four focus groups with policymakers and teachers, which will be conducted online to achieve greater geographic diversity in the perspectives among participants.

The focus group method is a qualitative group approach to data collection, which involves holding group sessions guided by a moderator, who in turn follows a topical outline containing questions or topics focused on a particular issue. The questions are purposefully open-ended in nature, incorporating several prompts to encourage a deeper, multifaceted exploration of a topic. The method is useful for going beneath the surface of a response. As a research tool, focus groups are useful for understanding a targeted group’s views on complex social topics such as issues surrounding cultural practices and education.

The ideal focus group size is 8-10 people. Having a group this size enables discussion and provides for a diversity of ideas to be heard. With more than 10 participants, it becomes less possible to ensure that all views are heard, that a discussion flows in a desirable manner, or that all of the questions are addressed in the time allotted. Having a group with less than 8 individuals limits the number of perspectives that are heard, and may also impede discussion. To ensure that appropriate and relevant information is obtained, it is anticipated that the focus groups will have 9 participants each, for a total of up to 80 respondents across the respondent types and grades, as shown in Table 1.

Table 1. Sample Size for Focus Groups

Location	Respondent Group	Eligibility Criteria	Number of Participants	Timing
Baltimore, MD	Parents/guardians of children in grades 3-12 in public schools	Lower income (Household income less than $40K) Diversity in race/ethnicity	Recruit 10 for 9 to show	120 minutes; 4:30-6:30 p.m.
		Mid/high income (Household income $50K or greater) Diversity in race/ethnicity	Recruit 10 for 9 to show	120 minutes; 7:00-9:00 p.m.
Richmond, VA	Students attending public schools	Middle school students (grades 7-8) Diversity in race/ethnicity	Recruit 10 for 9 to show	90 minutes; 5:00-6:30 p.m.
		High school students (grades 9-12) Diversity in race/ethnicity	Recruit 10 for 9 to show	90 minutes; 5:30-7:00 p.m.
Online, via WebEx	Policymakers	Assistant school superintendents	Recruit 10 for 9 to show	90 minutes; 5:30-7:00 p.m.
		School board members	Recruit 10 for 9 to show	90 minutes; 7:30-9:00 p.m.
	Teachers	Teachers in grades 7-8	Recruit 10 for 9 to show	90 minutes; 5:30-7:00 p.m.
		Teachers in grades 9-12	Recruit 10 for 9 to show	90 minutes; 7:30-9:00 p.m.

Recruitment of In-person Focus Groups

Participants will be recruited in each location by local recruiting resources (see section 5). Participant positions will be filled through the use of opt-in databases of local residents that are maintained by the local focus group facilities engaged for this project—Baltimore Research in Baltimore, MD, and Alan Newman Research in Richmond, VA. Both vendors are experienced in recruiting and hosting focus groups for federal studies. To develop their databases, recruiters call local residents at random from telephone listings to ask them to participate in the database. The criteria for inclusion in the databases are generally very broad. Participants are then screened more narrowly for specific studies, based on the objectives and screening criteria of each study.

As is typical with focus group research, the sampling for the sessions is purposive in that recruiters are asked to find participants that meet specific criteria (e.g., parents/guardians with children in public schools, grades 3-12). To identify participants, we will use two sets of screening questionnaires, one for parents/guardians (appendix A) and one for students (appendix B). Based on experience with similar groups, up to 90 people will be screened for each focus group to achieve nine participants. The screening questionnaires also include instructions for those selected to participate.

Recruiters will ensure that they meet the specific target audience requirements needed for the research. In addition, recruiters will ensure diversity among participants regarding socioeconomic and other demographic factors. Separate moderator’s scripts/guides—one for parents/guardians and one for students will be used in the focus group sessions to guide conversations (see Volume II). Each focus group is designed to be a one-time information collection activity with participants specifically recruited to inform the assessment literacy education effort.

Parents/guardians sought for the 2 parent/guardian focus groups (to take place in Baltimore, MD) will be identified through Baltimore Research’s local database. Parents of students sought for the 2 student focus groups (to take place in Richmond, VA) will be identified through Alan Newman Research’s local database. The initial contact will take place via phone. The recruiter will briefly describe the study, the types of information sought, and will ask the parent/guardian to provide demographic information about themselves (for the parent/guardian focus groups) or their child (for the student focus groups) to ensure a diverse sample is selected per the aforementioned criteria (see appendix A and B). During that call, for student focus groups, the parent/guardian’s willingness to allow their child to participate in the focus group session will also be confirmed. If unsuccessful in contacting them on the first call, the recruiter will identify the parent/guardian as a non-respondent and make no further attempts at contact.

As participants are recruited, they will be scheduled in a focus group at a particular date and time. Once they are scheduled, the recruiter will send a confirmation e-mail and/or letter (see Appendix E) and for student participants a parental consent form (see appendix L). Students will need to bring the signed parental consent form to the session to be allowed to participate. Additionally, written consent will be obtained from student participants who are over the age of 18 (see Appendix K). Parent/guardian participants will review and sign a consent form (appendix H) at arrival.

Recruitment of Online Focus Groups

Hager Sharp will conduct two focus group sessions via WebEx with education policymakers and two focus group sessions with teachers. Recruiters will ensure they meet the specific target audience requirements needed for the research and ensure diversity among participants. Two separate moderators’ scripts/guides—one for policymakers (assistant superintendents and school board members) and one for teachers (see Volume II)—will be used in the focus group sessions to guide conversations. Each focus group is designed to be a one-time information collection activity with participants specifically recruited to inform the assessment literacy education effort.

Assistant superintendents, school district board members, and teachers will be sought for their respective focus group through Alan Newman Research’s national database. The initial contact will take place via phone, during which the recruiter will briefly describe the study; the types of information sought; and will ask participants to provide their demographic information to ensure a diverse sample is selected as per the aforementioned criteria (see appendix C and D).

If the participant qualifies for the focus group, he or she will be invited to attend the WebEx discussion on the specified date. Once respondents agree to participate, Alan Newman Research will follow up via email or web form to provide a copy of the consent form (see appendix I and J) and provide additional information about the WebEx (see appendix F and G). Consent forms will be obtained prior to the start of the interviews.

Consent Forms

The individual consent forms outline the name of the project and its purpose; the criteria and procedure for participation, including ability to withdraw from participation at any point during the process without negative consequence; the process for handling personal information; any monetary token of appreciation given for participation; and the project director’s contact information. All participants will receive a copy of the signed consent form for their records. The purpose of the research and confidentiality statement will be reiterated during introductory remarks by the focus group moderator.

No further preparation prior to the group discussion will be required. The focus groups with parents/guardians will be 2 hours in length and with all other participants 1.5 hours in length to allow for adequate discussion and testing of prototype messages and materials. They will take place during afternoon/evening hours on midweek dates to ensure that the maximum number of individuals will be available to attend without interfering with their hours of employment or school.

4. Data Collection Process

Participants will first be welcomed, introduced to the moderator, and told they are there to help answer questions about assessment literacy and the role of assessments in learning/instruction (see Volume II). The moderator from Hager Sharp will be tasked with keeping participants engaged by asking probe questions, soliciting responses from less talkative participants, and asking follow-up questions directed to the group (e.g., “what do you all think about… [repeat focus group probe]”, or “that’s interesting, could you tell me a little bit more about that”). Interviewers may also take additional notes during the focus groups, including behaviors (e.g., the participant’s facial expressions indicating that they are confused), and if extra time was needed to answer certain questions.

For the online focus groups, the moderator will explain the technology and describe the tools the participants may use, such as muting their phone or asking questions. The moderator will use the focus group WebEx guide (see Volume II) for each of the sessions.

Participants will be reassured that their participation is voluntary and that their responses will be used for research purposes only (see section 6). As part of the introduction process, the moderator will explain to participants that their responses will be recorded. A video recording will provide opportunity to playback the sessions to ensure accurate capture of verbatim quotes. In the analysis of each focus group, the senior researcher will be able to use both the transcript of the discussion and the video recordings. After the session, the notes and video and/or audio recording will be summarized into a final report.

5. Consultations Outside the Agency¹

Hager Sharp will moderate the focus groups, analyze the findings, and prepare preliminary and final reports. Hager Sharp has worked as a contractor for NCES since 2002, providing publication, outreach, and dissemination support for NAEP. The communications firm is familiar with the processes of NCES, IES, and the Governing Board.

Baltimore Research is a full-service marketing research firm located in Baltimore, MD that provides a broad portfolio of research services, including focus groups, consumer surveys, and data collection analytics to a wide variety of industries. Baltimore Research will work to identify and recruit parents for the parent focus groups.

Alan Newman Research is a Virginia-based market research company. It has been in the market research field for over 25 years providing quantitative, qualitative, and field services. Alan Newman Research will work to identify and recruit students, teachers, and policymakers for the focus groups.

6. Assurance of Confidentiality

Prior to the start of the focus groups, participants will be notified that their participation is voluntary and that their answers may be used only for research purposes, and may not be disclosed or used in identifiable form for any other purpose except as required by law (20 U.S.C. §9573).

Written consent will be obtained from participants who are over the age of 18 and from parents or legal guardians of students who are under the age of 18. During the informed consent process, participants (or parents of student participants) will be asked to read and sign the consent form. After the participant signs the consent forms, copies will be made for the participants to have for their records. Only individuals who sign the consent form will be eligible to participate in the focus group.

Participants will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form. The consent forms, which include the participant name, will be separated from the participant interview files, secured for the duration of the study, and will be destroyed after the final report is completed.

The interviews will be recorded². The only identification included on the files will be the unique ID assigned to each participant by the interviewer. The video recordings will only be used for research purposes, and no identifying information will be included in the data analyses. The recorded files will be secured for the duration of the study and will be destroyed after the final report is submitted.

7. Justification for Sensitive Questions

This study does not involve sensitive questions.

8. Estimate of Hourly Burden

This study includes eight focus groups. The maximum number of focus group participants will be approximately 80 (ten per group). Table 2 presents the estimated respondent burden hours associated with this study. This burden estimates is based on other NCES focus groups conducted for similar programs.

Table 2. Estimated Burden Hours

Respondent Type	Maximum Number of Respondents	Number of Responses	Burden Per Respondent	Total Hours
Parent/Guardian of student in public schools, grades 3-12
Screening/Recruitment	180	180	10 minutes	30
Focus Group	20*	20	2 hours	40
Student in public school, grades 7-12
Screening/Recruitment	180	180	10 minutes	30
Focus Group	20*	20	1.5 hours	30
Education policymakers
Screening/Recruitment	180	180	10 minutes	30
Focus Group	20*	20	1.5 hours	30
Teachers, grades 7-12
Screening/Recruitment	180	180	10 minutes	30
Focus Group	20*	20	1.5 hours	30
Total	720	800		250

* Subset of initial contact group.

9. Recruitment Costs

The estimated incentive costs for respondents are $4,100. To encourage participation and thank each respondent for his or her time and effort, VISA gift cards will be offered to each respondent.

For the student focus groups, a $25 VISA gift card will be offered to each participating student, plus $25 VISA gift card to a parent or legal guardian for bringing the student to and from the focus group facility.

For the focus groups among parents, we will provide an incentive of a $75 VISA gift card. For the teacher focus groups, a $45 VISA gift card will be offered to each participating teacher. For the groups among policymakers, we will offer an incentive of $40 VISA gift card with an option to donate the incentive to a charity of their choice. The incentive amounts are being offered to improve the quality of the information being collected, such as through obtaining respondents who otherwise would not participate and reducing bias in participation. The parents will be given a higher amount than the teachers and policymakers given that the parent focus group will occur in-person. The amounts provided for this study are consistent with previous focus groups performed for NAEP³.

10. Costs to Federal Government

The estimated cost to the government of all recruitment and data collection, analysis, and reporting activities for this study is $18,700.

11. Schedule

Table 3 depicts the high-level schedule for the focus group activities. The commencement of activities is contingent upon OMB approval.

Table 3. High-Level Schedule of Milestones

Dates	Activity
October 2014	Recruitment process
October 2014 - March 2015	Data collection, analysis, and reporting

1 Please see appendix M for contact information for each of the contractors.

2 Recordings may be audio and/or video, as described in the specific interview sections.

3 See (2014 NAEP) Focus Groups for Transitioning Reading and Math Survey Questionnaires to Technology Based Assessments (OMB# 1850-0803 v.96).