Justification

This material is being submitted under the generic U.S. Department of Education clearance agreement (OMB# 1880-0542), which provides for the Department to conduct surveys and other studies regarding customer satisfaction.

Background and Study Rationale

National Assessment of Educational Progress (NAEP) is administered by the National Center for Education Statistics (NCES), part of the Institute for Education Sciences, within the U.S. Department of Education. In 2013, a major assessment-literacy initiative was undertaken by the National Assessment Governing Board (NAGB) to explore current perception and knowledge of educational assessments. The purpose of this initiative is to promote greater understanding of assessments and the role of NAEP in assessments for three key target audiences: students, parents/guardians, and policymakers.

In October 2014, parents were selected as the first audience for the assessment initiative. To build on the information obtained in that study, and further develop communication materials, which will educate parent audiences about assessment literacy, NCES now proposes to conduct a focus group study with parents and teachers. Research has shown that teachers are instrumental in building assessment literacy among parents, serving a critical role as intermediaries and influencers. Thus, teachers also will be a focus of outreach and materials development to equip them to support assessment literacy goals.

Hager Sharp, a communications firm experienced in conducting focus groups has been selected by NCES to conduct the NAEP Assessment Literacy focus groups. Also, it is important to note that the focus groups will not inform policy – they are intended to inform education and communication efforts only. Findings from these focus groups will be summarized in a report.

Hager Sharp will conduct two focus group sessions in person with parents and two via WebEx with teachers. The goal of the focus group sessions is to get their feedback on a tool that has been created to help parents discuss their concerns and questions about assessments with teachers. Parents and teachers are likely to have valuable insights into whether the Discussion Guide (see Appendix D: Print Mockup of Parent Discussion Guide) will be useful to parents in addressing their typical questions about this issue. To optimize the likelihood that teachers will be able to share their professional expertise and opinions with us while minimizing inconvenience to them, we will use a web-based format to share the draft tools and gather their feedback.

In this manner, the proposed focus groups, with parents and teachers, will evaluate three different communication prototypes (shown in Appendices G thru I) and will provide needed feedback on the content, presentation, and usability of these developed materials. The results will be used to finalize these materials.

Volume I of this submittal contains descriptions of the design and sampling, estimated respondent burden, cost, and schedule information for the study. Volume II contains the screening call scripts and focus group protocols for the various focus groups. The appendices contain consent forms, screening checklists, and phone scripts.

Study Design and Context

Focus groups with parents will be conducted face-to-face, and focus groups with teachers will be web-based. The focus group method is a qualitative, group approach to data collection. It involves holding group sessions guided by a moderator, who follows a topical outline containing questions or topics focused on a particular issue. The questions are purposefully open-ended in nature, incorporating several prompts to encourage a deeper, multifaceted exploration of an issue or topic. The method is useful for going beneath the surface of a response. As a research tool, focus groups are useful for understanding a targeted group’s views on complex social topics such as issues surrounding cultural practices and education.

The ideal focus group size is 8-10 people. Having a group this size enables discussion and provides for a diversity of ideas to be heard. With more than 10 participants, it becomes less possible to ensure that all views are heard, that a discussion flows in a desirable manner, or that all of the questions are addressed in the time allotted. Having a group with less than 8 individuals limits the number of perspectives that are heard, and may also impede discussion. To ensure that appropriate and relevant information is obtained, it is anticipated that the focus groups will have 9 participants each, for a total of up to 40 respondents across the respondent types and grades, as shown in Table 1.

Table 1. Sample Size for Focus Groups

RESPONDENT GROUP	ELIGIBILITY CRITERIA	NUMBER OF PARTICIPANTS	TIMING
Parents/guardians of children in grades 3-12 in public schools	Low Economic Status, Racial/Ethnic Diversity	Recruit 10 for 9 to show	90 minutes
	Mid-to-High Economic Status, Racial/Ethnic Diversity	Recruit 10 for 9 to show	90 minutes
Teachers	Teachers in grades 6-8	Recruit 10 for 9 to show	90 minutes
Teachers	Teachers in grades 9-12	Recruit 10 for 9 to show	90 minutes

The initial contact will take place via phone. The recruiter will briefly describe the study; the types of information sought; and will ask the respondent to provide demographic information about themselves to ensure a diverse sample is selected per the aforementioned criteria (see Appendices A-B for Sample Recruitment Scripts). If unsuccessful in contacting them on the first call, the recruiter will identify the respondent as a non-respondent and make no further attempts at contact.

As participants are recruited, they will be scheduled in a focus group at a particular date and time. Once they are scheduled, the recruiter will send a confirmation e-mail and/or phone call (see Appendices C-D: Sample Follow-up Email/Call) that outlines the logistics of their focus group participation, and is accompanied by the Informed Consent Form (see Appendix C: Adult Informed Consent Forms).

The Confirmation Letter will ask participants to read and bring the consent form to the focus group. The individual consent form outlines the name of the project and its purpose, the criteria and procedure for participation, including ability to withdraw from participation at any point during the process without negative consequence. All participants will receive a copy of the signed consent form for their records. The purpose of the research and confidentiality statement will be reiterated during introductory remarks by the focus group moderator.

Recruitment of Parent Focus Groups

Focus groups with parents will be conducted face-to-face. Participant positions will be filled through the use of opt-in databases of local residents that are maintained by the local focus group facility engaged for this project—Baltimore Research in Baltimore, MD. This vendor is experienced in recruiting and hosting focus groups for federal studies. Recruiters call local residents at random from telephone listings to ask them to participate in the database. The criteria for inclusion in the databases are generally very broad. Participants are then screened more narrowly for specific studies, based on the objectives and screening criteria of each focus group session.

As is typical with focus group research, the sampling for the sessions is purposive in that recruiters are asked to find participants that meet specific criteria (e.g., parents/guardians with children in public schools). To identify participants, we will use a screening questionnaire, (see Volume II (2): Phone Script and Screening Questionnaire for Parent/Guardian Focus Group). Based on experience with similar groups, up to 600 people will be screened for each focus group category (parent/teacher) to achieve twenty participants per category. The screening questionnaires also include instructions for those selected to participate.

Recruitment of Teacher Focus Groups

C² Research (see section 5) will perform the teacher recruitment activity. Recruiters will ensure they meet the specific target audience requirements needed for the research and ensure diversity among participants. A separate moderators’ script/guide will be used in the focus group sessions to guide conversations (see Volume II (4)). Each focus group is designed to be a one-time information collection activity with participants specifically recruited to inform the assessment literacy efforts. Teachers will be sought for their respective focus group through C² Research’s national database. The initial contact will take place via phone, during which the recruiter will briefly describe the study; the types of information sought; and will ask participants to provide their demographic information to ensure a diverse sample is selected as per the aforementioned criteria (see Volume II (3): Phone Script and Screening Questionnaire for Teacher Focus Group). If the participant qualifies for the focus group, he or she will be invited to attend the WebEx discussion on the specified date (see Appendix B: Sample Follow-Up for Teachers). Once respondents agree to participate, C² Research will follow up via email or web form to provide a copy of the consent form and provide additional information about the WebEx. Consent forms will be obtained prior to the start of the interviews.

Data Collection Process

At the chosen time, the group will convene where the focus group moderator will introduce the process, collect consent forms, and conduct the focus group discussion. The focus groups will be moderated by an experienced Hager Sharp researcher trained in conducting focus groups. This individual has an appreciation and knowledge of how and when to use prompts as well as how to keep a discussion moving and on target. This person is also knowledgeable about how to clarify questions without directing respondents toward a particular response. Because the focus group touches on areas that could be considered sensitive, it is extremely important that the focus group be conducted with sensitivity such as not to present bias or opinion during the session. The moderator will use the appropriate focus group guide for each of the sessions (see Volume II).

For the online focus groups, the moderator will explain the technology and describe the tools the participants may use, such as muting their phone or asking questions. The moderator will use the focus group WebEx guide (see Volume II) for each of the sessions. Each focus group session will take approximately 90 minutes to complete.

Participants will be reassured that their participation is voluntary and that their responses will be used for research purposes only. As part of the introduction process, the moderator will explain to participants that their responses will be recorded. An audio and/or video recording will provide opportunity to playback the sessions to ensure accurate capture of verbatim quotes, and to aid in the analysis of each focus group by a senior researcher. The final report will provide conclusions and recommendations from the focus groups, and the presented findings will be supported with verbatim comments from respondents.

Consultations Outside of the Agency

Hager Sharp will moderate the focus groups, analyze the findings, and prepare preliminary and final reports. Hager Sharp has worked as a contractor for NCES since 2002, providing publication, outreach, and dissemination support for NAEP. The communications firm is familiar with the processes of NCES, IES, and the Governing Board.

Baltimore Research is a full-service marketing research firm located in Baltimore, MD that provides a broad portfolio of research services, including focus groups, consumer surveys, and data collection analytics to a wide variety of industries. Baltimore Research will work to identify and recruit parents for the parent focus groups.

C² Research, Inc. is a consumer market research firm with a strong focus and expertise in business-to-business and hard to reach target audiences. C²Research will work to identify and recruit teachers for the teacher focus groups.

Assurance of Confidentiality

Prior to the start of the focus groups, participants will be notified that their participation is voluntary and that their answers may be used only for research purposes and will not be disclosed or used, in identifiable form, for any other purpose except as required by the Education Sciences Reform Act of 2002 [(ESRA 2002) 20 U.S.C., § 9573].

Written consent will be obtained from all participants. During the informed consent process, participants will be asked to read and sign the consent form. After the participant signs the consent forms, copies will be made for the participants to have for their records. Only individuals who sign the consent form will be eligible to participate in the focus group.

Participants will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form. The consent forms, which include the participant name, will be separated from the participant interview files, secured for the duration of the study, and will be destroyed after the final report is completed.

The interviews will be audio and/or video recorded. The only identification included on the files will be the unique ID assigned to each participant by the interviewer. The video recordings will only be used for research purposes, and no identifying information will be included in the data analyses. The recorded files will be secured for the duration of the study and will be destroyed after the final report is submitted.

Justification for Sensitive Questions

Throughout the focus group development process, effort has been made to avoid asking for information that might be considered sensitive or offensive.

Estimate of Hourly Burden

This study includes four focus groups. The parent and teacher focus group participants’ total response burden is comprised of approximately 10-15 minutes for recruitment efforts and 1.5 hours for the focus group interviews.

Table 2. Estimated Burden Hours

Respondent Type	Number Of Respondents	Number Of Responses	Burden Hours Per Respondent	Total Hours
Parents/Guardians of Children in Grades 3-12 in Public Schools
Screening/Recruitment	600	600	.167	100
Follow-up email/call and consent form	*20	20	.083	2
Focus Group	*20	20	1.5	30
Total	600	640		132
Teachers
Screening/Recruitment	600	600	.167	100
Follow-up email and consent form	*20	20	.083	2
Focus Group	*20	20	1.5	30
Total	600	640		132
Total Respondents	1,200	1,280		264

Subset of the contacted group

Estimate of Costs for Recruiting and Paying Respondents

To encourage participation and thank the participants for their time and effort, a $75 gift card will be offered to each participant of the focus groups. The total estimated participant incentive cost for four focus groups is $3,000.

For this study, we are proposing the $75 incentive (consistent with past in-person focus group incentives) for both parents and teachers. We propose the same incentive for the teachers even though their focus groups will be conducted via web-ex. Based on our experience with the first phase of the Assessment Literacy focus group study in October, we were unable to recruit teachers for the approved lower incentive ($40). The recruitment contractors indicated that they regularly incentivize teachers $100-$125 and were unable to perform the desired recruitment for $40. Thus, the higher teacher incentive is being offered to achieve effective recruitment so that the goals of the study can be achieved.

Cost to Federal Government

The estimated cost to the government of all recruitment and data collection, analysis, and reporting activities for this study is $44,543.

Schedule

The following table depicts the schedule for efforts related to the implementation of the focus groups. The commencement of activities is contingent upon OMB approval.

Table 3. Project Schedule

Activity	Date
Recruit participants	May-June 2015
Conduct focus groups	June 2015
Complete transcription of focus groups	July 2015
Draft and submit a summary report on focus group studies	July-August 2015

Appendix A - Sample Follow-up Email/Call for Parents

EMAIL:

Hello <PARTICIPANT’S NAME>:

If you recall, you spoke with one of my colleagues from <RESEARCH COMPANY> on <DATE> and answered a few survey questions. We would like to remind you of the group discussion that you agreed to attend at <TIME> on <DATE>.

Please arrive at <ARRIVAL TIME>. The group discussion will last 1½ hours. You do not need to prepare anything prior to the discussion. As indicated when we invited you, you will be given a $75 VISA gift card as a token of our appreciation for your participation in the discussion. The discussion will be audio and video recorded and may be observed, but no names will be used in our report.

Please confirm that you will be attending the discussion by contacting <CONTACT NAME> at <CONTACT PHONE NUMBER> or by email at <CONTACT EMAIL>.

We have <ATTACHED/INCLUDED> a consent form <TO/WITH> this confirmation. Please sign, date, and return the form when you come to the focus group session. Without your written consent, you will not be able to participate in this research study.

If you have any questions before the group discussion, please contact <CONTACT NAME> at <CONTACT PHONE NUMBER> or by email at <CONTACT EMAIL>.

Thank you very much for agreeing to participate, and we look forward to seeing you on <DATE>.

Sincerely,

<NAME>

BY PHONE:

Hello, my name is <INTERVIEWER’S NAME> from <RESEARCH COMPANY>. If you recall, you spoke with one of my colleagues on <DATE> and answered a few survey questions. We would like to remind you of the group discussion that you agreed to attend at <TIME> on <DATE>.

Are you still planning to attend on <DATE>?

Please arrive at <ARRIVAL TIME>. The group discussion will last 1½ hours. You do not need to prepare anything prior to the discussion. Prior to the start of the focus groups, all participants will be asked to read and sign the consent form and send it back to [Focus Group Facility Mailing Address].

As indicated when we invited you, you will be given a $75 VISA gift card as a token of our appreciation for your participation in the discussion. The discussion will be audio and video recorded and may be observed, but no names will be used in our report.

If you have any questions before the group discussion, please contact <CONTACT NAME> at <CONTACT PHONE NUMBER> or by email at <CONTACT EMAIL>.

Thank you very much for agreeing to participate, and we look forward to seeing you on <DATE>.

Appendix B - Sample Follow-up Email/Call for Teachers

<DATE>

Dear <NAME>,

If you recall, you spoke with one of my colleagues from <RESEARCH COMPANY> on <DATE> and answered a few survey questions. You have been scheduled to participate in a focus group study at <TIME> on <DATE>. The focus group will last no more than 90 minutes. To thank you for participating, you will be mailed a $75 VISA gift card after participating. The discussion will be audio recorded and may be observed, but no names will be used in our report.

All the information you provide will be used only for research purposes and will not be disclosed or used, in identifiable form, for any other purpose except as required by the Education Sciences Reform Act of 2002 [ESRA 2002; 20 U.S.C., § 9573)].

To participate in the focus group, click on the following link: <LINK>

We have <ATTACHED/INCLUDED> a consent form <TO/WITH> this confirmation. Please sign, date, and return the form to [STAFF NAME] via email, [STAFF EMAIL], before the beginning of the focus group. Without your written consent, you will not be able to participate in the focus group.

If you have any questions before the group discussion, please contact <CONTACT NAME> at <CONTACT PHONE NUMBER> or by email at <CONTACT EMAIL>.

Sincerely,

<NAME>

BY PHONE:

Hello, my name is <INTERVIEWER’S NAME> from <RESEARCH COMPANY>. If you recall, you spoke with one of my colleagues on <DATE> and answered a few survey questions. We would like to remind you of the WebEx discussion that you agree to attend at <TIME> on <DATE>.

Are you still planning to attend on <DATE>?

Please sign-in at <ARRIVAL TIME>. The group discussion will last 1½ hours. You do not need to prepare anything prior to the discussion. Prior to the start of the focus groups, all participants will be asked to read and sign the consent form and email it back to [Focus Group Facility EMAIL Address] if you have not already done so.

As indicated when we invited you, you will be given a $75 VISA gift card as a token of our appreciation for your participation in the discussion. The discussion will be audio recorded, but no names will be used in our report.

If you have any questions before the group discussion, please contact <CONTACT NAME> at <CONTACT PHONE NUMBER> or by email at <CONTACT EMAIL>.

Thank you very much for agreeing to participate, and we look forward to your active participation on <DATE>.

Appendix C - Adult Informed Consent Form

STUDY TITLE: Key Concepts for Assessment Literacy Focus Group Studies

WHAT IS THE STUDY ABOUT?

The National Assessment of Educational Progress (NAEP) is the largest nationally representative assessment of what America's students know and can do in various subject areas.

This study is being carried out by Hager Sharp on behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education. The focus groups will be used to obtain feedback from parents and teachers to help inform efforts to educate audiences about assessment literacy. The results of this research will be used to develop communication materials to encourage assessment literacy and engagement.

WHAT IS INVOLVED IN THE STUDY?

Participation in this study is voluntary. If you agree to take part in this study, you will participate in a focus group with other [parents/guardians of children or teachers] representing grades 3 through 12. Each focus group discussion will have the same data collection structure. The focus group will be led by moderator, <NAME>.

The focus group will be held <in-person/via WebEx> and will be <audio/video> recorded in order to accurately capture what is said. You may choose how much or how little you want to speak during the group. You may also choose to leave the focus group at any time.

The focus group process will take approximately 90 minutes.

If you participate in the study, you will receive a $75 VISA gift card, to thank you for your time and input.

CONFIDENTIALITY

Recordings of the focus groups will be transcribed and kept in a password-protected file. Any printed copies will be kept in a secured and locked office. Only project staff will serve as note takers and observers, and will be able to view the recording or read the transcription.

QUESTIONS AND SUBMISSION OF THIS FORM

If you have any questions about this study, please contact Task Lead, [STAFF NAME], ([PHONE NUMBER] or [STAFF EMAIL]). Upon signing the form, please return it to [PROJECT LEAD NAME] at [PROJECT LEAD MAILING ADDRESS].

ADULT INFORMED CONSENT

I have read the consent form and all of my questions about the study have been answered. I have been provided with a copy of the Adult Informed Consent document. I understand that the focus group will be recorded. I agree to participate in this study.

Print Name: _____________________________________________

Signature: ______________________________________________

Date: _______________________

Appendix D - Print Mockup of Parent Discussion Guide

Appendix E - Storyboard of Assessment Literacy 101 Video

Appendix F - Design Mockup of Assessment Literacy Microsite

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS
Author	jeanette.hassin
File Modified	0000-00-00
File Created	2021-01-27