Justification

National Center for Education Statistics

National Assessment of Educational Progress

Volume I

Supporting Statement

National Assessment of Educational Progress (NAEP)

NAEPQ Giving Back Website 2018 Focus Groups

OMB# 1880-0542

October 2018

Table of Contents

1) Submittal-Related Information 3

2) Background and Study Rationale 3

3) Recruitment and Data Collection 3

4) Consultations Outside the Agency 5

5) Justification for Sensitive Questions 5

6) Paying Respondents 5

7) Assurance of Confidentiality 5

8) Estimate of Hourly Burden 5

9) Costs to Federal Government 6

10) Schedule 6

1. Submittal-Related Information

This material is being submitted under the generic U.S. Department of Education clearance agreement (OMB# 1880-0542), which provides for the Department to conduct surveys and other studies regarding customer satisfaction. This request is to conduct focus groups to obtain teachers’ and school principals’ feedback on the NAEPQ Giving Back Website.

2. Background and Study Rationale

The National Assessment of Educational Progress (NAEP) is a federally authorized survey by the National Assessment of Educational Progress Authorization Act (20 U.S.C. §9622) of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, civics, geography, economics, and the arts. NAEP is conducted by the National Center for Education Statistics (NCES), which is part of the Institute of Education Sciences, within the U.S. Department of Education. NAEP’s primary purpose is to assess student achievement in the different subject areas and collect survey questionnaire (i.e., non-cognitive) data from students, teachers, and principals to provide context for the reporting and interpretation of assessment results. Since 1969, the National Assessment of Educational Progress (NAEP) has been a major part of the country’s fabric of reporting on what students know and can do and, since 1969, there has been a growing need for innovative reporting geared toward teachers and schools. To better meet the needs of these constituencies, NAEP developed the NAEPQ Giving Back Tool as a way to “give back” useful information and activities to engage teachers and schools in NAEP results.

In 2015, NAEP launched an interactive website that provides interested teachers and schools an opportunity to access preliminary NAEP teacher and school survey questionnaire results as a way to “give back” to them for participating in NAEP. Upgrades to the site were made in 2017, and additional minor changes were made in 2018. However, the number of teachers and schools that request access, and then go to the website to create reports remains low. The goal of this study is to employ focus groups to gain insight about why the usage remains low and to learn from teachers and school principals about the type of NAEP data that would be of greatest value to them. The results of the focus groups will be used to determine useful information and activities to engage teachers and schools in NAEP results and to determine the value of the current Giving Back website.

A total of 18 teachers and school principals will be recruited to participate in one of three focus groups. The focus groups have been designed to explore what kind of data and activities teachers and schools would find beneficial and useful, and to evaluate how teachers and principals view the current Giving Back website. Each session will have a minimum of three participants to provide a dialogue rather than a conversation. There will be a single moderator and a dedicated note-taker in order to allow the discussion to flow. Outreach efforts will be made to recruit approximately eight participants per focus group to account for possible no-shows and cancellations. To assure a diverse sample of participants, we will utilize a screener [Appendix F & G] to identify grades taught, years of teaching, race and ethnicity, gender, and school location.

3. Recruitment and Data Collection

Recruitment

Under a contract to NCES, the Educational Testing Service (ETS) develops survey questionnaire for NAEP and will be responsible for the overall management of the focus group. EurekaFacts, a subcontractor to ETS, will be responsible for focus group recruitment and administration (see Section 4, Consultations outside the Agency).

Participants will be recruited from the Washington, DC, Maryland, and Virginia areas. We will pursue a participant recruitment strategy that integrates multiple contact methods and resources. These methods include wide-scale digital and print advertising, along with outreach to educational organizations (e.g., unions, associations, universities, learning centers, etc.), and mass media recruiting (e.g. postings on Craigslist, etc.). EurekaFacts has an existing large database of diverse contacts, which will assist in the effort to recruit groups with mixed demographics.

All initial contact to teachers and principals will be generated from an existing database of individuals who are interested in taking part in research. EurekaFacts will contact potential participants via postal mail and follow-up with them via e-mail [Appendix A, B]. In addition, telephone outreach will be performed using targeted contact lists purchased from a reputable third-party vendor using the recruitment phone script and screener [Appendix G]. Furthermore, educational organizations/associations will be contacted via e-mail to communicate details about the study [Appendix B, D]. For both organizations and educators, about two days after the initial e-mail contact, EurekaFacts will make a second contact effort via phone [Appendix G]. Based on their experience, outreach efforts are more effective and response rates improve with initial written contact followed by secondary phone contact as the person of contact already has some preliminary information about the research effort and the reason for the call. EurekaFacts will contact organization personnel and individuals once by e-mail and up to three times by phone before terminating contact to avoid any potential disturbance. Follow-up contact via email and/or letter will be made to administer the screener [Appendix F & G] and with potential respondents who may be interested in assisting with outreach or in participating, but have requested more information about EurekaFacts, or about the study in question. The follow-up to information requests will be accompanied by the Frequently Asked Questions document [Appendix E].

For those potential respondents who are recruited by digital means (e.g., list-serves, email blasts, social media campaigns, etc.) and those who prefer the online medium, an electronic self-screening intake form will be provided [Appendix F]. However, we will complete the bulk of teacher and principal recruiting through postal mail, email, and telephone.

The recruitment procedure will ensure that:

• All recruiting staff receive project-specific training before the start of any recruitment effort, including a discussion regarding the overall recruitment objectives, specific instructions on the critical importance of complete adherence to all OMB guidelines, protocols and restrictions, a section-by-section review of the screening instrument, and mock-interviews that cover a variety of screening scenarios.

• During screening, participants will be informed of the objectives, purpose, and participation requirements of the data collection effort, as well as the activities that it entails, as stated within the Interview Protocol [Volume II]. The consent form [Appendix H] clearly identifies the objectives of the research, including several examples and descriptions of the items being asked, indicate that audio of the session will be recorded for analysis purposes only, whether any other individuals will be observing the interviews, and describe the reasons for such observations.

• Participants will be screened using the screener script [Appendix F & G] to be programmed into a CATI-like software (e.g., Vovici) to ensure that the screening procedure is uniformly conducted and instantly quantifiable throughout the recruitment effort. Its use will also help to ensure that qualified and screened participants that fit the needed criteria are then scheduled for a focus group.

• Along with identifying, contacting, and screening qualified research participants, ensuring their participation is equally as important. After screening participants, their contact information including telephone numbers and email and postal contact information will be collected, depending on the screening medium. After confirming that a participant is qualified, willing, and available to participate in the research project, they will receive a confirmation via email that includes the date, time, and location of the focus group, along with a map, directions, and any other information that may be required [Appendix I]. To supplement the email confirmation, participants will also receive confirmation via postal mail [Appendix I].

• Participants will receive a telephone reminder at least 24 hours prior to their appointment to confirm participation and to field any questions they may have regarding their participation [Appendix J].

• Participants will receive a thank you email following their focus group session [Appendix K].

The focus groups will most likely be conducted at EurekaFacts facilities in Rockville, MD, which are located near mass transit, but alternative locations will be offered if needed to secure desired participation, given that traveling outside one’s home/school area can be a limitation for some teachers and principals. If an off-site facility will be used, EurekaFacts will ensure that an appropriate, safe, and easily accessible facility is secured and offered to participants as a way to reduce their burden and increase the likelihood of expressed interest and participation.

Data Collection

Three, 90-minute focus group sessions (with a minimum of three and a maximum of six participants) will be convened to determine teachers’ and principals’ engagement with data in general, their familiarity with NAEP data, and value of the current NAEPQ Giving Back Website. The focus group moderators will use the interview protocol provided in Volume II. During the sessions, teachers and principals will have the opportunity to engage with the website and with screenshots of the 2015, 2017, 2018, and 2019 NAEPQ opt-in questions from the teacher and school questionnaires¹ [Volume II, Figure 1].

After the three focus group sessions, findings from the notes taken and from recordings will be compiled and included in a comprehensive report that will describe the methodology used and the implications and limitations of the study’s results.

4. Consultations Outside the Agency

Educational Testing Service (ETS) will be responsible for management of all activities described in this package.

EurekaFacts will offer facilities, tools, and staff (senior qualitative methodologists) to collect and analyze qualitative data, will recruit participants, moderate the focus groups, and prepare a comprehensive report.

5. Justification for Sensitive Questions

Throughout the item and debriefing question development processes, effort will be made to avoid asking for information that might be considered sensitive or offensive.

6. Paying Respondents

To encourage participation in the 90 minutes session by busy teachers and principals, we will offer each participant a $100 gift card from a major credit card company (e.g., Visa) to thank them for their time and effort.

7. Assurance of Confidentiality

The study will not retain any personally identifiable information. Prior to the start of the study, teachers and principals will be notified that their participation is voluntary and that the information they provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). Written consent will be obtained from teachers and principals before their participation.

8. Estimate of Hourly Burden

The estimated burden for recruitment assumes attrition throughout the process, including an estimated 75 percent attrition rate during direct participant recruitment from initial contact to follow-up, and a 50 percent rate from follow-up to confirmation. All focus groups will be scheduled for no more than 90 minutes. Table 1 details the estimated burden for the focus groups.

Table 1. Response Burden for Teacher and Principal Focus Groups

Respondent	Number of respondents	Number of responses	Hours per respondent	Total hours (rounded up)
Teacher and Principal Recruitment
Initial contact	120	120	0.05	6
Follow-up contact	36*	36	0.15	6
Confirmation and Consent form completion and return	18*	18	0.15	3
Sub-Total	120	174	-	15
Participation (Focus Groups)
Teachers	9*	9	1.5	14
Principals	9*	9	1.5	14
Sub-Total	18*	18	-	28
Total Burden	120	192		43

* Subset of initial contact group, not double counted in the total number of respondents.

Note: Numbers have been rounded and therefore may affect totals

9. Costs to Federal Government

The estimated cost for the focus group described in this submittal is $89,326 as delineated in Table 2.

Table 2. Estimate of Costs

Activity	Provider	Estimated Cost
Focus group development, project oversight, and review of all study materials and data	ETS	$52,644
Focus group design, preparation, administration (including recruitment, incentive costs, data collection, and documentation), data analysis, and comprehensive report	EurekaFacts	$36,682
Total		$89,326

10. Schedule

Table 3 depicts the start and end dates of the focus group phases.

Table 3. High-Level Schedule of Milestones

Activity	Dates
Recruitment and administration of focus groups	October 2018 -December 2018
Study summary and recommendations report	December 2018 - January 2019

1 NCES decided not to provide access to the Giving Back Website in 2016 because the national sample was too small to provide meaningful disaggregated data.