Justification

National Center for Education Statistics

National Assessment of Educational Progress

Volume I

Supporting Statement

National Assessment of Educational Progress (NAEP)

2015 National Indian Education Study (NIES) Report Focus Group

OMB# 1880-0542

April 2019

Table of Contents

1) Submittal-Related Information 3

2) Background and Study Rationale 3

3) Recruitment and Data Collection 4

4) Consultations Outside the Agency 5

5) Justification for Sensitive Questions 5

6) Paying Respondents 5

7) Assurance of Confidentiality 5

8) Estimate of Hourly Burden 6

9) Costs to Federal Government 6

10) Schedule 6

1. Submittal-Related Information

This material is being submitted under the generic U.S. Department of Education clearance agreement (OMB# 1880-0542), which provides for the Department to conduct surveys and other studies regarding customer satisfaction. This request is to conduct a focus group to obtain Indian education stakeholders’ feedback on the National Indian Education Study (NIES) 2015 Report (National Center For Education Statistics 2017-161).

2. Background and Study Rationale

NIES is sponsored by the Office of Indian Education (OIE) and conducted in conjunction with the National Assessment of Educational Progress (NAEP). The study is authorized under Executive Order 13592, Improving American Indian and Alaska Native (AI/AN) Educational Opportunities and Strengthening Tribal Colleges and Universities, which was issued in 2011. Executive Order 13592 is the most recent authorization of Executive Order 13336, AI/AN Education, issued in 2004. To date, NIES has been conducted six times, in 2005, 2007, 2009, 2011, 2015, and 2019.

NAEP is a federally authorized survey by the National Assessment of Educational Progress Authorization Act (20 U.S.C. §9622) of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, civics, geography, economics, and the arts. NAEP is conducted by the National Center for Education Statistics (NCES), which is part of the Institute of Education Sciences, within the U.S. Department of Education. NAEP’s primary purpose is to assess student achievement in the different subject areas and collect survey questionnaire (i.e., non-cognitive) data from students, teachers, and principals to provide context for the reporting and interpretation of assessment results. Since 1969, NAEP has been a major part of the country’s fabric of reporting on what students know and can do and, since 1969, there has been a growing need for innovative reporting geared toward teachers and schools.

NIES is administered as part of NAEP to allow more in-depth reporting on the achievement and experiences of AI/AN students in grades 4 and 8. NCES has produced NIES report(s) for each administration of the study. As part of the continuing effort to enhance the NIES reports, NCES conducted a focus group study on the NIES 2009 reports. The purpose of that focus group study was to evaluate NCES’s recommended enhancements for future NIES reports. To ensure that appropriate and relevant feedback was obtained from meaningful sources, participation was sought from attendees to the 2010 National Indian Education Association annual convention. The purpose of the upcoming focus group study is to review the NIES 2015: AI/AN Students at Grades 4 and 8 (NCES 2017-161) report and ensure that appropriate and relevant feedback is obtained from meaningful sources on the report regarding data gathered in 2015 and published in 2017.

This focus group will obtain relevant feedback from Indian education stakeholders, which may include school administrators, teachers, school board members, and other education stakeholders from high-density public schools (25 percent or more of all the students in the school are AI/AN) and Bureau of Indian Education (BIE) schools. Previous focus groups were designed to gather the knowledge, attitudes, and beliefs of the participants concerning how the data might be used, the utility of the presentation of the data (e.g., accessibility and purpose), how the report portrays AI/AN academic achievement, layout and data presentation, clarity of the information being reported, and the effect of the content organization (tables, graphs, and color.). In this manner, the focus groups were expected to provide important feedback on the content, presentation, and usability of the reports for end-users, including perceptions of specific items (graphs, examples, statistics, etc.), mode of presentation, and knowledge conveyed. The current focus group will have the same design.

The focus group session will have a minimum of 8 and a maximum of 12 participants to provide a dialogue. There will be a moderator and a stenographer in order to allow the discussion to flow. The stenographer will take notes as well as make an audio recording of the session. Outreach efforts will be made to recruit participants as well as alternates to account for possible no-shows and cancellations. To assure a diverse sample of participants, we will utilize a screener [Appendix E – mailed when contacting via mail and/or letter, G – read by interviewer during a follow-up call or when calling potential participants from a compiled list] to identify position or role in education (teacher, school administrator, tribal leader, parent), years in that role, race/ethnicity, sex, and school type (BIE or high-density public schools).

3. Recruitment and Data Collection

Recruitment

Under a contract to the U.S. Department of Education, Institute of Education Services, Tribal Tech, LLC developed screening questions for NAEP and will be responsible for the overall management of the focus group. Tribal Tech, LLC will be responsible for focus group recruitment and administration. Participants will be recruited from pre-registered individuals planning to attend the Tenth Annual American Indian/Indigenous Teacher Education Conference on June 21-22, 2019, in Flagstaff, Arizona, sponsored by Northern Arizona University. We will pursue a participant recruitment strategy that integrates multiple contact methods. These methods include digital and written outreach, with possible face-to-face recruitment, if necessary.

All initial contact to teachers, school administrators, and other educational stakeholders will be generated from the conference registration. An email will be sent to all registrants [Appendix A]. In the week following the initial email contact, Tribal Tech, LLC will make a second contact effort via phone [Appendix G] to those participants who have responded to the initial email. Based on experience, outreach efforts are more effective and response rates improve with initial written contact followed by secondary phone contact as the person contacted already has some preliminary information about the research effort and the reason for the call. Tribal Tech, LLC will contact individuals once by email and up to three times by phone before terminating contact. Follow-up contact via email and/or letter will be made to administer the screener [Appendix E] and with potential respondents who may be interested in assisting with outreach or in participating, but who have requested more information about Tribal Tech, LLC or about the study. The follow-up to information requests about the study or Tribal Tech, LLC will be accompanied by the Frequently Asked Questions document [Appendix F].

After a respondent agrees to participate, he/she will be sent a letter confirming their agreement that provides him/her with details about the report to view, time and venue of the focus group, and contact information [Appendix B]. Each intended participant will also receive a welcome letter with a copy of the report when they register for the conference [Appendix C]. They will be asked to spend up to 60 minutes reviewing the report prior to the focus group. The bulk of recruiting will be through email and telephone. However, if a sufficient number of participants have not been recruited by the time of the convention, postal mail and on-site recruitment will take place. The former contactor found that on-site recruitment was necessary and successful. Should too many people indicate interest in participating, some will receive an email notification that they were not selected to participate in study [Appendix K].

The recruitment procedure will ensure that:

• All recruiting staff receive project-specific training before the start of any recruitment effort, including a discussion regarding the overall recruitment objectives, specific instructions on the critical importance of complete adherence to all OMB guidelines, protocols and restrictions, a section-by-section review of the screening instrument, and mock-interviews that cover a variety of screening scenarios.

• During intake, participants will be informed of the objectives, purpose, and participation requirements of the data collection effort, as well as the activities that it entails, as stated within the Interview Protocol [Volume II]. The consent form [Appendix D] clearly identifies the objectives of the research, including several examples and descriptions of the items being asked, indicates that audio of the session will be recorded by the stenographer for analysis purposes only, whether any other individuals will be listening to the recording, and describe the reasons for this.

• Along with identifying, contacting, and screening qualified research participants, ensuring their participation is equally as important. After screening participants, their contact information including telephone numbers, email, and postal contact information will be collected. After confirming that a participant is qualified, willing, and available to participate in the research project, they will receive a confirmation via email that includes the date, time, and location of the focus group, along with a link to an interactive map, and any other information that may be required [Appendix H]. To supplement the email confirmation, participants will also receive confirmation via postal mail.

• Participants will receive a telephone reminder at least 24 hours prior to the focus group to confirm participation and to field any questions they may have regarding their participation [Appendix I].

• Participants will receive a thank you email following their focus group session [Appendix J].

The focus group will be conducted directly before the conference welcome reception at Northern Arizona University, Flagstaff, AZ, in cooperation with the Tenth Annual American Indian/Indigenous Teacher Education Conference.

Data Collection

One 90-minute focus group session with 8-12 participants will be convened to evaluate the NIES 2015 report and to recommend enhancements for future NIES reports. Participants will be asked to review the NIES report prior to the focus group, which should take no longer than 60 minutes. Specifically, the purpose is to gather the knowledge, attitudes, and beliefs of the participants concerning how the data might be used, the utility of the presentation of the data (e.g., accessibility and purpose), how the report portrays AI/AN academic achievement, layout and data presentation, clarity of the information being reported, and the effect of the content organization (tables, graphs, and color).

After the focus group session, findings from the notes taken and from the audio recordings will be compiled and included in a comprehensive report that will describe the methodology used and the limitations and implications of the study’s results.

4. Consultations Outside the Agency

Tribal Tech, LLC will be responsible for management of all activities described in this package. Tribal Tech, LLC will offer facilities, tools, and staff (senior qualitative methodologists) to collect and analyze qualitative data, will recruit participants, moderate the focus group, and prepare a comprehensive report.

5. Justification for Sensitive Questions

Throughout the item and debriefing question development processes, effort will be made to avoid asking for information that might be considered sensitive or offensive.

6. Paying Respondents

As in prior similar NAEP studies, to encourage participation in the 90-minute session by busy school administrators, teachers, school board members, and other education stakeholders, we will offer each participant a $100 gift card from a major credit card company (e.g., Visa) to thank them for their time and effort.

7. Assurance of Confidentiality

The study will not retain any personally identifiable information. Prior to the start of the study, participants will be notified that their participation is voluntary and that the information they provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). Written consent will be obtained from focus group participants before their participation.

8. Estimate of Hourly Burden

The estimated burden for recruitment assumes attrition throughout the process, including an estimated 75 percent attrition rate during direct participant recruitment from initial contact to follow-up, and a 50 percent rate from follow-up to confirmation. The focus group will be scheduled for no more than 90 minutes. Table 1 details the estimated burden for the focus group.

Table 1. Response Burden for Teacher and Principal Focus Group

Respondent	Number of respondents	Number of responses	Hours per respondent	Total hours (rounded up)
Teacher, School Administrator and Stakeholder Recruitment
Initial contact	150	150	0.05	8
Follow-up contact (screener script)	40*	36	0.15	6
Confirmation and Consent form completion and return	12*	12	0.15	2
Preliminary review of report prior to focus group	12*	12	1.0**	12
Sub-Total	150	210	-	28
Participation (Focus Group)
Teachers	5*	5	1.5	8
School Administrators	5*	5	1.5	8
Other stakeholders	2*	2	1.5	3
Sub-Total	0*	12	-	19
Total Burden	150	222		47

* Subset of initial contact group not double counted in the total number of respondents.

** Respondents will be asked to review report prior to the focus group – anticipated time for the task is up to 60 minutes.

Note: Numbers have been rounded and therefore may affect totals.

9. Costs to Federal Government

The estimated cost for the focus group described in this submittal is $13,659 as delineated in Table 2.

Table 2. Estimate of Costs

Activity	Provider	Estimated Cost
Focus group design, preparation, administration (including recruitment, incentive costs, data collection, and documentation), data analysis, and comprehensive report	Tribal Tech, LLC	$13,659
Total		$13,659

10. Schedule

Table 3 depicts the start and end dates of the focus group phases.

Table 3. High-Level Schedule of Milestones

Activity	Dates
Recruitment and administration of focus group	May 2019 - June 2019
Study summary and recommendations report	June 2019 - July 2019
Finalize and present findings to the NIES technical review panel	July 2019 - August 2019