PPSS_TO24_OMBClearance_PartB_062916

JANUARY 2015

Task Order 24: Study of the Title III Native American and Alaska Native Children in School (NAM) Program

Office of Management and Budget Clearance Request

Part B

Prepared for

U.S. Department of Education

Office of Planning, Evaluation and Policy Development

Policy and Program Studies Service

By

American Institutes for Research

June 29, 2016

Contents

Page

Introduction 1

Supporting Statement for Paperwork Reduction Act Submission 2

Justification (Part A) 2

A1. Circumstances Making Collection of Information Necessary 2

A2. Use of Information 6

A3. Use of Improved Technology to Reduce Burden 6

A4. Efforts to Avoid Duplication of Effort 7

A5. Efforts to Minimize Burden on Small Businesses and Other Small Entities 7

A6. Consequences of Not Collecting the Data 7

A7. Special Circumstances Causing Particular Anomalies in Data Collection 7

A8. Federal Register Announcement and Consultation 8

A9. Payment or Gift to Respondents 9

A10. Assurance of Confidentiality 9

A11. Sensitive Questions 10

A12. Estimated Response Burden 10

A13. Estimate of Annualized Cost for Data Collection Activities 12

A14. Estimate of Annualized Cost to Federal Government 12

A15. Reasons for Changes in Estimated Burden 12

A16. Plans for Tabulation and Publication 12

A17. Display of Expiration Date for OMB Approval 15

A18. Exceptions to Certification for Paperwork Reduction Act Submissions 15

Description of Statistical Methods (Part B) 16

B1. Sampling Design 16

B2. Procedures for Data Collection 16

B3. Methods to Maximize Response Rate 19

B4. Expert Review and Piloting Procedures 20

B5. Individuals and Organizations Involved in the Project 20

References 22

Appendix A. Interview and Focus Group Protocols 26

Appendix B. Notification Letters 84

Appendix C. Study Information Document 89

Appendix D. Institutional Review Board (IRB) Letter of Approval 92

Supporting Statement for Paperwork Reduction Act Submission

Description of Statistical Methods (Part B)

B1. Sampling Design

The overall purpose of the case studies is to provide detailed information regarding how current grantees implement NAM activities. The study team will collect data from all 22 current grantees (i.e., FY 2011 and FY 2013 grantees). In order to recruit all 22 sites, the study team has developed recruitment materials, including notification letters to the grant coordinators and tribal leaders from the 24 tribes in which there are current NAM grantees (Appendix B), and an overview of the study that describes the purpose of the study and activities requested of participants (Appendix C). The study overview will be included in the notification letters and also was distributed in January 2016 during a Department monthly consultation with tribes.

The Department will send the letters and study overview to grant coordinators and tribal leaders, and study team staff will follow up by phone. The study team will complete tribal research requirements, as needed, before collecting data.

B2. Procedures for Data Collection

Descriptions of the data collection procedures of each of the main components of the study (telephone interviews and site visits) follow.

Semi-Structured Telephone Interviews

The study team will conduct semi-structured telephone interviews with 22 grant coordinators. The purpose of these interviews is to gather contextual information about the grant sites before initiating site visits, as well as to collect data about the overall implementation of the NAM program at the site, school, district, tribal community, and/or community college levels. Interview protocols are included in Appendix A.

Site Visits

At each site, the study team will interview or conduct focus groups with the following NAM staff and stakeholders, as applicable:

• Tribal education director(s) in tribes in which there are current NAM activities

• District superintendent(s) in districts in which there are current NAM activities

• School principal(s) in schools in which there are current NAM activities

• Teachers or program instructional staff

• Tribal college instructors

• Parent or family coordinator(s) at the NAM sites in which there are family outreach activities

• Professional development provider(s)

• Language instruction or curriculum development specialist(s)

• Program administrator or evaluator(s)

Interviews or focus groups will be determined by the schedules and roles of the selected interviewees. Timing will be determined, for example, in consultation with each school’s principal, based on the teachers’ schedules.

The study team has developed a separate set of interview and focus group protocols for each respondent type (included in Appendix A). Each protocol includes questions about key constructs of interest, including goals and characteristics of the program; the resources and training that facilitate implementation of the program; challenges and lessons learned; and stakeholder perceived program outcomes. The study team has designed interview and focus group protocols to be semi-structured, guiding respondents to comment on questions of interest but also maintaining a conversational tone to capture breadth of information. To reduce the burden on respondents and to invite rich discussion and descriptions, the protocols have broad, open-ended questions. However, in order to capture all relevant information, a column on the right side of the protocol reminds the interviewer which information or details are critical and what information to probe for. Text at the beginning of each protocol also reminds the interviewer about the most critical pieces of information to learn from each participant.

All interviews will be audio-recorded and later transcribed to facilitate coding. A study team member will also take detailed notes during each interview or focus group.

IRB Approval

All data collection protocols were submitted to the study team organization’s IRB for approval. The IRB, which is registered with the Office of Human Research Protections and operates under a federalwide assurance comparable to many universities, is responsible for reviewing all research conducted by the study team’s organization and its subcontractors. The IRB ensures that projects involving human subjects comply with professional standards and government regulations designed to safeguard participants and that research team members, including subcontractor staff, are adequately trained. The criteria that a study must meet for IRB’s approval include assurances that risks to participants are minimized and balanced by benefits, subject selection is equitable, participants are informed about risks and give uncoerced consent, privacy of the subjects is adequately protected, and the rights and welfare of populations that might be vulnerable to coercion or undue influence are protected. The IRB’s letter of approval is included in Appendix D.

Conducting Site Visits

Data collection will take place over approximately a nine-month period from September 2016 to May 2017. Before data collection, the study leadership will train site visitors and carefully plan with each site the schedule and logistics of the visit.

Training of Site Visitors. Before site visits are conducted, all members of the case study team will participate in a one-day in-person training to ensure that they thoroughly understand the content of the protocols and site visit procedures, including culturally responsive data collection procedures in NA/AN communities. Study team members will review best practices for interviews and focus groups, including strategies for: (1) avoiding leading questions, (2) ensuring consistency, and (3) conducting interviews and focus groups in a way that is conversational yet still directed toward collecting the intended information systematically. Study team members will practice using the interview and focus group protocols, with careful monitoring from study leadership to ensure that they appropriately probe for information most relevant to the study. Important procedural issues to be addressed are the guidelines for ensuring respondent privacy, including use and protection of identification numbers.

Site Visit Logistics. Members of the case study team will conduct a two- to three-day visit, in pairs, to each of the 22 case study sites. Site visits will include: (1) any culturally appropriate or tribally mandated research kickoff activity, such as a meeting with tribal leaders, (2) and as applicable, on-site interviews of tribal education directors, district superintendents, principals, parent or family coordinators, professional development providers, teachers, instructors at tribal colleges (as applicable), language instruction and curriculum development specialists, and program administrators or evaluators. The length of each visit will be partially determined by how many separate locations within each grant site the study team must visit and the distance between locations. In some very rural areas, for example, the study team members may need a day to travel to different schools within a district.

A researcher and a junior staff member will visit each site. The junior staff member’s responsibilities will include assistance with logistics and note taking to ensure that each site visit is as productive as possible. All interviews and focus groups will be audio-recorded for later transcription. Site visit schedules will be developed in concert with the appropriate staff at the selected sites. The same pair of site visitors will be responsible for scheduling and conducting visits, thus developing a rapport with staff at each site. Exhibit 6 shows a sample site visit schedule.

Exhibit 6. Sample Site Visit Schedule

Day/Time	Activity
Before site visit, by phone	Conduct interview with: grant coordinator
	Day 1
8:00–10:00 a.m.	Attend kickoff meeting with tribal stakeholders at tribal office
10:00–11:00 a.m.	Conduct interview with tribal education director
11:00 a.m.–12:00 p.m.	Conduct interview with language instruction or curriculum development specialist
12:00–1:30 p.m.	Take lunch break and travel to district office
1:30–2:15 p.m.	Conduct interview with district superintendent
2:30–3:30 p.m.	Conduct focus group with principals
	Day 2
9:00–10:00 a.m.	Conduct interview with parent/family coordinator
10:15–11:15 a.m.	Conduct focus group with professional development providers
11:30 a.m.–12:30 p.m.	Conduct focus group with available program instructional staff
12:30–1:30 p.m.	Take lunch break
1:30–4:00 p.m.	Conduct focus groups with additional program instructional staff (as necessary)
	Day 3
9:00–10:00 a.m.	Conduct interview with administrators or other staff
10:15 a.m.–12:30 p.m.	Conduct additional focus group with professional development provider (as necessary)
12:30–1:30 p.m.	Take lunch break
1:00–3:00 p.m.	Conduct additional interviews if needed
After site visit, by phone	Conduct interview with program evaluator

B3. Methods to Maximize Response Rate

The study team has developed interview and focus group protocols that are appropriately tailored to the respondent group and are designed to place as little burden on respondents as possible. The team also will pilot core data collection instruments with similar respondents in a former NAM grantee site to ensure that the instruments are user friendly and easily understandable, all of which increases respondents’ willingness to participate in the data collection activities and thus increases response rates.

In addition to careful instrument design, a 100 percent response rate among the case study sites may be ensured through careful recruitment materials. These recruitment materials emphasize the social incentive to respondents by stressing the importance of the data collection to help the Department better support grantees. These recruitment materials also emphasize the larger social importance of such a study that is responsive to a long-standing need expressed by leaders in NA/AN communities for research about language instruction for NA/AN students.

The study team’s experience in past evaluations has demonstrated the importance and value of building a consensus of support with participating tribes. Site visit activities include time for an initial meeting with tribal members and leaders to establish communication about the site visits, which is important for building tribal stakeholder willingness and commitment to cooperate fully with the research and data collection responsibilities. Investing in site development at the front end reduces problems at the back end, helping to ensure smooth implementation of the study.

B4. Expert Review and Piloting Procedures

All study protocols were reviewed by the study’s technical working group in March 2016.

Following publication of this OMB submission for 60-day public comment, the study team will pilot test all site visit protocols by asking nine staff members in similar roles in a former NAM grantee (i.e., a non-study program) to participate in mock interviews (responses will not be included in the final study data). The study team will ask for feedback on questions that are confusing and will incorporate changes in a final version of the protocols before data collection in fall 2016. The study team also will note how long each interview takes. These pilot tests will determine whether the protocols are an appropriate length, which items need to be revised for clarity, and whether additional items are needed to develop a full picture of the program.

B5. Individuals and Organizations Involved in the Project

American Institutes for Research (AIR) is the contractor for Task Order 24: Study of the Title III Native American and Alaska Native Children in School (NAM) Program. The project director is Dr. Erin Haynes, who is supported by an experienced team of researchers that will guide and conduct the case study data collection and analysis (see Exhibit 7 for a list of key staff involved in the project, their roles, and contact information).

Exhibit 7. Key Staff Involved in the Project

Role	Organization	Contact Name	Telephone number
Project Director	AIR	Dr. Erin Haynes	(781) 373-7017
Case Studies Task Leader	AIR	Susan Cole	(650) 843-8187
Quality Assurance Monitor	AIR	Dr. Diane August	(202) 403-5949
Quality Assurance Monitor	AIR	Dr. Kerstin LeFloch	(202) 403-5649
Site Liaison	AIR	Stephanie Autumn	(202) 573-2125
Site Visitor	AIR	Traci Karageorge	(202) 403-6217
Site Visitor	AIR	Linda Cavazos	(512) 391-6523
Site Visitor	AIR	Jayne Sowers	(312) 288-7634
Site Visitor	AIR	Jenny Scala	(831) 621-4003
Consultant	Academic Transitions	Starla Bilyeu	(918) 346-1571

6290_06/16