TPP20 Tier 1 and Tier 2 Implementation Study Supporting Statement B
Supporting Justification for OMB Clearance of Teen Pregnancy Prevention FY20 Tier 1 and Tier 2 Implementation Study
Part B: Statistical Methods for TPP20 Tier 1 and Tier 2 Implementation Study Data Collection
June 2022
Department of Health and Human Services
Office of Population Affairs
1101 Wootton Pkwy
Suite 200
Rockville, MD 20852
Alexandra Osberg, 301-348-3537 [email protected]
1. Respondent Universe and Sampling Methods
For the TPP20 Implementation Study, we will document how each of the 75 grantees funded under these grant programs are implementing teen pregnancy prevention (TPP) programs in their respective communities or priority areas. Sixty-two grant projects are replicating evidence-based interventions to optimally change the map for teen pregnancy prevention (Tier 1). Thirteen grant projects are creating innovation and impact networks to develop, test, refine, and evaluate innovative interventions to reduce the rates of teen pregnancy and sexually transmitted infections and improve adolescent health (Tier 2).
The TPP20 Implementation Study starts with a systematic review of each grantee’s grant application to the Office of Population Affairs (OPA) to understand the variety of approaches each grantee originally planned to implement the grant requirements for Tier 1 or Tier 2. OPA anticipates that grantees will employ diverse strategies to meet the grant requirements and that these strategies will have changed and evolved since the grant application due to local needs and the ever-changing environment surrounding the COVID-19 pandemic. Because this information collection will contribute to the emerging knowledge base about (1) community-wide efforts to engage a systems-thinking approach to implementing evidence-based interventions (EBIs) and (2) leveraging multiple organizations to develop an innovation network to explore, develop, test and refine innovative new approaches to reduce teen pregnancy and STIs within select priority areas, it will be important to document the variety of grantee approaches and challenges they have encountered as a result of local conditions and strategies.
To document these features and experiences, each grantee will be asked to complete a short pre-interview informational form to provide up-to-date basic background information about their TPP project. (The pre-interview form was approved under OMB control number 0990-0379 and is therefore not included in this request.) Following the fielding of the pre-interview form, one to two lead staff members at each grantee organization will be interviewed (either in person—for up to six grantees—or over videoconference or telephone). In addition, staff from up to two key grantee partner organizations will be interviewed (e.g., for Tier 1, organizations that provide the EBIs to youth for 31 of the 62 grantees; for Tier 2, organizations that are significantly involved in the Innovation and Impact Network for all 13 grantees).
Partner organizations to be interviewed will be selected based on the prominence and variety of their roles within each initiative in order to provide multiple perspectives on implementation. To obtain more detail on implementation than can be gathered in an interview, in-person site visits with up to six grantees (four Tier 1 grantees and two Tier 2 grantees) will be conducted to collect data that will illustrate in detail a variety of approaches and strategies for implementing the grant strategies. The six grantees will be selected and recruited based on a review of the grantees’ semi-annual progress reports (submitted to OPA and shared with the Abt team), performance measures, and findings from the preliminary data review. The Abt team will stratify grantees that can accommodate an in-person site visit based on their program characteristics such as grantee organization type, grant structure, primary implementation setting, reach/saturation, and urbanicity to select a variety of grantees for the site visits. The Abt team will also solicit input from TPP Project Officers and OPA. Within each participating grantee community, data will be collected from grantee and partner staff with varying roles and responsibilities. They will have knowledge about the development and operations of the TPP project and the challenges it has encountered. Up to 238 participants will be included in the data collection overall (including videoconference/telephone interviews and site visits).
2. Procedures for the Collection of Information
The interview data will be collected through semi-structured interviews held over videoconference/telephone or in-person (for up to six grantees). The interview protocols are included in this package. We developed separate interview protocols for the Tier 1 and Tier 2 grantees (Attachments A and C), which will be fielded with lead staff members at each grantee organization to organize data collection and documentation of the grantees’ projects. These interview protocols identify the main questions that will be asked of each grantee to document the grantee strategies, challenges, and adaptations. We have also created separate interview guides for discussions with the Tier 1 and Tier 2 grant partners (Attachments B and D). The interview protocols will be tailored for each grantee prior to the data collection. This will include populating the interview protocols with data already known to the study team to reduce the burden on the interview respondents.
Topics to be explored in this data collection include: overall implementation of the TPP project, implementation context, implementation facilitators and barriers, systems thinking, community reach, promising strategies, selection of evidence-based interventions, partner and youth engagement, priority area strategies, flexibility and change, and program evaluation plans.
Data collection will occur during the second half of the grant projects’ final year of funding. This will allow grantees and grantees’ partners to reflect on their experiences with lessons learned over the majority of the grant’s lifespan. A two-person team will conduct all interviews. The team leader will be a senior researcher who is experienced in qualitative data collection. The two-person team will work together to ask follow-up probes to elicit the desired information and increase the accuracy of information obtained.
Estimation Procedures
The site visits and interviews are designed to provide in-depth qualitative information about the TPP projects; no estimation procedures will be used. The data analysis will be descriptive.
Communication with Grantees and Communities
Prior to conducting the interviews or site visits, the study team will send an introductory email to the project director for each of the Tier 1 and Tier 2 grants, explaining the purpose of the interviews, asking to schedule an interview or site visit, and asking about key partner staff with whom we should also speak. We will reach out to key partner staff with a similar introductory email explaining the purpose of the telephone interview and asking about their availability.
Once an interview or site visit is scheduled, the study team will provide interview topic areas to the respondent(s), which will allow respondents to feel more comfortable and prepared for the interview, and to think about questions prior to the interview or look up information, if desired.
Training and Data Collection
Prior to the interviews or site visits, the data collection team will participate in a group training to provide a study overview, an overview of the key components of the Tier 1 and Tier 2 grant strategies, and to review the data collection protocols in detail. Following the training, each telephone interview will be conducted by a two-person team: a lead interviewer and a support interviewer/note-taker (the interview team). Most interviewers will have participated in the grant application review and will already be familiar with the Tier 1 and Tier 2 programs. Before each interview, the lead interviewer and support interviewer will review the grantee’s grant application review summary and any additional information available on the TPP project. The lead interviewer will work with the support interviewer to pre-fill any questions or background on the interview protocol and tailor the probes for partner organizations as needed.
Each grantee interview will last two hours, on average and each partner interview will last one hour, on average. All interviews (in-person and virtual) will be recorded, with the consent of the interviewee(s). Immediately after each interview, the interview team will review and finalize notes. After each interview, or within 24 hours, the interview team will discuss interviews and observations to ensure that they captured all important information in the notes. The team will also resolve discrepancies regarding details of the interviews or site observations if needed. After completion of the site visit, the audio recordings from the interviews will be sent out for transcription and then annotated with any nuances captured in the interview’s notes.
3. Methods to Maximize Response Rates and Deal with Nonresponse
Participation in the federal evaluation conducted or commissioned by OPA was a condition of being awarded TPP Tier 1 or Tier 2 funding. It is anticipated that all grantees will agree to participate since OPA will request their compliance. Additionally, interviews and site visits will be planned well in advance so that all identified respondents can participate. Based on our experience in the previous TPP Tier 1B Scale-Up study, we anticipate that refusals to participate and absences will be rare.
The interview team will work closely with the primary contact for each TPP grantee to help in scheduling the interviews or site visit. One member of the two-person interview team will take responsibility for working with the primary contact person to handle the scheduling and logistics, e.g., identifying appropriate interview respondents. Interview appointments will then be confirmed via e-mail. Should a potential respondent not be available during the allocated time, the interview team will follow up with a time to interview the person, either during the site visit, if applicable, or later by phone.
4. Tests of Procedures or Methods to be Undertaken
Many of the topics and probes in the implementation study interview protocols have been tested and refined in a previous study with similar respondents (the TPP Scale-Up study, OMB Control Number 0990-0451). Additionally, three current TPP20 grantees will participate in exploratory or pilot site visits before end of September 2022 to facilitate interviewer training on customizing the interview guides and asking probes. The pre-testing will only include a maximum of nine people across the three grantees.
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or
Analyzing Data
The TPP20 Implementation Study interviews and site visits will be overseen by staff of Abt Associates, the contractor selected in September 2021 to conduct the study. Listed below are the individuals whom OPA consulted on the collection and/or analysis of the data described in this document. They include the contractor staff for the evaluation and the OPA Contract Officer Representative (COR) for the evaluation.
Kimberly Francis, Ph.D.
Abt Associates
10 Fawcett Street
Cambridge, MA 02138
(617) 520-2502
Tanya de Sousa, MPP
Abt Associates
6130 Executive Blvd.
Rockville, MD 20852
(301) 347-5695
Lesley Freiman, MPP
Abt Associates
6130 Executive Blvd.
Rockville, MD 20852
(617) 520-2334
Cynthia Castaldo-Walsh, Ph.D.
Decision Information Resources, Inc.
3900 Essex Lane, Suite 900
Houston, TX 77027
(832) 485-3777
Alexandra Osberg, MPH
Office of Population Affairs
Office of the Assistant Secretary for Health
U.S. Department of Health and Human Services
1101 Wootton Parkway, Suite 200
Rockville, MD 20852
(301) 348-3537
Ms. Osberg is the TPP20 Implementation Study project officer and has overseen the development of the current interview instruments. Inquiries related to the TPP Tier 1 or Tier 2 grant programs or the evaluation of them may be directed to Ms. Osberg.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | DHHS |
| File Modified | 0000-00-00 |
| File Created | 2022-10-14 |