Part B: Methods for the Collection of Cross-site Study Data for Improving Implementation Evaluation among Office of Population Affairs (OPA) Teen Pregnancy Prevention (TPP) Grantees to inform National Implementations (IMAGIN)
CONTENTS
Tables iv
ATTACHMENTS iv
INSTRUMENTS iv
PART B: INTRODUCTION 1
B1. Respondent Universe and Sampling Methods 2
B2. Procedures for Collection of Information 3
B3. Methods to Maximize Response Rates and Deal with Non-Response 4
Dealing with Nonresponse 4
Maximizing Response Rates 5
B4. Test of Procedures or Methods to be Undertaken 5
B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 6
Table B.1.1. Overview of data collection methods and respondents ………………………………………3
ATTACHMENT A: EMAIL INVITATION FOR GRANTEE LEADERSHIP INTERVIEW
ATTACHMENT B: EMAIL INVITATION TO BEGIN SITE VISIT PLANNING
ATTACHMENT C: FRONTLINE STAFF SURVEY INVITATION AND REMINDER EMAILS
ATTACHMENT D: PERSONS CONSULTED ON INSTRUMENT DEVELOPMENT AND/OR ANALYSIS
ATTACHMENT E: CONFIDENTIALITY PLEDGE
ATTACHMENT F: 60 DAY FEDERAL REGISTER NOTICE
INSTRUMENT 1: GRANTEE PROGRAM LEADERSHIP staff INTERVIEW GUIDE (Initial and follow up)
INSTRUMENT 2: KEY PROGRAM STAFF INTERVIEW GUIDE
INSTRUMENT 3: COMMUNITY STAKEHOLDER INTERVIEW GUIDE
INSTRUMENT 4: FRONTLINE STAFF SURVEY
The consequences of adolescent sexual activity remain a critical social and economic issue in the United States. Although births to teen mothers have dropped sharply over the past 25 years, the teen birthrate remains higher in the United States than in other industrialized countries and varies widely across geographic regions and racial and ethnic groups (Martin et al. 2017). Adolescents and young adults also account for half of all sexually transmitted infection (STI) cases each year (Centers for Disease Control and Prevention [CDC] 2017), and rates of STIs continue to rise (CDC 2018). Because adolescent sexual activity often occurs outside stable, long-term relationships, it is frequently linked to risk behaviors such as alcohol and substance use, teen dating violence, and sexual assault, and specific vulnerabilities, such as living in foster care or involvement with the juvenile justice system.
As part of the government’s ongoing efforts to support youth in making healthy decisions about their relationships and behaviors, Congress authorized the Teen Pregnancy Prevention (TPP) program in 2010. Administered by the Office of Population Affairs (OPA) (formerly the Office of Adolescent Health, which officially merged into the Office of Population Affairs, effective June 2019) of the Office of the Assistant Secretary for Health (OASH) at the U.S. Department of Health and Human Services (HHS), TPP provides funding to local organizations to implement evidence-based (Tier 1) and promising new programs (Tier 2). The TPP Tier 1 grantees (funded in 2010 and in 2015) focused on replicating and testing specific curricula; however, the evaluations of these programs raised more questions than they provided answers about how to design and deliver high quality and effective TPP programs.
With a new TPP funding opportunity announcement in spring 2018, OPA is addressing the knowledge gap. The first phase (Phase I) of these new grants have a specific focus on program improvement and formative evaluation to help grantees identify and integrate elements of effective programs, and facilitate readiness for implementation and summative evaluation. Summative evaluation will occur in Phase II for a subset of Phase I grantees that show readiness for summative evaluation. In 2018, OPA awarded two-year Phase I grants to 14 Tier 2 grantees (TPP18). An additional 30 two-year Phase I grants to support evidence based programming (Tier 1) are expected in 2019 (TPP19). Grantees are expected to develop or select programs that integrate sexual risk avoidance (SRA) and/or sexual risk reduction (SRR) approaches, as well as address youth, family, and systems-level protective factors.
OPA’s contract, Improving Implementation Evaluation among OPA TPP Grantees to Inform National Implementation (IMAGIN), supports the TPP program improvement agenda. The study will support grantees in two goals 1) assessing the quality and feasibility of their programming and using data to improve program delivery through continuous quality improvement, and 2) provide important information on the design and implementation of effective TPP programs to grantees and relevant efforts across the nation.
With this new ICR, OPA seeks approval for data collection activities for a cross-site study to address the second goal above. These activities, to be conducted with grantees and their partners, will include (1) interviews with leadership staff, such as grantee directors; (2) discussions with key program staff; (3) discussions with community stakeholders; and (4) a frontline staff survey. The cross-site study will document and describe the process, challenges, and successes related to getting programs ready for implementation and summative evaluation. The study will also assess how a multiphase grant effort supported grantees in implementing and preparing for rigorous evaluation of their program models.
This Information Collection Request (ICR) includes the following activities associated with the cross-site study: 1) initial and follow-up interviews with grantee directors; 2) site visits consisting of interviews with key program staff and community stakeholders; and 3) a frontline staff survey. The specific data to be collected per grantee is summarized in Table B1.1.
Grantee leadership interviews. The study team will interview leadership staff from all grantees in the TPP18 and 19 cohorts (44 grantees) towards the end of the first year of their grant period (pending OMB approval). The initial interview will last up to 90 minutes and will be conducted over the phone. The study team will also conduct 60-minute follow-up interviews with grantee leadership staff (up to 44 staff) during the second year of the grant period to discuss refinements to their programs and processes, challenges, successes, and lessons learned related to implementation readiness. The discussion guide lists the topics for the initial and follow-up leadership interviews (Instrument 1).
Site visits consisting of interviews with program staff and
community stakeholders. After the initial discussions with
grantee leaders, the study team will select a purposive sample of up
to 14 grantees from the two cohorts to conduct site visits for two
days each. During the visit, site visitors will meet with and
interview up to two program supervisors, up to eight frontline
staff, and up to two community stakeholders with knowledge of the
program (Instruments 2 and 3). Key program staff and community
stakeholders will include diverse staff and community members who
play substantive roles in program development or selection,
preparation for implementation and evaluation, who are knowledgeable
about the origins and operations of their program, and who can
discuss any challenges encountered and how they were resolved. Site
visitors will work closely with grantee leadership staff or a
designated contact to identify the appropriate respondents and
schedule the visits at a time that is convenient for them
(Attachment B).
The cross-site study team will select a
mix of grantees at different readiness stages and with varied
implementation contexts to gather insights into key facilitators and
challenges to implementation readiness. Criteria for purposive
selection include:
A mix of program models incorporating traditional classroom-based methods and those using innovative strategies (e.g. content delivered through apps, experiential learning, story-telling, etc.) where the lessons could be useful and meaningful to a variety of audiences (including grantees, organizations implementing similar programs, researchers, and funders)
A mix of interesting settings or populations (e.g. foster care, juvenile justice centers, high schools, app-based models) that could provide insights into specific challenges to readiness in working with these groups or in these settings
Use of similar components or methods whose process for readiness would be useful to explore across multiple programs
The study team will work with OPA and project officers to develop criteria and indicators of readiness for site selection that ensure an appropriate mix of grantees. The team will begin visits in fall 2019 (pending OMB approval), prioritizing a subset of three or four TPP18 grantees that are approved by OPA to begin full implementation in the second year of their grant. The study team will visit four TPP18, and up to an additional seven TPP19 grantees in summer and fall 2020, to give them time to reflect on their challenges, success, and lessons. The number of visits that can be conducted will ultimately depend on grantees’ progress toward implementation readiness and their status at the end of their first year.
Frontline staff survey. For all grantees, the cross-site study team will survey the frontline staff delivering the programs (Instrument 4). It is expected that up to eight respondents from each grantee will take the survey, but this could vary based on the scope of the program and number of staff. The 30-minute web-based survey is designed to collect information on staff backgrounds and roles, the training and preparation they received to deliver the program, their experiences with early implementation and data collection for evaluation, and key lessons related to program and organizational readiness. The survey draws on similar surveys from other projects and incorporates specific closed- and open-ended questions aligned with and designed to address the cross-site study research questions. The cross-site study team will coordinate with the program staff (and local evaluator, if any) to determine the final number of respondents for each grantee and decide on the best time to administer the survey based on staff convenience and schedules.
Table B.1.1. Overview of data collection methods and respondents
Instrument |
Number of grantees |
Mode |
Number of respondents |
Grantee Leadership Interview |
44 |
Phone interviews |
2 per grantee (initial and follow up) |
Key Program Staff Interview |
14 |
During site visit, in-person |
Up to 10 per grantee |
Community Stakeholder Interview |
14 |
During site visit, in-person |
2 per grantee |
Front line staff survey |
44 |
Web-based survey |
Up to 8 per grantee |
Interviewers with grantee leadership will be conducted over the phone following the discussion guide (Instrument 1). Key program staff and community stakeholder interviews will be conducted during planned site visits to 14 grantees. Grantees will be selected for site visits based on information collected from document review and the grantee leadership interviews. The data collection procedures for site visits include using a discussion guide (Instruments 2 and 3). We will conduct the initial leadership interviews and frontline staff survey in 2019 and 2020, follow up leadership interviews in 2020 and 2021, and the site visits in 2019 and 2020, upon OMB approval. The timing of data collection activities for each grantee will be based on when the grantee received funding (Table A.2.1).
For planning the grantee leadership interviews, members of the cross-site study team will make the initial contact with grantee project directors through email (Attachment A). The list of grantee directors and their contact information will be obtained from OPA. The study team will work closely with the director or a designated contact to minimize disruption and schedule all leadership phone interviews at a time that is convenient for the respondents. A team of two researchers working on the cross-site study team will conduct the interviews (one lead interviewer and one note-taker). Before each interview, the interviewers will ask for verbal consent from each respondent to participate in the interview and to record the interviews. Based on grantees’ progress towards readiness and to clarify or confirm particular questions, the study team will conduct 60-minute follow-up phone interviews with up to 44 grantee leadership staff in their second year.
Once a purposive sample of grantees has been selected for site visits, the designated site visitors (comprised of a lead visitor and a note-taker) will work with grantee staff to schedule the visit and identify the appropriate respondents for interviews (Attachment B). Up to two supervisors, eight frontline staff, and two community stakeholders will be interviewed at each grantee across 14 grantee sites. During the site visits, to the extent possible, the semi-structured interviews with program staff and community stakeholders will be held with small groups of staff at similar levels in their organizations. For example, a small-group interview may be held with implementation team members who are frontline staff, such as teachers at a school, or case managers at a social service agency. If a grantee has only a single staff member in a particular level, however, an individual interview will be held with that person.
The study team will also work with the grantee director or a designated contact to identify and contact the relevant frontline program staff to complete a 30-minute web-based survey about program and organizational readiness. Selected staff will be intimately involved in or have direct knowledge of program delivery and readiness. Staff will receive an emailed invitation letter with instructions for accessing the web-based survey (Attachment C). Both the email and initial log-in page of the survey will contain consent language, informing respondents that their participation is voluntary.
Data collected for the cross-site study will be descriptive.
The study team anticipates a high response rate based on experience with similar instruments on other studies. We expect that 80 percent or more of the selected participants will choose to participate in the cross-site study interviews and surveys; response rate is based on previous evaluation studies of TPP (Making Proud Choices! Federal Evaluation and Regional Partnership Grant Evaluation, where response rates of over 80% were achieved). Descriptions of the strategies for maximizing response in the data collection efforts follow.
The cross-site study is a descriptive study. We do not anticipate levels of nonresponse significant enough to affect data analysis.
We expect to obtain a high response rate, 80% or more, among grantees for the interviews and the frontline staff survey. We will take several steps to help ensure a high rate of cooperation from participants.
Conduct interviews with heavily invested stakeholders. We aim to interview stakeholders who are heavily invested in the grant activities. We anticipate that respondents will be eager to engage in these discussions.
Conduct interviews with key program staff and community-stakeholders during site visits. Interviews with key program staff and community stakeholders will occur during site visits. We anticipate that selected grantees will agree to participate, and to help ensure high participation, we will coordinate with the grantees, supervisors/managers, partners, and staff to determine convenient dates and schedules for these visits (Attachment B).
Design frontline staff survey in a manner that minimizes respondent burden. To minimize burden on participants, the survey is web-based, and structured such that respondents do not have to pay attention to routing and skip logic or view questions that do not apply to them. Participants will be able to complete the survey at a time that is convenient for them, on a computer or other mobile device.
Send advance and reminder emails to frontline staff for survey (Attachment C). We will send advance emails to frontline staff requesting their participation. If participants have not completed the survey within a certain amount of time, we will send reminder emails requesting them to complete the surveys.
Solicit the help of grantees to encourage completion of the frontline staff surveys. If response rates for individual grantees lag, the cross-site study team will work with grantee leadership staff to identify additional strategies for increasing completed surveys without compromising respondent confidentiality. For instance, grantee leadership staff may be asked to send an email to all the survey participants they had identified in their site, encouraging everyone’s response. In similar projects, this approach helped boost response rates, because grantee leadership staff had personal relationships with their staff and used their proximity to encourage responses. This approach of combining follow-up requests from the evaluator to people who have not completed the survey with general requests from the grantee to all desired respondents has proved effective in multiple Mathematica projects that involved collecting similar data through web-based surveys.
We will not conduct a separate pilot test of our data collection procedures or discussion guide. We have successfully used similar data collection procedures, discussion guides and staff surveys to collect comprehensive and reliable data in other studies with similar respondents such as the PREP (ACF), PAF (OPA), YARH (ACF) and PACT evaluations (ACF). The site visitor team will meet following the first two site visits to discuss any necessary refinements of the discussion guide, if needed. We do not anticipate needing to make significant changes, however, if changes are made to the discussion guides we will re-submit to OMB for approval.
The implementation study site visits and staff survey will be conducted by OPA’s contracting organization, Mathematica Policy Research. Mathematica will also conduct all analyses of the data. Attachment D lists the individuals whom OPA consulted on the collection of the implementation study instrument.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Jen Walzer |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |