SSB - Formative ACF Program Support GenIC - SRAE Grantee COVID Interviews

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Sexual Risk Avoidance Education (SRAE)

Grantee COVID-19 Interviews

Formative Data Collections for Program Support

0970 - 0351

Supporting Statement

Part B

October 2021

Submitted by:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Jessica Johnson

Calonie Gray

Part B

B1. Objectives

Study Objectives

The objective of the Sexual Risk Avoidance Education (SRAE): Grantee COVID-19 Interviews is to collect data from SRAE grantees about how COVID has affected their programming and operations and how it will shape programming in the upcoming year.

The data collected from the interviews will provide the Administration for Children and Families (ACF) with up-to-date feedback from grantees on their needs for training and technical assistance (T&TA) and program guidance due to the ongoing impacts of the COVID-19 pandemic. The information will inform ACF support for grantees.

Generalizability of Results

This study is intended to present an internally valid description of how the COVID-19 pandemic has affected program operations for SRAE grantees providing direct services, not to promote statistical generalization to other sites or service populations. Information will be used to inform support and programming at the sites from which it was collected.

Appropriateness of Study Design and Methods for Planned Uses

We will conduct one-on-one asynchronous virtual interviews. Participants will log in to an online platform and respond to interview questions moderated by the study team at Mathematica¹. The flexibility of a self-administered instrument combined with the benefits of study team moderation to probe and follow up on open-ended responses will enable us to collect qualitative data from a diverse set of grantees.

The study design lends itself well to this population, as grantee staff often have competing demands and might find it difficult to set aside time during the day for a phone interview. The chat board format allows respondents to complete the interview at their convenience over a five-day period. In addition, the design is appropriate to collect the data needed to meet the study objectives. The interview protocol (Instrument 1) includes a mix of closed- and open-ended questions; the open-ended questions enable the team to collect more in-depth qualitative data to capture the diversity of grantees’ experiences adapting programming and operations during the pandemic. The chat board format lets the study team moderate across interviews, following up with individual respondents as needed to ensure they collect the data necessary to address the study questions. We describe key limitations of the design in Supporting Statement Part A, Section A2, and will include them in internal written products associated with the study.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.  

B2. Methods and Design

Target Population

The population of interest for this study is the 150 SRAE grantees who are direct service providers, a subset of all organizations funded through the SRAE grant program who are providing programming in most states. Respondents of interest for the interviews are the grantee administrators or program directors, whomever the grantee identifies as the most knowledgeable respondent.

Sampling

ACF will supply the contractor, Mathematica, with the contact information for grant administrators. The contractor will field the Grantee COVID-19 Interview (Instrument 1) with a purposive sample of all Competitive and General Departmental SRAE grantees who provide services directly to youth. This sample differs from other grantees, such as the State SRAE grantees, that provide programming to youth through numerous subrecipients. The Competitive and General Departmental SRAE grantee organizations are located in most states, and therefore represent the experiences providing programming in different regions across the country who may have had varying experiences due to varying COVID-19 restrictions. We are sampling grantees that provide services directly to youth because the interviews focus on program operations and will capture information about adaptations to their workspace, workforce, and programming; youth engagement with programming; and potential youth needs resulting from the pandemic. These questions cannot be addressed by a grant administrator who is overseeing numerous subrecipients without the grant administrator conducting additional data collection. We will attempt to interview the entire subsample of grantees who provide services directly to youth so that we capture any variation in experiences across states and regions..

B3. Design of Data Collection Instruments

Development of Data Collection Instrument

The Grantee COVID-19 Interview protocol was developed in collaboration with ACF and Mathematica staff to address four main topic areas for program support: (1) changes to work space and work environments; (2) program design and adaptations; (3) changes to data collection and evaluation activities; and (4) the future of programming. During development, we reviewed additional sources of information and streamlined the protocol to include only questions that build on data collected through performance measures and a short FYSB grantee COVID survey administered in fall 2020 under the ACF Fast Track Generic Clearance for Collection of Qualitative Feedback on Agency Service Delivery (OMB #0970-0401; Family and Youth Services Bureau (FYSB) Grantee COVID Survey).

The interview protocol is designed to collect a combination of closed- and open-ended items to minimize respondent burden and to enable the study team to capture the data necessary to meet the study objectives. Each section starts with closed-ended items. We included additional open-ended questions, so grantees that modified operations and programming due to the pandemic can share feedback on their needs during the pandemic and their plans and needs for the upcoming year.

The study design prevents measurement error in two ways: (1) study leads can oversee all interview moderators in the chat board, ensuring consistency in grantee probes and follow-ups and clarifying unclear or ambiguous responses, and (2) the online platform exports a transcript of all comments to interviewers, preventing errors in notetaking and transcription that could occur during a telephone or in-person interview.

B4. Collection of Data and Quality Control

ACF is contracting with Mathematica for this data collection effort.

Mathematica will conduct the interview using QualBoard, an online bulletin board where participants can see and respond to questions. The interview will not be a real-time chat; participants can log on at any time convenient for them. Responses will be stored in a database only accessible to the study team and used as an interview transcript.

Data collection will occur on a rolling basis. We will administer the interviews each week in small groups of 20 to 30. Before data collection, the contractor will send an email inviting administrators or program directors of grantees to participate in an interview (Appendix A: Grantee Email Invitation). The invitation will explain the purpose of the interview and detail the data collection mode. It will include a proposed week of data collection, to ensure participants will be available during their data collection window. Following the invitation, participants will be invited in groups to access the QualBoard site for their interviews. Groups will be staggered over a five-week period to limit the number of grantees accessing the board at one time. Smaller groups will also enable the study team to more carefully review and moderate interviews. However, individual grantees will not be able to see other grantees responses.

Participants will receive log-in credentials via email to access the QualBoard site (Appendix B: QualBoard Invitation Email).

The contractor will follow up with interview nonrespondents to encourage their participation by highlighting the importance of the interviews. Federal project officers might also contact nonrespondents to encourage their participation. In addition, to encourage participation and to provide flexibility, upon request, we will allow participants to complete the protocol via telephone with a trained member of the study team.

A trained member of Mathematica’s study team will conduct each interview. Questions from the interview protocol will be programmed into chat boxes in QualBoard, and participants will type responses. Interview moderators will be trained on the interview protocol, and the project lead will review all moderators’ sessions for consistency in engagement with participants. Throughout the sessions, moderators will review responses. When necessary, they will encourage more in-depth responses and probe participants to elaborate on responses. When the moderator asks a follow-up question, the participant will receive an email notification that follow-up on their interview has been requested.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The interviews are not designed to produce statistically generalizable findings.

At the time of their application, grantees were notified that participation in any of the SRAE study activities is mandatory. However, respondents will be told that they can skip questions that they do not feel comfortable answering. Responses will be monitored and the study team and/or the FYSB project officers will send reminders to non-respondents. The study team will also offer telephone interviews for grantees if that is preferrable over the QualBoard interface. Based on early survey efforts with SRAE grantees, we can expect to exceed a 90 percent participation rate.

Nonresponse

As participants will not be randomly sampled, and findings are not intended to be representative, nonresponse bias will not be calculated. Respondent grantee type will be documented and reported in written data collection materials.

B6. Production of Estimates and Projections

The data will not be used to generate population estimates, either for internal use or dissemination.

B7. Data Handling and Analysis

Data Handling

Interview data retrieved from QualBoard will be saved on a secure drive accessible only to Mathematica study team members. Direct export of the QualBoard data to the secure drive will result in minimal processing. Interviewers will review transcripts to fix spelling and grammar issues, fill in missing words, and explain unclear terms or phrases in preparation for qualitative coding and analysis.

Data Analysis

Qualitative data from the interview transcripts will be reviewed for overarching themes and lessons on each of the key topics explored through the virtual discussions. The research team will develop a coding scheme based on the research objectives and interview topics. Moderators will apply the coding scheme to the transcripts and conduct a thematic analysis of the responses under each topic. The task lead for the interviews will monitor coding and thematic analysis across the team to ensure accuracy and consistency.

Data Use

The interview data will be used to inform ACF with up-to-date feedback from grantees on their needs for training, technical assistance, and program guidance due to the ongoing impacts of the COVID-19 pandemic. With this critical information, ACF will be able to provide focused training and technical assistance to better serve the grantees. As noted in Supporting Statement A, information will result in an internal memo that summarizes key impacts of the COVID-19 pandemic on the grantees, their program implementation, and their technical assistance needs. The memo will discuss limitations of the data (Supporting Statement Part A, Section A2) and include information on how to use and interpret the qualitative interview data. ACF may also elect to prepare a public facing brief on the findings if there are lessons learned that could be helpful for grantees’ continuous program improvement efforts.

B8. Contact Persons

Table B.1 lists the federal and contract staff responsible for the study, their affiliations, and their email addresses.

Table B.1. Individuals responsible for study

Name	Affiliation	Email address
Jessica Johnson	Family and Youth Services Bureau Administration for Children and Families U.S. Department of Health and Human Services	[email protected]
Calonie Gray	Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services	[email protected]
Susan Zief	Mathematica	[email protected]
Tiffany Waits	Mathematica	[email protected]
Jennifer Walzer	Mathematica	[email protected]
Sarah Wagner	Mathematica	[email protected]

Attachments

Appendix A. Grantee Email Invitation

Appendix B: QualBoard Invitation Email

Instrument 1: Grantee COVID-19 Interview Protocol

1 ACF has contracted with Mathematica to carry out this information collection.

5