SRAE PAS_ SSB_080519_clean_092619

Sexual Risk Avoidance Education Program Performance Analysis Study

(SRAE PAS)

OMB Information Collection Request

0970-New Collection

Supporting Statement

Part B

August 2019

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer:

Caryn Blitz

CONTENTS

B1. Respondent Universe and Sampling Methods 1

1. Youth Participants 1

2. Grantees and Sub-Awardees 1

B2. Procedures for Collection of Information 2

B3. Methods to Maximize Response Rates and Deal with Non-Response 3

B4. Test of Procedures or Methods to be Undertaken 4

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 4

TABLES

B.1 Annual Respondent Universe and Expected Response Rates for the Study of Performance Measures 2

FIGURES

1 Levels of Performance Measures Data Collection 3

INSTRUMENTS

INSTRUMENT #1 – PARTICIPANT ENTRY SURVEY

INSTRUMENT #2 – PARTICIPANT EXIT SURVEY

INSTRUMENT #3 – PERFORMANCE REPORTING SYSTEM DATA ENTRY FORM

INSTRUMENT #4 – SUBAWARDEE DATA COLLECTION AND REPORTING FORM

B1. Respondent Universe and Sampling Methods

The participant entry and exit surveys will provide aggregate data on the demographic and behavioral characteristics of program participants and aggregate data on participants’ perceptions of program influences on knowledge, attitudes, and behavior and their responses to the program. Administrative data reported by the grantees for performance measurement will include semi-annual data on program features and structure, allocation of funds, participant numbers, levels of participant engagement, fidelity to evidence-based program models, and staff perceptions of quality challenges and needs for technical assistance. Table B1.1 describes the respondents and expected response rates associated with the performance measures data collection.

1. Youth Participants

Instrument 1: Participant Entry Survey. We expect the Sexual Risk Avoidance Education (SRAE) grantees are expected to serve approximately 1,367,874 participants over the three-year OMB clearance period, for an average of about 455,958 new participants per year. Of those, we expect 95 percent, or approximately 433,160 participants, will complete the participant entry survey each year.

Instrument #2: Participant Exit Survey. It is estimated that about 20 percent of the participants will drop out of the program prior to completion, leaving approximately 364,766 (455,958 x .80) participants at the end of the program annually.¹ Of those, we expect 95 percent, or approximately 346,528 participants, will complete the participant exit survey each year.

2. Grantees and Sub-Awardees

Instruments 3-4: Performance Reporting System Data Entry Form and Subawardee Data Collection and Reporting Form. The 137 grantees² will report performance measures data into the SRAE Performance Measures Portal (SPMP) developed for the SRAE Performance Analysis Study (PAS). They will gather this information with the assistance of their sub-awardees (estimated to be 2,330 across all grantees). Because collecting and reporting data for performance measures is a funding requirement of all SRAE grants, we expect the grantee and sub-awardee response rates to be 100 percent.

Table B.1.1. Annual Respondent Universe and Expected Response Rates for the Study of Performance Measures

Data Collection	Type of respondent	Number of Potential Respondents	Expected response rate	Total expected responses
Instrument 1: Participant Entry Survey	Youth participant	455,958	95%	433,1606
Instrument 2: Participant Exit Survey	Youth participant	364,766	95%	346,528
Instrument 3: Performance Reporting System Data Entry Form	Grantee Administrator	137	100%	1377
Instrument 4: Sub-awardee Data Collection and Reporting Form	Sub-Awardee Administrator	2,330	100%	2,330

B2. Procedures for Collection of Information

Instruments 1-2: Participant Entry and Exit Surveys. Each grantee and their sub-awardees will make decisions regarding procedures for collecting the participant entry and exit surveys. Some grantees may have elected to work with local evaluators that will administer the surveys for performance measures purposes; the local evaluators could decide to use paper-and-pencil or web-based surveys. For those grantees not working with local evaluators, it is likely that the subawardees’ program facilitators will administer paper versions of the entry and exit surveys in group or individual settings. The contractor will provide training and technical assistance to all grantees with regard to privacy and security, which will inform their data collection and storage procedures (See Section A10 of Supporting Statement A). Grantees will inform their individual program participants that participation is voluntary and that they may refuse to answer any or all of the questions in the entry and exit questionnaires.

Instruments 3-4: Performance Reporting System Data Entry Form and Subawardee Data Collection and Reporting Form. Grantees will report separately on levels of participant attendance, reach and dosage (see Figure 1). Subawardees (i.e., providers and programs; see Figure 1) will collect data on these measures (Instrument 4). Administrative data on program features and structure, allocation of funds, and staff perceptions of quality challenges will be collected by grantees and sub-awardees through their administrators (Instruments 3 and 4). Grantees will prepare and submit their final data sets to ACF through the SRAE Performance Measures Portal (SPMP). The Performance Reporting System Data Entry Form (Instrument 3) contains the list of all data elements grantees will report, collected from among their sub-awardees.

Figure 1. Levels of Performance Measures Data Collection

The timing of participant survey data collections will be customized for each site depending upon the start and end dates of each cohort of participants. Administrative performance measurement data and participant information will be reported twice a year.

Data Analysis. The contractor will analyze SRAE performance data to generate performance measurement reports for ACF and other audiences. Core analyses will include computing means and sums of continuous numeric measures (such as number of participants served) and producing frequency distributions of categorical and character variables (such as program models implemented). We will use cross-tabulations of these statistics to explore potential relationships between variables, such as whether participants’ perceptions of the programs’ influences differ by participant’s age or other characteristics. However, we will not link the entry and exit survey data, so the analyses will not include any cross-tabulations of youths’ past behaviors with their perceptions of program influences. In later reporting years, we will examine changes in the performance measures data over time. We will conduct analyses separately for each of the SRAE program’s three funding streams (DSRAE, SSRAE, and CSRAE).

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.  

B3. Methods to Maximize Response Rates and Deal with Non-Response

Instruments 1-2: Participant Entry and Exit Surveys. We will maximize response rates for participant surveys through the administration of entry surveys to all participants at enrollment and administration of the exit surveys during final program sessions. Where feasible, exit surveys will be administered on an individualized basis to participants at program exit who are absent during final sessions when the surveys are completed. As noted previously, we expect the response rate for both surveys to be 95 percent. As indicated in Table B1.1, the estimated number of respondents is less for the exit survey because we expect about 20 percent of the participants to drop out of the program prior to completion.

Instruments 3-4: Performance Reporting System Data Entry Form and Subawardee Data Collection Reporting Form. To reduce grantee burden and maximize grantee response rates, ACF is providing common data element definitions across program models and collecting these data in a uniform manner through the SPMP (see Instruments 3-4). The submission of the performance measures data is a grant requirement, except in the cases when grantees receive waivers for the sensitive questions on the participant entry and exit surveys. As such, ACF does not expect problems with non-response. We expect the grantee and sub-awardee response rates to be 100 percent.

B4. Test of Procedures or Methods to be Undertaken

We conducted cognitive pretesting with nine youth ages 12 to 18 for the Participant Entry and Exit Surveys (Instruments 1 and 2). The cognitive pretest sample included males and females, as well as youth from a mix of racial and ethnic backgrounds. We revised the survey questions based on the results of these tests. The current submission reflects those revisions.

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Grantees and their subawardees will collect performance measures data. In some cases, grantees might engage local evaluators who will assist them in performance measure data collection. Grantees will report these data into the data warehouse that will be maintained by ACF’s contractor. The contractor will use these data to analyze SRAE performance data and to generate performance measurement reports for ACF.

In developing the performance measures, ACF consulted with FYSB program staff (grantees’ project officers), select SRAE grantees, and Administration for Children and Families and Office on Adolescent Health leadership to obtain their feedback on the proposed measures.

1 Based on our experience with similar performance analysis studies, we estimate that 80 percent of youth in SRAE programs will complete the program.

2 The 137 grantees include 50 DSRAE, 39 SSRAE, and 48 CSRAE grantees.