SSB - PREP Local Evaluation Support Instruments

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

PREP Local Evaluation Support Instruments

Formative Data Collections for Program Support

0970 - 0531

Supporting Statement

Part B

October 2024

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer:

Selma Caal (COR)

Part B

B1. Objectives

Study Objectives

The Office of Planning, Research, and Evaluation (OPRE) at the Administration for Children and Families (ACF) under the U.S. Department of Health and Human Services (HHS) proposes to conduct an online survey with up to 85 State PREP and Competitive Personal Responsibility Education Program (PREP) grantees, as well as to have all 12 PREP in Education Innovative Strategies program (PREIS) grantees complete an analysis plan template and a sample enrollment tracking tool. The purpose of these data collection instruments is to learn more about grantees’ evaluations in order to provide targeted technical assistance (TA). Information from the instruments will be used by OPRE’s PREP Local Evaluation Support team (Abt Global (Abt) and their subcontractor Child Trends) to understand grantees’ current and future evaluation activities, and to improve the program evaluation TA provided by Abt and their subcontractors.

Generalizability of Results

This study is not intended to promote statistical generalization to other programs or populations. The State PREP and Competitive PREP Grantee Evaluation Activities Survey (Instrument 1) is meant to understand State PREP and Competitive PREP grantees’ program evaluation plans and to understand grantees’ overall program evaluation TA needs. The PREIS Analysis Plan Template (Instrument 2) is to understand PREIS grantees’ planned analysis, including data preparation and analytical approach. The PREIS Sample Enrollment Progress Tracking Tools (Instruments 3 and 4) are designed to understand PREIS grantees’ current sample enrollment and survey completion numbers in order to track their overall progress towards achieving target sample sizes.

Appropriateness of Study Design and Methods for Planned Uses

A survey using a combination of multiple choice and open-ended questions is the best data collection approach to assess State PREP and Competitive PREP program evaluation plans and interest in receiving evaluation support. Multiple-choice questions will provide descriptive data that is collected in a consistent way across programs. Open-ended survey items provide programs the opportunity to include more detailed and nuanced responses that will add context to quantitative data. Surveys can be completed electronically and are of relatively low burden to respondents but allow for the perspectives of a variety of staff from many programs to be captured and used to inform future program evaluation support activities.

A template for PREIS grantees’ analysis plans is the best data collection approach for grantees to use to document their planned analysis to ensure that grantees are reporting information in a systematic way to allow TA to be conducted in a systematic, comprehensive way. This will help to ensure grantees receive relevant and timely support to allow them to conduct analyses in a rigorous fashion that meets the Standards of Rigor developed under this contract that serves as benchmarks for standards grantees should be achieving.

A tool for PREIS grantees’ to complete to document their sample enrollment is the best data collection approach to supporting grantees to collect information in a systematic way that allows grantees to appropriately track sample enrollment and follow-up. By having grantees use a systematic, uniform tool, this will aid grantees later when they are asked to compile their final results into a report. This will also help to ensure grantees have a systematic way of monitoring sample enrollment to ensure they have enough power to later detect impacts.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.  

B2. Methods and Design

Target Population

The target population for the State PREP and Competitive PREP Grantee Evaluation Activities Survey (Instrument 1) are all State PREP and Competitive PREP grant recipients who received funding as of September 30, 2024 (a maximum of 85 grantees).

The target population for the PREIS Analysis Plan template (Instrument 2) and the PREIS Sample Enrollment Progress Tracking Tools (Instruments 3 and 4) are all 12 PREIS grantees.

Respondent Recruitment and Sampling

There is no specific site selection for the instruments in this data collection request. For the State PREP and Competitive PREP Grantee Evaluation Activities Survey, the study team will contact program directors of all the newly funded State PREP and Competitive PREP programs about the survey. The survey will be administered remotely, and grantees are spread throughout the U.S. and its territories.

FYSB will provide contact information for each of the State PREP and Competitive PREP programs. The study team will then follow up with each grantee with an invitation to complete the survey (see Attachment A). Program directors will identify the appropriate staff to complete the survey. Only one (collective) response per grantee will be requested.

For the PREIS Analysis Plan template and The PREIS sample enrollment Progress Tracking Tools, all PREIS local evaluators will receive a copy of these instruments via email (see Attachment D and E). Additionally, on local evaluation support calls, these instruments will also be introduced to ensure all PREIS grantees have access to these template and tools.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

The State PREP and Competitive PREP Grantee Evaluation Activities Survey was developed by a team of researchers familiar with the current State PREP and Competitive PREP grantees and based on the team’s understanding of the different types of evaluation-related activities that are possible. The survey was reviewed by the study team, including OPRE project officers and FYSB program officers. The study team also conducted cognitive testing of the survey with 3 current State and Competitive grantees (see A8 for additional information pertaining to who reviewed). The survey was streamlined to include only questions necessary to achieve the objectives of the data collection. See Instrument 1 for the full survey instrument.

A team of researchers used preexisting templates and tools used by the government for other contracts as a starting point to design the PREIS Analysis Plan template and the PREIS sample enrollment Progress Tracking Tools. The template and tools were modified accordingly to improve the utility and efficiency of the government’s review of completed templates. The template and tools were reviewed by a subset of PREIS grantees to ensure they were not too burdensome and were useful (see A8 for additional information pertaining to who reviewed). See Instruments 2 – 4 for the full instruments.

B4. Collection of Data and Quality Control

Abt, the prime contractor for this study, will be responsible for the administration and data monitoring of data collection instruments included in this information request.

State PREP and Competitive PREP Grantee Evaluation Activities Survey

Outreach to State PREP and Competitive PREP program staff:

All respondent recruitment will take place via email (see Appendix A for a copy of the email text). A representative from FYSB will email State PREP and Competitive PREP program directors to provide information about the survey. Abt will follow up with the program directors to provide additional information and instructions for completing the survey. Ultimately, each program director will be asked to identify one person from their program to complete the survey using the online survey platform, such as Confirmit. The individual who is responsible for completing the online survey should seek input from additional staff (e.g., local evaluation partners, data entry staff) as needed. The study team will plan to send up to two additional emails to program directors reminding them to complete the survey.

Survey administration:

Abt will share the survey link with State PREP and Competitive PREP staff via email (see Appendix A for a copy of the email text). The survey will be programmed into a secure, online platform, such as Confirmit. Program staff will have the opportunity to complete the survey at their convenience within a one-month timeframe. Contact information for the study will be shared with the survey to allow respondents to reach out with any clarifying questions regarding survey completion.

Data monitoring:

The survey will be programmed to safeguard against major data quality issues. For example, for multiple choice questions where there is only one appropriate response, the survey will not allow more than one response option to be selected. Skip patterns will also be programmed so that respondents are not asked questions that are not relevant to their project.

PREIS Analysis Plan template and the PREIS sample enrollment Progress Tracking Tools

Outreach to PREIS grantees:

All respondent recruitment will take place via email (see Appendix D and E for a copy of the two emails). The grantees’ local evaluation support point of contact from Abt will email the PREIS local evaluations to provide information about the template and tools. Ultimately, the local evaluators are responsible for completing the template and tools, and they will be encouraged to reach out to their local evaluation support point of contact with any questions.

Completion of the template and tool:

Abt will walk the grantees’ through the template and tools both through a prerecorded webinar, as well as on local evaluation support calls to help assist the grantees in completing the template and tools. PREIS local evaluators will ultimately complete the full template and tools and email them to their local evaluation support of contact.

Data monitoring:

Once grantees have completed the template and tools, the local evaluation support point of contact will review the information and then follow-up with the grantees’ for any additional information needed or any clarification needed to ensure the information provided is complete, accurate and of high quality.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

None of the instruments in this information request are designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Response rates will not be reported publicly. For the State and Competitive PREP Grantee Activities Survey, response rates will be calculated and reported internally to OPRE and FYSB. Response rates will not be calculated for the PREIS template and tools, as we anticipate 100% completion.

NonResponse

As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated.

B6. Production of Estimates and Projections

The data will not be used to generate population estimates, either for internal use or dissemination. The data collected will only be used to describe evaluation-related activities and interest in future program evaluation support and to develop plans to provide evaluation support.

B7. Data Handling and Analysis

Data Handling

All survey responses will be collected using a secure, online platform, such as through Confirmit. The survey will be programmed with validation checks to reduce errors (e.g., for questions where respondents should select only one answer, the survey will be programmed such that only one response can be selected). Data will be downloaded directly and imported into a data analysis software, Stata, reducing any errors associated with manually transferring data.

The PREIS Analysis Plan template and the Sample Enrollment Progress Tracking Tools will be emailed to the local evaluation point of contact. Requested revisions to the template or tools will be sent back to the grantee via email.

Data Analysis

For quantitative survey items on the State and Competitive PREP Grantee Activities Survey, the study team will run descriptive statistics using the Stata software. Where appropriate, the study team will also run cross-tabulations to examine any differences across funding stream, program characteristics, or other meaningful subgroups. The study team will then review and summarize program responses to open-ended questions.

For the PREIS Analysis Plan template and the Sample Enrollment Progress Tracking Tools, no data analysis will occur.

Data Use

The research team will not release any data from this collection to the public. The data collected will be for internal use only (either internal to the project team and ACF or internal to the grantees via the secure website) only to inform real time TA. The findings from the State and Competitive PREP Grantee Activities Survey will be shared with ACF in a memo that will include descriptive statistics and summarize qualitative responses. Upon receiving written guidance from ACF, Abt will destroy all data using an electronic file shredder.

B8. Contact Persons

Team members from Abt and Child Trends will lead the data collection and analysis for this information request. The Project Director for the contract is Michelle Blocklin ([email protected]). The federal point of contact from OPRE is Selma Caal ([email protected]).

.

Attachments

Instrument 1 - State PREP and Competitive PREP Grantee Evaluation Activities Survey

Instrument 2 – PREIS Analysis Plan Template

Instrument 3 – Sample Enrollment Progress Tool – Individual Assignment

Instrument 4 – Sample Enrollment Progress Tool – Cluster Assignment

Appendix A - Invitation email for SPREP/CPREP grantees

Appendix B – Individual Assignment Tool Instructions

Appendix C – Cluster Assignment Tool Instructions

Appendix D – PREIS Analysis Plan Template email

Appendix E – Sample Enrollment Progress Tool email

7