OMB supporting statement part A OF STREAMS pre-test (DraFT)
Pre-testing Request for Strengthening Relationship Education and Marriage Services (STREAMS)
Generic Clearence for Pre-testing of Evaluation Data Collection Activities
0970 - 0355
Supporting Statement
Part A
August 2018
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
330 C Street, SW, 4th Floor
Washington, DC 20201
The Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services seeks approval to conduct pre-tests with staff at two Healthy Marriage and Relationship Education (HMRE) programs funded by the Office of Family Assistance (OFA) within ACF. This information collection is being carried out as part of the Strengthening Relationship Education and Marriage Services (STREAMS) project. This part of the project aims to test and refine training and coaching strategies to address common needs of HMRE programs working with youth. This information collection (IC) will involve pre-testing a data collection instrument with two HMRE programs serving youth to develop and refine it for use in a potential future evaluation of the training approaches and materials. Approval for this data collection is requested under ACF’s generic clearance for pre-testing (0970-0355).
The Deficit Reduction Act of 2005 created the HMRE grant program, which authorized $150 million over five years to support program activities aimed at promoting and sustaining healthy marriages, providing relationship education services to youth, and fostering economic stability. The Claims Resolution Act of 2010 re-authorized this grant program, and three-year grants totaling $150 million were awarded in September 2011 (and subsequently extended through September 2015). In October 2015, ACF awarded five-year grants to 46 HMRE grantees, including 31 grantees serving youth ages 14-24. Youth-serving grantees focus on teaching skills to promote healthy relationships, including conflict resolution, problem solving, goal-setting, and communication skills, and may also integrate job readiness and financial management skills such as budgeting, resume writing, and interviewing skills.
Despite widespread implementation of HMRE programs for youth, relatively little research has been conducted on training or other approaches to improving the quality of program delivery. HMRE programs for youth offer group-based programming using a standard curriculum in both school- and community-based settings. Some programs also offer case management services. Programs typically provide training on the HMRE curricula, but vary in how they monitor and support facilitators. In a recent study of HMRE programs for youth, most program directors reported that facilitators needed more training on group facilitation skills, program delivery skills, and youth engagement strategies (Scott et al. 2017).
As part of STREAMS, they study team is developing and piloting training approaches and materials to enhance the facilitation and youth engagement skills of HMRE youth facilitators. As part of the pilot, program facilitators in two sites will receive up to 15 hours of classroom training on group facilitation strategies across three training sessions. Each training session will be delivered at the start of a school semester and will include such topics as participant-centered facilitation; building trust and safety; debriefing/drawing out teachable moments; managing energy in the classroom; trauma-information facilitation; addressing personal bias and cultural sensitivity; climate-building in the classroom; and managing personal disclosure. After receiving classroom training, facilitators will receive feedback and coaching to help facilitators learn to use the strategies.
STREAMS will use an iterative process to develop the training approaches and materials and pilot them in a series of three trainings with two HMRE programs serving youth. This data collection effort will pre-test a data collection instrument to develop and refine it for use in a potential future evaluation of the training approaches and materials developed under STREAMS. ACF contracted with Mathematica Policy Research and Public Strategies to conduct the STREAMS project.
This is a discretionary data collection authorized under Sec. 811 (b) Healthy Marriage Promotion and Promoting Responsible Fatherhood Grants of the Claims Resolution Act of 2010, Pub. L. No. 111-291, 124 Stat. 3064 (Dec. 8, 2010). A copy of the legislative authority is included as Attachment A.
The STREAMS team will develop and pre-test one data collection instrument and procedures for use in a potential future evaluation of the training approaches and materials. The purpose of the pre-test is to evaluate and improve the quality of the data collected using the instrument. ACF/OPRE will use the results internally to inform subsequent information collection requests. The results of these pre-tests may be used in reports on instrument development or instrument user guides. The data collected will not be presented as findings on implementation or effectiveness of the training approaches and materials.
The pre-test will be conducted with purposive samples of ACF-funded HMRE program staff. All data collection activities conducted under this generic clearance will be voluntary and low burden.
Under this clearance, ACF will pre-test one data collection instrument with staff from two HMRE programs for youth. We will conduct pre-tests of a preliminary version of a facilitator self-assessment form after delivery of each of three training workshops, each focusing on a different facilitation strategy. The results of these pre-tests will be used to refine questionnaire items and scales to prepare instruments for a possible future evaluation.
We are submitting a master version of the instrument. Grantees will only need to respond to a subset of the questions, corresponding to the training topics they choose along with the study team once the study is underway. If this iterative pre-testing results in changes to the instrument, ACF will upload revised materials to ROCIS as a non-substantive change between each round of testing and will provide a memo describing the rationale for the changes.
Clearance is being requested to pre-test the Facilitator self-assessment form (Instrument 1). The purpose of this form, to be completed by facilitators after group workshops with youth, is to collect information about facilitator self-reported use of behaviors taught in the training to support effective facilitation, including frequency and ease of use. Facilitators will also be asked to identify areas in which they need additional training.
For ease of completion, the facilitator self-assessment form will be available to staff as an online form created using Survey Monkey. We will provide a link via email that HMRE program staff can use to access and complete the forms using a tablet, smartphone, or laptop. Survey Monkey is appropriate for this data collection effort because we are not collecting any personally identifiable information (PII) or sensitive information. All online forms will also be available as paper-and-pencil forms, in case data entry via a tablet, smartphone, or laptop is not feasible.
The STREAMS team is not collecting any information that is available elsewhere. None of the instruments ask for information that can be reliably obtained through other sources.
The HMRE grantees participating in the study are community-based organizations. The STREAMS team will only request information required for the intended use. We will minimize burden by restricting the surveys to only information that is critical to the pretest.
The purpose of this data collection is to pre-test a data collection instrument for use in a potential future evaluation of training approaches and materials with OFA grantees. Iterative pre-testing with small groups of staff is needed to test and refine the data collection instrument. Collecting the data less frequently would prevent the study team from being able to adequately test each instrument with small groups of staff from different HMRE programs serving youth in different program settings.
There are no special circumstances for the proposed data collection.
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13 and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995)), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of the overarching pre-testing generic information collection activities. This notice was published on October 20, 2017; Volume 82, Number 202, page 48820, and provided a sixty-day period for public comment. During the notice and comment period, no substantive comments were received.
STREAMS consulted with four experts on HMRE programs for youth in May 2015, during the first year of the project.
Table A.1 STREAMS expert consults on youth programming
Name |
Affiliation |
Shari Miller |
RTI International |
Mindy Scott |
Child Trends |
Renee Sieving |
University of Minnesota |
David Wolfe |
University of Toronto |
No incentives for respondents are proposed for this information collection.
Information collected will be kept private to the extent permitted by law. Only staff from Mathematica will handle data collected under this clearance. Public Strategies staff will not be involved in data collection or analysis. All study team staff involved in the project will receive training on (1) limitations of disclosure; (2) safeguarding the physical work environment; and (3) storing, transmitting, and destroying data securely. All Mathematica staff sign the Mathematica Confidentiality Agreement (see Attachment B), complete online security awareness training when they are hired, and receive annual refresher training thereafter. Training addresses security policies and procedures found in the Mathematica Corporate Security Manual.
The instrument to be pre-tested will not collect any PII. To further protect the privacy of the participants, included at the beginning of each questionnaire is a reminder for respondents to not include any personal information such as names and contact information in their responses. A study identification number will be used to identify each HMRE program staff member participating in the study. Staff will provide this identification number when they complete the facilitator self-assessment form. All analysis files will contain only the study identification numbers and no identifying information.
There are no sensitive questions in this data collection.
Table A.2 summarizes the estimated reporting burden and costs for the facilitator self-assessment form. After each of three training sessions, we expect HMRE program facilitators to complete the form 3 times per week for 6 weeks (3 forms * 6 weeks * 3 training sessions = 54 forms). There will be 8 facilitators per site and 2 sites (54 forms * 8 facilitators * 2 sites = 864 forms). Each form will take 12 minutes to complete. Total burden for the facilitator self-assessment form is 173 hours.
We estimate the average hourly wage for staff at the HMRE programs is the average hourly wage of “social and service managers” taken from the U.S. Bureau of Labor Statistics, National Compensation Survey, 2010 ($27.86).
Table A.1 Total burden requested under this information collection
Instrument |
Total annual number of respondents |
Number of responses per respondent |
Average burden hours per response |
Annual burden hours |
Average hourly wage |
Total annual cost |
|
Pre-Test 1 |
|||||||
1. Facilitator self-assessment form |
16 |
54 |
0.2 |
173 |
$27.86 |
$4819.78 |
|
There are no additional costs to respondents.
The cost for data collection under this current request will be $134,311.
This is an information collection request under generic clearance 0970-0355.
The purpose of this clearance request is for pre-testing a data collection instrument and procedures to evaluate and improve their quality for use in a potential future evaluation of training approaches and materials with OFA grantees.
The project team will summarize quantitative data from the facilitator self-assessment form using basic descriptive methods such as frequencies and means. Analysis of open ended questions will involve coding and theme identification. The results will be used to assess the suitability of the data collection instrument and procedures for studying the implementation and effectiveness of the training materials.
The information collected under this clearance will not be the primary subject of any published ACF reports; however information may be made public through methodological appendices or footnotes, reports on instrument development, or instrument user guides. When necessary, results will be labeled as exploratory in nature.
The pre-testing will begin in September 2018, after obtaining OMB approval, and continue through December 2019. Pre-test results will be reported to OPRE in an internal memo in April 2020.
All instruments will display the expiration date for OMB approval.
No exceptions are necessary for this information collection.
Scott, Mindy E., Elizabeth Karberg, Ilana Huz, and Maryjo Oster. Healthy Marriage and Relationship Education Programs for Youth: An In-depth Study of Federally Funded Programs. OPRE Report #2017-74. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2017.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Diane Paulsell |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |