Alternative Supporting Statement for Information Collections Designed for Research, Public Health Surveillance, and Program Evaluation Purposes
Success Sequence Interviews – Pre-Test
Pre-Testing of Evaluation Data Collection Activities
0970 – 0355
Supporting Statement
Part A
September 2021
Submitted by:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officer:
Caryn Blitz
Type of Request:
This information collection request (ICR) is for a generic information collection under the umbrella generic, Pre-Testing of Evaluation Data Collection Activities (0970-0355).
Description of Request:
The goal of this generic ICR is to support pre-testing of interview instruments for use in the forthcoming Success Sequence Interviews study. We will submit a full ICR for the study to the Office of Management and Budget (OMB). This proposed pre-test focuses on reaching a diverse sample of individuals, including Spanish speakers, to ensure the interview questions are clear, uniformly understood, and not overly sensitive. Pre-test data are not intended to produce generalizable statistical information.
Time Sensitivity:
ACF will use the pre-test data from this generic information collection to refine the interview protocol and data collection design before submitting a full request for this study. To remain on track for the study timeline, pre-testing must be completed in September 2021. This schedule will enable us to begin the full request process in fall 2021.
The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seeks approval to pre-test screening and interview instruments related to the Success Sequence Interviews study. This proposed generic information collection meets the primary goals of ACF’s generic clearance for pre-testing (0970-0355): to develop and test information collection instruments and procedures.
As part of the federal government’s ongoing efforts to support youth in making healthy decisions about their relationships and behavior, Congress updated Title V, Section 510 of the Social Security Act in February 2018 to authorize the Sexual Risk Avoidance Education (SRAE) grant program. SRAE—which is administered by the Family and Youth Services Bureau of ACF, HHS—funds programs that teach adolescents to refrain from sexual activity. One topic SRAE-funded programs are required to cover is the “success sequence.”1 The success sequence is a set of life milestones that, if followed in the prescribed order, are theoretically associated with economic self-sufficiency in young adulthood.2 The milestones listed in their prescribed order are high school graduation, full-time employment, and getting married before having children.
To expand the limited available evidence on the success sequence pathways and their effect on economic self-sufficiency, ACF contracted with Mathematica to conduct a literature review followed by analyses of longitudinal data. The purpose of the literature review was to summarize the current state of research related to the success sequence, including how researchers define the success sequence, success sequence milestones and the order in which they are achieved, and the relationship between the ordering of the milestones (pathways) and economic outcomes.3 The literature review demonstrated a need for more research on the success sequence—in particular, how different pathways shape outcomes such as economic self-sufficiency.
Subsequent analyses used data from the National Longitudinal Survey of Youth’s 1997 Cohort to identify the pathways young adults followed most commonly, variations by demographic characteristics, and associations between those pathways with economic self-sufficiency.4 Findings revealed that, for some individuals, completing the success sequence milestones in the prescribed order is associated with economic self-sufficiency in young adulthood. However, the economic analyses also pointed to other pathways associated with economic self-sufficiency. The findings demonstrate that adolescents’ trajectories to adulthood are diverse and complex, signaling a need for more research in this area.
The Success Sequence Interviews study is descriptive and will help fill gaps in the findings of the literature review and the analyses of the longitudinal data set. Data for the study will be collected through two instruments: (1) a screener and (2) an interview and debriefing protocol. This request is for approval to pre-test the two instruments. ACF will use the pre-test data to refine the instruments before submitting a full ICR for the Success Sequence Interviews study to OMB. This proposed pre-test is necessary to ensure we develop and test the instruments and data collection procedures with a diverse sample. The pre-test sample will be diverse demographically and across life experiences, and it will include Spanish-speaking participants. ACF has contracted with Mathematica to conduct the Success Sequence Interviews study.
There are no legal or administrative requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.
The primary goals of the pre-test are to ensure that questionnaire items are appropriate for a diverse audience of adults ages 30 to 35 with a range of life experiences related to the success sequence milestones, and that the items can elicit a diverse range of responses. The instrument is designed to collect data on a wide range of participants’ possible life experiences, requiring a number of different questions to capture the variations in pathways found in the economic analysis report.5 The report shows that participants followed nearly all (64 out of 65) of the possible sequences of milestone completion. Moreover, the report found that the most common combinations and sequences of milestones vary by gender, race, ethnicity, and parents’ education. For example, the proportion of young adults who reported having completed all four milestones by age 30 was higher for females than males (33.9 percent versus 27.7 percent), higher for non-Hispanic Whites (34.7 percent) than non-Hispanic Blacks (19.4 percent) and Hispanics (28.5 percent), and higher for young adults whose parents had some college education (32.8 percent) than those whose parents did not attend college (29.0 percent).
In addition to ensuring a diverse sample tests the clarity and understandability of the interview questions, the pre-testing will also enable the research team to test the flow and administration of the interview protocol across a large enough pool of participants to ensure respondents have pursued different pathways and have achieved varying economic self-sufficiency to address the study research questions. We will use this information to improve the wording and ordering of items in the interview protocol. ACF will incorporate findings from the pre-test into instruments submitted to OMB for a full ICR for the Success Sequence Interviews study.
The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision maker, and it is not expected to meet the threshold of influential or highly influential scientific information.
ACF proposes to examine the following guiding questions for this pre-test:
Are the instructions to participants clear on how to use the virtual platform?
How long does it take respondents to respond to the interview and subsequent probes from the interviewer?
Are the questions on the instruments appropriate for a diverse population?
Are the questions administered in a way that captures rich and detailed data on the various pathways?
Are the questions and response options clearly written, easy to understand, and free of bias?
Are the Spanish-language questions clearly written, easy to understand, free of bias, and capturing the intended information?
If questions are misunderstood, how can we improve the wording so participants better understand what we are asking?
The research team at Mathematica will work with market vendor the Schlesinger Group to recruit pre-test participants. To fully test the success sequence milestones described in Section A1, participants will be recruited to ensure variation in race, ethnicity, gender, and geographic location (information on participant selection is detailed in Supporting Statement Part B, Section B2). The Mathematica research team will test all instrument sections and questions with individuals experiencing lower levels of economic self-sufficiency and education (for example, participants who did not finish high school), and implement and test a Spanish version of the instrument with Spanish-speaking participants. (Supporting Statement Part B, Section B3 provides more details on the initial instrument development.)
The design of this two-phase pre-test consists of (1) pre-testing the study instruments through virtual asynchronous interviews with as many as 20 participants, (2) gathering feedback from participants throughout and at the end of the interview, (3) revising the instrument and procedures based on that feedback, and (4) repeating the Steps 1 through 3 with 20 additional participants. Appendix A provides the materials for pre-test recruitment, and Appendix C provides an invitation email that we will send to screened participants.
See Supporting Statement Part B for additional information about the study design, including appropriateness for intended use. Results from the pre-test will be used to improve and update the study instruments and are not intended to be representative of or generalizable to a given subpopulation.
Table A.1 summarizes the study design, including the two data collection instruments, their content and respondent types, and the mode and duration of each data collection activity.
Table A.1. Study design summary
Data collection activity |
Instruments |
Participant, content, purpose of collection |
Mode and duration |
Pre-testing administration of Success Sequence instruments |
Instrument 1: Success Sequence Pre-Test Screener |
Participants: Adults ages 30–35 who speak English and Spanish Content: Focuses on demographics, life milestones, and income Purpose: To screen potential participants for the pre-test of the interview protocol to ensure that it meets objectives for respondent diversity (see Supporting Statement Part B, Section B2 for details about objectives) |
Mode: Telephone Duration: 5 minutes |
Pre-testing administration of Success Sequence instruments |
Instrument 2: Success Sequence Pre-Test Interview and Debriefing Protocol |
Participants: Total sample of 40 adults ages 30–35, across two phases of pre-test; includes English and Spanish speakers; the pre-test focuses on testing the protocol questions, and specifically questions on life milestones to ensure the items are clear, uniformly understood, and not overly sensitive Content: Focuses on life history, education, employment, cohabitation, marriage, childbearing, economic self-sufficiency, and rationale for decision making Purpose: To develop and test virtual administration of data collection instruments |
Mode: Individual chat board interviews and debriefings Duration: 1 hour |
This pre-test will inform data collection instruments for a future Success Sequence Interviews study. No other data sources or information will be used for this process.
The pre-test interview protocol (Instrument 2: Success Sequence Pre-Test Interview and Debriefing Protocol) is programmed through QualBoard, a chat board platform that imposes a smaller burden than other interview methods.6 In addition, the online message board permits respondents to maintain anonymity when discussing their life histories with the interviewer and allows more flexibility in scheduling the interview. The anonymous nature of the message board will allow pre-test participants to provide honest feedback when responding to the interview protocol.
The Success Sequence Interviews study, forthcoming in a full OMB ICR, will be the first effort to collect qualitative data on the success sequence model. ACF has carefully reviewed the information collection requirements, and no other federal or non-federal studies are collecting the same type of data.
No small businesses will be involved with this information collection.
This is a one-time data collection.
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and OMB regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to submit a request to OMB to review the overarching generic clearance for pre-testing activities. This notice was published January 5, 2021, in Volume 86, Number 2, on page 308, and provided a 60-day period for public comment. During the notice and comment period, no substantive comments were received. A second notice in the Federal Register announced the agency’s submission of the overarching generic clearance for pre-testing activities for OMB’s review. This notice was published May 21, 2021, in Volume 86, Number 97, on page 27624, and provided a 30-day period for public comment. During the notice and comment period, no comments were received.
For these pre-testing activities, we will not be seeking consultation from experts outside of the study.
To recruit participants and to conduct the pre-testing activities efficiently, we plan to offer pre-test participants a $40 Visa gift card. These tokens of appreciation will help ensure participants reflect backgrounds and characteristics similar to those who will eventually participate in the Success Sequence Interviews study, and we will purposefully select participants with varying income levels. The research team plans to offset the direct costs of participating in the interviews, such as child care expenses, to decrease the risk that only individuals able to overcome financial barriers will be able to participate in the pre-test. The proposed data collection design, which includes a pre-test that uses a virtual asynchronous method, is expected to offset some of these costs by giving participants flexibility in when they can respond. However, respondents might have little intrinsic motivation to participate in the interview, answer sensitive and personal questions, and provide feedback on the process. Our goal is for the proposed token of appreciation to give them incentive to participate.
The ACF study team for the Success Sequence Interviews pre-test will not collect personally identifiable information (PII). As part of recruitment, the market research vendor will record each participant’s name and email and use this contact information to send the participants invitation emails. The vendor serves as a protective intermediary for PII and responses to interview questions. Neither Mathematica nor ACF will have access to the PII; the vendor will provide Mathematica with unique identifying numbers for recruited interviewees and their responses to the screener questions (Instrument 1: Success Sequence Pre-Test Screener). Using Instrument 1, the vendor will also collect demographic information, such as age, race, and ethnicity, to screen a diverse set of participants. When participants log in to the virtual chat board, they will be instructed to not share their full name, but rather to use their first name, an initial, or an alias.
Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. As the contract specifies, the contractor shall protect respondent privacy to the extent permitted by law and will comply with all federal and departmental regulations for private information. Appendix B provides a pre-test consent form containing assurances of privacy; all participants must read and acknowledge the form before participating in the data collection.
The contractor has developed a data safety and monitoring plan that assesses all protections of respondents’ information. The contractor shall ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract and subcontract, are trained on data privacy issues and comply with the above requirements. All research team staff involved in the study will receive training in (1) limitations of disclosure; (2) safeguarding the physical work environment; and (3) storing, transmitting, and destroying data securely. All Mathematica staff must sign a confidentiality agreement, complete online security awareness training when they are hired, and participate in refresher training annually.
As specified in the evaluator’s contract, the contractor shall use encryption compliant with the Federal Information Processing Standard (Security Requirements for Cryptographic Module, as amended) to protect all sensitive information during storage and transmission. The contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Information Processing Standard. The contractor shall ensure that this standard is incorporated into its property management and control system and establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology requirements and other applicable federal and departmental regulations. In addition, the contractor’s data safety and monitoring plan includes strategies for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive information that ensures secure storage and limits on access.
Some of the interview questions from the interview protocol that we will pre-test might be perceived as sensitive, such as those pertaining to relationship history and childbearing. These questions are necessary for the Success Sequence Interview study to address ACF’s informational needs, namely, to improve SRAE program content related to the success sequence (Section A2). This pre-test is intended to improve the quality of the interview and ensure respondents understand each question, not to capture specific answers.
Before the start of the pre-test, participants will read the consent form (Appendix B), which describes the purpose of the overarching study and the goals of this request to pre-test the interview. The consent form indicates the rights of the research participants and acknowledges their consent before starting the interview. All participants will be informed that they do not have to respond to any questions that make them uncomfortable and that their participation is voluntary.
The protocol and all related materials, such as the consent form, have been approved by an institutional review board.
Table A.2 lists sensitive questions that will be in the interview and the justification for asking them.
Table A.2. Justification for sensitive questions in the interview protocol
Topics and interview questions |
Justification |
Income and economic support
|
A primary goal of success sequence research is to determine whether adults who follow the sequence are economically self-sufficient. These questions are the primary outcome variables and provide context for the variations in income and support as they relate to the sequence of life milestones. |
Relationship history and childbearing
|
To understand participants’ rationale for completing or not completing the success sequence milestones in the prescribed order, it is necessary to understand the circumstances that led to participants’ decisions about relationships, cohabitation, marriage, and childbearing. |
In Table A.3, we summarize the estimated reporting burden and costs for each instrument.
Screener. To reach our sample objectives, we estimate completing an eligibility screener with approximately 120 respondents. Of those, we estimate that 50 percent, or 60 participants, will meet our eligibility criteria for age range and English or Spanish speaker. The eligibility screener will take approximately five minutes (0.083 hours), for a total estimated annual burden of 9.96 hours.
Interview. We estimate that as many as 40 of the eligible online research panel participants will complete the pre-test interview. We estimate that the interview will take one hour for completion via an online chat board, for an estimated annual burden of 40 hours.
We estimate the average hourly wage for the participants of this study is the average hourly wage of “all occupations” taken from the May 2020 National Occupation Employment and Wage Estimates, U.S. Bureau of Labor Statistics ($27.07).8 The estimated burden results appear in Table A.3.
Table A.3. Total burden requested under this information collection
Instrument |
No. of participants (total over request period) |
No. of responses per participant (total over request period) |
Avg. burden per response (hours) |
Total/annual burden (hours) |
Average hourly wage rate |
Total annual participant cost |
Instrument 1: Success Sequence Pre-Test Screener |
120 |
1 |
0.083 |
10 |
$27.07 |
$269.62 |
Instrument 2: Success Sequence Pre-Test Interview and Debriefing Protocol |
40 |
1 |
1.00 |
40 |
$27.07 |
$1,082.80 |
Estimated total annual burden |
|
|
|
50 |
|
$1,352.42 |
There are no additional costs to participants.
The total cost to the federal government for this study is $81,500 (Table A.4). This includes the costs incurred for designing and administering all collection instruments, processing and analyzing the data, and preparing reports.
Table A.4. Estimated total cost by category
Cost category |
Estimated costs |
Instrument development and OMB clearance |
$50,000 |
Data collection |
$24,000 |
Analysis/reporting |
$7,500 |
Total/annual |
$81,500 |
This is for an individual information collection under the umbrella generic clearance for pre-testing (0970-0355).
Pre-testing is expected to begin immediately after OMB approval, and the first round of pre-testing data collection will continue for one week. Using feedback from this initial round of pre-testing, the study team will revise the instrument and conduct a second round of pre-test data collection within two weeks of the first round to test the revisions. The study team anticipates completing both rounds of pre-test data collection within three weeks of OMB approval. We will analyze results immediately following both rounds of pre-testing and report them internally. We will incorporate final revisions into the instruments in preparation for a full ICR.
No exceptions are necessary for this information collection.
Appendices
Appendix A. Success Sequence Pre-Test Recruitment Materials (English and Spanish)
Appendix B. Pre-Test Consent Form (English and Spanish)
Appendix C. QualBoard Invitation Email Sample Screen Shot (English and Spanish)
Instrument
Instrument 1: Success Sequence Pre-Test Screener (English and Spanish)
Instrument 2: Success Sequence Pre-Test Interview and Debriefing Protocol (English and Spanish)
1 Family and Youth Services Bureau. “State Personal Responsibility Education Program Fact Sheet.” Washington, DC: Family and Youth Services Bureau, 2020. Available at https://www.acf.hhs.gov/fysb/fact-sheet/title-v-state-sexual-risk-avoidance-education-fact-sheet.
2 Haskins, R., and I.V. Sawhill. Creating an Opportunity Society. Washington, DC: Brookings Institution Press, 2009.
3 Goesling, B., H. Inanc, and A. Rachidi. “Success Sequence: A Synthesis of the Literature.” OPRE Report #2020-41. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2020.
4 Inanc, H., A. Spitzer, and B. Goesling. “Assessing the Benefits of the Success Sequence for Economic Self-Sufficiency and Family Stability.” OPRE Report #2021-XX. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
5 Inanc, H., A. Spitzer, and B. Goesling. “Assessing the Benefits of the Success Sequence for Economic Self-Sufficiency and Family Stability.” OPRE Report #2021-XX. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
6 Opdenakker, R. “Advantages and Disadvantages of Four Interview Techniques in Qualitative Research.” Qualitative Research in Ibero America, vol. 7, no. 4, 2006, p. 7.
7 Examples of sensitive topics include Social Security numbers; sex behavior and attitudes; illegal, anti-social, self-incriminating, and demeaning behavior; critical appraisals of other individuals with whom participants have close relationships (for example, family, students and teachers, employees and supervisors); mental and psychological problems; religion and indicators of religion; community activities that indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those with lawyers, physicians, and ministers; records describing how an individual exercises their First Amendment rights; receipt of economic assistance from the government (for example, unemployment insurance, or WIC or SNAP benefits); and immigration or citizenship status.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Jen Walzer |
| File Modified | 0000-00-00 |
| File Created | 2024-07-25 |