Evaluation of Adolescent Pregnancy Prevention Approaches (PPA)
Pre-testing of Evaluation Surveys
(OMB 0970‑0355)
Supporting Statement Part A
This section provides support for an early stage of information collection in the Evaluation of Adolescent Pregnancy Prevention Approaches (PPA) demonstration and evaluation. Specifically, this submission seeks clearance for an expanded pretest of the two planned baseline survey instruments, one for sexually active and one for sexually inactive youth. Under routine OMB rules, the Administration for Children and Families (ACF) would conduct a pretest of each instrument with nine individuals. ACF seeks approval for a pretest of each baseline instrument with 25 youth and the use of cognitive interview guides to debrief the pretest participants. The expanded pretest and cognitive interviews will allow us the opportunities to talk with a total of 50 youths of different ages and genders, individually and in a group setting, 25 who are sexually active and 25 who are not. The expanded pretest and cognitive interviews will allow us the opportunities to talk with youths of different ages and genders, individually and in a group setting, and to include youths with different levels of sexual experience. As indicated in the clearance for pre-testing, 0970-0355, all information gathered from these testing activities will be for the purpose of improving data collection instruments and procedures, not for the purpose of generating findings on the substantive topic under study. No information collected in this pretest will be released to the public. ACF will make a separate submission for clearance of the baseline survey instruments after they are revised based on this pretest, as well as for all other instruments to be used in subsequent follow-up data collection with the evaluation sample.
The Administration for Children and Families (ACF) in the U.S. Department of Health and Human Services (HHS) seeks to identify and evaluate promising approaches to reduce teen pregnancy. Preliminary data for 2005 show that teen birth rates have recently increased after several years of decline. The negative outcomes associated with teen pregnancy for both parents and their children are well documented in the research literature. The PPA evaluation will assess the effectiveness of a range of promising approaches to pregnancy prevention, primarily among high-school-age teens. The evaluation will employ an experimental design, and be carried out in up to eight sites with a total sample of approximately 10,800 youth. The overall goal of the Pregnancy Prevention Approaches (PPA) evaluation is to determine the effectiveness of these approaches in affecting key outcomes related to pregnancy prevention (e.g., sexual debut, pregnancy, STD infection). Ultimately, the purpose of the evaluation is to provide stakeholders—including practitioners and federal and other policymakers—with sound scientific findings on a range of existing and promising approaches to preventing teen pregnancy.
The evaluation will involve collection of data from youth at baseline, when they are enrolled in the sample, and at two follow-up points after assignment to a research group to measure outcomes. Requests for clearance and justification statements will be submitted for all of these instruments in stages. Development of the baseline instruments is now under way, and ACF wishes to conduct a thorough pretest of them with a respondent sample larger than nine subjects. This justification statement seeks to apply to this particular circumstance the general approval received from OMB (0970-0355) for such expanded pretests. ACF seeks approval for an expanded pretest with 25 sexually active youth and 25 non-sexually-active youth and the use of cognitive interview guides to debrief the pretest participants. The expanded pretest and cognitive interviews will allow us the opportunities to talk with youths of different ages and genders, individually and in a group setting, and to include youths with different levels of sexual experience.
Public Law 110-161, which set FY 2008 appropriations levels, included the following language: “$4,500,000 shall be available from amounts available under section 241 of the Public Health Service Act to carry out evaluations (including longitudinal evaluations) of adolescent pregnancy prevention approaches.” The proposed pretest activity will enable ACF to refine the data collection instruments and protocols related to the survey effort for this study, in support of the goals of section 241.
As noted above, the overall objective of the PPA is to test selected promising approaches to preventing teen pregnancy, primarily among high-school-age teens, building on past research to expand our knowledge of program effectiveness. For decades, policymakers as well as the general public have remained concerned about the prevalence of sexual intercourse among adolescents. Although adolescents today are waiting somewhat longer before having sex than they did in the 1990s, 60 percent of teenage girls and more than 50 percent of teenage boys report having had sexual intercourse by their 18th birthday.1 Approximately one in five adolescents has had sexual intercourse before his or her 15th birthday.2 While rates of teenage pregnancy declined by 38 percent from 1990 to 2004, the rate of teen births followed a similar decline3 until recently, when the rate of teen births rose by 3 percent from 2005 to 2006.
From 1999 to 2007, the federal Abstinence Education program funded an evaluation of programs funded by Title V, Section 510 of the Social Security Act; this evaluation was overseen by the Office of the Assistant Secretary for Planning and Evaluation (ASPE) in HHS and was carried out by Mathematica Policy Research, Inc. (MPR). This study concluded that students who participated in the four abstinence education programs evaluated were no more likely to abstain from sexual intercourse than those who received services as usual (i.e., the control group). The study also found that students who participated in the abstinence education programs were no more likely to engage in unprotected sex than those who received services as usual.4
The Evaluation of Adolescent Pregnancy Prevention Approaches would build on the earlier Title V study in at least three ways. First, the sites selected for the Title V study focused on elementary and middle school-age youth; the sites selected for the current PPA study will focus primarily on high-school-age youth. Second, the earlier study focused solely on programs funded through Title V, Section 510, often called “abstinence-only” or “abstinence-until-marriage” (AUM) programs; the current PPA study will focus on a range of programs whose goal is to reduce teen pregnancy, including some AUM programs but focusing at least as much attention on comprehensive sex education (programs that instruct students on safer sex practices and abstinence) and STD/HIV education and prevention programs. Lastly, the earlier study involved four sites; ACF expects to select up to eight sites for this evaluation.
The current proposed data collection will be used solely for refinement of the baseline survey instruments and the protocols for their administration. (The draft instruments for sexually active and inactive youth, not yet fully formatted, are attached as two separate parts of Appendix A.) This will be achieved through two debriefing formats (see Appendix B for the debriefing protocol, which will accommodate both formats and both instruments):
Cognitive Debriefing Interviews in 1:1 format. A participant will complete the questionnaire and make notations on any items or instructions that caused confusion or concern. A trained interviewer will record observations of the individual during administration such as: areas where he/she paused, showed confusion, or concern, and whether or not interview instructions were followed correctly. Immediately following completion of the questionnaire, the pair will then go through the instrument and protocol together to debrief on specific items.
Cognitive Debriefing Interview in a group format. Participants will first be asked to complete the questionnaire, making notations on any items or instructions causing confusion or concern. A trained focus group moderator will then collect the completed questionnaires and guide participants through a group discussion on selected topics. Questionnaires will be reviewed at a later time, when participants are not present.
The purpose of this information collection is to help ACF refine the survey instruments that will be used to gather baseline data about the youth enrolled in the PPA evaluation sample. This information will be used by contractor staff to make recommendations to ACF on: (1) specific items where wording needs to be refined or revised, (2) ways to improve the survey protocols to improve the quality of the responses provided by the target population of primarily high-school-aged teens, and (3) areas of importance for training of data collection staff.
The items in this questionnaire were gathered from a variety of sources,5 all deemed valid measures for their topics. However, not all these measures have been fielded in the past with a population of high-school-aged teens, nor have all been fielded in a self-administered format. Furthermore, individual items were modified to meet the needs of the current survey and for consistency within the questionnaire. Therefore, this pretest will identify items that require further refinement and test alternate ways of wording specific items in order to create a final version of the instrument that will provide the information necessary to answer the research questions being addressed by this survey.
The responses provided in the written questionnaires will not be used for any substantive analysis and, as such, will not be entered into an electronic format. This pretest is also designed to mirror the process for the full-scale study, which will be in a self-administered format (using pencil and paper versions of the questionnaire). The information collected through one-on-one or group debriefing sessions are not conducive to information technology, such as computerized interviewing, and thus improved information technology will not be used.
The purpose of the pretest is to evaluate instruments created for this study. While many of the items come from other national studies, the modifications and combination of items and self-administered format have not been tested together before. No existing data sources can provide the complete information needed.
No small businesses are impacted by data collection in this pretest.
During this step of the evaluation, information will be collected only once, thus no repetition of effort is planned. Not collecting the information at all would substantially limit the value of the investment ACF will make in this study. Pretesting the survey with the target population is crucial to ensuring the final instruments gather high quality data—by making the protocols logical and easy-to-follow, and wording the items in ways that yield data that answer the research questions. Given the likely variety of prior sexual experience of evaluation sample members, their age, and their gender, it is important to expand the pretest as proposed to ensure a thorough understanding of how different subgroups will understand and respond to the questionnaire items.
There are no special circumstances requiring deviation from these guidelines.
The first Federal Register notice for ACF’s generic clearance for pretesting was published in the Federal Register, volume 73, no. 39, p. 21957 on April 23, 2008. The second Federal Register notice was published in the Federal Register, volume 73, no. 137, p. 44271 on July 30, 2008.
We will provide an honorarium of $50 to each participant for their time and participation in the individual or group interview. The funds will be dispersed in cash, immediately following the 1.5 hour session.
This level of incentive is appropriate for several reasons. First, the pretest will require a substantial time commitment and impose burden. The pretest activity is expected to take one and one-half hours; it will require that parents/guardians provide transportation to the pretest site; and it comes in the midst of the summer vacation period. Second, pretest participants will have to take the initiative to respond to recruitment flyers, which makes it particularly important that the incentive be adequate to spur this initiative. Third, the pretest is to be conducted with the assistance of nonprofit organizations whose help in recruitment can be provided only in the summer vacation period. Setting the incentive at this level maximizes the attractiveness of participating, and our chances of recruiting the pretest group within the allotted time. Lastly, prior experience with similar pretests confirms that an incentive of this size is necessary. A similar pretest was conducted for a project conducted for Girls, Inc., eight years ago, and a $45 incentive was needed to gain cooperation for a pretest among high school students.
Participants’ names will not be associated with any of the findings from the individual or group debriefing interviews. All participants will be informed at the start of their session that the information they share will be private and the feedback they provide on both the questionnaire and survey protocol will be presented only in summary format to ACF. Participants will be told that, to the extent allowable by law, individual identifying information will not be disseminated publicly.
The pretest will involve the use of consent and assent forms (Appendix C). Because pretest participants will be minors, we will require that a consent form be completed by their parent or guardian prior to participation. Parents will receive information about privacy protections when they consent for their child to participate in the pretest. The consent form makes it clear that parents may withdraw their consent at any time. We will also request that participants complete an assent form. These forms will be collected for each of the participants prior to starting the session. No personal identifying information beyond name will be sought in these forms. The consent form will specify that no identifying information will be collected during the interview in either the one-on-one or group formats. Names are being collected on the consent/assent forms but will not be connected with any information provided during the debriefing sessions.
Interview and data management procedures that ensure security of data and privacy information will be employed by the contractor. All contractor staff working on this pretest will also be required to sign a privacy statement (Appendix D).
ACF does not plan to produce a report for public dissemination using the information obtained from this pretest. It is ACF’s intention that the information gathered through the proposed information collection will be used only for internal purposes related to refinement of the baseline survey instruments and protocols used in the evaluation.
The topic of the baseline questionnaires deals with the sexual activity of these youth. Therefore, it contains several items that are sensitive in nature. In the course of providing feedback on the items in the instrument, participants may be invited to share their opinions about items that are particularly sensitive. Survey data pertaining to the sexual history of evaluation sample members will be critical to the analytical work of this evaluation. Discussion of items addressed in the questionnaires is critical to the success of the pretest, as we seek to explore: how the teens interpret key terms; the extent to which they follow the survey protocol accurately; and whether the wording of items has been successfully modified both to an age-appropriate level and for a self-administered format.
This proposed information collection does not impose a financial burden on respondents. Respondents will not incur any expenses other than the time spent answering the questions in the questionnaire (30 minutes) and in the debriefing interview (60 minutes). Exhibit A12.1 summarizes the reporting burden on participants. Response times were estimated from informal pretests with contractor staff and prior experience. The annual burden is estimated from the total number of completed parental consent forms for parents and the total number of completed debriefing sessions for teens. The total annual burden is expected to be 75 hours.
EXHIBIT A12.1
ANNUAL INFORMATION COLLECTION BURDEN ESTIMATE
Instrument |
Number of Participants |
Number of Responses Per Participant |
Average Burden Hours Per Response |
Total Burden Hours |
Average Hourly Wage |
Total Annual Cost |
Pre-test Survey Instrument |
||||||
Pre-Test for Non-Sexually Active Youth (First Half of Appendix A) |
25 |
1 |
0.5 |
12.5 |
n/a |
n/a |
Pre-Test for Sexually Active Youth (Second Half of Appendix A) |
25 |
1 |
0.5 |
12.5 |
n/a |
n/a |
Debriefing Interviews |
||||||
Debriefing Interviews (Individual or Group) for both Non-Sexually Active and Sexually Active Youth (Appendix B) |
50 |
1 |
1 |
50 |
n/a |
n/a |
Total |
50 |
- |
- |
75 |
- |
- |
There are no direct monetary costs to participants; they spend only their time to participate in the pretest.
The total cost to the federal government of recruiting pretest respondents, obtaining consents, conducting the pretest, and modifying the instruments, as part of the baseline instrument development task in the Evaluation of Pregnancy Prevention Approaches contract is estimated to be $128,860.
This submission to OMB is a new request for approval.
There are no plans for tabulating and publishing the information gathered in this pretest. The information collected will be for internal use only. We will not be analyzing or reporting on substantive data provided by participants through their completed questionnaires. Rather, the findings from this pretest effort will all be related to question wording and survey protocol.
We will prepare a summary of key findings from feedback provided on the instruments and the protocols during the cognitive debriefing interviews. From this feedback, we will make revisions to the survey instrument and protocols.
All instruments and consent/assent forms will display the OMB number and expiration date.
No exceptions are necessary for this information collection.
1Abma, J.C., et al. “Teenagers in the United States: Sexual Activity, Contraceptive Use, and Childbearing, 2002.” Vital and Health Statistics, series 23, no. 24, 2004.
2Albert B., S. Brown, and C. Flannigan (eds.). 14 and Younger: The Sexual Behavior of Young Adolescents. Washington, DC: National Campaign to Prevent Teen Pregnancy, 2003.
3Teen birth rates declined by 34 percent from 1991 to 2005. See: Hamilton, B.E., J. A. Martin, and .J. Ventura. “Births: Preliminary Data for 2006.” National Vital Statistics Reports, vol. 56, no. 7. Hyattsville, MD: National Center for Health Statistics, 2007.
4Trenholm, C., et al. Impacts of Four Title V, Section 510 Abstinence Education Programs: Final Report. 2007. Retrieved June 2, 2008 from http://www.mathematica-mpr.com/publications/PDFs/impactabstinence.pdf
5This includes the National Survey of Family Growth (NSFG); National Longitudinal Study of Adolescent Health (Add Health); National Longitudinal Survey of Youth 1997 (NLSY97); America’s Promise Polls; Wave 4 Survey of Teen Activities and Attitudes (Mathematica Policy Research, Inc., 2005); All About Youth: Evaluation of Sexual Risk Avoidance and Risk Reduction Programs for Middle School Students (AAY); Monitoring the Future (8th Grade); National Survey on Drug Use and Health (NSDUH); and Silverberg and Small’s Parental Monitoring Scale.
| File Type | application/msword |
| File Title | Draft Justificiation Part A for Enhanced BL Pretest |
| Author | Jill Miller |
| Last Modified By | Seth F. Chamberlain |
| File Modified | 2009-08-26 |
| File Created | 2009-08-26 |