Evaluation of Adolescent Pregnancy Prevention Approaches (PPA)
Pre-testing of Evaluation Surveys
(OMB 0970‑0355)
Supporting Statement Part B
This section provides supporting statements for each of the five points outlined in Part B of the OMB guidelines, pertaining to the proposed expanded pretest of the baseline survey instrument and protocols for the Evaluation of Adolescent Pregnancy Prevention Approaches (PPA). Subsequent OMB submissions will seek clearance for the baseline instruments themselves, and later for follow-up data collection instruments.
The Evaluation of Adolescent Pregnancy Prevention Approaches (PPA) is designed to test promising approaches to preventing teen pregnancy. One of the major tasks in the evaluation entails a longitudinal survey of primarily high-school-aged teens to estimate program impact. To produce survey findings that will be of maximum use in the subsequent reports, the items in the questionnaire must be easily and consistently understood by sample members. The survey protocol should also provide directives for completion that are easy to follow, clearly written, and consistently followed with accuracy.
We have designed two baseline survey instruments, one for sexually active youth and one for sexually inactive youth. The instruments draw on items from multiple sources, many of which were targeted to older populations and designed for different modes of administration. For these reasons, it is essential to pilot test the new instruments. Pilot testing will enable the contractor to provide feedback to ACF on item wording or suggest changes to the administration protocol. The goal is to design a final version of the surveys that will minimize unit and item nonresponse and capture high quality data that will best respond to the evaluation’s research questions.
We request to conduct the pre-test of each baseline instrument with 25 primarily high-school-aged teens, or a total of 50 for the two instruments. To mirror the population for the study, these teens will primarily be between 14 and 16 years old. We hope to include students from a range of socio-economic backgrounds as well. We will draw a convenience sample of teens from the local Princeton, New Jersey, area to minimize travel burden to attend the session, which will be held at the contractor’s office or a nearby location. Potential participants will be recruited in partnership with local community-based organizations. Working with local organizations, we will distribute flyers with information about the pretest to youth attending community programs. Interested youth will sign up for the pretest. Those interested will be asked to provide contact information and be given a consent form for their parent/guardian to sign. Once consent is obtained, youth will be contacted and asked a few screening questions to schedule inclusion in a one-on-one or group session. If needed, we will also use a “snowball” sampling technique that relies on referrals from initial participants, to generate additional potential participants.
The sample for the one-on-one debriefing interviews will include four boys and four girls. To help in scheduling individuals for debriefing sessions, we will include a screening question asking youth to indicate whether they prefer to participate in a session for youth who are not yet sexually active or a session for youth who have been sexually active.
Each of the four group debriefing interviews will have up to seven participants. For the group interviews, we will separate the groups based on gender and also based on sexual activity to the extent the participating youth indicate their preference.
Each teen recruited for the pretest will be scheduled for a specific date and time for his/her session, depending upon whether he/she is in a group or individual interview. At the beginning of the session, the contractor will check participants in and ensure they have both consent and assent forms completed. The contractor will then provide a brief overview of the activities of the pre-test, reiterate the key elements of informed consent, and distribute the questionnaires. Participants will be given 30 minutes to complete the questionnaire on their own and will be asked to circle any items or instructions that they found were confusing, used out-dated terminology, were unclear, or were particularly sensitive. Screening questions will guide the participants to complete the appropriate questionnaire.
Those who have been recruited for a one-on-one interview will then pair off with a researcher who will debrief with the participant for an additional 60 minutes. The one-on-one debriefing will include a review of items circled by the participant, as well as additional items from the interview guide (Appendix B). At the end of the 60 minutes, the participant will be provided with the $50 honorarium and will be asked to sign a receipt of payment slip.
Those recruited for a group debriefing interview will pass in their completed questionnaire to the contractor, who will place them in an envelope for review at a later time. In contrast with the one-on-one interview, these participants’ responses to specific items will not be reviewed while the participant is present. This allows for more time to be spent in the group discussion on key topics (the moderator’s guide is found in Appendix B). The group debriefing will begin by setting the ground rules for the group discussion, related to maintaining privacy, encouraging participants to voice their opinions, and allowing one person to speak at a time. Following the 60-minute discussion, participants will be provided with the $50 honorarium and will be asked to sign a receipt of payment slip.
Contractor staff will takes notes during each type of session and prepare written summaries of key findings for submission to ACF.
Because the pretest is drawing a convenience sample, the issue of response rates does not apply. Our goal is to conduct debriefing interviews with 50 teens, primarily ages 14 to 16 years, 25 who are sexually active and 25 who are not. We will establish quotas to generate a diverse sample based on: gender, age, whether or not the teen is sexually active (self-reported), and socio-economic background.
The debriefing will improve our understanding of important issues critical to the construction of a high quality questionnaire: whether the questions have been worded in such a way that the participants define/understand key terminology the way the researchers intended; whether the reading level is accessible to this population; and whether the instructions on skip patterns are conveyed clearly enough to ensure participants follow them accurately. In particular, the current design calls for all respondents to complete the same “front” section of the questionnaire then select one of two “private” versions of the back half of the questionnaire based on their answer to the last question in the front section of the questionnaire, concerning whether they have ever had sex. Understanding how respondents interpreted this question and our instructions on which private section to complete will be an important part of this pretest. In addition, this pretest will also explore what may be gained from alternate ways of wording specific items. For example, we are interested in knowing whether or not teens can give specific information in response to a question about the “month and year” of first sex, instead of a question asking “how old were you when you first had sex.” We want to learn how they arrived at their answers for each question wording approach.
We are utilizing both individual and group debriefing sessions to maximize the benefits gained from each approach. In the one-on-one sessions, the contractor will be able to carefully review the completed questionnaire with the participant to probe specific issues in greater depth. This setting may also make it easier for teens to divulge information they may not feel comfortable sharing in a group setting. In contrast, the group debriefing allows an opportunity for teens to generate ideas or feedback they may not have otherwise arrived at as an individual. All of the debriefing interviews will be conducted by the contractor’s professionally trained research staff.
The information will be collected by the contractor selected to conduct the Evaluation of Adolescent Pregnancy Prevention Approaches (PPA), and its subcontractors, on behalf of the U.S. Department of Health and Human Services (HHS), Administration for Children and Families (ACF). With ACF oversight and approval, the contractor will be responsible for finalizing the study design, data collection, analysis, and report preparation. Key input to the current data collection instruments was received from the following individuals in the Department of Health and Human Services:
Administration for Children and Families, Family and Youth Services Bureau, Division of Abstinence Education
Stan Koutstaal, Director
Melodye Watson, Program Analyst
Administration for Children and Families, Office of Planning, Research, and Evaluation
Naomi Goldstein, Director
Mark Fucello, Director, Division of Economic Independence
Nancye Campbell, Senior Research Analyst
Seth Chamberlain, Research Analyst
Office of the Assistant Secretary for Planning and Evaluation
Lisa Trivits, Research Analyst
Inquiries regarding statistical aspects of the study design should be directed to:
Seth Chamberlain
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
370 L'Enfant Promenade, S.W.
Washington, DC 20477
202-260-2242
Mr. Chamberlain is the project officer and has overseen the design of the data collection instruments.
| File Type | application/msword |
| File Title | Draft Justification Part B for Enhanced BL Pretest |
| Author | Jill Miller |
| Last Modified By | Seth F. Chamberlain |
| File Modified | 2009-08-26 |
| File Created | 2009-08-26 |