Pretest memo

ATTACHMENT H
PRETEST SURVEY MEMO

This page has been left blank for double-sided copying.

ATTACHMENT H: PRETEST SURVEY MEMO

MATHEMATICA POLICY RESEARCH

955 Massachusetts Avenue, Suite 801
Cambridge, MA 02139
Telephone (617) 491-7900
Fax (617) 491-8044
www.mathematica-mpr.com

MEMORANDUM

TO:

Amy Farb, Office of Adolescent Health

FROM:

Sarah Forrestal, Laura Kalb, Jennifer Walzer, and Andrea Bucciarelli
7/1/2016
TP3 FADS 26
DATE:

SUBJECT:

Pretest Findings for the MPC! Evaluation Baseline Questionnaire

In this memorandum, we report on the pretest of a baseline questionnaire for the Making
Proud Choices! (MPC!) Evaluation. Mathematica Policy Research conducted the pretest in May
and June 2016 in order to assess how long it took to administer the instrument and to improve the
questions we developed for the baseline data collection.
In the following sections, we describe the (1) pretest participants, (2) pretesting and
debriefing procedures, (3) pretest findings, and (4) proposed revisions to the instrument in
addition to those informed by the pretest. Appendix A includes the draft of the pretest
questionnaire (A.1) and a question-by-question documentation of the issues raised during the
pretest (A.2). Appendix B includes a revised draft of the instrument, with suggested revisions
noted in Track Changes. Appendix C includes the same revised draft with the revisions
incorporated and no tracking of the changes.
A. Recruiting process and final pretest sample

We pretested the instrument with two different groups of youth in Chicago, Illinois. One
group was presumed not to be sexually active (Group 1) and the other was presumed to be
sexually active (Group 2). Having two groups enabled us to test the two alternate paths in the
instrument; the first group answered questions on pre-coital sexual behaviors, whereas the
second group answered more detailed questions on their sexual behavior.
We recruited Group 1 through Erie Neighborhood House, a community-based organization
serving low-income families. Agency staff agreed to coordinate the pretest by distributing and
collecting signed consent forms from parents of 7th and 8th grade students and hosting the
pretest at their facility. Both English and Spanish versions of the consent forms were provided.
Staff recruited 11 participants to help ensure we would have at least nine youth available for the
pretest.
We recruited Group 2 through Options for Youth, a community-based organization the
serves at-risk pregnant or parenting young women in sites throughout Chicago. We worked with
a site located in a public high school. The site program director agreed to coordinate the pretest
An Affirmative Action/Equal Opportunity Employer
H.3

ATTACHMENT H: PRETEST SURVEY MEMO

MATHEMATICA POLICY RESEARCH

MEMO TO: Amy Farb, Office of Adolescent Health
FROM:
Sarah Forrestal, Laura Kalb, Jennifer Walzer, and Andrea Bucciarelli
DATE:
7/1/2016
by recruiting young women in grades 9 and 10 and distributing and collecting signed parental
consent forms. She recruited nine participants.
Participants signed assent forms in which they agreed to participate in the pretest and have
the debriefing audio-recorded. After completing the pretest, each participant received a $25 or
$50 gift card. 1 We also distributed gift cards to additional youth who were recruited but could
not participate due to OMB restrictions. In total, nine youth who were presumed not to be
sexually active and nine youth who were presumed to be sexually active participated in the
pretests. Table 1 summarizes participants’ characteristics. Because four Group 2 participants
self-reported as non-sexually active, we tabulated them separately.
Table 1. Pretest participants’ characteristics
Group 1

Grade

Group 2

Presumed non-sexually active

Self-reported
non-sexually
active

Self-reported
sexually
active

Total

Male

Female

Female

Female

7th

3

3

0

0

6

8th

1

2

0

0

3

9th

0

0

3

1

4

10th

0

0

1

3

4

11th

0

0

0

1

1

Total

4

5

4

5

18

B. Pretest and debriefing process

The process was similar for both groups. We gathered participants in a room and handed
each one a packet containing an assent form and a hard copy of the survey. A member of the
study team described the pretest procedures and read the assent form aloud. Participants signed
the assent form and completed the survey. They were instructed to mark any questions or words
that were difficult to understand and to record their start and end times so we could determine
how long it took them to complete the survey. We reminded participants that their actual survey
responses were less important than understanding the process necessary to answer the questions,
and that they would not have to reveal their responses to the group. We reviewed the completed
survey of each participant as he or she finished it in order to note points for follow-up during the
discussion. Participants were divided into smaller groups of two or three for a debrief on their

1

We discussed offering $50 gift cards with Erie Neighborhood House staff, but they recommended offering the $25
amount instead because it is consistent with amounts offered to participants in similar activities they have hosted.
H.4

ATTACHMENT H: PRETEST SURVEY MEMO

MATHEMATICA POLICY RESEARCH

MEMO TO: Amy Farb, Office of Adolescent Health
FROM:
Sarah Forrestal, Laura Kalb, Jennifer Walzer, and Andrea Bucciarelli
DATE:
7/1/2016
reactions to the instrument, with males and females kept apart. Each youth received his or her
own completed survey to refer to during the discussion, and our team collected the surveys from
the youth again at the end of the discussion.
Before the Group 1 pretest, the study team attended a two-hour training on logistics, best
practices for speaking with youth about sensitive topics, subjects to prioritize during the
debriefing, the best way to address any issues that might arise, and the debriefing interview
guide. Although the guide included specific probes for many survey items, researchers were
given the latitude to rephrase the questions as needed and to choose which items to ask about if
time ran short. The guide focused on (1) how respondents came up with their answers; that is, the
process they went through to interpret and formulate responses to questions; (2) whether
respondents followed instructions and completed the survey as expected; and (3) whether any
question or wording was confusing or outdated. In a few cases, researchers gave respondents
alternate versions of question wording during the debriefing and asked them which version they
preferred.
To avoid asking more than nine pretest participants the same question, we followed different
protocols for the two groups. In administering the survey to the second group, we dropped some
of the questions we asked the first group, followed up on different aspects of the questions based
on feedback from the first group, and asked about different items in the survey based on the
sexually active vs. non-sexually active paths. In addition, we made a few minor changes to
questions based on the first pretest to assess the effect of alternate wording in the second pretest.
C. Key findings of the pretest

As noted, we were particularly interested in learning more about how long the survey took
to administer, including whether the length of time was similar for both groups and whether the
questions were clear and understandable. Overall, no major issues were identified:
•

Administration time. The five Group 2 pretest participants who self-reported having been
sexually active took an average of 21.2 minutes to complete the survey. The Group 1
(presumed non-sexually active) participants took an average of 20.1 minutes. 2 The total
average time for both groups was 20.5 minutes. We recommend maintaining a 30-minute
burden estimate in the OMB Information Collection Request because, as we discuss in
Section D, we recommend adding new items that will be asked of all survey respondents.

2

In calculating the estimated time it took to complete the survey, we excluded the four Group 2 participants who
self-reported not having been sexually active, because their hard-copy questionnaires did not include the set of
questions designed for the non-sexually active group. As a result, their administration time is lower than it should be
and cannot be used to estimate burden for the actual survey.
H.5

ATTACHMENT H: PRETEST SURVEY MEMO

MATHEMATICA POLICY RESEARCH

MEMO TO: Amy Farb, Office of Adolescent Health
FROM:
Sarah Forrestal, Laura Kalb, Jennifer Walzer, and Andrea Bucciarelli
DATE:
7/1/2016
•

Sensitivity. Several questions in the survey are potentially sensitive (for example, questions
on gender identity, sexual orientation, sexual behaviors, and substance use). Pretest
participants differed on whether they thought other youth would find these topics sensitive,
but they generally agreed that survey respondents would answer the questions truthfully,
because the mode of administration would help keep the answers from being seen by other
students or school staff.

•

Question comprehension. Overall, few pretest participants had difficulty understanding the
questions in the survey.

Question-by-question issues that were identified during the pretest informed our
recommended revisions to the instrument. These findings are presented in Appendix A. For
questions whose wording came from another survey, such as the Personal Responsibility
Education Program (PREP) survey or the Concordance survey, or are about one of the Office of
Adolescent Health’s recommended behavioral outcome measures, we considered how changing
the wording of the questions might affect our ability to compare findings across studies.
Appendix B presents the revised questionnaire with changes tracked.
D. Additional revisions to the instrument

In addition to the proposed changes informed by the pretest, we recommend some other
changes to the baseline questionnaire. First, before the pretest we changed “sexual intercourse,”
which in previous instruments such as PREP was defined specifically as vaginal sex, to “sex”
and used the term “vaginal sex” when applicable. The reason for the change was twofold: it both
simplified the terminology and expanded the types of behaviors respondents might think of as
“sex” to be more inclusive of sexual behavior in people with different sexual orientations.
Second, we reviewed the draft instrument’s contents and added or revised a few items to
better align the survey contents with the evaluation research questions or otherwise improve the
questions. We added items to assess attitudes about condoms; these items had been used in the
Evaluation of Adolescent Pregnancy Prevention Approaches Ohio baseline survey. We also
added questions from the Fog Zone survey measuring knowledge about pregnancy and longacting, reversible contraceptives, and revised the response options on the marijuana use question
to be more consistent with the drinking question.
Finally, we selected a participant-facing name for the study: the Attitudes, Behaviors, and
Choices (or ABC) Study. This is different from the evaluation’s name, yet it still accurately
describes the survey contents without including language that potentially could embarrass
participants if their peers or others were to see consent forms or study recruiting materials. In
addition, not all participants in the evaluation will be exposed to the MPC! curriculum.

H.6