15. 48-Month Survey Pretest Findings 11/2014 (Appendix G2)

APPENDIX G2:
48 MONTH SURVEY PRETEST MEMO
11/2014

This page left intentionally blank for double-sided copying.

111 East Wacker Drive, Suite 920
Chicago, IL 60601-4303
Telephone (312) 994-1002
Fax (312) 994-1003
www.mathematica-mpr.com

MEMORANDUM

TO:

Cynthia Miller, MDRC

FROM:

Lisbeth Goble, Lauren Maul, Sandra Mukasa, and Lisa Schwartz

SUBJECT:

YouthBuild 48-Month Youth Survey Round 3 Pretest
Findings

DATE:

11/24/2014

YB-71

A. Introduction

In preparation for the 48-month survey for the YouthBuild Evaluation, Mathematica Policy
Research, under subcontract to MDRC, conducted a third round of pretests focused on the series
of questions that aims to capture attributes generally associated with one’s character, referred to
as the “personal attributes questions.” These questions originated from a formative evaluation of
YouthBuild (Ferguson et al. 1996). The evaluation included one-on-one interviews with
YouthBuild participants. At the end of these interviews, respondents were asked: “If someone
asked you to describe yourself, to say who you are, what would you say?” The question was
open-ended and responses were coded into the categories that were ultimately used to develop
the first version of the personal attributes questions. (See Appendix A.)
The original version of the personal-attributes questions were tested in two previous rounds
of pretests. 1 The first round included a card-sort activity with debriefing. In the second round,
youth completed either interviewer-administered or self-administered questionnaires and then
participated in a cognitive interview using a retrospective protocol. The first two rounds of
pretests were conducted with current or former YouthBuild participants. Findings are
summarized below.
The first two rounds of the pretest uncovered several issues with the original version of the
personal attributes questions:

1

In addition to the questions about personal attributes, the first two rounds of pretesting included questions
about various activities in which youth may participate. Details of the methodology and findings from the earlier
pretests were included in our June 6, 2014, memo to MDRC.

An Affirmative Action/Equal Opportunity Employer

MEMO TO: Cynthia Miller, MDRC
FROM:
Lisbeth Goble, Lauren Maul, Sandra Mukasa, and Lisa Schwartz
DATE:
11/24/2014
PAGE:
2
• Participants interpreted the phrase “the people you hang out with” inconsistently. 2
Participants included friends, family members, and others as people they hang out
with and responded to the question series differently depending upon those included
in their reference group.
• Participants were unfamiliar with some of the terms (for example, “social dexterity”).
• Participants considered some of the terms redundant (for example, “Helpful” and
“Friends and family come to me for advice.”).
• The words and phrases used to describe personal attributes were strongly associated
with those common among YouthBuild programs.
Based on the findings from the first two rounds of testing, we revised the personal attributes
questions in several ways for the third round of pretesting. First, we modified the question stem
so that it no longer referenced “people you hang out with.” Second, we separated items grouped
together into single constructs. For example, the items “Intelligent/trying to learn/ hardworking”
were separated into “I make an effort to learn” and “Hardworking.” And last, we tested words
that were synonymous or nearly synonymous with the original words to reduce the potential bias
associated with using “program language” to measure potential program impacts. The modified
version of the question is in Appendix A.
The three main objectives of the third round of pretest were to (1) assess the consistency and
accuracy with which participants interpreted the revised personal attributes, (2) see if an openended version of the question, similar to the question from the 1996 Ferguson study, was a viable
alternative to the close-ended question, and (3) assess any differences in results when testing the
series with non-YouthBuild participants. This memo provides an overview of the pretest design
and findings, and presents recommendations based on those findings.
B. Pretest methodology

Our approach allowed us to assess both closed and open-ended approaches to measuring
personal attributes. This section reviews our recruitment strategy and pretest protocols.
Recruitment. To address concerns that the original pretests did not capture responses
typical of non-YouthBuild participants, we conducted this round of pretesting with students from
two adult literacy programs in Chicago. These participants were in the same age span as

2

The original question series asked, “Please indicate how the people you hang out with would say that each of
the following describes you. Would you say the following are ‘not at all like you,’ ‘kind of like you,’ ‘like you,’ or
‘very much like you’?”

MEMO TO: Cynthia Miller, MDRC
FROM:
Lisbeth Goble, Lauren Maul, Sandra Mukasa, and Lisa Schwartz
DATE:
11/24/2014
PAGE:
3
respondents in our survey and self-identified as young adults. We had six female and three male
participants. All were given a $25 gift card for their participation after completion of the
interviews. Trained Mathematica staff conducted all pretest interviews. Staff from
Mathematica’s Chicago office conducted the first five interviews, and staff from various offices
held the remaining four interviews.
Pretest protocol. The open-ended question was included as the first item in the protocol to
eliminate potential priming effects. Its inclusion allowed us to assess the viability of
incorporating an open-ended approach in the final version of the 48-month survey. It also
allowed us to see if the words in the close-ended list were adequately capturing what participants
say about themselves when not constrained by a list. Participants were then asked to complete
the revised personal attributes series as they would see it in the final version of the survey.
To assess participants’ interpretation of the adjectives given in the open-ended question and
list of personal attributes, participants were asked to provide definitions of the terms and
examples of the ways in which they demonstrate the given attributes. They were also asked
whether they would categorize the adjectives they provided in the open-ended question with any
of the words or phrases in the list of personal attributes. Participants were then asked how hard or
easy it was to provide a list of three attributes they would use to describe themselves. The
protocol is in Appendix B.
C. Findings

The main findings are documented below. For clarity of presentation, we first focus on
findings from the revised closed-ended questions and then on the open-ended questions.
1.

Close-ended personal attributes question

Limited variation in responses. Participants reported that the attributes were “very much
like them” or “like them” over 90 percent of the time. (See Table 1.) In fact, only one participant
selected the response category, “Not at all like me,” in response to the item, “Friends and family
come to me for advice.” Although we saw a reasonable distribution of responses to the item that
asked about providing advice, overall, the lack of variation raises two important concerns. First,
the pattern of responses suggests that social desirability bias affects responses, and second, as
written, the questions are unlikely to detect meaningful differences among participants. To
illustrate both issues, during the debriefing, one participant who reported that “honest” is “very
much like me” stated, “I'm somewhat honest, to a certain point. I won't betray nobody, but if I
was to find some money laying in the street, I wouldn't go through the neighborhood trying to
find who it belonged to. So I'm kind of honest.”

MEMO TO: Cynthia Miller, MDRC
FROM:
Lisbeth Goble, Lauren Maul, Sandra Mukasa, and Lisa Schwartz
DATE:
11/24/2014
PAGE:
4
Table 2. Frequency of responses for each personal attribute
Not at all like
me

Kind of like
me

Like me

Very much
like me

Helpful
1
8
Caring
1
8
Friends and family come to you for
1
2
3
3
advice
I make an effort to learn
2
7
Hardworking
2
7
1
Optimistic
1
1
5
Determined
9
1
This row does not sum to nine due to comprehension issues with the term “optimistic.” See discussion below.

Interpretation of terms. For the most part, participants were able to complete this series of
questions without difficulty. The only word that caused obvious confusion was “optimistic.”
Four of the first five pretest participants were uncertain about what this word meant and asked
for clarification while completing the question. For the remaining four interviews, we revised
this item to “I have a positive attitude.” This revision made it easier for participants to rank how
much it was like them. However, when asked to describe what they were thinking of when asked
if they have a positive attitude, some responses were not necessarily in line with the intended
meaning. For example:
• “The way I carry myself, I keep myself up, I keep my hygiene up, I dress well, I'm
not an expensive dresser, and I try to look nice.”
• “When I'm around sophisticated people, educated people, [I show a positive attitude]
by observing.”
• “Friends and family come to me for advice” also posed some challenges for
participants. The participants had difficulty responding to this item because of the
double-barreled phrase. Although this was the only item that picked up a reasonable
distribution of responses (previously discussed), the variation depended, in part, on
whether respondents were thinking about friends and/or family in formulating their
responses. Participants indicated that both friends and family did not necessarily
come to them for advice. In most cases, they reported that friends are more likely to
come to them for advice than are family members. For example, one participant
indicated that both family and friends come for advice, but when probed on how
often, he stated, “[Family] not so much because if they can work it out for
themselves, they will.”
Despite the relative ease with which participants completed the questions, their debriefing
comments suggest that they did not necessarily understand the words as intended. Specifically,
when asked to provide definitions for each item, participants failed to distinguish between many

MEMO TO: Cynthia Miller, MDRC
FROM:
Lisbeth Goble, Lauren Maul, Sandra Mukasa, and Lisa Schwartz
DATE:
11/24/2014
PAGE:
5
of the terms on the list. For example, many participants were unable to distinguish between
“helpful,” “caring,” and “providing advice.” They also did not clearly differentiate among “I
make an effort to learn,”, “being “hardworking,” and being “determined.” The lack of clarity and
inconsistency of interpretation suggests that, as written, the questions are not suitable for survey
inclusion.
2.

Open-ended personal attributes

When asked an open-ended version of the personal attributes question, most participants
offered favorable terms about themselves. Some words that were repeatedly provided were
“intelligent,” “nice,” and “trustworthy.” Of the 27 words provided by participants, 2 have
negative connotations, “impatient” and “annoying,” and a few were neutral (for example,
“independent” and “plain”). Table 2 includes all words that were provided in response to the
open-ended question categorized by whether they have a positive, negative, or neutral
connotation.
Table 2. List of words provided to open-ended personal attributes question
Positive
Beautiful
Caring
Dependable
Eagerness
Hard worker
Honest
Intelligent or smart (5)
Nice (2)
Reliable
Respectable
Respectful
Trustworthy (2)

Neutral
Independent
Mom
Outspoken
Plain

Negative
Annoying
Impatient

When asked if their self-descriptions fit with the list of close-ended items, most said yes,
initially. However, we found that participants struggled when asked to think about whether the
three words they provided could be grouped with any of the words in the adapted list of personal
attributes. They attempted to do so in order to engage in the task, but struggled to provide logical
explanations for their categorizations; for example:
• “Intelligent fits with hardworking and caring. If you're intelligent and work hard, you
care about what you're doing.”
• “Impatient goes with honest. I'm honest in saying I'm impatient.”

MEMO TO: Cynthia Miller, MDRC
FROM:
Lisbeth Goble, Lauren Maul, Sandra Mukasa, and Lisa Schwartz
DATE:
11/24/2014
PAGE:
6
• “Outspoken fits with helpful. Outspoken is being helpful; when someone asks you a
question, you respond to them, be outspoken with whatever you say.”
• “Annoying goes with honest.”
• “Honest fits with determined and having a positive attitude. If you don't have a
positive attitude, how can you be honest about anything? If you're going to do
something, you've got to be honest; you show that you're going to do what you say
you're going to do.”
• “Eagerness fits with determined because when you are eager to do something you are
determined. Eagerness is when you are going to do something no matter what.
Determined is that you'll try to do something up to a certain point. Eagerness is an
emotion, there are no boundaries.”
Although some of the words in Table 2 align with words in the close-ended version, many
do not, suggesting that the list does not adequately reflect the ways youth describe themselves. It
would be difficult to develop a comprehensive list of attributes, and if we were to do so, the list
would have to balance positive and negative attributes. Even with a more balanced list, it seems
likely that participants will mainly provide socially desirable responses and say that the positive
items are like them regardless of the “truth” of this assertion.
To assess the difficulty in completing the open-ended question, we asked participants how
hard or easy it was to come up with the three terms. All of participants said it was “easy” to
come up with three words to describe themselves because, as one participant stated, “It’s simple
enough to talk about yourself.” One participant said it was very hard to come up with just three
words because there are so many that could be used to describe them. These responses suggest
that the open-ended question would be a better way to measure personal attributes.
D. Recommendations

Based on the findings from the third round of pretesting, we strongly recommend reverting
back to the open-ended question originally used in Ferguson’s 1996 study. We recognize this
will require additional coding of responses; however, we believe the quality of the data received
from the open-ended question will be more informative than data from the close-ended version
of the question.
We see a number of benefits to taking this approach. The open-ended approach would
accomplish the following:
• Retain consistency across studies and allow for cross-study comparisons of responses
to the original Ferguson questions (assuming they are kept verbatim).
• Reduce respondent burden overall. The open-ended question takes, on average, no
more than a minute. This is a much shorter period of time in comparison to the

MEMO TO: Cynthia Miller, MDRC
FROM:
Lisbeth Goble, Lauren Maul, Sandra Mukasa, and Lisa Schwartz
DATE:
11/24/2014
PAGE:
7
closed-ended list, which can take several minutes to complete, especially if
respondents are not familiar with all the words.
• Eliminate most of the social desirability bias that appeared in responses to the closedended list. However, we expect some social desirability bias would clearly remain as
people naturally want to present themselves in a positive light.
• Eliminate the difficulty of measuring abstract concepts (such as “determined”) with a
single item. To date, we have not identified single terms that adequately convey the
intended construct and are understood consistently across participants.
• Prevent the introduction of bias that could be introduced by using words that are part
of the YouthBuild program culture.
Although it will take more time for coding on the back end, the open-ended format allows
for more analytical flexibility by allowing researchers to create meaningful categories based on
respondents’ self-reports.

APPENDIX A
ORIGINAL AND REVISED FERGUSON QUESTIONS

APPENDIX A – ORIGINAL AND REVISED FERGUSON QUESTION

MATHEMATICA POLICY RESEARCH

Original Ferguson Questions

1. Please indicate how the people you hang out with would say that each of the following
describes you. Would you say the following are “not at all like you,” “kind of like you”
“like you” or “very much like you?”
Not at all
like me
a.
b.
c.
d.
e.
f.
g.
h.
i.

Helpful/caring/loving/I can
give advice
Intelligent/trying to learn/hardworking
Nice/friendly/easy to get
along with
Honest/direct/sincere
Trying to make something of
myself/determined
Fun/carefree/easy-going/like
to party
Faith in God
Down-to-earth
Social dexterity/I can fit in

A.2

Kind of
like me

Like
Me

Very much
like me

APPENDIX A – ORIGINAL AND REVISED FERGUSON QUESTION

MATHEMATICA POLICY RESEARCH

Revised Ferguson Questions
Would you say that that is very much like you, like you, kind of like you, or not at all like you?
SELECT ONE RESPONSE PER ROW

NOT AT
ALL LIKE
ME
a. Helpful ................................................................
b. Caring..................................................................
c. Friends and family come to me for advice .........
d. I make an effort to learn ......................................
e. Hard-working .......................................................
f. Honest .................................................................
g. I have a positive attitude .....................................
h. Determined .........................................................

A.3

1

1

1

1

1

1

1

1

□
□
□
□
□
□
□
□

KIND OF
LIKE ME
2

2

2

2

2

2

2

2

□
□
□
□
□
□
□
□

LIKE ME
3

3

3

3

3

3

3

3

□
□
□
□
□
□
□
□

VERY
MUCH
LIKE ME
4

4

4

4

4

4

4

4

□
□
□
□
□
□
□
□

APPENDIX B
YOUTHBUILD 48-MONTH FOLLOW UP SURVEY:
PRETEST QUESTIONS

Mathematica Reference No: 06796

YouthBuild 48-Month Follow Up Survey:
Pretest Questions
Round 3: Pretest Protocol
November 2014

APPENDIX B - YOUTHBUILD 48-MONTH FOLLOW UP SURVEY: PRETEST QUESTIONS

MATHEMATICA POLICY RESEARCH

YouthBuild 48-Month Pretest Protocol: Round 3

Materials:
•
•

Pretest protocol for each respondent
Digital recorder

Administer Survey:
Thank you for your time and willingness to help us out today.
First, let me introduce myself and tell you a little bit about what we are going to be doing
today.
•
•

•

My name is [NAME] and I’m from Mathematica Policy Research, a research
company based in Princeton, New Jersey.
Today we need your help to test a few questions that will be part of a
questionnaire that will eventually be given to a larger group of young people,
like yourself. Your input is very valuable and will help us collect high quality
data.
Today’s session should take about 30 minutes. Do you have any questions
before we begin?
I would like to record discussion so that I can go back and listen to it later if I
need to. Is that ok?

B.3

YOUTHBUILD 48-MONTH FOLLOW UP SURVEY: PRETEST QUESTIONS

MATHEMATICA POLICY RESEARCH

YouthBuild Phone Pretest Timing Sheet
APPOINTMENT DATE:
TIME:
INTERVIEW START TIME:
INTERVIEW END TIME:

Comments
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________

B.4

YOUTHBUILD 48-MONTH FOLLOW UP SURVEY: PRETEST QUESTIONS

MATHEMATICA POLICY RESEARCH

Pretest Protocol
These first questions are about the way you might describe yourself.
If someone asked you to describe yourself in three words, what would you say? I’m interested in
the words you would use to describe who you are as a person, not things like how tall you are,
your age or what you look like. Which three words would you say describe who you are?
1. ___________________________________________
2. ___________________________________________
3. ___________________________________________
Now I’m going to read you some words that can be used to describe people. For each one,
please tell me how well that word describes you.
[Read each item]
Would you say that that is very much like you, like you, kind of like you, or not at all like you?
SELECT ONE RESPONSE PER ROW

NOT AT
ALL LIKE
ME
a. Helpful ....................................................................
b. Caring.....................................................................

1

1

c. Friends and family come to me for advice ............
1

d. I make an effort to learn .........................................
1

e. Hard-working ..........................................................
1

f. Honest ....................................................................
g. I have a positive attitude ........................................
h. Determined ............................................................

B.5

1

1

1

□
□
□
□
□
□
□
□

KIND OF
LIKE ME
2

2

2

2

2

2

2

2

□
□
□
□
□
□
□
□

LIKE ME
3

3

3

3

3

3

3

3

□
□
□
□
□
□
□
□

VERY
MUCH
LIKE ME
4

4

4

4

4

4

4

4

□
□
□
□
□
□
□
□

YOUTHBUILD 48-MONTH FOLLOW UP SURVEY: PRETEST QUESTIONS

MATHEMATICA POLICY RESEARCH

Debriefing Protocol
Before we move on let’s talk about these first questions.
1) Let’s go back and look at the first questions where you gave me three words that
you think describe you. Let’s start with the first word. [STATE WORD] Why did you
choose that word? When you say that you’re ___, what does that mean to you? In
what ways do you demonstrate ____? Are there other words you might use that
mean the same thing?

2) Let’s move on to the second word. [STATE WORD] Why did you choose that
word? When you say that you’re ___, what does that mean to you? In what
ways do you demonstrate ____? Are there other words you might use that
mean the same thing?
3) Let’s move on to the third word. [STATE WORD] Why did you choose that
word? When you say that you’re ___, what does that mean to you? In what
ways do you demonstrate ____? Are there other words you might use that
mean the same thing?
4) How hard or easy was it to come up with three words that describe you?
What made it easy/hard?
5) What about the next set of questions – where you told me how well certain
words describe you. I would like to know a little bit more about what you were
thinking of when you selected your answer for each of these words.
Let’s start by just talking about what each word means to you. [Interviewer –
go through each word one at a time]
• If you had to come up with a definition for the word [FILL: e.g. HELPFUL], what
would you say? How would you define it?

(Go through each word/phrase in the table and get definitions).
A. [If respondent selected response options 2-4 for the item]: Can you
give me some examples of the ways in which you are [FILL: e.g.
HELPFUL].
B. [If respondent selected response option 1 for the item]: Why do you
think this word would not be used to describe you?
C. [For friends and family ?]:
D. If I said friends and/or family, would that change how you answered
the question? What does the phrase and/or mean to you?

B.6

YOUTHBUILD 48-MONTH FOLLOW UP SURVEY: PRETEST QUESTIONS

MATHEMATICA POLICY RESEARCH

E. Do both friends and family come to you for advice ? How often
would you say friends come to you for advice? What about family
members?
6) Let’s go back and look at the three words you came up with to describe you.
[START WITH FIRST WORD]. Do you think [WORD] fits with any of the
words or phrases I asked about in the second question
In-person: Hand show card to respondent with words from second question;
By phone: Review the words again.
•

Where does it fit? Can you tell me why you would include it there? In
what ways do the words mean the same thing? In what ways are they
different from each other?

•

(If self-described word fits with more than one option in second
question) Does [WORD] fit better with one of these? Which one?
Why?

7) If this was the only question I asked you, do you think I would have a pretty
good sense of the kind of person you are? What’s missing?

Wrap-Up
We are coming to the end of our discussion. Thank you for sharing your experiences
and opinions. Is there anything else you would like to add about anything we discussed
today?

B.7