PRe-Test Cognitive Report

YBP-71_Grantee Pretest Memo_Final.pdf

Impact Evaluation of the YouthBuild Program

PRe-Test Cognitive Report

OMB: 1205-0488

Document [pdf]

Download: pdf | pdf

111 East Wacker Drive, Suite 920
Chicago, IL 60601-4303
Telephone (312) 994-1002
Fax (312) 994-1003
www.mathematica-mpr.com

MEMORANDUM

TO:

Eileen Pederson, YouthBuild Project Officer, ETA

FROM:

Lisbeth Goble, Lindsay Wood, Kim Mook, and Lisa Schwartz

DATE: 3/7/2012

YBP - 71
SUBJECT:

YouthBuild Grantee Survey Pretest Findings

A. INTRODUCTION
In preparation for the YouthBuild Evaluation, Mathematica Policy Research, under
subcontract to MDRC, conducted a pretest of the grantee survey instrument during March and
April 2011. The pretest used a mixed-method approach that included in-person cognitive
interviews using a think-aloud protocol, and telephone-administered cognitive interviews that
used a retrospective protocol.1 A sample of program directors from the 2010 YouthBuild
grantees responded via one of three pretest modes: (1) in-person cognitive interviewing using a
think-aloud protocol, (2) self-administered paper-and-pencil interview with a telephone
debriefing, or (3) telephone interview with a telephone debriefing. In total, six pretests took
place, including two in each mode. The average length of administration was 41 minutes.
The goal of the pretest was to assess respondents’ understanding of key survey terms and
questions, the accuracy and relevance of the questions, and the completeness of the information
captured about YouthBuild programs. Pretesting the survey in multiple modes was important
because we expect to administer the survey via the web with some telephone follow-up. As a
result, we needed to assess how well the questions functioned in both self- and intervieweradministered modes and obtain respondent burden estimates by mode.
This memo provides an overview of the pretest design and presents findings from the
cognitive interviews and the respondent debriefings.
B. GRANTEE SURVEY PRETEST METHODOLOGY
The pretest was designed in three phases: (1) in-person cognitive interviews, followed by (2)
interviewer and self-administered survey administration followed by respondent debriefings, and
(3) a second round of in-person cognitive interviews. Trained Mathematica staff conducted all
pretest interviews. Potential respondents were identified using a list of 2010 YouthBuild grantee
sites provided by ETA.

1 Throughout this memo, we refer to the cognitive testing that was done using a retrospective protocol as “respondent
debriefings.”

An Affirmative Action/Equal Opportunity Employer

MEMO TO: Eileen Pederson, YouthBuild Project Officer, ETA
FROM:
Lisbeth Goble, Lindsay Wood, Kim Mook, and Lisa Schwartz
DATE:
3/7/2012
PAGE:
2
Cognitive interviews were conducted first to identify issues with question comprehension.
During the preliminary design of the survey, we identified questions and terms that could be
ambiguous or unclear. Appendix A is the cognitive interview protocol, which was created with
specific probes for these questions and used to conduct the first phase of the in-person cognitive
interviews.
Upon completion of the cognitive interviews, the survey instrument was updated for the
interviewer- and self-administered phase of the pretest. The interviewer-administered mode was
conducted over the telephone and the respondent debriefing was conducted immediately after the
survey was completed. For the self-administered mode, hard copy surveys were sent to
respondents. They were asked to complete the survey and then return it to us. They were told to
keep a copy to refer to during the respondent debriefing, which was conducted upon receipt of
their completed hard copy. The debriefing protocol is included in Appendix B.
To ensure that question changes resulting from the interviewer- and self-administered
interviews did not create new comprehension issues, the pretest concluded with another
cognitive interview following similar procedures to the first one and using a modified protocol.
C. COGNITIVE INTERVIEW AND DEBRIEFING FINDINGS
The findings from the initial round of cognitive interviews and respondent debriefings were
consistent across modes. Based on these findings, we revised the questionnaire to improve the
clarity of questions about: funding sources, staff counts, applicant counts, Mental Toughness
Orientation (MTO), and hours in program activities. We also changed response categories to
more accurately capture the range of possible responses to survey questions.
1.

Funding Sources

The questionnaire initially included several questions about program operating budgets, the
organization’s primary funder, and program capacity. Some pretest respondents reported
confusion over how to report these figures because they believed that this DOL-funded
evaluation was primarily focused on program elements tied directly to DOL funding. In fact,
these questions are intended to measure all sources of program funding. We clarified this issue
by adding language to the introduction and the first question specifying that we are interested in
learning about budget and operations related to all funding sources. The revised introduction is
shown below with clarifying text shown in italics.

MEMO TO: Eileen Pederson, YouthBuild Project Officer, ETA
FROM:
Lisbeth Goble, Lindsay Wood, Kim Mook, and Lisa Schwartz
DATE:
3/7/2012
PAGE:
3

MDRC and its research partners Mathematica Policy Research
(Mathematica) and Social Policy Research Associates are conducting an
evaluation of the YouthBuild program on behalf of the U.S. Department of
Labor, Employment and Training Administration (ETA) and the Corporation for
National and Community Service (CNCS). As part of the evaluation, we are
asking YouthBuild grantees to complete a short survey about their program.
The survey covers several topics including the organizational structure of your
program, participant characteristics, the recruitment and enrollment
processes, and program services. Please note that, except where specified, we
are interested in learning about your YouthBuild program as a whole, not only
the components of your program that are tied directly to DOL funding. Your
participation will help ETA and CNCS better understand how YouthBuild
programs function and is mandatory for anyone receiving a grant

Pretesting also revealed that the order in which we asked these questions made it more
difficult for respondents to answer. The original version of the survey asked first asked whether
DOL was the predominant funder and then collected all funding sources. Reordering the
questions provided several opportunities to clarify that we wanted all funding sources included in
responses. The revised question order asks first about all funders, then about the predominant
funder, and then about the program’s operating budget.
Similarly, for questions about the services their programs provide, (Q34-41), respondents
were unsure whether to report only the services that are funded by DOL. We added introductory
language in this section to indicate that respondents should include all services, regardless of the
funding source.
Table 1 shows the original funding sources and service provision questions, and the
revisions we implemented based on the cognitive interviews.

MEMO TO: Eileen Pederson, YouthBuild Project Officer, ETA
FROM:
Lisbeth Goble, Lindsay Wood, Kim Mook, and Lisa Schwartz
DATE:
3/7/2012
PAGE:
4
TABLE 1. FUNDING SOURCE QUESTIONS
Original Funding Source Questions
Q1. How long has your YouthBuild program
been in operation?
Q4.

Is your program completely funded by a
Department of Labor grant?
1
0

Q5.

Q6.

Yes
No

Revised Funding Source Questions
Q1. How long has your YouthBuild program been in
operation under any agency funding?
Q2.

Which organization(s) fund your program?
SELECT ALL THAT APPLY

SKIP TO Q.7

Department of Labor (DOL)
Corporation for National and Community Service
(CNCS)

(IF Q.4=No) What other organization(s)
fund your program?

Private foundation funding
State or local funds

Corporation for National
Service (CNCS)

Private foundation
funding

State or local funds

(Please specify
________________________________________)
(STRING 255)

Something else? (Specify)

NO RESPONSE

Who is the predominant funder for your
organization?
1

U.S. Department of Labor

Another Public Source

Private Sources

Other (Specify)

Some other organization

Q3.

Who is the predominant funder for your
YouthBuild program?
SELECT ONE ONLY
Department of Labor (DOL)
Corporation for National and Community Service
(CNCS)
Private foundation funding
State or local funds
[FILL OTHER SPECIFY RESPONSE / Some other
organization]
NO RESPONSE

[Intro to Q31-41] Next, we would like to ask
you some questions about the different types
of services your YouthBuild program offers to
its participants.

[Intro to Q31-41] Next, we would like to ask you some
questions about the different types of services your
YouthBuild program offers to its participants. For the
following questions, please answer including all funding
services, not just those from DOL.

MEMO TO: Eileen Pederson, YouthBuild Project Officer, ETA
FROM:
Lisbeth Goble, Lindsay Wood, Kim Mook, and Lisa Schwartz
DATE:
3/7/2012
PAGE:
5

Respondents were consistently confused about if and how to report matching funding related
to DOL grants. To provide more clarity, we eliminated a filtering question that asked whether
their program was completely funded by DOL. We replaced this with the series of questions
shown in Table 1, which collect information on all funders and the organization’s predominant
funder. We included the Department of Labor as a response option to the question about the
predominant funder.
2.

Staff Counts

The original version of the survey asked respondents to provide the number of full- and parttime staff members employed in different components of their program (for example, the number
of people employed as educational instructors, case managers, worksite coordinators, and so on).
Respondents found these questions challenging in three ways. First, respondents interpreted
“employed” as meaning “compensated by.” They reported that some staff members were
compensated by an outside agency, such as AmeriCorps, but worked at their YouthBuild
program. Respondents did not know whether they should count these staff as “employed” by the
program. Second, some full- and part-time staff worked across multiple program components,
making it difficult to accurately and discretely count the number of staff providing different
types of program services. Third, some staff members worked in a specific program component
but did not have titles consistent with their staffing assignment; for example, a person who
worked on case management might not have the title of case manager. When the title and role
differed, respondents had difficulty deciding where to count the staff member. To resolve these
three issues, we revised this question to ask about the number of full-time equivalent (FTE)
positions instead of the number of people employed, and changed the program components from
job titles (such as case manager) to descriptors of the work itself (such as case management).
Table 2 shows the original staff count questions and the revisions we implemented based on
the cognitive interviews.
TABLE 2. STAFF COUNT QUESTIONS
Original Staff Count Questions
Q14.
How many of the following
types of staff does your program
currently employ?
a. Educational instructor
b. Vocational instructor
c. Case manager…

Revised Staff Count Questions
Q7. How many full-time equivalent (FTE) positions of the
following types currently work for your program? Please include
full-time and part-time staff …
a. Education instruction
b. Vocational instruction
c. Case management…

MEMO TO: Eileen Pederson, YouthBuild Project Officer, ETA
FROM:
Lisbeth Goble, Lindsay Wood, Kim Mook, and Lisa Schwartz
DATE:
3/7/2012
PAGE:
6
3.

Applicant Counts

Respondents reported that it was challenging to keep track of the responses they provided
for the number of applicants they received, number of applicants accepted into MTO, and the
final number of participants they enrolled in the program. We addressed this issue by adding Q23
(below), a grid that appears after this series of questions and shows the numbers the respondents
reported at each stage of the program recruitment and enrollment process. Respondents can
revise the numbers in the grid if necessary.

Q23

You have reported the following information about the individuals your program serves. Please
review and confirm your responses. Keep in mind we are interested in individuals who have
applied since you received your DOL grant in May 2011.
If your responses are correct, please continue. If you want to change a response, you may enter
it in the Updated Responses column below.

Your Reported Responses
1.

Number of applicants ...............................................................

Number of applicants invited to MTO or your similar
component.................................................................................

3.
4.

Number of applicants currently enrolled ................................
Maximum enrollment in your program ....................................

Updated
Responses

[FILL Q18]

[FILL Q20]

[FILL Q21]

[FILL Q22b]

Mental Toughness Orientation

The original survey instrument included a series of specific questions about Mental
Toughness Orientation. Several respondents reported that they had programs similar to MTO;
however, they did not feel comfortable referring to their program as such. To address this, we
modified the language in MTO-related questions to “MTO or similar component.”
5.

Program Hours

Originally, we asked respondents to provide the number of hours that participants spend in
various program activities (such as academic classes or construction work) over the course of the
entire program. Respondents found this calculation difficult. We first revised this series of
questions to capture the number of hours that participants spent in specific program activities
during an average month. However, respondents reported that this calculation was difficult
because some programs do not have consistent schedules from month to month, making
responses for a “typical” month complicated. In the end, we agreed that the intent of the question

MEMO TO: Eileen Pederson, YouthBuild Project Officer, ETA
FROM:
Lisbeth Goble, Lindsay Wood, Kim Mook, and Lisa Schwartz
DATE:
3/7/2012
PAGE:
7
is to discover the balance of vital program activities. The question was changed to ask whether
educational activities or workforce activities took more time, or if they were approximately
equal. Table 3 shows the original program hours questions and the revisions we made based on
the pretest.
TABLE 3. PROGRAM HOURS QUESTIONS
Original Program Hours Questions
Of all the activities and services you listed above
how much time do participants spend on each
during the course of your YouthBuild program?
Workforce/Job-related activities…
Educational/Academic Activities…

Revised Program Hours Questions
Now we would like to ask some more specific
questions about some of the services you offer and
how time is allocated among these various activities.
Q42. How often do youth participate in construction
activities—daily (for some portion of the day);
alternating days within a one-week period; weekly (one
week on/one week off); or something else?
Q43. How often do youth participate in academic
classes of some sort—daily (for some portion of the
day); alternating days within a one-week period; weekly
(one week on/one week off); or something else?
Q44. On average, do your participants spend more
time in academic activities or construction activities?

Additional Response Categories

For questions that contained a list of response categories, we probed respondents to identify
any missing categories that they thought should be included in the list. In most cases, we added
the suggested categories. The response options that we added based on our pretest findings are
highlighted in yellow in the questionnaire (Appendix C).
D. ADDITIONAL FINDINGS FROM RESPONDENT DEBRIEFINGS
The respondent debriefing protocol included several broad questions related to the survey.
Specifically, we asked:
• Were you right person to answer this survey?
• Was there anything important about the program that we didn’t address?
• How long did the survey take?

MEMO TO: Eileen Pederson, YouthBuild Project Officer, ETA
FROM:
Lisbeth Goble, Lindsay Wood, Kim Mook, and Lisa Schwartz
DATE:
3/7/2012
PAGE:
8
In designing the survey, we assumed that the program director would be the most
appropriate person to complete the survey. However, we wanted to find out if there were any
other administrative officials or program staff who would be better suited to be the primary
respondent. The program directors who completed the survey universally agreed that they were
the right individuals to respond, as they felt they had the best understanding of how their
programs operate. Some respondents used other resources or collaborated with colleagues to
complete some of the questions.
We also asked respondents if there were any topic areas we overlooked in the content of the survey.
In general, respondents felt the survey was comprehensive. They had a few suggestions for topic areas
they would be interested in, such as the role of charter schools and local non-governmental organizations
in supporting YouthBuild. These topics are outside the scope of this survey and were not added.

Finally, we documented the duration of each pretest survey administration to ensure that the
average time burden was appropriate for respondents. These documented survey durations are
shown in Table 4.
TABLE 4. PRETEST SURVEY ADMINISTRATION TIMES
Pretest Mode

Duration (Minutes)

Self-administered 1

Self-administered 2

Interviewer-administered 1

Interviewer-administered 2

Cognitive think-aloud 1

111*

Cognitive think-aloud 2

140*

Average Administration Time

*Think-aloud interviews were not included in the calculation of average questionnaire duration because they were
intended to gather data on respondent perceptions and opinions during the course of the interview. As a result, these
interviews were artificially lengthy.

The average administration time was 41 minutes excluding the think-aloud interviews. This
estimate is somewhat high due to one lengthy self-administered survey that lasted 55 minutes.
During this interview, the respondent asked for input from another staff member at the site and
they ended up discussing most of the terms in the survey. The respondent indicated that she was
very interested in the survey, and that her discussion with her colleague reflected their
intellectual curiosity about the study rather than confusion about the terms in the questionnaire.
The respondent did not think the survey would have taken that long if she hadn’t discussed so
many of the items with her colleague. We anticipate that administration time for the web version
of the survey will be shorter, approximately 30 minutes.
cc: Cynthia Miller

APPENDIX A:
YOUTH BUILD – GRANTEE SURVEY
PRETEST COGNITIVE INTERVIEW PROTOCOL

YOUTHBUILD – GRANTEE SURVEY
PRETEST COGNITIVE INTERVIEW PROTOCOL
INTERVIEWER: READ CONSENT SCRIPT AT NORMAL INTERVIEWING PACE, THEN ASK
FOLLOW-UP QUESTIONS.
Phone Interview Script:
(Hello, my name is [NAME] and I’m calling from Mathematica Policy Research (Mathematica), a research company
in Princeton, New Jersey.) I’m calling you regarding the evaluation of the YouthBuild program sponsored by the
U.S. Department of Labor, Employment and Training Administration. You should have received a letter from us
describing the study.
As we mentioned in our letter, we are currently testing a questionnaire that will be used as part of the evaluation of
the YouthBuild program. We expect that the final questionnaire will take about 10 minutes to complete. Today, we’ll
need about 90 minutes of your time to help us test the questionnaire. As we go through the questionnaire, I’ll ask
you if you think the questions are clear and easy to understand and answer, and whether or not the response
options are appropriate. I really want to hear your opinions and reactions to the questionnaire, so don’t hesitate to
speak up whenever something is unclear, hard to answer, or doesn’t seem to apply to your program.
There are no right or wrong answers, I just want to find out as much as possible about what you are thinking.
Everything that you tell me is confidential. You don’t have to answer any questions that make you feel
uncomfortable.
I will be tape-recording the interview so that I can go back and listen to it later if I need to when I’m summarizing the
results.
Do you have any questions before we start?
Let’s begin.
In Person Interview Script:
We are currently testing a questionnaire that will be used as part of the evaluation of the YouthBuild program. We
expect that the final questionnaire will take about 10 minutes to complete. Today, we’ll need about 90 minutes of
your time to help us test the questionnaire. As we go through the questionnaire, I’ll ask you if you think the questions
are clear and easy to understand and answer, and whether or not the response options are appropriate. I really
want to hear your opinions and reactions to the questionnaire, so don’t hesitate to speak up whenever something is
unclear, hard to answer, or doesn’t seem to apply to your program.
There are no right or wrong answers, I just want to find out as much as possible about what you are thinking.
Everything that you tell me is confidential. You don’t have to answer any questions that make you feel
uncomfortable.
I will be tape-recording the interview so that I can go back and listen to it later if I need to when I’m summarizing the
results.
Do you have any questions before we start?
Let’s begin.

Prepared by Mathematica Policy Research

INTRODUCTION
MDRC and its research partners Mathematica Policy Research (Mathematica) and Social Policy Research Associates are
conducting an evaluation of the YouthBuild program on behalf of the U.S. Department of Labor, Employment and Training
Administration (ETA) and the Corporation for National and Community Service (CNCS). As part of the evaluation, we are
asking YouthBuild grantees to complete a short survey about their program. The survey covers several topics including
the organizational structure of your program, participant characteristics, the recruitment and enrollment processes, and
program services. Your participation will help ETA and CNCS better understand how YouthBuild programs function and is
mandatory for anyone receiving a grant.
We have designed a web-based survey to help you provide the requested data in a simple, user-friendly format and ask
that your program director take a few minutes to fill out this survey.
This study will help us better understand the impact of YouthBuild on participants’ post-program employment and
education outcomes as well as gain a better understanding of how YouthBuild is implemented in programs around the
country. Individual responses will not be attributed to specific individuals or organizations. Responses to this data
collection will be used only for statistical purposes. The reports prepared from this survey will summarize findings across
all YouthBuild sites and individual forms will not be available to anyone outside the study team, except as required by law.
The survey should take approximately 30 minutes to complete. If you have any questions as you complete this
questionnaire, please contact Shawn Marsh at Mathematica Policy Research at 1-877-894-4589 or
[email protected]
INTERVIEWER PROBES:
Do you have any questions about what I’ve read so far?
Tell me in your own words what this study is about.
Now think about everything I read to you (from the consent script). Do you think it was too much information, too little, or
the right amount?
• (If too much) Let’s read it through together one more time. As I read through the script again, please tell me
which parts you would cut out and why.
• (If too little) What kind of things do you think are missing? What would you like to know that isn’t covered in
the script?
To start, we would like to ask you some questions about the structure and funding of your organization.
1.

How long has your YouthBuild program been in operation?
1

□

Less than 1 year

□

1-3 years

□

4-6 years

□

7-10 years

□

More than 10 years

Prepared by Mathematica Policy Research

REMOVED FROM THIS VERSION OF THE SURVEY

3a.

Please provide the start and end dates of the fiscal year during which you received the 2011 YouthBuild
grant:

3b.

START DATE: |

| |/| | |/|
Month
Day

| |
Year

END DATE:

| |/| | |/|
Month
Day

| |
Year

What is your YouthBuild program’s total operating budget for that fiscal year? (Please round to the nearest
dollar, no decimals.)
$|

|,|

| AMOUNT

INTERVIEWER PROBES: How did you come up with your answer?
What does the phrase “fiscal year” mean to you? How does the DOL grant cycle fit in
with your program’s fiscal year?
(If YouthBuild is part of a broader agency), When answering this question, were you
thinking about the operating budget for your agency overall or just for the YouthBuild
program? Does YouthBuild have its own, separate operating budget?
4.

Is your program completely funded by a DOL grant?
1
0

□
□

Yes

SKIP TO Q.7

(IF Q.4=No) What other organization(s) fund your program?
MARK ALL THAT APPLY
1
2
3
4

□
□
□
□

Corporation for National and Community Service (CNCS)
Private foundation funding
State or local funds
Other (Specify)

INTERVIEWER PROBES: Was it easy or hard to decide which answer(s) to choose?
Are there any categories that are missing or that you did not understand?

Prepared by Mathematica Policy Research

Who is the predominant funder for your YouthBuild program?
1

□

DOL

□

Another public source

□

Private sources

□

Other (Specify)

INTERVIEWER PROBES: What does ‘predominant funder’ mean to you in the context of this question?
Can you give me an example of a public funding source? What about a private
source?
Please answer the following questions thinking about all the staff employed by your YouthBuild program.
7.

How long has your program director worked at this YouthBuild site?
MARK ONE ONLY
1

□

Less than 1 year

□

1-3 years

□

4-6 years

□

7-10 years

□

More than 10 years

INTERVIEWER PROBES: What does the term “program director” mean to you in the context of this question?
Does the title ‘program director’ describe the senior executive for this program, or is
there some other title that you use?
8.

What is your program director’s highest level of education?
1

□
□

Less than high school
High school diploma or GED equivalency

□

Some college, no degree

□

Associate’s degree

□

Bachelor’s degree

□

Master’s degree or more

□

Something else (Specify)

Prepared by Mathematica Policy Research

How many full-time equivalent (FTEs) staff of the following types does your program currently employ?
a. Educational instructor ............................................... |

b. Classroom aide ......................................................... |

c. Vocational instructor ................................................. |

d. Case manager .......................................................... |

e. Leadership development instructor........................... |

Volunteer coordinator................................................ |

g. Other (Specify) .......................................................... |

INTERVIEWER PROBES: Are there any staff types that are missing or do not make sense?
What does “full-time equivalent (FTE) staff” mean to you? How easy or difficult was it
to calculate the number of “full-time equivalent (FTEs) staff” your program currently
employs? What did you consider “full time?”
Let’s go back through the answer categories. For each one, tell me what that person
does and whether you use a different title to describe that person’s role in your
program. For example, at your program, what would an “educational instructor” do?
Do you call that person an educational instructor or something else? (go through the
whole list)
What about the person who directs the construction component of your program?
What do you call that person? Did you count them in any of the categories that we
listed? (If so,) which one? (If not), should we add that person here?
10.

How many part-time staff of the following types does your program currently employ?
a. Educational instructor ............................................... |

b. Classroom aide ......................................................... |

c. Vocational instructor ................................................. |

d. Case manager .......................................................... |

e. Leadership development instructor........................... |

Volunteer coordinator................................................ |

g. Other (Specify) .......................................................... |

Prepared by Mathematica Policy Research

11.

[Ask only if Q.10a does not equal 0] How many of the educational instructors you currently employ have
the following amount of YouthBuild experience?
a. Less than 1 year ....................................................... |

b. 1-3 years ................................................................... |

c. 4-6 years ................................................................... |

d. 7-10 years ................................................................. |

e. More than 10 years ................................................... |

INTERVIEWER PROBES: Tell me in your own words, what was this question asking?
Do these response options make sense?
How easy or difficult was it for you to come up with the amount of experience of your
educational instructors? Why is that?
12.

13.

[Ask only if Q.10a does not equal 0] How many of the educational instructors you currently employ have
completed the following education levels?
a. Less than high school ............................................... |

b. High school diploma or GED equivalency ................ |

c. Some college, no degree .......................................... |

d. Associate’s degree.................................................... |

e. Bachelor’s degree ..................................................... |

Master’s degree or more ........................................... |

g. Something else (Specify) .......................................... |

[Ask only if Q.10d does not equal 0] How many of your case managers have the following years of
YouthBuild experience?
a. Less than 1 year ....................................................... |

b. 1-3 years ................................................................... |

c. 4-6 years ................................................................... |

d. 7-10 years ................................................................. |

e. More than 10 years ................................................... |

Prepared by Mathematica Policy Research

14.

[Ask only if Q.10d does not equal 0] How many of your case managers have the following education levels?
a. Less than high school ............................................... |

b. High school diploma or GED equivalency ................ |

c. Some college, no degree .......................................... |

d. Associate’s degree.................................................... |

e. Bachelor’s degree ..................................................... |

Master’s degree or more ........................................... |

g. Something else (Specify) .......................................... |

INTERVIEWER PROBES: How easy or difficult was this series of questions about your YouthBuild employees?
What would make it easier to answer these questions?
These next questions are about the worksites you use for the construction component of the YouthBuild
program. If your program uses more than one worksite, please answer these questions based on your
PRIMARY worksite.
INTERVIEWER PROBES: What does the word ‘own’ mean to you in the context of this question?
What does the phrase ‘primary worksite’ mean to you in the context of this question?
15.

Does your program have a construction component?
1
0

16.

Yes
No

SKIP TO Q.21

(IF Q.15=Yes) Does your program own its worksite?
1
0

17.

□
□
□
□

Yes

SKIP TO Q.18

(IF Q.16=No) Who owns your (primary) worksite?

INTERVIEWER PROBES: What does the word ‘own’ mean to you in the context of this question?
What does ‘primary worksite’ mean to you in the context of this question?

Prepared by Mathematica Policy Research

18.

Do you manage the worksite(s)?
INTERVIEWER NOTE: IF YOU HAVE MULTIPLE WORKSITES, DO YOU MANAGE THE PRIMARY
WORKSITE?
1
0

19.

□
□

Yes

SKIP TO Q.20

(IF Q.18=No) Who manages your (primary) worksite?

INTERVIEWER PROBE:
20.

What does the word ‘manage’ mean to you in the context of this question?

Does your construction site(s) focus on rehabilitation of existing homes or apartments, new construction
or both?
MARK ALL THAT APPLY
1
2
3

□
□
□

Rehabilitation
New construction
Something else (Specify)

INTERVIEWER PROBES: What does it mean to say that your construction site focuses on these kinds of
activities? Would the question mean something different if we asked you, “Does the
construction component of your program focus on …?” What’s the difference?
Which is the better way of asking this question and why?
Do these response options make sense to you? What did you include as
“rehabilitation”? What did you include as “new construction”
Are there any response options missing? Do you ever demolish buildings as part of
your program? (If so), which answer choice would you pick for “demolition.”
Now, we would like to ask you some questions about the individuals your program serves.
21.

Since you received your 2011 DOL grant, how many individuals have applied to be in your YouthBuild
program? (By apply, we mean individuals who try to enroll in the program but who have not yet been determined
to be eligible for the program.)
|

| NUMBER OF INDIVIDUALS

INTERVIEWER PROBES: Is the definition of ‘apply’ that we provide in this question the same as you would
define it?
For the purposes of the study, it’s important that we be able to distinguish between
people who apply to the program and those who enroll and receive YouthBuild
services. How do you describe these two groups?
How easy or difficult was it for you to come up with number of people who have
applied to your program? Why is that?

Prepared by Mathematica Policy Research

22.

How many of the individuals who applied are currently enrolled in your program? (By enroll, we mean
individuals who completed Mental Toughness Orientation, if you have it, and satisfied all other requirements in
order to start your program.)
|

| NUMBER OF INDIVIDUALS

INTERVIEWER PROBES: What does the word ‘enrolled’ mean to you in the context of this question?
How easy or difficult was it for you to come up with number of people enrolled in your
program? Why is that?
In answering this question, were you thinking of only those participants who are
covered by DOL funds or did you include all participants regardless of the source of
funding?
23.

What is the maximum number of individuals that could be enrolled in your program between April 2011 and
March 2012?
|

| MAXIMUM NUMBER OF INDIVIDUALS

INTERVIEWER PROBES: How easy or difficult was this series of questions about the individuals your program
serves?
How did you figure out what the maximum number of individuals would be? Talk me
through what went through your mind when you tried to answer this question. (You
may need to demonstrate how to “think aloud” here. You might say something like,
“For example, if I were thinking about the maximum number of people I could have at
a dinner party, I might say, “Well, my dining room table can seat 10 people pretty
comfortably but if we just did a buffet, we could probably have more people, so
maybe the maximum is 20 people because I think it will feel too crowded with more
than that.” I’d like you to do the same kind of thinking out loud so that I can
understand how you came up with your maximum number of participants.”

Prepared by Mathematica Policy Research

Next we have some questions about how you recruit
and enroll individuals into your program.
24.

How do you recruit individuals into your
program? Do you…

During which months of the year do you enroll
young adults in your program? (By enroll, we
mean young adults who complete MTO, if you have
it, and satisfy all other requirements in order to start
your program.)

INTERVIEWER NOTE: READ ALL RESPONSE
OPTIONS

MARK ALL THAT APPLY

1
2

5
6
7

□
□
□
□
□
□
□

26.

Get referrals from high schools

Get referrals from courts or corrections
agencies

Get referrals from outreach that other agencies
do in the community?

Get word-of-mouth referrals from people in the
community or former participants?

Actively reach out in the community yourselves

Receive self-referrals or walk-ins
7

Recruit individuals in some other
way? (Specify)

INTERVIEWER PROBES: Thinking of the response
categories, are there any missing?

25.

Of the recruiting methods listed below, which
method is your main source of recruiting
individuals into your program? Do you…

27.
INTERVIEWER NOTE: READ ALL RESPONSE
OPTIONS
MARK ONE ONLY
1
2

5
6
7

□
□
□
□
□
□
□

□
□
□
□
□
□
□
□
□
□
□
□

April 2011
May 2011
June 2011
July 2011
August 2011
September 2011
October 2011
November 2011
December 2011
January 2012
February 2012
March 2012

At what point in your application process do you
determine whether the applicant complies with
DOL requirements for eligibility in the
YouthBuild program?

Get referrals from high schools

MARK ALL THAT APPLY

Get referrals from courts or corrections
agencies

1
2

Get referrals from outreach that other agencies
do in the community?
Get word-of-mouth referrals from people in the
community or former participants?

□
□
□

At initial application
During, or immediately following, an
orientation meeting
After Mental Toughness Orientation

Receive self-referrals or walk-ins

INTERVIEWER PROBES: How did you interpret the
phrase “complies with DOL requirements for eligibility?”
Does it make sense to talk about DOL-specific
requirements? Why or why not?

Recruit individuals in some other
way? (Specify)

Do the response options for this question make sense or
are there other points when you determine eligibility?

Actively reach out in the community yourselves

Prepared by Mathematica Policy Research

28.

Do you have a formal screening process prior to
Mental Toughness Orientation (MTO) for those
who apply to your program?
1
0
2

□
□
□

Now we have some questions about your Mental
Toughness Orientation (MTO), if you have one.
30.

Yes
No

We do not have an MTO

INTERVIEWER PROBES: What does ‘formal screening
process’ mean to you in the context of this question?

31.

Suggested addition—I noticed that you hesitated. Tell me
what you were thinking.
If they pick “We do not have MTO”—Would you have
answered this differently if we said, “Other than MTO, do
you have a formal screening process prior to enrolling
someone in your program?”
(IF YES): How would you have answered that question?
29.

Which of the following are a part of your
application screening process?
INTERVIEWER NOTE: READ ALL RESPONSE
OPTIONS
MARK ALL THAT APPLY
1

□
□
□
□
□
□
□

Placement test
Test of basic skills

Does your program have a Mental Toughness
Orientation (MTO)?

□
□

Yes
No

SKIP TO Q.32

(IF Q.30=Yes) Of the individuals who apply to
your program, how many do you typically invite
to MTO?
|

| INDIVIDUALS INVITED TO
PARTICIPATE
SKIP TO Q.33

INTERVIEWER PROBES: How did you come up with your
answer?
What timeframe are you thinking of when you hear the
phrase ‘this year’?
How does your program enrollment relate to your MTO, if
you have one? If more clarification needed: In other words,
in your program are youth considered enrolled before
MTO, only after MTO, or throughout the entire process?
What was your reaction to the phrase “invite to MTO.” Is
that how you would describe it?
32.

Interview with staff member

(IF Q.30=No) On average, how many individuals
who apply to the program are invited to
participate in your YouthBuild program?

Staff observation

Paper application
Personal statement
Other (Specify)

INTERVIEWER PROBES: Are there any categories that
are missing or do not make sense?
Let’s go back through the list. For each item, please tell me
if it’s something you do prior to MTO, during MTO, or after
MTO, if your program has MTO.

INTERVIEWER PROBES: How easy or difficult was it to
calculate the number of individuals invited to participate in
your YouthBuild program? Tell me a little more about how
you arrived at your answer.
33.

(IF Q.30=Yes) How long is your MTO, in days?
|

| NUMBER OF DAYS

INTERVIEWER PROBES: How easy or difficult was it to
calculate the length of your MTO in days? Tell me a little
more about how you arrived at your answer.

(If they do something at more than one point in time): tell
me how this fits in with your screening process.
(If they only do something during or after MTO): if I wanted
to only pick up those things that you do before you decide
to invite someone to MTO or to enroll in your program, how
should I ask this question?

Prepared by Mathematica Policy Research

| INDIVIDUALS INVITED TO
PARTICIPATE
SKIP TO Q.38a

34.

On average, how many hours per day do
candidates spend in MTO?
1
2
3
4
5

□
□
□
□
□

Less than 2 hours a day

Once participants have completed MTO, on
average, how long does your YouthBuild
program last?
1

Between 2 and 4 hours a day

Between 4 and 6 hours a day

3
4

Between 6 and 8 hours a day

More than 8 hours a day

INTERVIEWER PROBES: What does the word ‘candidate’
mean to you in the context of this question?
How easy or difficult was it to calculate the number of
“hours per day?” Tell me a little more about how you
arrived at your answer.
35.

37.

Do the response options for this question make sense to
you?
38a. Do participants who enroll in your YouthBuild
program receive wages or stipends?
1

MARK ALL THAT APPLY

2
3
4
5
6
7

□
□
□
□
□
□
□

Team building activities

Yes

GO TO Q.38b

SKIP TO Q.39

Academic preparation
Workforce preparation
Goal-setting exercises

INTERVIEWER PROBES: What does the term “wages”
mean to you? What about “stipend?”
Do participants get monetary compensation for any other
reason while enrolled in the program? Do you consider this
compensation a part of the ‘wage’ or ‘stipend?’

Physical fitness activities
Other (Specify)

38b. How often do participants receive wages or
stipends?

INTERVIEWER PROBES: Are there any categories
missing or do not make sense?

Let’s go back through the answer categories. For each
one, tell me what types of activities would be included and
whether you use a different title to describe that category of
activities. (Go through the whole list).

On average, how many individuals who
participate in MTO are invited to participate in
your YouthBuild program?
|

□
□

Orientation to YouthBuild program

36.

Less than 6 months
6-9 months
10-12 months
13-18 months
More than 18 months

INTERVIEWER PROBES: For your program, what does it
mean to complete MTO?

What activities are offered during MTO?

□
□
□
□
□

| YOUNG ADULTS INVITED
TO PARTICIPATE

INTERVIEWER PROBES: During MTO, do people decide
that the program is not for them and drop out? Does your
program use MTO to determine whether someone is ready
for YouthBuild? How did you calculate the number of
people who are “invited to participate in your program?” If
needed: Did you exclude drop outs, people you decide
aren’t ready, or both?

□
□
□
□

Daily
Weekly
Monthly
Other (Specify)

INTERVIEWER PROBES: Are there any categories that
are missing (ex. One lump sum, bi-weekly) or does
something else make sense to you?
38c. How much do participants receive [FILL
ANSWER CHOICE FROM Q.38b] as wages or
stipends?
$|

|,|

| AMOUNT

INTERVIEWER PROBES: Do all participants receive the
same amount of wages or stipends?
Are there any reasons or situations in which someone
might get a pay increase or decrease? Can you describe
those?
If yes – What are examples of these situations?

Prepared by Mathematica Policy Research

Next, we would like to ask you some questions about the different types of services your YouthBuild program
offers to its participants.
39.

Which of the following workforce or job related services does your program offer, either on site or as a
referral? Does your program offer…
MARK ALL THAT APPLY

Offered
On-Site
a. Vocational/occupational skills training services ....................................

b. Pre-apprenticeship programs ...............................................................

On-the-job training (OJT) ......................................................................

d. (Does your program offer) Subsidized work experience/internships ....

e. Unsubsidized work experience .............................................................

Workforce information services ............................................................

g. Work readiness training ........................................................................

h. Career/Life skills training ......................................................................

Does your program offer some other type of workforce or job related
training (Specify) ...................................................................................

□
□
□
□
□
□
□
□
□

Offered as
Referral
2

□
□
□
□
□
□
□
□
□

Not
Offered
3

□
□
□
□
□
□
□
□
□

INTERVIEWER PROBES: What does “on-site” mean to you in the context of this question?
What does “referral” mean to you in the context of this question?
If they offer referral services – Are these services funded by your YouthBuild grant? Do you consider these services part
of your YouthBuild program, or as referrals to outside organizations?
Let’s go back through the answer categories. For each one, tell me what types of services would be included and whether
you use a different title to describe that category of services. For example, at your program, what would an example of
“vocational/occupational skills training services” be? Do you call that vocational/occupational skills training services or
something else? (go through the whole list)
Do these response options make sense in the context of this question; are there any response options missing?

Prepared by Mathematica Policy Research

40.

Which of the following educational or academic services does your program offer, either on site or as a
referral? Does your program offer…
MARK ALL THAT APPLY

Offered
On-Site
a. GED preparation ............................................................................................

b. Language arts (reading, writing, literacy) remediation (independent of
GED preparation)...........................................................................................

Math remediation (independent of GED preparation) ...................................

d. Does your program offer other academic tutoring services ..........................

e. Does your program offer a High school diploma program.............................

Preparation for standardized achievement tests, such as state or
local tests .......................................................................................................

g. ESL training ...................................................................................................

h. Does your program offer some other type of education activities or
academic services (Specify) ..........................................................................

□
□
□
□
□
□
□
□

Offered as
Referral
2

Not
Offered

□
□
□
□
□
□
□
□

INTERVIEWER PROBES: Let’s go back through the answer categories. For each one, tell me what types of services
would be included and whether you use a different title to describe that category of services. For example, at your
program, what would an example of “GED preparation” be? Do you call that GED preparation services or something else?
(go through the whole list).
Do these response options make sense in the context of this question; are there any response options missing?

Prepared by Mathematica Policy Research

41.

Which of the following post-secondary education preparation services does your program offer, either on
site or as a referral? Does your program offer…
MARK ALL THAT APPLY

Offered
On-Site
a. College awareness or college guidance activities ..........................................

b. Preparation for college entrance examinations ..............................................

College application assistance .......................................................................

d. Financial aid assistance .................................................................................

e. College preparation or transition programs ....................................................

Courses for college credit ...............................................................................

g. Other post-secondary preparation (Specify) ..................................................

□
□
□
□
□
□
□

Offered as
Referral
2
2
2
2
2
2
2

Not
Offered

□
□
□
□
□
□
□

3
3
3
3
3
3
3

□
□
□
□
□
□
□

INTERVIEWER PROBES: Let’s go back through the answer categories. For each one, tell me what types of services
would be included and whether you use a different title to describe that category of services. (go through the whole list).
What does “college awareness or college guidance activities” mean to you in the context of this question? Is this different
from “college preparation or transition programs”?
Do these response options make sense in the context of this question; are there any response options missing?
42.

Which of the following community service opportunities, not including your site’s construction work, does
your program offer, either on site or as a referral? Does your program offer…
MARK ALL THAT APPLY

Offered
On-Site
a. Public beautification projects (park or highway cleanup, mural painting,
etc.) ................................................................................................................

b. After-school homework help or mentoring (for example, Big Brother
program) ........................................................................................................

Activities with people with disabilities ............................................................

d. Elderly care activities .....................................................................................

e. Community activism.......................................................................................

Other community services opportunities (Specify) ........................................

□
□
□
□
□
□

Offered as
Referral
2

Not
Offered

□
□
□
□
□
□

INTERVIEWER PROBES: Let’s go back through the answer categories. For each one, tell me what types of services
would be included and whether you use a different title to describe that category of services. (go through the whole list).
Do these response options make sense in the context of this question; are there any response options missing?

Prepared by Mathematica Policy Research

43.

Which of the following leadership opportunities does your program offer, either on site or as a referral?
Does your program offer…
MARK ALL THAT APPLY

Offered
On-Site
a. Youth Policy Council .............................................................................

b. In-class leadership roles .......................................................................

Construction site leadership roles ........................................................

d. Opportunities to speak at public/governmental meetings.....................

e. Other leadership opportunities (Specify) ..............................................

□

Offered as
Referral
2

□

Not Offered

□

□
□
□
□
□

INTERVIEWER PROBES: Let’s go back through the answer categories. For each one, tell me what types of services
would be included and whether you use a different title to describe that category of services. (go through the whole list).
Do these response options make sense in the context of this question; are there any response options missing?
44.

Which of the following health services does your program offer, either on site or as a referral? Does your
program offer…
MARK ALL THAT APPLY

Offered
On-Site
a. Substance abuse services ....................................................................

b. Mental health treatment ........................................................................

Preventative medicine ..........................................................................

d. Emergency medical care ......................................................................

e. Pregnancy leave/prenatal care .............................................................

Other health services/activities (Specify)..............................................

□
□
□
□
□
□

Offered as
Referral
2

Not Offered

□
□
□
□
□
□

INTERVIEWER PROBES: Let’s go back through the answer categories. For each one, tell me what types of services
would be included and whether you use a different title to describe that category of services. (go through the whole list).
Do these response options make sense in the context of this question; are there any response options missing?

Prepared by Mathematica Policy Research

45.

Which of the following support services does your program offer, either on site or as a referral? Does your
program offer…
MARK ALL THAT APPLY

Offered
On-Site
a. Mentors to students ...................................................................................

b. Visits to the homes of students participating in your program ...................

A process to connect students with social services, such as
social workers ............................................................................................

d. Therapeutic services for socio-emotional and/or behavioral problems .....

e. Activities to support students’ acquisition of “life skills” .............................

Cultural enrichment activities .....................................................................

g. Activities to increase and/or improve parental and family involvement in
students’ school lives .................................................................................

h. Case management ....................................................................................

Other support services not listed (Specify) ................................................

□
□

Offered as
Referral
2

□
□
□
□

□
□
□

Not Offered

□
□

□
□
□
□

□
□
□

□
□
□
□
□
□
□
□
□

INTERVIEWER PROBES: Let’s go back through the answer categories. For each one, tell me what types of services
would be included and whether you use a different title to describe that category of services. (go through the whole list).
Do these response options make sense in the context of this question; are there any response options missing?
46.

Which of the following post-program services does your program offer, either on site or as a referral?
Does your program offer…
MARK ALL THAT APPLY

Offered
On-Site
a. Additional skills training .............................................................................

b. Subsidized work experience ......................................................................

Internships .................................................................................................

d. Job preparation classes or activities..........................................................

e. GED preparation or mentoring ..................................................................

Mentoring ...................................................................................................

g. Other post-program services (Specify) ......................................................

□
□
□
□
□
□
□

Offered as
Referral
2
2
2
2
2
2
2

Not Offered

□
□
□
□
□
□
□

3
3
3
3
3
3
3

□
□
□
□
□
□
□

INTERVIEWER PROBES: Let’s go back through the answer categories. For each one, tell me what types of services
would be included and whether you use a different title to describe that category of services. (go through the whole list).
Do these response options make sense in the context of this question; are there any response options missing?

Prepared by Mathematica Policy Research

Now we would like to ask some more specific questions about some of the services you offer and how time is
allocated among these various activities.
47.

Of all the activities and services you listed above how much time do participants spend on each during the
course of your YouthBuild program?
Number of Hours

a. Workforce/Job-related Activities ................................................................................................

b. Educational/Academic Activities ................................................................................................

Post-secondary Education Preparation Activities ......................................................................

d. Community Service Activities ....................................................................................................

e. Leadership Activities ..................................................................................................................

Health Services ..........................................................................................................................

g. Support Services .......................................................................................................................

h. Post-Program Services ..............................................................................................................

INTERVIEWER PROBES: Does measuring time spent in these activities in hours make sense, or would it make more
sense to measure time in some other way?
Would it be easier to report the time allocation in percentages or numbers? Why is that?
Would it be easier to report the time spent in various services immediately after we ask about those services? For
example, let’s go back to Q.39. Would it be easier or harder if we asked about time spent in workforce/job-related
activities right after this question?
Do most of the individuals who participate in your program use the same services or does service use vary by individual
need? (If varied), how did you account for this variation when you figured out how much time participants spend in various
services?
Did you include time spent in services that are offered as a referral? (IF YES) how do you know how much time is spent in
services that are not provided on site?
48.

How often do youth participate in construction activities?
1
2
3
4

□
□
□
□

Daily (for some portion of the day)
Alternating days within a one-week period
Weekly (one week on/one week off)
Other (Specify)

INTERVIEWER PROBES: Do these response categories make sense?

Prepared by Mathematica Policy Research

49.

How often do youth participate in academic classes of some sort?
1
2
3
4

□
□
□
□

Daily (for some portion of the day)
Alternating days within a one-week period
Weekly (one week on/one week off)
Other (Specify)

INTERVIEWER PROBES: What does the phrase “academic classes of some sort” mean to you in the context of this
question? What are the types of classes you would include here?
Do these response categories make sense?
50.

On average, do your participants spend more time in academic activities or construction activities?
1
2
3

□
□
□

Academic or education activities
Construction activities
About the same for each

INTERVIEWER PROBES: Do these response categories make sense?
Now, thinking more about your academic services…
51.

Do you offer academic services that happen in a group or classroom setting?
1
0

□
□

Yes
No

INTERVIEWER PROBES: What does the phrase “academic services” mean to you in the context of this question? Is this
different from academic classes?
52.

How often do these classes meet?
1
2
3
4

□
□
□
□

Daily
Every other day
Every other week
Other (Specify)

INTERVIEWER PROBES: Is this question asking you the same thing or something different from Q.49?
53.

What is the average size of these classes?
|

| # OF ATTENDEES

INTERVIEWER PROBES: How easy or hard is it for you to estimate class size?

Prepared by Mathematica Policy Research

54a. Are students divided into separate classes or groups depending on some criteria (academic, social, or
something else)?
1
0

□
□

Yes
No

SKIP TO Q.55

54b. How are students divided among classes?
1
2
3
4

□
□
□
□

Initial academic testing
Classroom observation
Student special needs
Other (Specify)

INTERVIEWER PROBES: Do these response categories make sense?
What does the term “student special needs” mean to you?
55.

Which of the following teaching methods are used in your YouthBuild program?
MARK ALL THAT APPLY
1
2
3
4
5
6
7
8
9
10
11

□
□
□
□
□
□
□
□
□
□
□

One-on-one assistance
Traditional whole group instruction or lecture
Small group project work
Independent work paired with individual tutoring
Computer instruction and exercises
Distance learning
Team teaching
Peer teaching
Self-paced instruction
Incorporation of “real world” implications and rewards in the classroom
Other (Specify)

INTERVIEWER PROBES: Thinking of the response options for this question, how would you define each of these? What
types of teaching methods would be included in each of these categories? (go through the whole list).
Do these response categories make sense?

Prepared by Mathematica Policy Research

56.

[AUTOFILL THE OPTIONS INDICATED IN Q.55] Of the teaching methods you indicated, please rank the three
methods used most often.
|

| [FILL FROM ABOVE]

INTERVIEWER PROBES: How easy/hard is it for you to rank the three methods used most often? OR How did you
determine which three methods are used most often?
57.

How often do you monitor student progress using the following strategies, either daily, weekly, monthly,
or something else? Do you monitor student progress through…
MARK ALL THAT APPLY

Daily
a. Formal evaluations .............................................

b. Meetings .............................................................

Testing, or ...........................................................

d. Do you use some other type of monitoring of
student progress? ...............................................

□
□
□
□

Weekly
2
2
2

□
□
□
□

Monthly
3
3
3

□
□
□
□

Somethin
g Else
4
4
4

□
□
□
□

Never
5
5
5

□
□
□
□

57e. (IF Q.57d=1-4, ASK) What other type of monitoring do you use?

INTERVIEWER PROBES: What does the word ‘evaluation’ mean to you in the context of this question? Do you consider
this to be different from testing?
Do the response options for the question make sense or is there some other way you monitor student progress?
58.

What is the ultimate educational goal of your YouthBuild program? Is it to obtain…
MARK ONE ONLY
1
2
3
4
5

□
□
□
□
□

a GED certificate,
a High school diploma,
basic academic improvement,
post-secondary education preparation, or
something else? (Specify)

INTERVIEWER PROBES: What does the phrase ‘ultimate educational goal’ mean to you in the context of this question?
What would you include as “basic academic improvement?” what about “post secondary education preparation – what did
you include there?

Prepared by Mathematica Policy Research

(ASK Q.59-Q.62 ONLY IF Q.43a = 1 OR 2) You mentioned that your YouthBuild program offers a Youth Policy
Council (YPC). We would like to ask a few questions about this activity.
59.

How often does your YPC meet? (Choose the statement that best describes the frequency.)
1
2
3
4
5

□
□
□
□
□

Less than once a month
1-2 times a month
3-4 times a month
More than 4 times a month
It varies

INTERVIEWER PROBES: Do these response categories make sense?
60.

How are individuals selected to be part of your YPC?
MARK ALL THAT APPLY
1
2
3
4
5

□
□
□
□
□

Youth are recommended by program staff
Youth are recommended by their peers
Youth are recommended by someone other than program staff or their peers
Youth apply to join the council, or
Are youth selected in some other way? (Specify)

INTERVIEWER PROBES: Do these response categories make sense; are there any response categories missing?
61.

Does your YPC have input into your program structure?
1
0

□
□

Yes

GO TO Q.62

SKIP TO Q.63

INTERVIEWER PROBES: What does “program structure” mean to you in the context of this question?

Prepared by Mathematica Policy Research

62.

What type of input does your YPC have into the program structure?
MARK ALL THAT APPLY
1
2
3
4
5
6
7
8
9
10

□
□
□
□
□
□
□
□
□
□

Program design
Staff hiring
Decisions about community service activities
Size of the youth stipends
Payment schedule of the youth stipends
Disciplinary actions
Selection of youth to present at community or governmental meetings
Design or decor of your program’s common spaces
Dress code for the program
Other (Specify)

INTERVIEWER PROBES: Thinking of the response options for this question, how would you define each of these? What
types of activities or decisions would be included in each of these categories?
Do these response categories make sense in the context of this question; are there any response categories missing?
Lastly, we have a few questions about parental involvement in your YouthBuild program.
63.

Thinking about all of the participants in the program, are their parents/guardians required to be involved in
the program?
1
0

64.

□
□

Yes
No

SKIP TO Q.65

(IF Q.63=YES) In which of the following activities are parents/guardians required to participate?
MARK ALL THAT APPLY
1
2
3
4
5

□
□
□
□
□

Student orientation
Volunteering at program site
Volunteering at construction site
Donating supplies to program
Other (Specify)

INTERVIEWER PROBES: What does “required to participate” mean to you in the context of this question?
Are there any response categories missing?

Prepared by Mathematica Policy Research

65.

Are there other ways that parents/guardians can volunteer to be involved in your program?
1
0

66.

□
□

Yes
No

SKIP TO END

(IF Q.65=YES) In which of the following activities can parents/guardians voluntarily participate?
MARK ALL THAT APPLY
1
2
3
4
5

□
□
□
□
□

Student orientation
Volunteer at program site
Volunteer at construction site
Donate supplies to program
Other (Specify)

Those are all the questions we have for you. Thank you so much for your time and information.

Persons are not required to respond to this collection of information unless this survey displays a currently valid OMB control number
(OMB xxxx-xxxx, expires xx/xx/xxxx). Responding to this questionnaire, which seeks to help the U.S. Department of Labor
understand how YouthBuild programs are serving disadvantaged youth, is voluntary. Public reporting burden for this collection of
information is estimated to average 30 minutes per response, including the time for reviewing instructions, searching existing data
sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments
regarding this burden estimate to Eileen Pederson, U.S. Department of Labor, Employment and Training Administration, Office of
Policy Development and Evaluation, Room N-5641, 200 Constitution Avenue, NW, Washington, DC 20210.
Do NOT send the completed questionnaire to this address.

INTERVIEWER PROBES: Based on what you know about the study so far, are you, the program director, the appropriate
person to be speaking with?
Were there any questions in this interview that you did not know how to answer? Tell me more about that.
Was it easy or difficult to answer questions about the application and enrollment process?
Were there any questions in this interview that made you feel uncomfortable? Which questions?
This was the last question I had. Did you have any other comments about the questionnaire?
Thank you for helping us test the questionnaire.

Prepared by Mathematica Policy Research

MPR Reference No.: 06796

YouthBuild
Grantee Survey
March 6, 2012

ALL

INTRODUCTION
MDRC and its research partners Mathematica Policy Research and Social Policy Research
Associates are conducting an evaluation of the YouthBuild program on behalf of the
U.S. Department of Labor, Employment and Training Administration (ETA) and the
Corporation for National and Community Service (CNCS). As part of the evaluation, we are
asking YouthBuild grantees to complete a short survey about their program. The survey
covers several topics, including the organizational structure of your program, participant
characteristics, the recruitment and enrollment processes, and program services. Please note
that, except where specified, we are interested in learning about your YouthBuild program as
a whole, not only the components of your program that are tied directly to DOL funding. Your
participation will help ETA and CNCS better understand how YouthBuild programs function
and is mandatory for anyone receiving a grant.
We have designed a web-based survey to help you provide the requested data in a simple,
user-friendly format and ask that your program director take a few minutes to fill out this
survey.
This study will help us better understand the impact of YouthBuild on participants’ postprogram employment and education outcomes as well as gain a better understanding of how
YouthBuild is implemented in programs across the country. Individual responses will not be
attributed to specific individuals or organizations. Responses to this data collection will be
used only for statistical purposes. The reports prepared from this survey will summarize
findings across all YouthBuild sites and individual forms will not be available to anyone
outside the study team, except as required by law.
The survey should take approximately 30 minutes to complete. If you have any questions,
please contact Shawn Marsh at Mathematica Policy Research at 1-877-894-4589 or
[email protected].

ALL
1

How long has your YouthBuild program been in operation under any agency funding?
Select one only

Less than 1 year ......................................................................................1
1-3 years .................................................................................................2
4-6 years .................................................................................................3
7-10 years ...............................................................................................4
More than 10 years..................................................................................5
NO RESPONSE ......................................................................................Empty

ALL
2

Which organization(s) fund your program?
Select all that apply

Department of Labor (DOL) .....................................................................1
Corporation for National and Community Service (CNCS) .......................2
Private foundation funding .......................................................................3
State or local funds ..................................................................................4
Some other organization .........................................................................5
(Please specify_______________________________) (STRING 255)
NO RESPONSE ......................................................................................Empty SKIP TO Q4a

SOFT CHECK: IF Q2.1 NE 1 (yes) AND Q2.2 NE 2 (yes) (neither DOL nor CNCS selected): Please
verify which organization(s) fund your program. Please confirm and keep your response or
change your response to continue.

PROGRAMMER SKIP BOX 2.1
IF NUMBER OF ITEMS SELECTED AT Q2 IS GT 1, CONTINUE TO Q3.
ELSE, SKIP TO Q4A.
IF EMPTY, SKIP TO Q4A

NUMBER OF ITEMS SELECTED AT Q2 IS GT 1
FILL OTHER SPECIFY RESPONSE FROM Q2.5 IF Q2.5 IS SELECTED;
IF NO OTHER SPECIFY TEXT IS ENTERED AT Q2.5, FILL Some other organization
3

Who is the predominant funder for your YouthBuild program?
Select one only

PROGRAMMER: ONLY DISPLAY RESPONSES SELECTED AT Q2.
Department of Labor (DOL) .....................................................................1
Corporation for National and Community Service (CNCS) .......................2
Private foundation funding .......................................................................3
State or local funds ..................................................................................4
[FILL OTHER SPECIFY RESPONSE / Some other organization] ............ 5
NO RESPONSE ......................................................................................Empty
PROGRAMMER DISPLAY BOX 4.1
DISPLAY Q4A AND Q4B ON THE SAME SCREEN

ALL
4a

Please provide the start and end dates of your fiscal year during which you received
the 2011 YouthBuild grant:
START DATE:

|/|

| (MM/DD/YYYY)

END DATE:

|/|

| (MM/DD/YYYY)

SOFT CHECK: IF START_YEAR LT 2010; You responded that the start date of your fiscal year
is before 2010. Please confirm and keep your response or change your response to continue.
SOFT CHECK: IF START_YEAR GT 2012; You responded that the start date of your fiscal year
is after 2012. Please confirm and keep your response or change your response to continue.
SOFT CHECK: IF END_YEAR LT 2011; You responded that the end date of your fiscal year is
before 2011. Please confirm and keep your response or change your response to continue.
SOFT CHECK: IF END_YEAR GT 2012; You responded that the end date of your fiscal year is
after 2012. Please confirm and keep your response or change your response to continue.
SOFT CHECK: IF START_YEAR GT END_YEAR; You responded that the start date of your
fiscal year is after the end date of your fiscal year. Please confirm and keep your response or
change your response to continue.
SOFT CHECK: IF END_YEAR EMPTY; Your response to this question is important. Please confirm
and keep your response or change your response to continue.

ALL
IF Q2 = MORE THAN 1 RESPONSE, FILL Please include all funding sources.
4b

What is your YouthBuild program’s total operating budget for that fiscal period?
[Please include all funding sources.] Please round to the nearest dollar.
$|

|,|

| AMOUNT (8 NUM; 0 – 99,999,999)

NO RESPONSE ......................................................................................Empty
SOFT CHECK: IF Q4b LT $450,000 OR GT $1,000,000; You reported that your total operating
budget for that fiscal period is $[fill from 4b]. Please confirm and keep your response or
change your response to continue.
PROGRAMMER DISPLAY BOX 5.1
DISPLAY Q5 AND Q6 ON THE SAME SCREEN

ALL
You reported that your YouthBuild program has been in operation for [FILL as specified
below] under any agency funding IF Q1 NE EMPTY; ELSE DO NOT FILL
less than 1 year IF Q1 = 1; 1-3 years IF Q1 = 2; 4-6 years IF Q1 = 3; 7-10 years IF Q1 = 4; more
than 10 years IF Q1 = 5
Please answer the following questions thinking about all the staff who currently work for your
YouthBuild program.
5

[You reported that your YouthBuild program has been in operation for [FILL as
specified above] under any agency funding.] How long has your program director
worked at this YouthBuild site?
Select one only

ALL
6

What is your YouthBuild program director’s highest level of education?
Select one only

Less than high school ..............................................................................1
High school diploma or GED equivalency ................................................2
Some college, no degree .........................................................................3
Associate’s degree ..................................................................................4
Bachelor’s degree ...................................................................................5
Master’s degree or more .........................................................................6
Something else........................................................................................7
(Please specify ______________________________) (STRING 255)
NO RESPONSE ......................................................................................Empty
PROGRAMMER RANGE BOX 7.1
RANGE FOR ALL ITEMS IN Q7 THROUGH Q11 IS 000 – 999 (ALLOW 3 NUMERIC CHARACTERS
AND 2 DECIMAL PLACES);
ALLOW EMPTY ITEMS UNLESS SPECIFIED IN CHECK BOXES BELOW

ALL
7

How many full-time equivalent (FTEs) positions of the following types currently work
for your program? Please include full-time and part-time staff. Either a whole number or
one with a decimal will be accepted (i.e. 15, 15.0, 15.5). Please mark N/A if this position
does not exist at your site. If this position exists but is not currently filled, please enter
“0.”
Number of Full-Time
Equivalent Staff

a. Educational instruction

|.|

b. Vocational instruction

|.|

c. Case management

|.|

d. Leadership development instruction

|.|

e. Volunteer coordination

|.|

Career counseling

g. Other
(Please specify ________________) (STRING 255)

SOFT CHECK: IF Q7a = EMPTY OR 0; You reported no FTEs for educational instruction. Please
confirm and keep your response or change your response to continue.
SOFT CHECK: IF Q7c = EMPTY OR 0; You reported no FTEs for case management. Please
confirm and keep your response or change your response to continue.
SOFT CHECK: IF Q7g GT 0 AND Q7g_othspec = EMPTY; Please specify the other type of fulltime equivalent staff currently working for your program.

PROGRAMMER SKIP BOX 7.2
IF Q7a GE 1 CONTINUE TO Q8;
IF Q7a = (EMPTY OR 0) AND Q7c GE 1; GO TO Q10;
ELSE, GO TO Q12.

PROGRAMMER DISPLAY BOX 8.1
DISPLAY Q8 AND Q9 ON THE SAME SCREEN
ADD TOTAL CALCULATOR TO EACH GRID AT Q8 AND Q9

Q7a GE 1
8

Thinking of the individual educational instructors who currently work for your program,
how many have the following years of experience in any YouthBuild program or site?
Please provide a response for each level of experience. If your program has no
educational instructors with a specific level of experience, please enter “0”.
Number of Educational
Instructors

a. Less than 1 year

c. 4-6 years

d. 7-10 years

e. More than 10 years

b. 1-3 years

Total Instructors

[FILL TOTAL]

Q7a GE 1
9

Of the individual educational instructors who currently work for your program, what is
the highest level of education each has completed? Please include counts for each
level below. If your program has no educational instructors in a specific level, please
enter “0”.
Number of Educational
Instructors

a. Less than high school

c. Some college, no degree

d. Associate’s degree

e. Bachelor’s degree

b. High school diploma or GED equivalency

Master’s degree or more

g. Something else
(Please specify __________________) (STRING 255)
Total Instructors

[FILL TOTAL]

SOFT CHECK: IF Q8_TOT NE Q9_TOT; The total number of educational instructors for these
two questions should be the same. Please confirm and keep your responses or change your
responses to continue.
SOFT CHECK: IF Q9g GT 0 AND Q9g_othspec = EMPTY; Please specify the other type of
education level.
PROGRAMMER SKIP BOX 9.1
IF Q7c GE 1; CONTINUE TO Q10;
ELSE, GO TO Q12.

PROGRAMMER DISPLAY BOX 10.1
DISPLAY Q10 AND Q11 ON THE SAME SCREEN
ADD TOTAL CALCULATOR TO EACH GRID AT Q10 AND Q11

Q7c GE 1
10

Thinking of the individual case managers who currently work for your program, how
many have the following years of experience in any YouthBuild program or site? Please
provide a response for each level of experience. If your program has no educational
instructors with a specific level of experience, please enter “0”.
Number of Case
Managers

a. Less than 1 year

c. 4-6 years

d. 7-10 years

e. More than 10 years

b. 1-3 years

Total Case Managers

[FILL TOTAL]

Q7c GE 1
11

Of the individual case managers who currently work for your program, what is the
highest level of education each has completed? Please include counts for each level
below. If your program has no case managers in a specific level, please enter “0”.
Number of Case
Managers

a. Less than high school

c. Some college, no degree

d. Associate’s degree

e. Bachelor’s degree

b. High school diploma or GED equivalency

Master’s degree or more

g. Something else
(Please specify ____________________) (STRING 255)
Total Case Managers

[FILL TOTAL]

SOFT CHECK: IF Q10_TOT NE Q11_TOT; The total number of case managers for these two
questions should be the same. Please confirm and keep your responses or change your
responses to continue.
SOFT CHECK: IF Q11g GT 0 AND Q11g_othspec = EMPTY; Please specify the other type of
education level.

ALL
These next questions are about the worksites you use for the construction component of the
YouthBuild program. If your program uses more than one worksite, please answer these
questions based on your PRIMARY worksite.
12

Does your program have a construction component?
Select one only

SKIP TO Q18

NO RESPONSE ......................................................................................Empty

SKIP TO Q18

Q12 = 1
13

Does your program own its (primary) worksite?
Select one only

YES .........................................................................................................1

SKIP TO Q15

NO...........................................................................................................0
NO RESPONSE ......................................................................................Empty

SKIP TO Q15

Q13 = 0
14

Who owns your (primary) worksite?

_______________________ (STRING 255)
NO RESPONSE ......................................................................................Empty
Q12 = 1
15

Does your program manage the (primary) worksite?
Select one only

YES .........................................................................................................1

SKIP TO Q17

Q15 = 0
16

Who manages your (primary) worksite?
_______________________ (STRING 255)
NO RESPONSE ......................................................................................Empty

Q12 = 1
17

Does your construction component focus on rehabilitation of existing homes or
apartments, new construction, or both?
Select one only

REHABILITATION ...................................................................................1
NEW CONSTRUCTION ..........................................................................2
BOTH ......................................................................................................3
NO RESPONSE ......................................................................................Empty
ALL
Now, we would like to ask you some questions about the individuals your program serves.
18

Since you received your DOL grant in May 2011, how many individuals have applied to
be in your YouthBuild program? (By apply, we mean individuals who try to enroll in the
program but who have not yet been determined to be eligible for the program.)
|

| NUMBER OF INDIVIDUALS (4 NUM; 0 – 999)

NO RESPONSE ......................................................................................Empty
ALL
19

Does your program have a Mental Toughness Orientation (MTO) or a similar
component?
Select one only

SKIP TO Q21

NO RESPONSE ......................................................................................Empty

SKIP TO Q21

SOFT CHECK: IF Q19 = Empty; Your response to this question is important. Please provide a
response and continue.

Q19 = 1
20

Of those who have applied since you received your DOL grant in May 2011, how
many have you invited to MTO or your similar component?
|

| INDIVIDUALS INVITED TO PARTICIPATE (4 NUM; 0 – 999)

NO RESPONSE........................................................... Empty
SOFT CHECK: IF Q20 GT Q18; You reported that more individuals were invited to MTO or your
similar component than applied to your program. Please confirm and keep your response or
change your response to continue.

ALL
21

Of those who applied since you received your DOL grant in May 2011, how many are
currently enrolled in your program? (By enroll, we mean individuals who have
completed Mental Toughness Orientation, if you have it, and satisfied all other
requirements in order to start your program.)
|

| NUMBER OF INDIVIDUALS (4 NUM; 0 – 999)

NO RESPONSE ......................................................................................Empty
SOFT CHECK: IF Q21 = Empty; Your response to this question is important. Please provide a
response and continue.
PROGRAMMER SKIP BOX 21.1
IF Q21 IS GT 0, CONTINUE TO Q22a;
IF Q21 IS 0 OR EMPTY, SKIP OT Q22b

Q21 GT 0
FILL RESPONSE FROM Q21
22a

Given your current operating budget, could your program serve more than the [FILL
Q21] participants currently enrolled?
Select one only

SKIP TO Q23

NO RESPONSE ......................................................................................Empty

SKIP TO Q23

(Q21 = 0 OR EMPTY) OR Q22a = 1
22b

Given your current operating budget, what is the maximum number of individuals
that could be enrolled into your program?
|

| MAXIMUM NUMBER OF INDIVIDUALS (3 NUM; 0 – 9999)

NO RESPONSE........................................................... Empty
PROGRAMMER DISPLAY BOX 23.1
DISPLAY ROW 2 BELOW ONLY IF Q19 = 1;
DISPLAY ROW 4 BELOW ONLY IF (Q21 = 0 OR EMPTY) OR Q22a = 1
FILL NUMBER OF INDIVIDUALS REPORTED. IF THE NUMBER OF
INDIVIDUALS REPORTED IS EMPTY, FILL 0.
PROVIDE NUMERIC FIELDS FOR UPDATING RESPONSES. EACH FIELD
IS 3 NUM WITH RANGE OF 000 – 999.

ALL
DISPLAY ROW 2 BELOW ONLY IF Q19 = 1
DISPLAY ROW 4 ONLY IF (Q21 = 0 OR EMPTY) OR Q22A = 1
FILL Q18, Q20, Q21, AND Q22b RESPONSES; IF RESPONSES ARE EMPTY, FILL “0”
23

You have reported the following information about the individuals your program
serves. Please review and confirm your responses. Keep in mind we are interested
in individuals who have applied since you received your DOL grant in May 2011.
If your responses are correct, please continue. If you want to change a response,
you may enter it in the Updated Responses column below.
Your Reported
Responses

Updated
Responses

a. Number of applicants

[FILL Q18]

b. Number of applicants invited to MTO or your similar
component

[FILL Q20]

[FILL Q21]

[FILL Q22b]

c. Number of applicants currently enrolled
d. Maximum enrollment in your program

ALL
Next, we have some questions about how you recruit and enroll individuals into your program.
24a

How do you recruit individuals into your program?
Select all that apply

Referrals from high schools .....................................................................1
Referrals from courts or corrections agencies..........................................2
Referrals from outreach that other agencies do in the community ........... 3
Word-of-mouth referrals from people in the community or former
participants ..............................................................................................4
Community outreach conducted by your site ...........................................5
Self-referrals or walk-ins ..........................................................................6
Recruit individuals in some other way ......................................................7
(Please specify _____________________________) (STRING 255)
NO RESPONSE ......................................................................................Empty
SOFT CHECK: IF 0 ITEMS SELECTED AT Q24a; Your response to this question is important.
Please provide a response and continue.
SOFT CHECK: IF Q24a.7 = 1 and Q24a.7_othspec = EMPTY; Please specify how else you recruit
individuals into your program.

PROGRAMMER SKIP BOX 24.1
IF NO ITEMS ARE SELECTED AT Q24a AFTER SOFT CHECK AND Q19 = 1, SKIP TO Q25;
IF NO ITEMS ARE SELECTED AT Q24a AFTER SOFT CHECK AND Q19 = 0 OR EMPTY, SKIP
TO Q26;
IF ONLY ONE ITEM IS SELECTED AT Q24a AND Q19 = 1, SKIP TO Q25;
IF ONLY ONE ITEM IS SELECTED AT Q24a AND Q19 = 0 OR EMPTY, SKIP TO Q26;
ELSE, CONTINUE TO Q24b

NUMBER OF ITEMS SELECTED AT Q24a IS GT 1
FILL OTHER SPECIFY RESPONSE FROM Q24a7 IF Q24a7 IS SELECTED;
IF NO OTHER SPECIFY TEXT IS ENTERED AT Q24a7, FILL Recruit individuals in some other
way
24b

Of the recruiting methods listed below, which is your main source of recruiting
individuals into your program?
PROGRAMMER: ONLY DISPLAY RESPONSES SELECTED AT Q24a.
Select one only

Referrals from high schools .....................................................................1
Referrals from courts or corrections agencies..........................................2
Referrals from outreach that other agencies do in the community ........... 3
Word-of-mouth referrals from people in the community or former
participants ..............................................................................................4
Community outreach conducted by your site ...........................................5
Self-referrals or walk-ins ..........................................................................6
[FILL OTHER SPECIFY RESPONSE / Recruit individuals in some
other way] ................................................................................................7
NO RESPONSE ......................................................................................Empty

PROGRAMMER SKIP BOX 24b.1
IF Q19 = 1, CONTINUE;
IF Q19 = 0 OR EMPTY, SKIP TO Q26

Q19 = 1
25

Do you have a formal screening process prior to MTO (or a component similar to MTO)
for those who apply to your program?
Select one only

YES .........................................................................................................1
NO...........................................................................................................0
NO RESPONSE ......................................................................................Empty

ALL
26

Which of the following are part of your application screening process?
Select all that apply

Placement test.........................................................................................1
Test of basic skills ...................................................................................2
Interview with staff member .....................................................................3
Staff observation .....................................................................................4
Paper application .....................................................................................5
Personal statement..................................................................................6
Drug testing .............................................................................................7
None of the above ...................................................................................8
Other .......................................................................................................9
(Please specify_______________________________) (STRING 255)
NO RESPONSE ......................................................................................Empty

PROGRAMMER SKIP BOX 26.1
IF Q19 = 1, CONTINUE;
IF Q19 = 0 OR EMPTY, SKIP TO Q30

Q19 = 1
Now, we have some questions about your Mental Toughness Orientation (MTO) or similar
component.
27

How long is your MTO, or your similar component, in days?
|

| NUMBER OF DAYS (2 NUM; 0 – 99)

NO RESPONSE........................................................... Empty

Q19 = 1
28

On average, how many hours per day do candidates spend in MTO or in your similar
component?
Select one only

Less than 2 hours a day ..........................................................................1
Between 2 and 4 hours a day ..................................................................2
Between 4 and 6 hours a day ..................................................................3
Between 6 and 8 hours a day ..................................................................4
More than 8 hours a day ..........................................................................5
NO RESPONSE ......................................................................................Empty
Q19 = 1
29

What activities are offered during MTO or your similar component?
Select all that apply

Orientation to YouthBuild program...........................................................1
Team building activities ...........................................................................2
Academic preparation..............................................................................3
Workforce preparation .............................................................................4
Goal-setting exercises .............................................................................5
Physical fitness activities .........................................................................6
Other .......................................................................................................7
(Please specify_______________________________) (STRING 255)
NO RESPONSE ......................................................................................Empty

ALL
Once participants have completed MTO or your similar component, on IF Q19 = 1;
On IF Q19 = 0 OR EMPTY
30

[Once participants have completed MTO or your similar component, on / On] average,
how long does your structured YouthBuild program last?
Select one only

Less than 6 months .................................................................................1
6-9 months ..............................................................................................2
10-12 months ..........................................................................................3
13-18 months ..........................................................................................4
More than 18 months...............................................................................5
NO RESPONSE ......................................................................................Empty
ALL
31

How often do participants who enroll in your YouthBuild program receive wages or
stipends?
Select one only

Daily ........................................................................................................1
Weekly ....................................................................................................2
Every 2 weeks .........................................................................................3
Monthly....................................................................................................4
Participants do not receive wages or stipends .........................................5

SKIP TO Q34

Other .......................................................................................................6
(Please specify ______________________________) (STRING 255)
NO RESPONSE ................................................................................ Empty

SKIP TO Q34

Q31 = 1, 2, 3, 4, 6
32

On average, how much do participants receive per pay period as wages or stipends?
$|

|, |

|.|

| AMOUNT (4 NUM; 0.00 – 9999.99)

NO RESPONSE ......................................................................................Empty

SKIP TO Q34

PROGRAMMER SKIP BOX 32.1
IF (Q31 = 1 (DAILY) AND Q32 GT $100) OR (Q31 = 2 (WEEKLY) AND Q32 GT $500) OR
(Q31 = 3 (EVERY 2 WEEKS) AND Q32 GT $1,000) OR (Q31 = 4 (MONTHLY) AND Q32 GT $2,000);
GO TO Q33 FOR VERIFICATION.
IF (Q31 = 1 (DAILY) AND Q32 LT $10) OR (Q31 = 2 (WEEKLY) AND Q32 LT $50) OR
(Q31 = 3 (EVERY 2 WEEKS) AND Q32 LT $100) OR (Q31 = 4 (MONTHLY) AND Q32 LT $200);
GO TO Q33 FOR VERIFICATION.
ELSE, CONTINUE TO Q34

SEE BOX 32.1
daily IF Q31 = 1; weekly IF Q31 = 2; every 2 weeks IF Q31 = 3; monthly IF Q31 = 4
DROPDOWN OPTIONS: Daily, Weekly, Every 2 weeks, Monthly, Other
33

You indicated that participants receive $[fill Q32] [daily / weekly / every 2 weeks /
monthly] as wages or stipends.
If your responses are correct, please continue. If you want to change a response, you
may enter it in the field below.
On average, how much do
participants receive as
wages or stipends?

Select a frequency
[INSERT DROP DOWN]
Enter amount below
$|

|, |

|.|

(4 NUM; 0.00 – 9999.99)

PROGRAMMER RANGE BOX 34.1
GRIDS 34 – 41 ALLOW EMPTY ITEMS, BUT THEY WILL BE CHECKED

ALL
Next, we would like to ask you some questions about the different types of services your
YouthBuild program offers to its participants. For the following questions, please answer
including all funding services, not just those from DOL.
34

For each workforce or job related training listed, please select if the service is offered
on-site, as a referral, or both.
Select one per row

OFFERED
ON-SITE

OFFERED
AS
REFERRAL

OFFERED
ON-SITE
AND AS
REFERRAL

NOT
OFFERED

a. Vocational/Occupational skills training

b. Pre-apprenticeship programs

c. On-the-job training (OJT)

d. Subsidized work experience/internships

e. Unsubsidized work experience

g. Career/Life skills training

h. Job certification program

i . First aid or CPR training

Work readiness training

Other workforce or job related training
(Please specify _______________) (STRING 255)

SOFT CHECK: IF ANY ROWS ARE EMPTY AT Q34a – Q34j; You did not provide a response to
one or two questions on this page. Unanswered questions are indicated by an asterisk (*).
Please review your answers and provide appropriate responses.
SOFT CHECK: IF (Q34j = 1, 2, OR 3) AND (Q34j_othspec = EMPTY); Please specify the other
types of workforce or job related training you provide.

ALL
35

For each post-secondary education preparation service listed, please select if the
service is offered on-site, as a referral, or both.
Select one per row

OFFERED
ON-SITE

OFFERED
AS
REFERRAL

OFFERED
ON-SITE
AND AS
REFERRAL

NOT
OFFERED

a. College awareness or college guidance activities

b. Preparation for college entrance examinations

c. College application assistance

d. Financial aid assistance

e. College preparation or transition programs

Courses for college credit

g. Other type of post-secondary education
preparation
(Please specify ______________) (STRING 255)

SOFT CHECK: IF ANY ROWS ARE EMPTY AT Q35a – Q35g; You did not provide a response to
one or two questions on this page. Unanswered questions are indicated by an asterisk (*).
Please review your answers and provide appropriate responses.
SOFT CHECK: IF (Q35j = 1, 2, OR 3) AND (Q35j_othspec = EMPTY); Please specify the other
types of post-secondary education preparation you provide.

ALL
36

For each education or academic service listed, please select if the service is offered onsite, as a referral, or both.
Select one per row

OFFERED
ON-SITE

OFFERED
AS
REFERRAL

OFFERED
ON-SITE
AND AS
REFERRAL

NOT
OFFERED

a. GED preparation

b. Language arts (reading, writing, literacy)
remediation (independent of GED preparation)

c. Math remediation (independent of GED
preparation)

d. Other academic tutoring services

e. A high school diploma program

g. ESL training

h. Other type of education activities or academic
services

Preparation for standardized achievement tests,
such as state or local tests

(Please specify ______________) (STRING 255)

SOFT CHECK: IF ANY ROWS ARE EMPTY AT Q36a – Q36h; You did not provide a response to
one or two questions on this page. Unanswered questions are indicated by an asterisk (*).
Please review your answers and provide appropriate responses.
SOFT CHECK: IF (Q36h = 1, 2, OR 3) AND (Q36h_othspec = EMPTY); Please specify the other
types of education or academic services you provide.

ALL
37

For each community service opportunity listed, please select if the service is offered
on-site, as a referral, or both.
Select one per row

OFFERED
ON-SITE

OFFERED
AS
REFERRAL

OFFERED
ON-SITE
AND AS
REFERRAL

NOT
OFFERED

a. Public beautification projects (park or highway
cleanup, mural painting, etc.)

b. After-school homework help or mentoring (for
example, Big Brother program)

c. Activities with people with disabilities

d. Elder care activities

e. Community activism

Other community services opportunities
(Please specify ______________) (STRING 255)

SOFT CHECK: IF ANY ROWS ARE EMPTY AT Q37a – Q37f; You did not provide a response to
one or two questions on this page. Unanswered questions are indicated by an asterisk (*).
Please review your answers and provide appropriate responses.
SOFT CHECK: IF (Q37f = 1, 2, OR 3) AND (Q37f_othspec = EMPTY); Please specify the other
types of community service opportunities you provide.

ALL
38

For each leadership opportunity listed, please select if the service is offered on-site, as
a referral, or both.
Select one per row

OFFERED
ON-SITE

OFFERED
AS
REFERRAL

OFFERED
ON-SITE
AND AS
REFERRAL

NOT
OFFERED

a. Youth Policy Council

b. In-class leadership roles

c. Construction site leadership roles

d. Opportunities to speak at public/governmental
meetings

e. Attending youth conferences

Other leadership opportunities
(Please specify ______________) (STRING 255)

SOFT CHECK: IF Q38a = EMPTY; Your response to question 38a is important. Please provide a
response and continue.
SOFT CHECK: IF ANY ROWS ARE EMPTY AT Q38b – Q38f; You did not provide a response to
one or two questions on this page. Unanswered questions are indicated by an asterisk (*).
Please review your answers and provide appropriate responses.
SOFT CHECK: IF (Q38f = 1, 2, OR 3) AND (Q38f_othspec = EMPTY); Please specify the other
types of leadership opportunities you provide.

ALL
39

For each health service listed, please select if the service is offered on-site, as a
referral, or both.
Select one per row

OFFERED
ON-SITE

OFFERED
AS
REFERRAL

OFFERED
ON-SITE
AND AS
REFERRAL

NOT
OFFERED

a. Substance abuse services

b. Mental health treatment

c. Preventative medicine

d. Emergency medical care

e. Pregnancy leave or prenatal care

Other health services/activities
(Please specify ______________) (STRING 255)

SOFT CHECK: IF ANY ROWS ARE EMPTY AT Q39a – Q39f; You did not provide a response to
one or two questions on this page. Unanswered questions are indicated by an asterisk (*).
Please review your answers and provide appropriate responses.
SOFT CHECK: IF (Q39f = 1, 2, OR 3) AND (Q39f_othspec = EMPTY); Please specify the other
types of health services you provide.

ALL
40

For each support service listed, please select if the service is offered on-site, as a
referral, or both.
Select one per row

OFFERED
ON-SITE

OFFERED AS
REFERRAL

OFFERED
ON-SITE
AND AS
REFERRAL

a. Mentors to students

b. Visits to the homes of students participating in your
program

c. A process to connect students with social service

d. Therapeutic services for socio-emotional and/or
behavioral problems

e. Activities to support students’ acquisition of “life
skills”

g. Activities to increase and/or improve parental and
family involvement in students’ school lives

h. Case management

Housing Assistance

Other support services not listed

Cultural enrichment activities

NOT
OFFERED

(Please specify _______________) (STRING 255)

SOFT CHECK: IF ANY ROWS ARE EMPTY AT Q40a – Q40j; You did not provide a response to
one or two questions on this page. Unanswered questions are indicated by an asterisk (*).
Please review your answers and provide appropriate responses.
SOFT CHECK: IF (Q40j= 1, 2, OR 3) AND (Q40j_othspec = EMPTY); Please specify the other
types of support services you provide.

ALL
41

For each post-program service listed, please select if the service is offered on-site, as a
referral, or both. Post-program services refer to services provided after your structured
YouthBuild program has ended.
Select one per row

OFFERED
ON-SITE

OFFERED
AS
REFERRAL

OFFERED
ON-SITE AND
AS
REFERRAL

NOT
OFFERED

a. Additional skills training

b. Subsidized work experience

c. Internships

d. Job preparation classes or activities

e. GED preparation

g. Career counseling

h. Other post-program services

Mentoring

(Please specify ______________) (STRING 255)

SOFT CHECK: IF ANY ROWS ARE EMPTY AT Q41a – Q41h; You did not provide a response to
one or two questions on this page. Unanswered questions are indicated by an asterisk (*).
Please review your answers and provide appropriate responses.
SOFT CHECK: IF (Q41h = 1, 2, OR 3) AND (Q41h_othspec = EMPTY); Please specify the other
types of post-program services you provide.

PROGRAMMER SKIP BOX 41.1
IF Q12 = 0 (NO CONSTRUCTION COMPONENT) SKIP TO Q43;
IF Q12 = EMPTY OR 1, CONTINUE TO Q42

Q12 NE 0
Now, we would like to ask some more specific questions about some of the services you offer
and how time is allocated among these various activities.
42

How often do youth participate in construction activities?
Select one only

Daily (for some portion of the day) ...........................................................1
Alternating days within a one-week period ...............................................2
Weekly (one week on/one week off) ........................................................3
Other .......................................................................................................4
(Please specify _____________________________) (STRING 255)
NO RESPONSE ......................................................................................Empty

ALL
43

How often do youth participate in academic classes of some sort?
Select one only

Daily (for some portion of the day) ...........................................................1
Alternating days within a one-week period ...............................................2
Weekly (one week on/one week off) ........................................................3
Other .......................................................................................................4
(Please specify______________________________) (STRING 255)
NO RESPONSE ......................................................................................Empty

PROGRAMMER SKIP BOX 43.1
IF Q12 = 0 (NO CONSTRUCTION COMPONENT) SKIP TO Q45;
IF Q12 = EMPTY OR 1, CONTINUE TO Q44

Q12 NE 0
44

On average, do your participants spend more time in academic activities or
construction activities?
Select one only

Academic or education activities..............................................................1
Construction activities ..............................................................................2
About the same for each .........................................................................3
NO RESPONSE ......................................................................................Empty

ALL
Now, thinking more about your academic services…
45

Do you offer academic services that happen in a group or classroom setting?
Select one only

SKIP TO Q49

NO RESPONSE ......................................................................................Empty

SKIP TO Q49

PROGRAMMER DISPLAY BOX 45.1
DISPLAY Q46 AND Q47 ON THE SAME SCREEN

Q45 = 1
46

How often do these classes meet?
Select one only

Daily ........................................................................................................1
Every other day .......................................................................................2
Every other week .....................................................................................3
Other .......................................................................................................4
(Please specify)_____________________________) (STRING 255)
NO RESPONSE ......................................................................................Empty
Q45 = 1
47

What is the average size of these classes?
|

| # OF ATTENDEES (2 NUM; 0 – 99)

NO RESPONSE........................................................... Empty

Q45 = 1
48

How are students divided among classes?
Select one only

Initial academic testing ............................................................................1
Classroom observation ............................................................................2
Student special needs .............................................................................3
Students are not divided among classes .................................................4
Other .......................................................................................................5
(Please specify_____________________________) (STRING 255)
NO RESPONSE ......................................................................................Empty
ALL
49

Which of the following teaching methods are used in your YouthBuild program?
Select all that apply

One-on-one assistance ...........................................................................1
Traditional whole group instruction or lecture...........................................2
Small group project work .........................................................................3
Independent work paired with individual tutoring .....................................4
Computer instruction and exercises .........................................................5
Distance learning .....................................................................................6
Team teaching.........................................................................................7
Peer teaching ..........................................................................................8
Self-paced instruction ..............................................................................9
Incorporation of “real world” implications and rewards in the classroom .. 10
Other .......................................................................................................11
(Please specify______________________________) (STRING 255)
NO RESPONSE ......................................................................................Empty

PROGRAMMER SKIP BOX 49.1
IF NO ITEMS ARE SELECTED AT Q49, SKIP TO Q51;
IF ONE ITEM IS SELECTED AT Q49, SKIP TO Q51;
IF MORE THAN ONE ITEM IS SELECTED AT Q49, CONTINUE TO Q50

SKIP TO Q51

NUMBER OF ITEMS SELECTED AT Q49 GT 1
FILL OTHER SPECIFY RESPONSE FROM Q49.11 IF Q49.11 IS SELECTED;
IF NO OTHER SPECIFY TEXT IS ENTERED AT Q49.11, FILL Other teaching method
50

Of the following teaching methods, which is the main method you use?
PROGRAMMER: ONLY DISPLAY RESPONSE OPTIONS SELECTED FROM Q49
Select one only

One-on-one assistance ...........................................................................1
Traditional whole group instruction or lecture...........................................2
Small group project work .........................................................................3
Independent work paired with individual tutoring .....................................4
Computer instruction and exercises .........................................................5
Distance learning .....................................................................................6
Team teaching.........................................................................................7
Peer teaching ..........................................................................................8
Self-paced instruction ..............................................................................9
Incorporation of “real world” implications and rewards in the classroom .. 10
[FILL OTHER SPECIFY RESPONSE / Other teaching method] .............. 11
NO RESPONSE ......................................................................................Empty
ALL
51

The ultimate educational goal of your YouthBuild program is to obtain…
Select one only

A GED certificate .....................................................................................1
A high school diploma .............................................................................2
Basic academic improvement ..................................................................3
Post-secondary education preparation ....................................................4
Something else........................................................................................5
(Please specify_______________________________) (STRING 255)
NO RESPONSE ......................................................................................Empty
PROGRAMMER SKIP BOX 51.1
IF Q38a = 1 – 3 (OFFERS YPC) CONTINUE;
IF Q38a = 4 OR Empty, SKIP OT Q55

Q38a = 1, 2, 3
You mentioned that your YouthBuild program offers a Youth Policy Council (YPC). We would
like to ask a few questions about this activity.
52

How often does your Youth Policy Council meet?
Select one only

Less than once a month ..........................................................................1
1-2 times a month ....................................................................................2
3-4 times a month ....................................................................................3
More than 4 times a month ......................................................................4
Some other frequency .............................................................................5
(Please specify_______________________________) (STRING 255)
NO RESPONSE ......................................................................................Empty
Q38a = 1, 2, 3
53

How are individuals selected to be part of your Youth Policy Council?
Select all that apply

Youth are recommended by program staff ...............................................1
Youth are recommended by their peers ...................................................2
Youth are recommended by someone other
than program staff or their peers..............................................................3
Youth apply to join the council .................................................................4
Youth are selected in some other way .....................................................5
(Please specify_______________________________) (STRING 255)
NO RESPONSE ......................................................................................Empty

PROGRAMMER RANGE BOX 54.1
ALLOW EMPTY RESPONSES FOR 54a – 54j

Q38a = 1, 2, 3
54

Does your Youth Policy Council provide input in any of the following areas?
Yes

a. Program design

b. Staff hiring

c. Community service activities

d. Size of the youth stipends

e. Payment schedule of the youth stipends

g. Selection of the youth to present at community or governmental
meetings

h. Design or décor of your program’s common spaces

Dress code for the program

Other

Disciplinary actions

(Please specify ___________________________) (STRING 255)

SOFT CHECK: IF ANY ROWS ARE EMPTY AT Q54a – Q54j; You did not provide a response to
one or two questions on this page. Unanswered questions are indicated by an asterisk (*).
Please review your answers and provide appropriate responses.
SOFT CHECK: IF Q54j = 1 AND Q54j_othspec = EMPTY; Please specify the other areas your
Youth Policy Council provides input.
ALL
Lastly, we have a few questions about parental involvement in your YouthBuild program.
55

Are there any circumstances under which parents or guardians are required to be
involved in the program?
Select one only

SKIP TO CONCLUDE

NO RESPONSE ......................................................................................Empty

SKIP TO CONCLUDE

Q55 = 1
56

Under what circumstances are parents or guardians required to participate?
Select all that apply

Parents are always required to participate ...............................................1
Parents of participants under age 18 .......................................................2
Parents of participants with juvenile records ............................................3
Parents of participants with a history of substance abuse ........................4
Some other circumstance ........................................................................5
(Please specify_______________________________) (STRING 255)
NO RESPONSE ......................................................................................Empty
Q55 = 1
57

In which of the following activities are parents or guardians required to participate?
Select all that apply

Student orientation ..................................................................................1
Volunteering at program site....................................................................2
Volunteering at construction site ..............................................................3
Donating supplies to program ..................................................................4
Other .......................................................................................................5
(Please specify_______________________________) (STRING 255)
NO RESPONSE ......................................................................................Empty
ALL
CONCLUDE
Those are all the questions we have for you. Thank you so much for your time and
information.
Persons are not required to respond to this collection of information unless this survey displays a currently valid OMB
control number (OMB xxxx-xxxx, expires xx/xx/xxxx). Responding to this questionnaire, which seeks to help the U.S.
Department of Labor understand how YouthBuild programs are serving disadvantaged youth, is voluntary. Public
reporting burden for this collection of information is estimated to average 30 minutes per response, including the time
for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and
completing and reviewing the collection of information. Send comments regarding this burden estimate to Eileen
Pederson, U.S. Department of Labor, Employment and Training Administration, Office of Policy Development and
Evaluation, Room N-5641, 200 Constitution Avenue, NW, Washington, DC 20210.

APPENDIX B:
YOUTH BUILD RESPONDENT DEBRIEFING
PROTOCOL PAPI/PHONE RESPONDENTS

YouthBuild Respondent Debriefing Protocol PAPI/Phone Respondents
1.

Overview

We are speaking with respondents after the interview is complete to assess their general perceptions
of the survey procedures, difficulties or sensitive questions.
•

For the phone interviews we are conducting a debriefing with the respondent
immediately following the phone interview

•

For the self-administered questionnaire
- The respondent are mailed a hardcopy of the survey and asked to complete it
by a specified date
- We will follow-up with the respondent at a specified time to complete the
debriefing.
- We will conduct in-depth one-on-one follow-up interviews with all
respondents.

Moderator Protocol

INTRODUCTION (1 minute)
Thank you for taking the time to speak with us today. First, let me introduce myself and tell you a
little bit about what we are going to be doing here today.
•

Introduce self and Mathematica: Hello, my name is [name] and I’m calling from
Mathematica Policy Research, Inc., a research company in Princeton New Jersey.

•

Explain the conversation will be recorded: I will be tape-recording the interview so
that I can go back and listen to it later if I need to when I’m summarizing the
results.

•

Give an approximate time frame: This session will last about XX minutes

•

Thank participants for their participation: Thank you again for your time, opinion,
and willingness to help us out.

GROUND RULES (2 minutes)

•

No wrong answers

THE SURVEY (15 minutes)
We want to get your feedback on several aspects of the survey that you took. I have some specific
questions that I would like to ask, but if you have any other comments, please feel free to share.
•

Did you feel that you were the appropriate person to answer these questions? Why
or why not?

•

Were there any terms or words that you did not understand?

•

Were there any terms or words that you feel were not used correctly in the context
of the questions? (For example, “apply,” or “enroll.”)

•

Were there any questions you found especially difficult to answer?

•

Do you think there is anything important about your program that we did not
address in this questionnaire?

•

How long do you think it took you to complete the survey?

WRAP-UP (5 minutes)
We are coming to the end of our discussion. Thank you for sharing your experiences and opinions.
Is there anything else that anyone would like to add about anything we discussed today?
Thanks again for your participation.

APPENDIX C:
YOUTH BUILD GRANTEE SURVEY