Download:
pdf |
pdfAPPENDIX F
PILOT TEST RESULTS FOR INTAKE FORMS
MEMORANDUM
TO:
Sheena McConnell
FROM:
Julita Milliner-Waddell and Pat Nemeth
SUBJECT:
WIA Pilot Test
P.O. Box 2393
Princeton, NJ 08543-2393
Telephone (609) 799-3535
Fax (609) 799-0005
www.mathematica-mpr.com
DATE: 4/26/2010
WIA-42
A. INTRODUCTION
Mathematica conducted a pilot test of the three intake forms that will be used to
enroll participants in the Workforce Investment Act Gold Standard Evaluation of the
Adult and Dislocated Worker Programs (WIA Evaluation). The goal of the pilot test was
to identify and eliminate problems with layout, wording, and skip errors and to determine
the time burden on respondents. The pilot test also provided early insights about the
interest and willingness of WIA customers to enroll in a program like the WIA
Evaluation.
The pilot test was conducted at the Middlesex County One Stop Career Center in
New Brunswick, New Jersey on March 29, 2010. Pat Nemeth, the Survey Director for the
evaluation, and Julita Milliner-Waddell, Deputy Survey Director, administered the pilot
test. Customers of the Center were approached and asked if they would be willing to
complete the intake forms. Each pilot test participant was paid $25 for completing the
forms and providing feedback about their experience.
Replicating the plan for the main study, the three intake forms were packaged
together—a Consent to Participate form, a baseline information form, and a contact
information form. The forms were modified slightly for the pilot test. Specifically, spaces
to record start and end times were added to each form and questions requesting a full
Social Security Number (SSN) or the last four digits of the SSN were shaded and not
collected as part of the pilot test.
A total of seven WIA customers completed the pilot test. All were utilizing Resource
Center services. Four of participants were male; four were black; two were white; and
one was Asian. As shown in Table 1, the pilot test participants took approximately 13
minutes to complete all three forms, on average. At the end of each session, the pilot test
participants were debriefed to ask for their overall impression of the forms, and any
specific questions or problems encountered. A standard debriefing protocol was used for
this purpose (see Attachment A).
An Affirmative Action/Equal Opportunity Employer
MEMO TO: Sheena McConnell
FROM:
Julita Milliner-Waddell and Pat Nemeth
DATE:
4/26/2010
PAGE:
2
TABLE 1
WIA INTAKE FORMS ‐ AVERAGE COMPLETION TIMES (minutes)
R1 R2 R3 R4 R5 R6 R7 TOTAL
AVERAGE
Consent Form
3 3 3 3 5 5 3
25
3.57
BIF
3 5 4 6 7 8 5
38
5.43
Contact Form
3 3 3 5 4 4 7
29
4.14
All 3 Forms
9 11 10 14 16 17 15
92
13.14
Each intake form is discussed separately below.
B. CONSENT FORM
Participants took between three and five minutes to complete the consent form, with
an average of 3.57 minutes overall. For the most part, the language on the consent form
did not present a problem for any of the participants who seemed to understand the
concept of random assignment and the assignment to groups. One participant
misunderstood the three groups described on the consent, interpreting that people selected
would work in groups. The only other problem was the acronym “WIA”. The
parenthetical reference did not seem to work.
The current wording is as follows:
The national study, called the Workforce Investment Act Gold
Standard Evaluation (WIA Evaluation), is being conducted by a team
of researchers at Mathematica, Social Policy Research Associates, and
MDRC.
We recommend that this wording be changed to read:
The national study, called the Workforce Investment Act (WIA) Gold Standard
Evaluation is being conducted by a team of researchers at Mathematica, Social
Policy Research Associates, and MDRC. This study is also called the WIA
Evaluation.
A copy of the consent form is attached as Attachment B.
C. BASELINE INFORMATION FORM (BIF)
Participants took an average of 5.43 minutes to complete the BIF, with a range of
three to eight minutes. Overall, participants did not have problems with the BIF, but
MEMO TO: Sheena McConnell
FROM:
Julita Milliner-Waddell and Pat Nemeth
DATE:
4/26/2010
PAGE:
3
provided helpful feedback on ways to improve several items. The recommended changes
to the BIF are presented below by question number.
Question 2:
Question 7:
Question 8:
Question 9:
Question 10:
Question 15:
Question 17:
Question 24:
Question 25:
Question 25:
Question 26:
Change the format to ask for first name, first.
Add an option to check “none” if the respondent does not have a
home phone.
Add an option to check “none” if the respondent does not have a
cell phone.
Provide more space to record an e-mail address.
Consider dropping this question which asks for a third phone
number. Only two of the seven participants provided a response.
Two others provided a name in the “this number belongs to” line
even though no number was provided, likely referring to the cell
phone entry at question 8.
The only two people who apparently live alone, entered “0” here,
even though the question asks “including yourself”. We may need
to ask if you live alone first.
Provide more space to record “other kinds of degrees, diplomas, or
certificates.
Add “at your main job” to this question to focus respondents with
more than one employer.
Change rate of pay boxes to a straight line.
Add a note to provide an average if wages vary.
Add an option to check “none” at this question which asks about
public assistance receipt.
The section of the BIF to be completed by a counselor was not tested. This section is
study-specific and would not have been meaningful to counselors at the New Brunswick
One Stop Career Center. Attachment C provides a copy of the BIF.
D. CONTACT INFORMATION FORM
The contact information form took participants an average of 4.14 minutes to
complete and ranged from 3 to 7 minutes. Respondents did not have any problems
completing this form. All but one was able to provide three contacts. Most respondents
provided only one number for each contact. While most participants were able to recall
this information, a couple needed to look the information up. The contact information
form is included as Attachment D.
MEMO TO: Sheena McConnell
FROM:
Julita Milliner-Waddell and Pat Nemeth
DATE:
4/26/2010
PAGE:
4
E. CONCLUSIONS
The pilot test of the intake forms for the WIA Evaluation provided useful
information for improving the format, wording and content of the three intake forms
planned for the study. The forms will be revised to reflect the information learned.
cc: Karen Needels, Linda Rosenberg
06503.610
Mathematica Policy Research
ATTACHMENT A
WIA PRETEST DEBRIEF QUESTIONS
1. What is your overall reaction to the forms you just completed? (PROBE for length of time it took,
willingness to provide the needed information, any questions or words that were hard to understand,
etc.)
____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
2. The consent form explains that participants would be randomly assigned to one of three groups. In
your own words, please tell me what that meant to you. (PROBE: Did you understand what was
explained? What, if anything, was confusing or difficult to understand in these descriptions?)
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
3. How do you feel about the questions on the Baseline Information Form? (PROBE: Were any
questions difficult to answer? Which ones? Did you dislike anything on the form?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
4. You did not have to provide your social security number for this pretest, but would be asked to do so
if you were at a participating OneStop Center. How would you feel about that? (PROBE: What about
providing the last 4-digits only?)
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
5. How do you feel about providing contact information of friends and relatives? (PROBE: Would you be
willing to provide this kind of information if you were at a participating OneStop Center? / Which
particular pieces of information would you be most likely to provide/not provide? Were you able to
recall address and telephone numbers for your contacts? Do you know why we ask for this
information?)
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
6. If this program were being offered at this One Stop, would you have consented to participate?
Why/Why not?
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
5
06503.610
Mathematica Policy Research
START TIME: |
|
|:|
|
|
ATTACHMENT B
CONSENT FORM
CONSENT TO PARTICIPATE IN THE WORKFORCE INVESTMENT ACT GOLD
STANDARD EVALUATION PRETEST VERSION
The U.S. Department of Labor is sponsoring a study about the nation’s employment and training programs to learn how
well these programs are working and how they can be improved. The national study, called the Workforce Investment Act Gold
Standard Evaluation (WIA Evaluation), is being conducted by a team of researchers at Mathematica, Social Policy Research
Associates, and MDRC.
By signing this consent form, you are agreeing to take part in this very important study. As a participant in this study, the
following will happen:
• A computer will assign you to one of three groups. Your placement in one of these groups is like a lottery. The
group you end up in will be decided completely by chance and will not be affected by any of your characteristics.
The group you are assigned to will affect the services you can access for up to three years. The three groups are:
• Group 1: If you are assigned to Group 1, you will have access to all of the WIA services available. This includes
access to WIA training funds to help pay for training at a state-approved provider, if appropriate. Most people will
be assigned to this group.
• Group 2: If you are assigned to Group 2, you will have access to all of the WIA services, except WIA-funded
training. You will be able to access intensive staff-assisted services, which may include career interest and skills
assessments, job counseling, and longer-term workshops on further developing work skills, and core services
which include services in the Resource Room such as job listings and access to the Internet.
• Group 3: If you are assigned to Group 3, you will have access to core services, which may include job listings and
access to the Internet. You will not have access to WIA intensive staff-assisted services or to WIA-funded
training.
• The decision to participate in the study is up to you. If you decide not to be in the study, you will only have access
to core services. Core services include services in the Resource Room such as access to job listings and the
Internet.
• You may be contacted by an interviewer from Mathematica to complete two interviews by telephone over the
next few years. These interviews are voluntary, but they are very important to the success of the study. You will
receive a payment for each survey you complete.
• Government agencies such as the Social Security Administration, Employment Service, and agencies that
administer the Temporary Assistance for Needy Families (TANF), Supplemental Nutrition Assistance Program
(SNAP), and Workforce Investment Act (WIA) programs may share information with the research team about
your earnings and government services and benefits you receive.
• All information that is collected about you through interviews or agency records will be used for research
purposes only. All information will be kept strictly confidential, unless the law requires otherwise, or you request
release of your information in writing. Your name will never be used in any reports and no information will be
reported in any way that can identify you.
I have read this consent form (or it has been read to me). I have also been given written information about this
study. I understand the information provided in these materials and voluntarily agree to participate in the WIA
Evaluation. If I have questions I can contact the study hotline toll free at 1-xxx-xxx-xxxx.
__________________________________________
__
CUSTOMER’S NAME (Printed)
SOCIAL SECURITY NUMBER—LAST 4 DIGITS ONLY
| X | X | X | X |
__________________________________________
_______
CUSTOMER’S SIGNATURE
DATE
END TIME: |
|
ELAPSED TIME: |
6
|:|
|
|
|
|:|
|
START TIME: |
|
|:|
|
ATTACHMENT C
BASELINE INFORMATION FORM
Background
12.
What is your race?
CHECK ONE OR MORE BOXES
1.
Today’s Date: |
|
|/|
|
Month
2.
|/| 2 | 0 |
Day
|
|
1
Year
2
3
Name:
_____________________
__________
____
Last Name
First Name
M.I.
4
5
6
2a.
White
Black or African American
American Indian or Alaska Native
Asian
Native Hawaiian or Pacific Islander
Some Other Race (Write in):
Maiden Name:
13.
What is your primary spoken language?
CHECK ONE BOX
3.
4.
Address:
English
Spanish
Other (Write in):
1
_____________________________
_________
2
Street
Apt. #
3
|
____________________
|
City
State
Date of Birth: |
|
|/|
Month
|
|
_________
14.
ZIP Code
|/| 1 | 9 |
Day
|
What is your marital status right now?
CHECK ONE BOX
|
Married
Living together, unmarried
Separated
1
Year
2
5.
Social Security Number:
3
| X | X | X |-| X | X |-| X | X | X | X |
Male
6.
Gender:
7.
Home Phone Number:
|
(|
1
|
|) - |
|
2
|
15.
Female
|-|
|
|
|
5
6
Divorced
Widowed
Never married
Including yourself, how many people live with
you? (Please include babies, small children,
people who are not related to you, and people
who are temporarily away.)
|
|
4
|
| # OF PEOPLE LIVING WITH YOU
Area Code
16.
Are you a U.S. Veteran?
Under whose name is that phone listed?
1
2
Yes
1
My own name
Someone else’s name (Write in):
8.
First Name
17.
Cell Phone Number:
|
(|
|
|) - |
|
|
|-|
|
|
|
Which of the following degrees, diplomas, or
certificates have you received?
CHECK ALL THAT APPLY
|
Area Code
1
2
9.
Email Address:
10.
Other Contact Phone Number:
(|
|
|
|) - |
3
|
|
|-|
4
|
|
|
5
|
Area Code
6
7
This number belongs to:
8
9
10
11.
Are you of Hispanic, Latino, or Spanish origin?
1
Yes
0
11
No
12
13
Prepared by Mathematica Policy Research
No
Education
_________________________ _____________
Last Name
0
7
None
Elementary, Middle, or Junior High diploma
High School Diploma
Adult Basic Education (ABE) certificate
General Educational Development (GED)
Vocational/Technical degree or certificate
Business degree/certificate
Associates degree (AA)
Bachelor’s degree or equivalent (BA/BS)
Master’s degree or equivalent (MA/MS)
Doctor’s degree (MD, Ph.D.)
Other professional degree/certificate
Other (Write in):
(3/23/10)
|
Employment
18.
25.
What was your current or most recent rate of pay,
before taxes and deductions?
0
19.
|
Yes
No
0
Yes
No
|
|
|.|
|
| PER
Cents
CHECK ONE BOX
1
3
Hour
Week
Every 2 weeks
4
5
6
Twice per month
Year
Other (Write in):
____________________
Program Participation
Have you had a job in the past five years?
1
| ,|
Dollars
2
1
|
$|
Do you have any health problems—mental,
physical, or emotional—or substance abuse
problems that limit the kind or amount of work
or training that you can do?
26.
Do you or anyone in your household currently
receive assistance from any of the following
programs?
GO TO #26
CHECK ALL THAT APPLY
20.
1
Are you currently working?
2
1
0
Yes
No
3
GO TO #22
4
5
6
21.
In what month and year did your last job end?
|
|
Month
| /| 2 | 0 |
|
|
27.
TANF (Cash assistance)
SSI or SSDI
General Assistance
SNAP (Food Stamps)
Unemployment Compensation
Other (Write in):
Have you ever used services at a One-Stop Career
Center in the past?
GO TO #22
Year
1
Yes
0
No
ANSWER QUESTIONS 22-25 ABOUT YOUR CURRENT OR
MOST RECENT JOB. (If you currently have more than one
Thank you for completing this form. Please return it to
your WIA counselor.
job or had more than one job recently, give answers about
your job with the most hours.)
A.
FOR COUNSELOR USE ONLY
B.
22.
23.
Customer’s WIA Registration or Other Local Identifier:
|
What is the name of your current or former
employer?
1
WIA Site: [PREPRINTED]
Self-employed
What are (or were) your main duties at this
company? PLEASE BE SPECIFIC
|
|
|
|
|
|
D.
WIA Counselor’s ID: |
E.
WIA Counselor’s Name: ____________________ __________ ____
Last Name
First Name
M.I.
F.
Likelihood of participation in WIA-Funded Intensive Services
in absence of study:
1
Very Likely
3
Somewhat Unlikely
2
Somewhat Likely
4
Very Unlikely
G.
Likelihood of participation in WIA-Funded Training in absence of study:
| HOURS PER WEEK
|
|
3
Somewhat Unlikely
GO TO I
4
Very Unlikely
GO TO I
Likely Provider of WIA-Funded Training:
I.
BIF Entered Into PTS:
1
Yes
0
No
J.
Contact Form Submitted:
1
Yes
0
No
K.
Participant Consent Submitted: 1
Yes
0
No
END TIME: |
Prepared by Mathematica Policy Research
|
1
2 year or Community College
2
Private/Proprietary School
3
Other (Write in):__________________
How many hours per week do (or did) you usually
work?
|
1
Dislocated Worker
2
Adult
Customer’s qualification status:
H.
|
|
C.
1
Very Likely
2
Somewhat Likely
24.
|
8
|
|:|
|
| ELAPSED TIME:|
|
|:|
|__|
(3/23/10)
START TIME: |
|
ATTACHMENT D
WIA EVALUATION
CONTACT INFORMATION FORM—PRETEST VERSION
Please print clearly. Use pen only
APPLICANT INFORMATION
1. Name:
_________________________
2. Social Security Number—Last 4 Digits only:
_________
|_X_|_X_|_X_|_X_|
____________________________
Last Name
First Name
Middle Initial
CONTACT INFORMATION - RELATIVES AND FRIENDS
INSTRUCTIONS: In the space below, please provide the name, address, email address, and phone number(s) of three close relatives or friends
who do not live with you but who are likely to know how to contact you in the next year. We will only contact these people if we cannot reach you
directly. Please complete all three sections.
3. NAME AND ADDRESS OF A CLOSE FRIEND OR RELATIVE WHO DOES NOT LIVE WITH YOU
____________________________________ _________________________________________________
Last Name
______________
First Name Middle Initial
____________________________________ _________________________________________________
_______________
Street Address
Apt. No.
TELEPHONE AND E-MAIL:
___________________________________
City
|___|___| ______________
Home (______) _______-_________________
State Zip Code
Area Code
Number
Cell (______) _______-_________________
RELATIONSHIP TO APPLICANT: 5 CHECK ONE
Area Code
1 □ Parent
4 □ Friend/Neighbor
2 □ Grandparent 5 □ Employer
3 □ Brother/Sister 6 □ Other_________________
Number
Work (______) _______-_________________
Area Code
Number
Email Address _________________________
4. NAME AND ADDRESS OF A CLOSE FRIEND OR RELATIVE WHO DOES NOT LIVE WITH YOU
____________________________________ _________________________________________________
Last Name
______________
First Name
Middle Initial
____________________________________ _________________________________________________
_______________
Street Address
Apt. No.
TELEPHONE AND EMAIL:
___________________________________
City
|___|___| ______________
State Zip Code
Home (______) _______-_____________________
Area Code
Number
Cell
RELATIONSHIP TO APPLICANT: 5 CHECK ONE
1 □ Parent
2 □ Grandparent
3 □ Brother/Sister
(______) _______-_________________
Area Code
4 □ Friend/Neighbor
5 □ Employer
6 □ Other_________________
Number
Work (______) _______-_________________
Area Code
Number
Email Address _________________________
5. NAME AND ADDRESS OF A CLOSE FRIEND OR RELATIVE WHO DOES NOT LIVE WITH YOU
____________________________________ _________________________________________________
Last Name
______________
First Name Middle Initial
____________________________________ _________________________________________________
Street Address
_______________
Apt. No.
TELEPHONE AND EMAIL:
___________________________________
City
|___|___| ______________
State Zip Code
Home (______) _______-_________________
Area Code
Number
Cell
RELATIONSHIP TO APPLICANT: 5 CHECK ONE
1 □ Parent
2 □ Grandparent
3 □ Brother/Sister
4 □ Friend/Neighbor
5 □ Employer
6 □ Other_________________
Number
Work (______) _______-_________________
Area Code
Number
Email Address _________________________
Prepared by Mathematica Policy Research
Prepared by Mathematica Policy Research
(______) _______-_________________
Area Code
END TIME: |
|
ELAPSED TIME: |
9
|:|
|
|
|:|
|
|
|
(3/23/10)
| File Type | application/pdf |
| Author | Computer & Network Services |
| File Modified | 2011-09-20 |
| File Created | 2010-03-15 |