LEAP Pretest Memo

^Attachment 16: Protocols for Pretest of Consent and Respondent Information Forms

Linking to Employment Activities Pre-Release (LEAP) Evaluation

Site Visit Protocols

March 2016

This page has been left blank for double-sided copying.

MEMORANDUM

P.O. Box 2393 Princeton, NJ 08543-2393 Telephone (609) 799-3535 Fax (609) 799-0005 www.mathematica-mpr.com

TO: Megan Lizik and Christina Yancey

FROM: Jillian Stein, Jeanette Holdbrook, and Jeanne Bellotti DATE: 3/14/2016

SUBJECT: LEAP Pretest of Consent and Respondent Information Forms

I. Introduction

As part of The Linking to Employment Activities Pre-Release (LEAP) implementation study, Mathematica Policy Research and Social Policy Research Associates will collect consent¹ and respondent information forms (RIFs) at the start of focus groups with: (1) LEAP frontline staff, (2) LEAP partner staff, (3) LEAP pre-release participants, and (4) LEAP post-release participants. The RIFs are brief, two-page questionnaires that are designed to gather demographic characteristics and other relevant information from focus group participants. To identify instrumentation problems, measure the response burden, and prevent any unforeseen difficulties in administration, Mathematica pretested the following instruments:

• LEAP frontline staff /partner RIFs. The study team will collect data from LEAP frontline staff and LEAP partner staff through a single version of the RIF that is designed to gather respondents’ demographic characteristics and background information, such as their education, work experience, and program roles and activities.

• LEAP participant consent form and RIF. The study team will collect active consent and RIF data from two types of LEAP participants—those who are currently receiving LEAP pre-release services and those who have been released from jail and are receiving post-release services. The focus group facilitator will provide the consent form to members of the focus groups and read it aloud at the start of the session to ensure that all participants, regardless of reading ability, understand the information about the risks and benefits of their participation. The LEAP participant forms will collect information about respondents’ demographic characteristics and background information, such as their work and criminal justice histories. The pre- and post-release consent and RIFs are very similar, with only slightly different wording at certain questions.²

In total, Mathematica conducted eight pretests across the three groups of respondents in February 2016. The pretest respondents included a LEAP frontline staff member (a LEAP case manager), a LEAP partner (an American Job Center [AJC] case manager), three pre-release LEAP participants, and three post-release respondents who were not involved with LEAP but were formerly incarcerated. Although we originally planned to conduct a third pretest with a LEAP staff member, we were unable to do so due to a miscommunication among program staff. We considered seeking out an additional staff member or partner to test the form but decided it was unnecessary, given the diversity of the two respondents who did test the form and the limited feedback they provided. Table 1 details the demographic characteristics of the pretest respondents.

Table 1. Demographic characteristics of the respondents

Respondent	Round	Instrument(s) tested	Title/role	Gender	Age	Race/ ethnicity
1	1	Staff/partner RIF	LEAP case manager	Male	38	White
2	1	Staff/partner RIF	AJC case manager	Female	58	Dominican Hispanic
3	1	Consent form and pre-release participant RIF	Pre-release participant	Male	35	White
4	1	Consent form and pre-release participant RIF	Pre-release participant	Male	35	White
5	1	Consent form and pre-release participant RIF	Pre-release participant	Male	57	White
6	2	Revised consent form and post-release participant RIF	Formerly incarcerated individual	Male	53	African American
7	2	Revised consent form and post-release participant RIF	Formerly incarcerated individual	Male	30	African American
8	2	Revised consent form and post-release participant RIF	Formerly incarcerated individual	Female	43	White

The remainder of this memo discusses the pretest methodology, findings for each respondent type, findings on respondent burden, and next steps.

II. Pretest methodology

The pretests were conducted in two phases so that information from the first round of pretests could inform revisions to the materials, and the revised instruments could be tested in the second round. Before the first round of pretests, survey staff met to discuss best practices and review the pretest protocol and procedures regarding the post-survey debriefing with respondents. Appendix A presents the pretest procedures and protocols that were used to conduct the structured debriefs with respondents during rounds one and two.

To mirror the implementation study design, the pretests were conducted face-to-face using paper and pencil forms. During each pretest, respondents received a brief introduction about the purpose of the activity and an opportunity to ask questions before the activity began. Next, respondents received the survey materials and were asked to read and complete them. Afterward, the pretest facilitator asked respondents a set of open-ended questions and probes to gauge the respondents’ understanding of the questions. The pretest facilitator also asked respondents targeted questions to assess their understanding of specific terms and ease of answering certain questions that the study team identified as potentially confusing.

Pretests with program participants also involved testing the consent form that will be administered to LEAP pre- and post-release participants involved in the focus groups. Before responding to the RIF, respondents received the written consent form but the pretest facilitator also read it aloud to them as is specified in the implementation study protocols. Afterward, the facilitator asked respondents a set of open-ended questions to gauge their comprehension of the form and to identify any areas that were ambiguous or unclear.

Because the pre- and post-release participant forms are so similar, the survey team intentionally staggered the effort so that the post-release pretests could act as the test for the revised consent form and revised participant RIF. After revising the instrument on the basis of pre-release participant responses (discussed below), the team administered the revised RIF and consent form to the post-release respondents. This test assessed whether the modifications resolved the issues the pre-release participants identified or whether the revisions caused any new confusion. Consistent with the study plans, each post-release respondent also received a $25 incentive for travel expenses and time.

During the pretests, the pretest facilitator ook notes on the responses to the questions, the timing of the activity, and any issues related to wording and general understanding of questions. Survey research staff discussed differences and commonalities across interviews and revised the survey materials as discussed in the following section.

III. Findings on instrument content and resulting modifications

The following summarizes the main findings from pretests with the staff and partners as well as pre- and post-release participants. It also describes the instrument modifications made as a result of these findings. Revised instruments will be included in the final evaluation design report and are included as Attachments 8 and 9 and Attachments 11–13 in the updated OMB package.

1. Frontline staff and partners

We conducted two pretests with frontline staff/partners associated with LEAP. One respondent worked directly with the program in a correctional facility and the other was minimally familiar with the program and worked in a community AJC.³ The two respondents differed in their demographics, program roles, education, and prior work experience. Overall, only three minor issues arose during the staff/partner pretests. Table 2 outlines these issues and describes how they were resolved.

Table 2. Issues identified during the staff/partner pretests and instrument modifications

Item	Issue	Instrument modification
A7. Primary responsibilities for LEAP	The original question asked “Which of the following activities represent your primary responsibilities as a staff member for the LEAP program?” Debriefs revealed inconsistencies in how respondents interpreted the concept of “primary responsibilities.” Additionally, it was unclear whether respondents should only include services they provided directly or whether they should include services to which they referred clients.	Because the study team is most interested in capturing the primary activities that staff and partners provide directly, we revised the question text to ask about primary “activities” rather than “responsibilities.” The revised question reads, “Which of the following represent your primary activities for the LEAP program?” We also added an instruction, “Please only mark services that you provide directly.”
A4. Years of experience with justice-involved individuals	One respondent questioned the phrase "justice involved individuals."	Replaced the phrase "justice involved individuals" with "individuals with criminal or delinquent backgrounds."
B2. Age	One respondent reported discomfort reporting an exact age and would have preferred to select an age range.	To be sensitive to this discomfort, we revised the question to ask for age ranges.

2. Program participants

In round 1, we conducted three pretests with pre-release participants in a LEAP program in a correctional facility. Respondents were all white men but they varied on a range of other characteristics such as age, educational level, offense and incarceration histories, as well as their disability and work statuses before their most recent incarceration. The post-release pretests involved three individuals who were previously incarcerated. Unlike the pre-release participants, the post-release respondents varied by age and gender.

Table 3 outlines the issues identified as part of the review of the consent form and RIF and discusses how the study team revised the forms to address those issues. Generally, the results from the second round of pretests of the participant consent and RIFs confirmed that our revisions improved the clarity of the instruments and eliminated the problems we identified in round 1. The one exception to this relates to question C8, regarding one’s physical and mental health. In this instance, all three respondents had difficulty answering the revised question. After consultation, the study team determined that the most important information to capture is whether respondents have physical or mental health conditions that prevent them from working. Hence, we removed question C8 and revised question C9 to make clear that respondents should include any physical or mental health conditions that limit their ability to work. Table 3 summarizes the issues and solutions identified through each round of participant pretests.

Table 3. Issues identified during the participant pretests and instrument modifications

Item	Issue	Instrument modifications
Round 1 – Pre-release participants
Consent form	Respondents indicated confusion about what information would be shared, what would be confidential, and what the risks of participating were.	We added some explanation about the RIF to the consent form before round 2 and changed the wording of the section on risks of participation for ease of understanding.
A1 and A3. Education and work experience	Respondents inconsistently included education and work experience they gained while incarcerated.	We clarified the question text so respondents would include education received while incarcerated but exclude work experience while incarcerated.
A2. Education and work credentials	Two respondents assumed that the question about work and education credentials and certificates was asking about general diplomas and degrees.	We revised the question text to include the word "specialized" to help clarify that this question is not asking about general education credentials and added a second sentence that explicitly states that respondents should not include general education credentials.
A4. Work experience before incarceration	The original question asked about the respondent’s employment status in the six months before incarceration but all respondents reported that their job status varied over that time, making it hard to select one response option.	Because we are most interested to know whether respondents had any recent work experience, we revised A4 by narrowing the reference window and asking whether respondents were working at any time during the month before their most recent incarceration.
A4. Amount of work experience	Originally, A4 asked about work status and quantity of work. The modifications to A4 as discussed above required us to add a question to capture quantity of work.	We added a question to capture whether the respondents who reported working at any time had worked 1–29 or 30 or more hours per week.
Section on criminal background	One respondent did not understand why we needed to know about criminal background, why we did not already have that information, and described the transition from Section A to B as “rough.”	We added a short introduction to Section B to explain why we are asking the criminal background questions.
B7. Criminal charges	All three pretest respondents selected "Other (specify)" for their criminal charge.	We removed categories of crimes that LEAP participants should not have due to eligibility requirements and added driving under the influence (DUI). Additionally, because individuals may be serving time for more than one charge, we added clarifying instructions: “Please select only the most serious one."
C8. Describe health	Originally, C8 asked respondents to describe their health, without specifying whether to consider physical or mental health. Through our debrief, we determined that some respondents were considering mental health whereas others were not.	To ensure consistency in responses, we revised the RIF to explicitly ask respondents to consider both their mental and physical health. The revised question reads, “How would you describe your overall physical and mental health?” NOTE: Question C8 as described here has been removed from the RIF.
Round 2 – Post-release respondents
B5. Total time incarcerated	One respondent was incarcerated for less than a month and was unsure how to answer B5.	Added clarification “If less than 1 month, please record 1 month”
C8. Describe health	Three respondents reported that they would rank their physical and mental health differently, making it hard to provide a single answer to the revised question that asked individuals to consider both their mental and physical health.	The most important information to capture is whether respondents have physical or mental health conditions that prevent them from working. Hence, we removed question C8 and we revised question C9 to make clear that respondents should include any physical or mental health conditions that limit their ability to work. NOTE: Former questions C9 and C10 have been renumbered on the RIF as C8 and C9, respectively, to reflect the removal of former question C8.

IV. Findings on respondent burden

The average administration time was three minutes across the staff/partner pretests and approximately five minutes across the participant pretests (with a range of three to six minutes). The draft OMB package included an estimated burden of seven minutes per form, which is higher than demonstrated in pretests. We will revise the burden tables in the OMB package to reflect three minutes for staff/partner RIFs and five minutes for pre- and post-release participant RIFs.

V. Next steps

The study team has updated the OMB package to indicate that the pretest was completed, to include the revised versions of the consent form and RIFs, and to adjust the burden tables to reflect the pretest results. We will include the revised instruments in our final design report and the applications for IRB approval and the NIH Certificate of Confidentiality. If DOL has any questions, we are happy to discuss the pretests further.

cc: Jillian Berk, Andrew Wiegand, Samina Sattar

This page has been left blank for double-sided copying.

Appendix A

Protocols for Pretest of LEAP Consent and Respondent Information Forms:

Pretest Questions – Round 1

February 2016

PART I – INTERVIEW SCRIPT: FRONTLINE/PARTNER STAFF

INTERVIEWER: READ CONSENT SCRIPT AT NORMAL INTERVIEWING PACE,
THEN START FIRST ACTIVITY.

In-Person Interview Script for Frontline Staff/Partners:

Hi, my name is [name] from Mathematica Policy Research, a social research company based out of Princeton, New Jersey. Thanks so much for being willing to help us out today.

As we go through this activity, you may notice that at some points I’m reading from my notes. I just want to make sure that I give each person the same information. There will be plenty of time for us to talk through any questions you might have after we finish the activity.

As you may know, we are currently testing a form that will be used to collect information from LEAP staff and partners before focus groups that will be held as part of an evaluation sponsored by DOL. We want your help to figure out how to make the questions and answer categories clear and easy to understand. To do this I am going to ask you to do one activity, and then we will discuss it. All together this will take about 30 minutes of your time.

Please keep in mind that there are no right or wrong answers to any of the questions I ask you. We really just want to find out as much as possible about what you are thinking, so don’t hesitate to speak up whenever something is unclear, hard to answer, or doesn’t seem to apply to you.

I will be taking notes during our activity [and IF ALLOWED: I will be recording the interview so that I can go back and listen to it later when I’m summarizing the results].

Everything that you tell me will be kept private. You don’t have to answer any questions that make you feel uncomfortable.

Do you have any questions before we start?

Let’s begin.

Please hit the record button on your digital voice recorder

PART I A – RESPONDENT INFORMATION FORMS: FRONTLINE/PARTNER STAFF

Materials:

Frontline/Partner Staff RIF

Pen or pencil

Tape recorder (batteries)

Timer

Step 1: Explain the purpose of the activity

I’d like for you to answer the questions on this paper. The questions are about you. I am not focused on what your answers are, but afterwards I would like to know how clear the questions were, how easy they were to answer, and if the answer choices made sense.

Step 2: Respondent completes RIF

• Start timer/Look at watch

• Hand RIF to participant

  • Note which questions respondent spends the most time reading and answering.

• Stop timer/ Look at watch

Step 3: Debrief RIF

1. Were there any questions on the form that you did not know how to answer? Tell me more about that.

2. Were there any questions where your answer didn’t fit the options?

3. Were there any questions that didn’t apply to you?

4. Were there any parts that you found confusing? Tell me more about that.

5. Were there any questions that made you feel uncomfortable or that you didn’t want to answer? Which questions? Can you elaborate on that a little?

6. For question A4, who do you consider to be “justice involved individuals?”

7. In question A5, what does “workforce development” mean to you?

8. For question A7, who do you consider to be a staff member of the LEAP program? Do you consider yourself a staff member?

   • If not, how would you characterize your role in the LEAP program? A partner?

9. For question A7, you checked X, Y, and Z. Do you do those all equally? Are some more primary responsibilities than others?

10. For question A8, how easy or difficult is it for you to estimate the percentage of time you spend on LEAP during a typical week? Why?

11. For question A9, how easy or difficult is it for you to estimate your average LEAP caseload size during a typical week? Why?

    • Does “caseload” apply? What wording would be clearer or make more sense for someone in a position like yours?

    • Would this be clearer: How many LEAP participants do you currently interact with? Is that typical?

This was the last question I had. Did you have any other comments about the questionnaire? Thank you for helping us test the questionnaire.

PART II – INTERVIEW SCRIPT: LEAP PARTICIPANTS

INTERVIEWER: READ CONSENT SCRIPT AT NORMAL INTERVIEWING PACE,
THEN START FIRST ACTIVITY.

In-Person Interview Script for Pre- and Post-Release Participants:

Hi, my name is [name] from Mathematica Policy Research, a social research company based out of Princeton, New Jersey. Thanks so much for being willing to help us out today.

As we go through this activity, you may notice that at some points I’m reading from my notes. I just want to make sure that I give each person the same information. There will be plenty of time for us to talk through any questions you might have after we finish each activity.

As someone may have explained, we are currently testing two forms that we will use before group that we’re holding as part of an evaluation sponsored by DOL. The first form is a consent form, and the second is a form that will be used to collect information from LEAP participants. We want your help to figure out how to make the consent form, questions, and answer categories clear and easy to understand. To do this I am going to ask you to do two different types of activities. We will discuss each activity after you do them. All together this will take about 30 minutes.

Please keep in mind that there are no right or wrong answers to any of the questions I ask you. We really just want to find out as much as possible about what you are thinking, so don’t hesitate to speak up whenever something is unclear, hard to answer, or doesn’t seem to apply to you.

I will be taking notes during our activity [and IF ALLOWED: I will be recording the interview so that I can go back and listen to it later when I’m summarizing the results].

Everything that you tell me will be kept private. You don’t have to answer any questions that make you feel uncomfortable.

Do you have any questions before we start?

Let’s begin.

Please hit the record button on your digital voice recorder

PART II A – CONSENT FORM: LEAP PARTICIPANTS ONLY

Materials:

Pre-release Participant Focus Group Consent Form

Post-release Participant Focus Group Consent Form

Pen or pencil

Tape recorder (and batteries)

Timer / stopwatch

Step 1: Give respondent instructions

Here’s how it works. The first thing I am going to do is give you a copy of the consent form we plan to use in our focus groups. Then I will explain what it’s for and read it out loud to you. You can follow along as I read, if you’d like to. As I read it, please note anything that doesn’t make sense or is unclear. If you want, you can mark or circle anything that’s unclear, to help you remember. When I’m done reading, I will ask you some questions about your reactions to the consent form and your thoughts about it.

Step 2: Read consent form

• Start timer/Look at watch

• Hand consent form to participant

• Read consent form aloud to participant

• Stop timer/Look at watch

Step 3: Debrief consent form

What questions do you have about this consent form?

Which parts were confusing or unclear? Was there any wording that could be more clear?

Are there changes we can make to it to help others understand it better?

To you, what is the purpose of this consent form?

• Why would we give it to someone before a group discussion?

PART II B – RESPONDENT INFORMATION FORMS: LEAP PARTICIPANTS

Materials:

Pre-Release Participant RIF

Post-Release Participant RIF

Pen or pencil

Tape recorder (and batteries)

Timer/ stopwatch

Step 1: Explain the purpose of the activity

I’d like for you to answer the questions on this paper. The questions are about you. I am not focused on what your answers are, but afterwards I would like to know how clear the questions were, how easy they were to answer, and if the answer choices made sense.

Step 2: Respondent completes RIF

• Start timer/Look at watch

• Hand RIF to participant

  • Note which questions respondent spends the most time reading and answering.

• Stop timer/ Look at watch

Step 3: Debrief RIF

1. Were there any questions on the form that you did not know how to answer? Tell me more about that.

2. Were there any parts that you found confusing? Tell me more about that.

3. Were there any questions where your answer didn’t fit the options?

   • Were there questions where you had to choose only one answer, but you wanted to choose more than one answer?

4. Were there any questions that made you feel uncomfortable or that you didn’t want to answer? Which questions? Can you elaborate on that a little?

5. For question A2, what do “education or work credentials or certificates” mean to you? What comes to mind?

6. In question A3, what does “work experience” mean to you? What counts as work experience to you?

7. For question A4, if you said you were working: what is the total number of jobs you had? [If more than one job reported: How many hours did you work per week (on average) across all jobs?]

8. In question A4, what does “looking for work” mean to you? What does it involve?

9. [FOR PRE-RELEASE PARTICIPANTS ONLY:] For A4, over the six months before your most recent incarceration, did your job status change?

10. For question B5, how difficult was it to come up with an answer? Please explain how you came up with the answer you wrote.

11. In question C8, were you thinking about your physical health, mental health, or both?

12. For question C9, what do you consider to be a disability that limits your ability to work?

This was the last question I had. Did you have any other comments about the questionnaire? Thank you for helping us test the questionnaire.

Protocols for Pretest of LEAP Consent and Respondent Information Forms:

Pretest Questions – Round 2

(Revised participant materials)

February 2016

PART I – INTERVIEW SCRIPT: LEAP PARTICIPANTS

INTERVIEWER: READ CONSENT SCRIPT AT NORMAL INTERVIEWING PACE,
THEN START FIRST ACTIVITY.

In-Person Interview Script for Pre- and Post-Release Participants:

Hi, my name is [name] from Mathematica Policy Research, a social research company based out of Princeton, New Jersey. As someone may have explained, we are currently testing two forms that are part of an evaluation we are doing on behalf of the Department of Labor. The first form is a consent form, and the second is a background information form that will be used to collect information from participants in the Linking to Employment Activities Pre-Release Program. We plan to use these forms with people who participate in group discussions about the program. We want your help to figure out how to make the consent form and background information form clear and easy to understand. Thanks so much for being willing to help us out today.

As we go through this activity, you may notice that at some points I’m reading from my notes. I just want to make sure that I give each person the same information. I am going to ask you to review the consent form first and then the background information form. We will discuss each form after you review them. There will be plenty of time for us to talk through any questions you might have after we finish each form. All together this will take no longer than 30 minutes.

Please keep in mind that there are no right or wrong answers to any of the questions I ask you. We really just want to find out as much as possible about what you are thinking, so don’t hesitate to speak up whenever something is unclear, hard to answer, or doesn’t seem to apply to you.

I will be taking notes during our activity [and IF ALLOWED: I will be recording the interview so that I can go back and listen to it later when I’m summarizing the results].

Everything that you tell me will be kept private. You don’t have to answer any questions that make you feel uncomfortable.

Do you have any questions before we start?

Let’s begin.

Please hit the record button on your digital voice recorder

PART I A – CONSENT FORM: LEAP PARTICIPANTS ONLY

Materials:

Pre-release Participant Focus Group Consent Form

Post-release Participant Focus Group Consent Form

Pen or pencil

Tape recorder (and batteries)

Timer / stopwatch

Step 1: Give respondent instructions

Here’s how it works. The first thing I am going to do is give you a copy of the consent form we plan to use in our discussion groups. Then I will read it out loud to you. You can follow along as I read, if you’d like to. As I read it, please note anything that doesn’t make sense or is unclear. If you want, you can mark or circle anything that’s unclear, to help you remember. When I’m done reading, I will ask you some questions about your reactions to the consent form and your thoughts about it.

Step 2: Read consent form

• Start timer/Look at watch

• Hand consent form to participant

• Read consent form aloud to participant

• Stop timer/Look at watch

Step 3: Debrief consent form

What questions do you have about this consent form?

Which parts were confusing or unclear? Was there any wording that could be more clear?

Are there changes we can make to it to help others understand it better?

To you, what is the purpose of this consent form?

• Why would we give it to someone before a group discussion?

PART I B – RESPONDENT INFORMATION FORMS: LEAP PARTICIPANTS

Materials:

Post-Release Participant RIF

Pen or pencil

Tape recorder (and batteries)

Timer/ stopwatch

Step 1: Explain the purpose of the activity

I’d like for you to answer the questions on this paper. The questions are about you. I am not focused on what your answers are, but afterwards I would like to know how clear the questions were, how easy they were to answer, and if the answer choices made sense.

Step 2: Respondent completes RIF

• Start timer/Look at watch

• Hand RIF to participant

  • Note which questions respondent spends the most time reading and answering.

• Stop timer/ Look at watch

Step 3: Debrief RIF

13. Were there any questions on the form that you did not know how to answer? Tell me more about that.

14. Were there any parts that you found confusing? Tell me more about that.

15. Were there any questions where your answer didn’t fit the options?

    • Were there questions where you had to choose only one answer, but you wanted to choose more than one answer?

16. Were there any questions that made you feel uncomfortable or that you didn’t want to answer? Which questions? Can you elaborate on that a little?

17. For question A2, what do education or work credentials or certificates mean to you? What comes to mind?

18. In question A4, what does “looking for work” mean to you? What does it involve?

19. For question B5, how difficult was it to come up with an answer? Please explain how you came up with the answer you wrote.

20. In question C8, how difficult was it to answer? If it asked you about only your mental health or only your physical health, would your answer be different?

This was the last question I had. Did you have any other comments about the questionnaire? Thank you for helping us test the questionnaire.

1 Verbal consent will be collected from frontline staff and partner focus group participants. Written consent will be collected from all LEAP pre- and post-release participants.

2 For example, instead of asking for respondents’ current living situation, the pre-release participant RIF asks respondents about their living situation before their most recent incarceration.

3 Considering the very minor and revisions made to the forms, the study team decided it was unnecessary to expend additional resources to conduct a second round of staff pretests.