Appendix D: Discussion of Survey Pre-Test

MEMO TO: David Butler, Charles Michalopoulos, Stephen Freedman, Dina Berin

FROM: Julia Milliner-Waddell, Stacie Feldman, Lisa Schwartz

DATE: 11/10/2008

PAGE: 0

MEMORANDUM

TO: David Butler, Charles Michalopoulos, Stephen Freedman,
and Dina Berin

FROM: Julia Milliner-Waddell, Stacie Feldman, and Lisa Schwartz DATE: 11/10/2008

ABD - 52

SUBJECT: Accelerated Benefits Demonstration 12-Month Follow-up Survey Pretest

A. INTRODUCTION

In preparation for the Accelerated Benefits Demonstration (AB) 12-month follow up survey, Mathematica Policy Research, Inc. conducted an internal pretest of the 12-month follow-up survey instrument. That is, instead of testing the questionnaire with SSDI beneficiaries, we used MPR on-call staff as both interviewers and respondents. Testing with the target population, AB demonstration participants, was not feasible. Similarly, we could not recruit uninsured SSDI beneficiaries for the pretest while the enrollment period is still underway for the demonstration. Also, since many of the questions were used at baseline and at the six-month follow up, this approach made sense. The same approach was used for the six-month pretest and worked well. A total of nine pretest interviews were conducted.

The goals of the pretest were to evaluate the clarity of the questions asked, identify possible modifications to either question wording or question order that could improve the quality of the outcome data, and identify and eliminate problems with specific questionnaire items, including skip errors prior to programming the instrument for Computer-Assisted Telephone Interviewing (CATI). In addition, we used the pretest to estimate respondent burden.

On average, the baseline questionnaire took 45 minutes to administer, with a range of 35 to 50 minutes. The pretest was conducted using paper and pencil administration.

B. 12-Month Follow-UP Instrument PreTest

Three interviewers with experience on prior rounds of AB data collection were trained to conduct the 12-month interview pretest. Nine on-call staff members who were not familiar with AB served as respondents.

Each respondent followed one of three scenarios that were designed to test different paths through the questionnaire. The scenarios tested each of the experimental conditions—AB, AB Plus, and control. The scenarios were designed to test all of the major skip patterns. Table 1, describing the scenarios tested is attached to this memo. Upon completion of the nine pretest interviews, we held a debriefing session to gather feedback and input from the interviewers.

Since many of the questions were used previously, the pretest uncovered few problems. We are recommending some relatively minor modifications to the questionnaire based on the findings from the pretest.

The issues uncovered by the pretest can be categorized as 1) minor wording changes, 2) skip instruction corrections, 3) the need to clarify whether specification is needed for certain “other specify” responses, and 4) question sequencing. Minor wording modifications and skip corrections are highlighted in track changes to the pretest questionnaire attached to this memo. The issues involving other specify questions and question sequencing are discussed below.

1. Other Specify Questions

There are several questions for which an “other” response is not further specified. We need to confirm that such specification is not required for analytic purposes. It should be noted that if these responses are specified, coding will be required. The following questions do not currently require specification for an “other” response.

B2a. What is the main reason {you think/(NAME) thinks} {your/his/her} health is [fill MUCH BETTER, SOMEWHAT BETTER, SOMEWHAT WORSE OR MUCH WORSE] now than it was in [fill RA MONTH YEAR]?

INTERVIEWER: RECORD VERBATIM, THEN CODE

CODE ALL THAT APPLY

PROGRESSION OF DISEASE 01

BETTER HEALTH CARE 02

MORE DOCTOR VISITS 03

MORE/BETTER PRESCRIPTIONS MEDICINES 04

PHYSICAL THERAPY HELPED 05

STOPPED PHYSICAL THERAPY 06

CAN’T AFFORD PRESCRIPTION MEDICINES 07

STOPPED GOING TO THE DOCTOR 08

HAD TO CHANGE DOCTORS 09

HAD SURGERY 10

OTHER 11

DON’T KNOW d

REFUSED r

C14b. What kind of insurance {have you/has (NAME)} had since [fill RA MONTH, YEAR]?

PROBE: If {you/he/she} had more than one type of insurance since that time, please answer about {your/his/her} most recent coverage.

INTERVIEWER: DO NOT ACCEPT LIMITED COVERAGE PLANS LIKE DENTAL ONLY, PRESCRIPTION ONLY, ETC.

READ LIST IF NECESSARY

CODE ONE ONLY

MEDICAID 01

MEDICARE 02

MEDI-GAP 03

VA/TRICARE/CHAMPUS/CHAMP-VA 04

INDIAN HEALTH SERVICE 05

WORKER’S COMPENSATION 06

COBRA 07

STATE GOVERNMENT PROGRAM 08

PRIVATE HEALTH INSURANCE PURCHASED

ON OWN 09

PRIVATE HEALTH INSURANCE PURCHASED

THROUGH EMPLOYER 10

PRIVATE HEALTH INSURANCE THROUGH NAME’S

SPOUSE/PARTNER’S PLAN 11

SOME OTHER KIND OF HEALTH INSURANCE 12

DON’T KNOW d

REFUSED r

Note: We collected the specification for this question at baseline. Responses frequently needed back coding, usually into the “state government program” answer category.

This question is asked a second time (see C14f) if the sample member has had more than one kind of insurance since random assignment.

G2. Where did {you/he/she} get help? Was it from . . .

IF NEEDED: A WIPA is a local organization that has arranged with Social Security to provide work incentive and planning services for Social Security SSDI and SSI beneficiaries. This program replaced the former Benefits Planning Assistance and Outreach (BPAO) Program

CODE ALL THAT APPLY

A Work Incentives Planning and Assistance (WIPA)

project or SSA, 01

An AB Plus coach or employment benefits counselor, or 02

Some other type of place? 03

DON’T KNOW d

REFUSED r

G4. Where did {you/he/she} receive those services? Was it from . . .

CODE ALL THAT APPLY

A Ticket to Work Employment Network? 01

A State Vocational Rehabilitation agency? 02

A State unemployment office? 03

A Center for Independent Living? 04

A Private rehabilitation provider? 05

Or some other type of place? 06

DON’T KNOW d

REFUSED r

G5. Which of the following employment services {have you/has (NAME)} received since [fill RA MONTH YEAR]? Did {you/he/she} receive . . .

	mark one per row
	YES	NO	DON’T KNOW	REFUSED
1. A work or job assessment?	01	00	d	r
2. Help to find a job?	01	00	d	r
3. Training to learn a new job or skill?	01	00	d	r
4. Advice about making modifications or accommodations to {your/(NAME)’s} job or workplace?	01	00	d	r
5. On-the-job training, job coaching, or support services?	01	00	d	r
6. Career counseling?	01	00	d	r
7. Some other employment service?	01	00	d	r

2. Question Sequence

We are recommending a change in sequence for some questions on the employment grid. Specifically, because self-employment is not widespread among this population, we would like to ask sample members if any of their employment since random assignment was self-employment. If the sample member has not been self-employed since random assignment, we will not need to re-ask this question for each employer listed. This avoids the awkwardness that is introduced when asking sample members if a job constituted self-employment when clearly it is not the case; for example, employment at Wal-Mart. We would also like to move up the question regarding whether a job is part of a special work program to precede other specific questions about each job. Finally, we have added a question about the main reason why a job ended. The recommended revised sequence is shown below.

(Asked at Baseline)

F1. These next questions are about employment.

{Are you/Is (NAME)} currently working at a job for pay? Include both part-time and full-time jobs, as well as any self-employment jobs you held for pay or profit.

YES 01 (F3)

NO 00

DON’T KNOW d

REFUSED r

(F1=00, d or r)

F2. Since [fill RA MONTH YEAR], {have you/has (NAME)} worked at a job for pay? Include both part-time and full-time jobs, as well as any self-employment jobs you held, for pay or profit.

YES 01 (F4)

NO 00 (F13)

DON’T KNOW d (F13)

REFUSED r (F13)

(F1=01)

(Asked at Baseline)

F3. How many jobs {do you/does (NAME)} currently have? Include both part-time and full-time jobs, as well as any self-employment jobs you held for pay or profit.

INTERVIEWER: TREAT JOBS WITH TEMPORARY AGENCIES AS ONE JOB, REGARDLESS OF THE NUMBER OF ASSIGNMENTS.

| | NUMBER OF JOBS

(1-5)

DON’T KNOW d

REFUSED r

(F1=01 or F2=01)

F4. PROGRAMMER: IF F1=01, USE FILL: (Including {your/(NAME’s)} {current job/(FILL NUMBER FROM F3) current jobs}, How many different jobs have {you/he/she} had since [fill RA MONTH YEAR]? Include both part-time and full-time jobs, as well as any self-employment, but only include jobs for pay or profit.

INTERVIEWER: IF A JOB THAT WAS INTERRUPTED BY TWO OR MORE UNPAID WEEKS, COUNT AS SEPARATE JOBS, EVEN IF IT IS WITH THE SAME EMPLOYER. IF THE SEPARATION WAS LESS THAN TWO WEEKS, COUNT IT AS ONE JOB.

INTERVIEWER: TREAT JOBS WITH TEMPORARY AGENCIES AS ONE JOB, REGARDLESS OF THE NUMBER OF ASSIGNMENTS.

| | NUMBER OF JOBS

(1-5)

DON’T KNOW d

REFUSED r

(F1=01 or F2=01)

(Asked at Baseline)

F4a. {(Were you/Was (NAME)} self-employed at any time since [fill RA MONTH YEAR]?

YES 01

NO 00

DON’T KNOW d

REFUSED r

	JOB 1	JOB 2
(F1 or F2=01) F5. Starting with {your/his/her} (current job/main job/the most recent job that {you/he/she} had), please tell me the name of the companies, organizations, and people {you have/(NAME) has} worked for since [fill RA MONTH YEAR]. PROBE: Your main job is the one with the most hours. PROBE: IF F3 OR F4 >1 SAY: What was the job before that?	(SPECIFY)[specify] 01 DON’T KNOW d REFUSED r	(SPECIFY) [specify] 01 DON’T KNOW d REFUSED r
(F1 or F2=01) F5a. Let me verify. Since [fill RA MONTH YEAR] {you/(NAME)} worked at [fill F5 NAMES]. Is this correct, or are there any other jobs {you/he/she} may have had? IF CORRECT, ENTER “1” AND CONTINUE. IF NOT CORRECT, GO BACK TO F3, F4, F4a ,AND F5 TO ENTER CORRECT NUMBER AND NAMES OF JOBS HELD.	CORRECT 01 NOT CORRECT 00 DON’T KNOW d REFUSED r	CORRECT 01 NOT CORRECT 00 DON’T KNOW d REFUSED r
(F4a=01) (Asked at Baseline) F5b. {(Are/Were) you/(Is/Was) (NAME)} self-employed at this job?	YES (F12) 01 NO 00 DON’T KNOW d REFUSED r	YES (F12) 01 NO 00 DON’T KNOW d REFUSED r

	JOB 1	JOB 2
(Asked at Baseline) F6. ( There are a number of special work programs available to people with disabilities.) {(Is/Was) your/ (NAME)’s} job at [fill JOB NAME] part of a sheltered workshop program, transitional employment program, the Business Enterprise Program for the blind, or a supported employment program? PROBE: A sheltered workshop is a program that provides employment with subsidized wages (or special wages that would not be available in a regular job) for people with disabilities. PROBE: A transitional employment program allows workers with disabilities to work at reduced levels while they ease back into the workplace. PROBE: The Business Enterprise Program for the blind offers legally blind persons the opportunity to own their own businesses. PROBE: Supported employment programs provide job coaches or other on-the-job supports to help individuals with disabilities get and keep jobs.	YES 01 NO 00 DON’T KNOW d REFUSED r	YES01 NO 00 DON’T KNOW d REFUSED r
(F1 or F2=01) (Asked at Baseline) F7. In what month and year did {you/(NAME)} start working for [fill JOB NAME]? PROBE: Since [fill RA DATE]. RECORD MONTH AND YEAR.	| | | / | | | | | (F9) MONTH YEAR (1-12) (2007-2010) DON’T KNOW d REFUSED r	| | | / | | | | | (F9) MONTH YEAR (1-12) (2007-2010) DON’T KNOW d REFUSED r

	JOB 1	JOB 2
(F7=d or r) (Asked at Baseline-reference period modified) F8. Would {you/(NAME)} say {you/he/she} began working at [fill JOB NAME] within the past six months, between six months and one year ago, between a year and a year and a half ago, between a year and a half and two years ago, or more than two years ago? PROBE: {Your/His/Her} best estimate is fine.	WITHIN THE PAST 6 MONTHS 01 BETWEEN SIX MONTHS AND A YEAR AGO, , 02 BETWEEN A YEAR AND A YEAR AND A HALF AGO, 03 BETWEEN A YEAR AND A HALF AND TWO YEARS AGO, OR 04 MORE THAN 2 YEARS AGO 05 DON’T KNOW d REFUSED r	WITHIN THE PAST 6 MONTHS 01 BETWEEN SIX MONTHS AND A YEAR AGO, , 02 BETWEEN A YEAR AND A YEAR AND A HALF AGO, 03 BETWEEN A YEAR AND A HALF AND TWO YEARS AGO, OR 04 MORE THAN 2 YEARS AGO 05 DON’T KNOW d REFUSED r
(F1 or F2=01) F9. In what month and year did that job at [fill JOB NAME] end (or are you still working there)? RECORD MONTH AND YEAR	| | | / | | | | | (F10) MONTH YEAR (1-12) (2007-2010) STILL AT JOB (F10) n DON’T KNOW d REFUSED r	| | | / | | | | | (F10) MONTH YEAR (1-12) (2007-2010) STILL AT JOB (F10) n DON’T KNOW d REFUSED r
(F9=d or r) F9a. Would {you/(NAME)} say {your/his/her} job at [fill JOB NAME] ended within the past month, between one and 3 months ago, between 3 and 6 months ago, between 6 and 12 months ago, or more than 12 months ago? PROBE: {Your/His/Her} best estimate is fine.	WITHIN THE PAST MONTH 01 BETWEEN 1 AND 3 MONTHS AGO 02 BETWEEN 3 AND 6 MONTHS AGO 03 BETWEEN 6 AND 12 MONTHS AGO, OR 04 MORE THAN 12 MONTHS AGO 05 DON’T KNOW d REFUSED r	WITHIN THE PAST MONTH 01 BETWEEN 1 AND 3 MONTHS AGO 02 BETWEEN 3 AND 6 MONTHS AGO 03 BETWEEN 6 AND 12 MONTHS AGO, OR 04 MORE THAN 12 MONTHS AGO 05 DON’T KNOW d REFUSED r

	JOB 1	JOB 2
(F9 NE n, d, or r) F9b. What was the main reason your job at [fill JOB NAME] ended? Was it because …. CODE ONE RESPONSE	{you were /(NAME was )} laid off…...01 (your /(NAME’s)} health worsened…02 {you/(NAME)} needed to take care of a family member……………………….03 {you/(NAME)} retired…………………04 {you/(NAME)} returned to school…...05 {you/(NAME)} moved………………..06 {you/(NAME)} got a better job……….07 {you/(NAME)} were discharged or fired; or………………………………...08 some other reason……………………09	{you were /(NAME was )} laid off.... 01 {your /(NAME’s)} health worsened…02 {you/(NAME)} needed to take care of a family member…………………..03 {you/(NAME)} retired………………...04 {you/(NAME)} returned to school…..05 {you/(NAME)} moved………………..06 {you/(NAME)} got a better job………07 {you/(NAME)} were discharged or fired; or………………………………..08 some other reason…………………..09
(F1 or F2=01) (Asked at Baseline) F10. How many hours per week, {(do/did) you/(did/does) (NAME)} usually work at [fill JOB NAME]? PROBE: Include overtime if {you/he/she} usually work(ed) overtime.	| | | (F11) (1-80) VARIES v DON’T KNOW d REFUSED r	| | | (F11) (1-80) VARIES v DON’T KNOW d REFUSED r
(F8=v, d or r) F10a. Would {you/(NAME)} say {you/he/she} (work/works/worked) less than 10 hours per week, between 10 and 14 hours per week, between 15 and 19 hours per week, between 20 and 24 hours per week, between 25 and 29 hours per week, between 30 and 34 hours per week, between 35 and 39 hours per week, or 40 or more hours per week?	LESS THAN 10 HOURS PER WEEK 01 BETWEEN 10 AND 14 HOURS PER WEEK 02 BETWEEN 15 AND 19 HOURS PER WEEK 03 BETWEEN 20 AND 24 HOURS PER WEEK 04 BETWEEN 25 AND 29 HOURS PER WEEK 05 BETWEEN 30 AND 34 HOURS PER WEEK 06 BETWEEN 35 AND 39 HOURS PER WEEK 07 40 OR MORE HOURS PER WEEK 08 DON’T KNOW d REFUSED r	LESS THAN 10 HOURS PER WEEK 01 BETWEEN 10 AND 14 HOURS PER WEEK 02 BETWEEN 15 AND 19 HOURS PER WEEK 03 BETWEEN 20 AND 24 HOURS PER WEEK 04 BETWEEN 25 AND 29 HOURS PER WEEK 05 BETWEEN 30 AND 34 HOURS PER WEEK 06 BETWEEN 35 AND 39 HOURS PER WEEK 07 40 OR MORE HOURS PER WEEK 08 DON’T KNOW d REFUSED r
(F1 or F2=01) (Asked at Baseline) F11. What kind of work {do/did} {you/(NAME)} do at [fill JOB NAME]? PROBE: That is, what (is/was) {your/his/her} occupation?	RECORD VERBATIM DON’T KNOW d REFUSED r	RECORD VERBATIM DON’T KNOW d REFUSED r

	JOB 1	JOB 2
(F1 or F2=01) (Asked at Baseline) F12. What kind of business is this? PROBE 1: For what type of organization or industry {do/did} {you/(NAME)} work? For example, accounting firm, daycare center, educational facility, food services. PROBE 2: What {does/did} the company {you work(ed)/he/she works(ed)} for make, sell, or do?	RECORD VERBATIM DON’T KNOW d REFUSED r	RECORD VERBATIM DON’T KNOW……………………….d REFUSED……………………………r
(F1 or F2=01) F12a. (My next questions are about earnings.) What (is/was) {your/(NAME’s)} usual pay, including tips and commissions at [fill JOB NAME] before taxes or other deductions (are/were) taken? PROBE: {Your/His/Her} best estimate is fine. INTERVIEWER: ACCEPT MOST CONVENIENT PAY PERIOD. IF NECESSARY, CONFIRM PAY PERIOD.	$ | | | |,| | | |.| | | (F12c) (5.00 – 300,000.00) PER HOUR 1 PER WEEK 2 ONCE EVERY TWO WEEKS TWICE A MONTH 4 PER MONTH 5 PER YEAR 6 OTHER (SPECIFY) [specify] 7 NOT YET PAID n DON’T KNOW d REFUSED r	$ | | | |,| | | |.| | | (F12c) (5.00 – 300,000.00) PER HOUR 1 PER WEEK 2 ONCE EVERY TWO WEEKS 3 TWICE A MONTH 4 PER MONTH 5 PER YEAR 6 OTHER(SPECIFY) [specify] 7 DON’T KNOW……………………….d REFUSED…………………………….r

	JOB 1	JOB 2
(F12a=n, d, or r) F12b. I’ll read some ranges. Please try to estimate {your/(NAME’s)} annual pay at [fill JOB NAME]. Would {you/he/she} say {your/his/her} annual earnings (are/were) . . . PROBE: Does this include tips and commissions?	Less than $10,000, 01 $10,000 or more, but less than $20,000, 02 $20,000 or more but less than $30,000, 03 $30,000 or more but less than $40,000, 04 $40,000 or more but less than $50,000, 05 $50,000 or more but less than $75,000, 06 $75,000 or more but less than $100,000, or 07 more than $100,000? 08 DON’T KNOW (F21) d REFUSED (F21) r	Less than $10,000, 01 $10,000 or more, but less than $20,000, 02 $20,000 or more but less than $30,000, 03 $30,000 or more but less than $40,000, 04 $40,000 or more but less than $50,000, 05 $50,000 or more but less than $75,000, 06 $75,000 or more but less than $100,000, or 07 more than $100,000? 08 DON’T KNOW (F21) d REFUSED (F21) r
(F12a NE n, d, or r OR F12b NE d or r) F12c. Is that before or after taxes?	BEFORE TAXES 01 AFTER TAXES 02 DON’T KNOW d REFUSED r	BEFORE TAXES 01 AFTER TAXES 02 DON’T KNOW d REFUSED r

Table 1

Pretesting Scenarios

		Control		AB Basic		AB Plus
Ability to complete interview		Requires a proxy		Willing and able to complete interview		Willing and able to complete interview
Insurance status		Does not currently have insurance		No insurance besides AB		Private insurance through employer, optical coverage
		Had COBRA 6/2008-9/2008
Costs for medical care		$65,000		$11,000		$4,000
Usual source of medical care		Cancer clinic		Psychiatrist		Specialist
Other medical providers SM saw		Internist, Specialist		Specialist, Physical therapist		General practitioner, Psychologist
Overnight stay in hospital		Yes		Yes		No
Visited emergency room		Yes		No		No
Didn't see or postponed seeing doctor		Yes		Yes		No
Reasons for not seeing or postponing seeing doctor		Couldn't afford it, Doctor didn't accept insurance		No transportation
Referred to doctor/sent for tests, x-rays/recommended procedure, surgery		Referred to specialist, Sent for tests		Referred to specialist		Sent for tests, Recommended surgery
Went to visits, tests, x-rays, procedures, surgery		No		No		Yes
Reasons for not going to visits, tests, x-rays, procedures, surgery		Couldn't afford it		Too sick to go
Prescription medicines		Yes, regularly takes 4 prescriptions		Yes, regularly takes 3 prescriptions		No
Current employment status		Unemployed		Unemployed		Employed
# Jobs since RA	0		3		2
	Received job offer
	Turned down job offer because worried about losing SSA benefits and the job didn’t offer health insurance
Job 1			Elementary School 4/2008 – 6/2008, worked 30 hrs/week as teacher’s assistant, not self-employed, not a special work program, $10/hr before taxes		Bank, 5/2008 – present, works 20 hrs/week as a bank teller, not self-employed, not a special work program, $12.50/hr before taxes
Job 2			Ice Cream Shop, 8/2008- 12/2008, worked 20 hrs/week as ice cream scooper, not self-employed, part of a special work program, $6/hr before taxes		Bowling Alley, middle of last month-present, 10hrs/week as concession stand attendant, not self-employed, not a special work program, $8/hr before taxes
Job 3			Grocery Store, 1/2009- 3/2009, worked 20 hrs/week as cashier, not self-employed, part of special work program, $8/hr before taxes
Find out how working would affect benefits	Searched internet
Received help to understand how working would affect SSA benefits	No		Yes, from employment benefits counselor		Yes, from AB coach
Received rehabilitation and/or employment services from ….	State vocational rehab agency, a private rehabilitation provider		No		Ticket to work, Center for independent living
Received which employment services	Job assessment, Advice about making modifications to job				Help to find job, Advice about making modifications to job, Support services
Enrolled in school or classes	No
Adults in household	2		1		3
Children in household	0		0		1
# in household who work	1		1		2

cc: David Wittenburg, Barbara Kolln, Amy Raduzycki, Rebecca Weiner