Attachment G
August 24, 2011
MEMORANDUM FOR CPS Disability Supplement Team
From: Brandon Kopp
Office of Survey Methods Research
Subject: Cognitive Interview Retest of Select Questions
TOP LINE RESULTS
Acceptance of proposed changes is recommended for Questions 6, 9, 10, 11, 16, 19, & 20
Rejection of proposed change to response options is recommended for Questions 6
Rejection of wording change is recommended for Question 8 (with additional changes)
Minor wording change is proposed for Questions 4
COGNITIVE TESTING REPORT
Following the initial cognitive testing report for the Current Population Survey (CPS) Disability Supplement questions, issued on July 8, 2011, the CPS Disability Supplement Team accepted several recommendations offered in that report and made several other revisions based on the data collected in the cognitive interviews. Table 1 shows a list of changes made to the original set of questions:
Table 1: List of changes made to original set of Disability Supplement questions.
Q |
Change Description |
Testing |
Sample |
|
4 |
|
Cognitive Interview |
Disabled |
Not in labor force or unemployed |
6 |
|
Cognitive interview |
Disabled |
All labor force categories |
8 |
|
Cognitive interview |
Disabled |
Employed & requested change in workplace |
9 |
|
Cognitive interview |
Disabled |
Employed & requested change in workplace |
10 |
|
Expert review* |
n/a |
n/a |
11 |
|
Expert review* |
n/a |
n/a |
16 |
|
Cognitive interview |
Disabled |
Employed & work at home |
19 |
|
Expert review* |
n/a |
n/a |
20 |
|
Cognitive interview |
Disabled |
All labor force categories Received financial assistance |
An additional round of testing was proposed to validate these changes. Jennifer Edgar revised and resubmitted the Office of Management and Budget (OMB) protocol and Brandon Kopp conducted 10 cognitive interviews once the protocol was approved. The following report is meant to supplement the earlier report and will not outline the rationale and design in detail as that report did. Please refer to the July 8th CPS Disability Supplement Cognitive Test Report for more details.
PARTICIPANTS
Because time to recruit and interview participants was limited to 7 working days (11 – 19 August, 2011), participants who are employed and have a disability were selectively recruited. The majority of the question changes were made to questions that apply to this group, thus recruiting from this population offered the most efficient means of meeting recruiting goals and testing deadlines. Table 2 outlines the labor force and disability status of participants in this study and for household members they offered proxy responses for. Table 3 presents the disabilities participants and their household members have.
Table 2: Number of participants by disability and labor force status and relationship to participant.1
|
|
Self |
Proxy |
Total |
Disabled |
Employed |
8 |
0 |
8 |
Unemployed |
0 |
1 |
1 |
|
Not in the Labor Force |
1 |
2 |
3 |
|
Non-Disabled |
Employed |
1 |
3 |
4 |
Unemployed |
0 |
1 |
1 |
|
Not in the Labor Force |
0 |
0 |
0 |
|
Total |
10 |
7 |
|
|
Table 3: Number of participants by disability.2
Disability |
Frequency |
Hearing |
1 |
Seeing |
4 |
Remembering, Concentrating |
6 |
Walking or Climbing Stairs |
5 |
Dressing or Bathing |
0 |
Doing Errands |
3 |
DETAILED FINDINGS FROM COGNITIVE INTERVIEWS
Question Number |
Question |
Response Options |
Frequency |
4 |
The purpose of this next question is to identify barriers to employment faced by persons with difficulties. What would you say the main barriers to employment are for (you/Name)? |
1.
Education or training 6.
Employer and coworker attitudes 12. Own Disability |
1 |
-- |
|||
-- |
|||
1 |
|||
-- |
|||
-- |
|||
-- |
|||
-- |
|||
-- |
|||
-- |
|||
-- |
|||
3 |
|||
|
|
Total |
5 |
Change from Round 1: The response option “Discrimination” was changed to “Employer and coworker attitudes.” Also, this question was changed to a mark-all-that-apply format.
Debriefing Responses: Four participants answered this question, and three of them answered on behalf of other household members. Only one participant experienced difficulty with this question, asking “What does that mean?” When the interviewer explained it to him, he was able to give an answer (he said his wife had back problems that interfered with her ability to work; coded as “Own Disability”).
None of the participants gave a response that would indicate “Employer or coworker attitudes” was a barrier to employment for them or their household members. When asked if they consider this to be a barrier to employment for others, all four said yes, and all four said they consider “Employer and coworker attitudes” distinct from “Discrimination.” Participants indicated that “discrimination” was something more malicious and willful than “employer and coworker attitudes.” This is not a serious issue since the answer categories are not read. Interviewers could be trained to classify both types of responses as “Employer and coworker attitudes.” Alternatively, “Discrimination” could be added as a separate category though this latter strategy would likely result in inconsistent classification of responses.
One of the four participants offered two answers to this question, saying another household member would lose government assistance if she started working and that her health issues kept her from working.
Question Performance in Round 2: The question performed similarly to Round 1. Once again, the majority of responses mentioned the participants’ or household members’ disability as the main barrier to employment.
Recommendation for Change: If you wish to keep this as a mark-all-that-apply question, I recommend removing the word “main” from the question. That better indicates that multiple answers are acceptable for this question. The question would then read: What would you say the barriers to employment are for (you/Name)? |
6 |
The purpose of this next question is to find out if (you have/Name has) taken advantage of any of the following sources that help people prepare for work or advance on the job. In the past 5 years, [(Have you)/(Has Name)] received assistance from: |
1. Vocational Rehabilitation Programs 2. One Stop Career Centers 3. Ticket to Work Program 4. Assistive Technology Act Program 5. Center for Independent Living for Individuals with Disabilities 6. Client Assistance Program 7. Other |
|
34 |
|||
1 |
|||
1 |
|||
4 |
|||
3 |
|||
2 |
|||
|
|
Total |
20 |
Change from Round 1: A five year reference period was added to this question and a response option for “Haven’t heard of this program” was added.
Debriefing Responses: Nine participants answered this question and nearly all of them had participated in at least one program. Only one participant had difficulty with this question and that difficulty wasn’t apparent until the debriefing. This participant answered “Yes” when asked whether she’d received assistance from a Vocational Rehabilition Program and a One Stop Career Center within the last 5 years, but, during the debriefing, said that she had not received what she would classify as “assistance” from those organizations. This participant had problems with several questions in the survey and may be an outlier, and so I would not recommend any changes based on her problem here.
Participants seemed to understand and give answers in line with the 5 year reference period. During the debriefing I asked participants when was the last time they received assistance from these organizations and answers ranged from “4 years ago” to “this week” but did not fall outside the reference period. Two participants mentioned particpating in programs more than 5 years ago but didn’t say “Yes” during the interview because they were outside the time period.
Performance in Round 2: The addition of a reference period helped focus participants on the recent past, which led to less burden for participants and more relevant answers for data users. I recommend keeping this version of the question.
Recommendation for Change: I recommend removing “Haven’t heard of this program” from the list of response options. Only two participants said this, or something similar, during the interview, though, almost all said there was at least one program they hadn’t heard of (the “Assistive Technology Act Program” was, once again, the least familiar program). Participants more often than not say “No” they have not received assistance rather than indicating they haven’t heard of the program. |
8 |
Have (you/Name) ever requested any change in your current workplace to help you do your job better? |
1. Yes 2. No 3. Don't Know 4. Refused |
3 |
9 |
|||
-- |
|||
-- |
|||
|
|
Total |
12 |
Change from Round 1: The phrase “…, for example in equipment or work processes,…” was removed from the question.
Debriefing Responses: While it’s difficult to tell from such a small sample, the removal of the example phrase appears to have decreased the rate of responding “Yes” to this question. In the first round of testing, participants responded “Yes” for 6 out of 13 eligible people (self and proxies), or 46%. In the second round of testing 3 participants said “Yes” to this question, or 25%. All three participants in the second round of testing gave affirmative answers for themselves. None responded “Yes” for another household member.
During the debriefing, participants were shown the list of possible changes (from Question 9) and asked if this prompted any memories for changes they had requested. After seeing the list, four participants who had originally responded “No” to Question 8 recalled changes they had requested (changes indicated were 1, 4, 5, & 7 from Question 9). One participant who had responded “Yes” recalled two additional changes she had requested (additional changes indicated were 2 & 6 from Question 9).
As you can see from the distribution of responses for Question 9, the types of requests that spontaneously came to mind shifted from “New or modified equipment” in Round 1 (which received 7 out of 8 responses, or 88%) to “Changes in work tasks…” in Round 2 (which received 4 out of 5 responses, or 80%).
Performance in Round 2: The examples given in Round 1(equipment or work processes) appear to have prompted participants’ memory, leading to more affirmative responses. However, the examples also directed participants’ memory search to a specific type of change (New or modified equipment). Removing the example phrase for Round 2 decreased the number of true positive responses (i.e., affirmative responses when the participant had requested a change) but led to a less biased memory search.
Recommendation for Change: I recommend re-inserting the example phrase and including more examples. For example, the question could read as follows: Have (you/Name) ever requested any change in your current workplace, for example in equipment, work tasks, schedule, or policies, to help you do your job better? |
9 |
What change did (you/Name) request? |
1.
New or modified equipment
8.
Training |
-- |
1 |
|||
-- |
|||
4 |
|||
-- |
|||
-- |
|||
-- |
|||
-- |
|||
-- |
|||
-- |
|||
|
|||
|
|
Total |
5 |
Change from Round 1: The addition of “Training” as a potential response option.
Debriefing Responses: Two of the three participants who said they had requested a change in the workplace indicated that they had requested multiple changes.
Interviewer Observations: One participant said she had requested “assistance with carrying things.” I found this difficult to classify during the interview. On the spot, I classified it as a “Physical change to the workplace” but then thought it may be better placed in “Changes in work tasks.”
During the debriefing, the woman who indicated that she requested assistance carrying things mentioned accommodations her coworkers had requested such as personal aides and translators. These too were difficult to classify without some thought.
Performance in Round 2: I recommend leaving this question as a mark-all-that-apply. I also recommend keeping the current set of response options and providing interviewer training or guidance for difficult to classify responses to ensure they classify responses in a systematic way.
10 |
Was the change granted? |
1. Yes 2. No 3. Partially 4. Don't Know 5. Refused |
5 |
-- |
|||
-- |
|||
-- |
|||
-- |
|||
|
|
Total |
5 |
Change from Round 1: The order of the “Partially” and “No” response options was reversed.
Interviewer Observations: All of the participants in this round of testing said their requests had been fulfilled, so I did not have the opportunity to test whether the change in the order of responses was helpful.
Performance in Round 2: I recommend keeping this ordering of response options.
11 |
How [(do you)/(does Name)] typically commute to work? |
1. Bus 2. Specialized bus or van service 3. Train/subway 4. Taxi 5. Own vehicle 6. Passenger in a family member's car 7. Passenger in friend's car 8. Passenger in carpool 9. Driver in carpool 10. Motorcycle 11. Bicycle 12. Walk 13. Other 14. Work from home 15. Don't Know 16. Refused |
7 |
-- |
|||
8 |
|||
-- |
|||
2 |
|||
-- |
|||
-- |
|||
1 |
|||
-- |
|||
-- |
|||
-- |
|||
-- |
|||
-- |
|||
-- |
|||
-- |
|||
-- |
|||
|
|
Total |
18 |
Change from Round 1: This question was changed to a mark-all-that-apply format.
Interviewer Observations: Nine out of 10 participants answered this question. Three said simply “public transportation” as an initial answer. After probing, I discovered that they meant they take both the bus and the Metro (sometimes in the same trip, sometimes alternatively) to get to work. Five participants gave more than one answer to this question. One woman said that she takes either the Metro, or a “company car” but didn’t elaborate on the latter answer. I coded this response as “passenger in a carpool.”
Performance in Round 2: From my experience talking to trained CPS interviewers, they understand the need for probing to get specific answers. I think they would probe as needed for this question without additional training or prompting. If this is a concern, emphasis could be placed on probing for this question.
16 |
What is the main reason why (you work/Name works) at home? |
1. Less commuting 2. Reduce expenses for transportation, food, clothing, etc. 3. Coordinate work schedule with work and family needs 4. More control over own life 5. Illness, disability, health reasons 6. Mandated by employer to reduce employer costs 7. Mandated by employer to meet local transportation management and pollution abatement requirement 8. More productive 9. Self employed/business at home 10. Other 11. Don't Know 12. Refused |
1 |
-- |
|||
-- |
|||
-- |
|||
1 |
|||
-- |
|||
-- |
|||
1 |
|||
-- |
|||
2 |
|||
-- |
|||
-- |
|||
|
|
Total |
5 |
Changes from Round 1: The addition of “Self employed/business at home” as a response option
Interviewer Observations: One respondent said that he and his girlfriend both work from home occasionally because they don’t get everything done during the day. I coded this other “Other.”
Performance in Round 2: I recommend keeping the current set of response options.
19 |
There
are a variety of programs designed to provide financial assistance
to people. In the PAST YEAR did (you/Name) receive assistance
from any of the following programs? |
1. Workers Compensation 2. Social Security Disability Income 3. Supplemental Security Income 4. Veterans Disability compensation 5. Disability Insurance Payments 6. Other disability payments 7. Medicaid 8. Medicare 9. Other |
15 |
46 |
|||
-- |
|||
-- |
|||
-- |
|||
1 |
|||
67 |
|||
2 |
|||
-- |
|||
|
|
Total |
13 |
Change from Round 1: Emphasis of the reference period by capitalizing “PAST YEAR.”
Debriefing Responses: Two participants had difficulty with this question. One participant said “Yes” when asked if he had received assistance from Social Security Disability Income (SSDI) but, during the debriefing, said that he does not receive SSDI, rather he receives regular Social Security. This participant was of retirement age. A second participant said “No” she did not receive assistance from Medicaid but, during the debriefing, changed her answer to “Yes.” When asked why she changed her answer the participant thought the question referred to only direct financial assistance.
As with Round 1, participants did not have any problems with the reference period. When asked during the debriefing when they had last received assistance from these organizations, all participants gave a time period within the past year.
Interviewer Observations: The participant who responded “Yes” to Other disability payments said she had received “Internal Disability Assistance or IDA.”
Performance in Round 2: I recommend keeping the current version of the question. I am hesitant to recommend changes based on the difficulties the two participants had with this question. The participant who failed to report Medicaid also had difficulty with Question 6 so she may be somewhat of an outlier.
20 |
Some financial assistance programs include limitations on the amount of work you can do. Did (this program/any of these programs) cause you to work less than you would otherwise? |
1. Yes 2. No 3. Don't Know 4. Refused |
48 |
8 |
|||
1 |
|||
-- |
|||
|
|
Total |
13 |
Change from Round 1: An extensive wording change was made to this question.
Debriefing Responses: Participants seemed to understand the purpose of this question. Seven of the ten participants received this question and six were able to adequately paraphrase the question, saying things like “will my husband’s SSDI be discontinued if he works full time,” “Does Medicaid allow me to work more than a certain number of hours,” and “Does how much you work affect how much assistance you receive.” One participant was not able to provide any paraphrase for the question. Participants who responded “No” to this question said they do not consider this when making decisions regarding their job.9 One of the participants who responded “Yes” to this question, for himself, said that he would work full time if it weren’t for the income limitations of SSDI. Two participants, who responded “Yes” to this question on behalf of other household members, said that their household member currently does not work but would work at least part time if it weren’t for the limitations of the program.
None of the participants who were asked this question said they found it personal or sensitive. This is not unusual for cognitive interviews where people who are willing to participate in these types of studies are often also willing to disclose information. Also, they receive payment for their participation and may be hesitant to say they are uncomfortable answering questions. Because of this, the additional question “Do you think others would find this question sensitive?” was asked. Three participants said they believed others would find this question sensitive.
Performance in Round 2: This question appears to have performed better in Round 2 than in Round 1. I recommend keeping the question as currently worded.
1 One participant reported having bipolar disorder and receiving assistance though did not respond affirmatively to any of the disability status questions. For the purpose of these interviews she was asked the questions for those without a disability.
2 This table includes self and proxy responses. Also, some participants reported more than one disability.
3 This column indicates the number of people who said “Yes” they had received assistance from this type of organization.
4 These numbers include a participant who said “Yes” during the interview but changed her answer to “No” during the debriefing.
5 This column indicates the number of people who said “Yes” they had received assistance from this type of organization.
6 Number does include one participant who, during the debriefing, changed his answer to “No” because he received regular Social Security and not SSDI.
7 Number does not include one participant who, during the debriefing, changed her answer to “Yes” because she thought this question was concerned solely with direct financial assistance. This is the same participant who changed her answer to Question 6.
8 This column shows the number of people who used the provided terms to describe any of the programs.
9 Nine out of ten of the participants in this round of testing were employed. Answers to this question may be different for those who are Not In the Labor Force.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | kopp_b |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |