Attachment H - Summary of Cognitive Interview of Questionnaire

Summary of Cognitive Tests of Veteran’s Supplement:

New Questions on Afghanistan and Iraq

Christine Rho

August 21, 2009

1. Purpose

The Current Population Survey (CPS) is a nationally-representative monthly sample survey of households that provides information on labor force activity in the United States. The CPS is the source of the monthly national unemployment rate, as well as other rich demographic details, including age, sex, race, Hispanic origin, educational attainment, marital status and family attributes, foreign-born status, veteran status, and other demographic characteristics.

The Veterans Supplement is now conducted every year and is asked at the end of the CPS. This supplement is sponsored by the Department of Veterans Affairs (VA) and the Veterans Employment and Training Services (VETS) at the Department of Labor.

In response to recent legislation, the July 2010 Veteran’s Supplement to the CPS will identify veterans who have served in Afghanistan and Iraq. Currently CPS only collects information about when a veteran served and not where. Two questions could be added to the July 2010 supplement in order to capture this information.

Before these new questions are to be fielded, they underwent cognitive testing by the Office of Survey Methods Research (OSMR). The primary goal of cognitive testing was to evaluate how well the new questions capture the information that is sought. Are the questions clear? Does it use any terminology that is unfamiliar or confusing to the respondents (who are veterans as well as civilians who respond as a proxy)? The new questions were tested for clarity, comprehension, length, and any sensitivity among questions.

2. Methodology

Participants. The OSMR conducted 13 cognitive interviews: 8 interviews with recent veterans and 5 interviews with spouses or other household members of recent veterans. In addition, there were two versions of the interviews which tested for different wording options for the question about Iraq. The first version contained the language “serve off the coast of Iraq” and the second version contained the language “serve in the waters around Iraq.” The participants were randomly assigned to either the Off the Coast version or the In the Waters version.

Each interview lasted approximately 20 minutes. Participants were interviewed in person in the OSMR lab. Participants were compensated $40 for their participation. One of the participants was a BLS employee and did not receive any compensation.

Procedure. Participants were given an Informed Consent to sign and apprised of their rights to privacy. They then were administered the paper version of an abbreviated CPS interview, which included questions on household roster, demographics, education, veteran status and labor force status. Then, they were administered an abbreviated version of the Veteran’s Supplement, which also included the two new questions on Afghanistan and Iraq (one of the two version). All responses were recorded on paper. Following the interview, the participants were asked general debriefing questions about the survey and retrospective probes on the two new questions of interest.

3. Results

A. Afghanistan Question. [(Did you/ (Did he/she)] serve in Afghanistan; or (did you)/(did he/she) fly missions over Afghanistan at anytime since October 2001?

This question was placed in the survey after the question about Vietnam/Cambodia/Laos (S1). For the participants in the cognitive test, this was the first question they received after the “Lead-In” statement, since they did not serve during the Vietnam War era.

Veterans

All eight veterans were able to adequately paraphrase the question in their own words, thus indicating that they comprehended the question. They were further probed about what “serve in Afghanistan” meant to them in this question. Most of the veterans said that they thought it meant to be deployed to Afghanistan and being on active duty. One veteran mentioned that it meant that you received hostile fire pay in Afghanistan. Another veteran noted that “serve in Afghanistan” covered all active duty and that it was not necessary to also ask about flying missions over Afghanistan since “to serve” is inclusive of flying missions.

Additionally, during the debriefing the veterans were asked about an alternative wording of the question, which included the phrase “on active duty in Afghanistan” in lieu of “serve in Afghanistan.” Among the veterans, four participants preferred the phrase “serve in Afghanistan.” Some of the reasons they cited for their preference were that it was more inclusive of reservists who were “called to serve.”

Three participants preferred the phrase “on active duty in Afghanistan” because they thought that “active duty” was more specific. For example, one participant said that “serve can be the whole gambit. You can be a civilian contractor and serve your country. Active duty is clearer that it’s military.”

The remaining two participants did not have any preference and thought that “serve” and “active duty” were interchangeable terms.

Non-Veterans/Proxy

Three out of five proxy participants were able to paraphrase the question without any problems. One proxy participant showed some confusion and thought that the question was asking if her husband (a "Gulf war era 2" veteran) went to Afghanistan in October 2001. Another proxy participant was unable to respond to the question because she was unsure if being in Afghanistan as a military contractor was considered as “serving in Afghanistan.”

When probed further about what she thought “serve in Afghanistan” meant to her in the question, she thought that it meant serving in a military capacity and that it could include a military contractor who has retired from the armed forces.

When given the alternate wording, which included the phrase “active duty in Afghanistan,” the proxies had a slight preference for this phrasing. They thought it was clearer that the question was only asking about military service. In contrast, “serve in Afghanistan” could encompass other things, such as military contractors or other civilian work abroad.

Summary Points

The veterans did not have any problems with the Afghanistan question and they tended to prefer the phrase “serve in Afghanistan” over “active duty in Afghanistan” because it was more inclusive of reservists who may have been called to serve. In contrast, some of the proxies had some difficulties with the question and had a slight preference for “active duty.” Overall, the “serve” language worked better and therefore, that is the language recommended.

During testing, some participants tended to respond before the interviewer finished reading the question in its entirety. In these cases, the question was usually interrupted after the phrase, “Did [you/name] serve in Afghanistan…” and thus, some participants may not have heard the part with the date, “…since October 2001.” If the date is critical to the question, it is recommended that the phrase “since October 2001” be moved to the beginning of the question, as follows:

At anytime since October 2001, [(did you)/ (did he/she)] serve in Afghanistan; or (did you)/ (did he/she) fly missions over Afghanistan?

2. Iraq Question – Off the Coast Version

[(Did you)/ (Did he/she)] serve in Iraq; off the coast of Iraq; or (did you)/ (did he/she) fly missions over Iraq at anytime since March 2003?

Iraq Question - In the Waters Version

[(Did you)/ (Did he/she)] serve in Iraq; in the waters around Iraq; or (did you)/ (did he/she) fly missions over Iraq at anytime since March 2003?

Approximately half of the participants received the Off the Coast Version of the Iraq question in the survey. During the debriefing, they were also given the alternate version (In the Waters Version) of the question and probed about the different connotations.

Veterans

Among the veterans, six out of eight participants preferred the Off the Coast Version. They thought the language was clearer and that it made them think of the serving in the navy, marines or coast guard. One participant said, “I prefer ‘off the coast’ because it sounds odd to say ‘in the waters.’ It dumbs it down too much.” Another participant said that “in the waters” sounded more vague and that he thought of parts of Africa which is around the waters of Iraq.

The two veterans who preferred the In the Waters Version thought that this language was “plainer.” One participant said, “You’re in the waters, it’s not any plainer than that.” One veteran who had served in Kuwait said that she initially thought of Kuwait as an area that is off the coast of Iraq. However, she correctly answered the question in responding “no” that she had not served in Iraq.

Non-Veterans/Proxy

Among the proxies, there was no clear preference for one version or the other. Two preferred Off the Coast Version and two preferred In the Waters Version and one said she had no preference for either version. However, one of the proxies who thought that the Off Coast Version was clearer said that “in the waters sounds like something someone who doesn’t speak English as their first language would say.”

Summary Points

Overall, the Off the Coast Version tested better than the In the Waters Version. Participants thought that “off the coast” was a more familiar phrasing. Most did not find the question confusing and they did not think of other land masses, with the exception of the one participant who thought of Kuwait. Another consideration was to put the phrase “off the coast” after “fly missions over Iraq” since flying missions may be more common. However, it is recommended to keep “off the coast” phrase in the middle, since people tend to remember the first and last parts of a sentence better than the middle of the sentence (known as the Primacy and Recency effects).

Appendix A: Retrospective Probes (Off Coast Version)

DEBRIEFING PROBES:

1. Overall, what is your impression of the survey? Did you think it was worded clearly for someone like you to answer?

2. What did you think about the flow of the survey?

3. Were there any questions in this survey that you think some people might find sensitive?

4. Were there any questions in this survey that you think some people might find difficult?

5. Is there anything you would like to see changed or improved in the survey?

SPECIFIC PROBES ABOUT THE NEW VETERAN’S SUPPLEMENT QUESTION

Let’s talk about some of the Armed Forces Questions. I’d like to review some of these questions and get your thoughts on them. Earlier I asked you the following question:

[For Afghanistan Veterans]

S1ai. [(Did you/ (Did he/she)] serve in Afghanistan; or (did you)/(did he/she) fly missions over Afghanistan at anytime since October 2001?

<1> YES

<2> NO

<D> Don’t know

<R> Refused

Probes S1ai:

• Can you repeat the question I just asked in your own words?

• What does “serve in Afghanistan” mean to you in this question?

• Can you think of other examples of serving in Afghanistan that is not covered by this question?

• I’m going to read you a slightly different wording of this question. Let me know if this version works better for you. [(Were you)/(Was he/she)] on active duty in Afghanistan; or (did you)/(did he/she) fly missions over Afghanistan at anytime since October 2001?

[For Iraq Veterans]

S1aii. [(Did you/ (Did he/she)] serve in Iraq; off the coast of Iraq; or (did you)/(did he/she) fly missions over Iraq at anytime since March 2003?

 

<1> YES

<2> NO

<D> Don’t know

<R> Refused

Probes S1aii:

• Can you repeat the question I just asked in your own words?

• What does “off the coast of Iraq” mean to you in this question? What areas did you think of?

  • How many miles?

• What does “serve in Iraq” mean to you in this question? What does it encompass?

• Can you think of other examples of serving in Iraq that is not covered by this question?

• I’m going to read you a slightly different wording of this question. Let me know if this version works better for you.

[(Did you)/(Did he/she)] serve in Iraq; in the waters around Iraq; or (did you)/(did he/she) fly missions over Iraq at anytime since March 2003?

END OF TESTING.

1. Thank the participant for their time.

2. Pay participant and have them sign the voucher.

Appendix B: Retrospective Probes (In Water Version)

DEBRIEFING PROBES:

6. Overall, what is your impression of the survey? Did you think it was worded clearly for someone like you to answer?

7. What did you think about the flow of the survey?

8. Were there any questions in this survey that you think some people might find sensitive?

9. Were there any questions in this survey that you think some people might find difficult?

10. Is there anything you would like to see changed or improved in the survey?

SPECIFIC PROBES ABOUT THE NEW VETERAN’S SUPPLEMENT QUESTION

Let’s talk about some of the Armed Forces Questions. I’d like to review some of these questions and get your thoughts on them. Earlier I asked you the following question:

[For Afghanistan Veterans]

S1ai. [(Did you/ (Did he/she)] serve in Afghanistan; or (did you)/(did he/she) fly missions over Afghanistan at anytime since October 2001?

<1> YES

<2> NO

<D> Don’t know

<R> Refused

Probes S1ai:

• Can you repeat the question I just asked in your own words?

• What does “serve in Afghanistan” mean to you in this question?

• Can you think of other examples of serving in Afghanistan that is not covered by this question?

• I’m going to read you a slightly different wording of this question. Let me know if this version works better for you. [(Were you)/(Was he/she)] on active duty in Afghanistan; or (did you)/(did he/she) fly missions over Afghanistan at anytime since October 2001?

[For Iraq Veterans]

S1aii. [(Did you/ (Did he/she)] serve in Iraq; in the waters around Iraq; or (did you)/(did he/she) fly missions over Iraq at anytime since March 2003?

 

<1> YES

<2> NO

<D> Don’t know

<R> Refused

Probes S1aii:

• Can you repeat the question I just asked in your own words?

• What does “in the waters around Iraq” mean to you in this question? What areas did you think of?

• What does “serve in Iraq” mean to you in this question? What does it encompass?

• Can you think of other examples of being on active duty in Iraq that is not covered by this question?

• I’m going to read you a slightly different wording of this question. Let me know if this version works better for you.

[(Did you)/(Did he/she)] serve in Iraq; off the coast of Iraq; or (did you)/(did he/she) fly missions over Iraq at anytime since March 2003?

• How many miles is off the coast?

END OF TESTING.

3. Thank the participant for their time.

4. Pay participant and have them sign the voucher.