Attachment H - Summary of Cognitive Interview of Questionnaire

Attachment H

Cognitive Tests of Veterans Supplement Questions

Final Report

Christine Rho

January 15, 2009

1. Purpose

The Current Population Survey (CPS) is a nationally-representative monthly sample survey of households that provides information on labor force activity in the United States. The CPS is the source of the monthly national unemployment rate, as well as rich demographic details, including age, sex, race, Hispanic origin, educational attainment, marital status and family attributes, foreign-born status, veteran status, and other demographic characteristics.

The Veterans Supplement is conducted every two years and is asked at the end of the CPS. August 2007 was the last time this supplement was fielded. This supplement is sponsored by the Department of Veterans Affairs (VA) and the Veterans Employment and Training Service (VETS) at the Department of Labor.

In 2007, the Veterans Supplement was updated to reflect the growing interest in learning about the labor market situation of veterans who are or were in the Reserve and National Guard. Of particular interest were those Reserve or National Guard veterans who served in combat during the Gulf War era (August 1990 to present). Five new questions were tested by OSMR and added to the supplement to capture this information.

The next supplement will be on August 2009. The Division of Labor Force Statistics (CLFS) and the VA identified three some questions that were potentially problematic and requested that these questions be evaluated by OSMR for the 2009 supplement. These questions were:

• (S4 – Presence of disability) “Has the Department of Veterans Affairs (VA) or Department of Defense determined that you have a service-connected disability; that is, a health condition or impairment caused or made worse by any of your military service?”

• (S10 – Year of separation) “In what year were you last separated from active duty in the Armed Forces?”

• (S11a – Combat) “”Did you EVER serve in a combat or war zone?” READ IF NECESSARY: Persons serving in a combat or war zone usually receive combat zone tax exclusion, Imminent Danger Pay, or Hostile Fire Pay.

Several wording changes to the three questions were proposed and tested through two rounds of cognitive interviews with veterans and proxies. The main purpose of the cognitive interviews was to test whether the new proposed wording changes obtained the information, improved comprehension, and resulted in more accurate responses. The cognitive interviews are summarized in the following sections.

2. Procedure

Two rounds of interviews were conducted by an OSMR researcher with one participant at a time. Each session lasted approximately 15-30 minutes. The interviews in the first round of testing were conducted in-person (N=8), in a private office at the office of Veterans Affairs in Washington, DC. The OSMR researcher administered the interviews and recorded all responses on paper. The sessions were also audio-taped to verify and supplement the notes kept by the researcher. Participants were given an informed consent form. The protocol of the interviews can be found in Attachment A. The participants were instructed to answer each question as if they were being interviewed for the survey. Then the participants were asked retrospective probes about question meaning and they were asked to comment about some of the major concepts studied in the survey. Participants were federal employees and did not receive any payment for their participation.

In the second round of testing, the interviews were conducted on the telephone (N=10) and were not audio-taped. One of the primary reasons why interviews were conducted on the telephone was to allow us to include proxy respondents (i.e., participants who have a household member that is a veteran) who could not be paid due to budgetary reasons. Another reason for the telephone mode was to reduce any potential privacy concerns the veterans had about being interviewed on location at the VA office.

3. Participants and Burden Hours

Participants were recruited from the VA office, with special care to recruit Gulf War veterans, female veterans, veterans who have been deployed to Iraq or Afghanistan, or veterans who have been called up from the Reserve or the National Guard. No burden hours were used. All participants (including proxies) were federal employees (from the VA, BLS and Federal Reserve). Participants did not receive any compensation.

Eight cognitive interviews were conducted in Round 1 with veterans. 10 interviews were conducted in Round 2: 6 interviews with veterans and 4 interviews with spouses or other household members of veterans (also called proxies). Of the proxy interviews, only one of them reported about a veteran that served during the Gulf War.

The characteristics of the participants are shown below in Table 1.

Table 1. Participant Characteristics

	Participant No.	Self or Proxy	Gulf War	Gender1
Round 1	1	Self	Yes	Female
	2	Self	Yes	Female
	3	Self	Yes	Male
	4	Self	Yes	Male
	5	Self	Yes	Female
	6	Self	Yes	Female
	7	Self	Yes	Male
	8	Self	Yes	Male
Round 2	1	Self	Yes	Female
	2	Self	Yes	Female
	3	Self	Yes	Male
	4	Self	Yes	Male
	5	Self	Yes	Male
	6	Self	Yes	Male
	7	Proxy	No	Female
	8	Proxy	Yes	Female
	9	Proxy	No	Female
	10	Proxy	No	Female

4. Findings

Round 1: General Findings

After the participants were administered the Veterans Supplement questions, they were asked some general debriefing questions, such as their overall impression of the survey and whether they found any of the questions to be difficult and/or sensitive. Overall, most of the participants thought the survey was fine and did not have any major problems with it. However, a couple issues came up during this debriefing. One reservist found the concept of “active duty” a little confusing because she felt there are different kinds of reserves: active reserve and individual ready-serve reserve. A reservist in the “active reserve” serves in monthly drills and assemblies and does 14 days of active duty for training per year. She felt that the concept of active reserves may be confused with active duty.

Most of the issues centered on the service-connected disability questions (section S4-S7). One participant thought that the question about receiving monthly check for a service-connected disability (question S6) may be confusing for retired veterans with 20 years of service or more. Because these veterans also receive benefits checks and it could be confused with the disability benefits checks.

One participant thought question S7 (“Did your service-connected disability ever present you from getting or holding a job in the past?”) was not clear. He thought it could be interpreted in two ways: 1) having to do with the ability to work or 2) having to do with the ability to obtain work.

Two out of eight participants thought that the questions about disability were sensitive and private and had issues with being asked about their disability. One participant refused to provide a service-connected disability rating (question S5) and only answered that it was above a certain percent. One of the issues that these participants had with the disability questions was that they worried about how the information might be used and what impact it might have on their disability benefits. They noted that it is especially sensitive because the VA is the survey’s sponsor and they are also the agency that gives the veterans the disability payments. On the other hand, a different participant specially noted that the disability questions are not sensitive at all, because he felt that veterans answer questions about their disability ratings all the time.

These are issues that warrant further evaluation in future studies. They were not specifically the focus of these set of evaluation, but they are noted here for future research.

Round 1: Specific Findings for Revised Questions

The focus of this round of cognitive interviews was to test the wording changes for questions S4 (Presence of disability), S10 (Year of separation) and S11a (Combat). These questions are discussed separately below.

S4 (Presence of disability): “Has the Department of Veterans Affairs (VA) or Department of Defense determined that you have a service-connected disability; that is, a health condition or impairment caused or made worse by any of your military service?”

This question immediately follows the set of questions about service in the Reserves or the National Guard. Given this context, the survey sponsors were concerned that respondents may not realize that the question refers to all of their military service and not just their time in the reserves. In order to remind respondents that this question refers to all military service, a phrase was added to the beginning of the question, as follows (new phrase is highlighted):

“Thinking about all of your military service, has the Department of Veterans Affairs (VA) or Department of Defense determined that you have a service-connected disability; that is, a health condition or impairment caused or made worse by any of your military service?”

This new wording change was tested in Round 1 and participants were probed about it retrospectively. In general, the participants did not have any difficulty with this question. They were able to paraphrase the question sufficiently (showing that they comprehended the question well). They also tended to think of all of their active duty service and also considered their time in the reserves (if applicable) when answering this question. However, one participant noted that the reserve time does not count towards a service-connected disability. In conclusion, this new wording tested well.

Another interesting finding was that one participant mentioned that the VA does not have a formal definition of a combat or war zone and that the VA does not recognize imminent pay. He also noted that he liked how it was defined in the survey because it made it very clear.

However, one caveat is that most of the participants (7 out of 8) had one continuous service in the armed forces and thus, it is easier for them to consider all of their military service, since they had no breaks in service. Only one participant had a break in service and he also did not have any difficulty with this question.

(S10 – Year of separation) “In what year were you last separated from active duty in the Armed Forces?”

One of the problems that this question encountered in the field was that the respondents did not seem to be giving their last year. Specifically, the year that the veterans provided as their last year did not correspond to their most recent reported period of service, so there was reason to suspect that they were correctly understanding this question. For example, in August 2007, there were 465,000 veterans who reported serving in “other service periods” only and who were discharged between 1990 and 1999. The Gulf War Era began in August 1990; thus, most of these veterans should have been classified in the gulf war era. The current question may not be as clear as possible, as the key words “last” and “separated” are buried in the middle of the question. In order to address this issue, two revised versions of this question were tested. They were as follows:

Version 1: “What was your LAST year on active duty in the Armed Forces?”

Version 2: “In what year were you separated from active duty in the Armed Forces for the LAST time?”

From the cognitive interviews, it seemed that the participant’s comprehension of this question was fine. For example, the year that they provided as their last year in active duty corresponded to their responses to question AF2_N (“When did you serve on active duty in the U.S. Armed Forces?”), which was asked earlier (as part of the CPS demographics question set).

When asked specifically which version of the question wording they preferred (i.e., thought was more clear), the results were mixed. Half of the participants (4 out of 8) preferred version 1 because they thought it was shorter and more concise. Three out of 8 participants preferred version 2, because they thought that “separated” was a good, explicit term. One participant also noted that version 2 (with the word separated) might work better for veterans who had “spurts” or breaks in service. However, they also noted that this version sounded grammatically awkward. Given the mixed findings, recommendation was made to revise the question to include the word “separated” but to make the question short with fewer clauses in the question’s syntax. New wordings were proposed for the second round of testing. This will be discussed in more detail in Round 2: Findings section of the report.

(S11a – Combat) “Did you EVER serve in a combat or war zone?” READ IF NECESSARY: Persons serving in a combat or war zone usually receive combat zone tax exclusion, Imminent Danger Pay, or Hostile Fire Pay.

The survey sponsor (VA) requested that this question be revised because they had concerns that there were many false negatives to this question. In other words, they were concerned that veterans who had in fact served in a combat zone would answer “no” to this question because they did not engage in combat, per se. In particular, they were concerned that female veterans would give more false negatives because they are prohibited by law from combat duty, even though they can serve in a combat or war zone (and meet the definition of this question). To address these concerns, the definition was included in the question, instead of being a “READ IF NECESSARY.” Two versions of this question was proposed and tested in Round 1. They were as follows:

Version 1: “Did you EVER serve in a combat or war zone; that is, did you ever receive combat zone tax exclusion, Imminent Danger Pay, or Hostile Fire Pay?”

Version 2: “Did you EVER serve in a combat or war zone? Persons serving in a combat or war zone usually receive combat zone tax exclusion, Imminent Danger Pay, or Hostile Fire Pay.”

Both versions tested well, but there was a stronger preference for Version 2 (5 out of 8 participants preferred it). They thought that this version flowed better and it also provided context of what is meant by combat or war zone, without limited it to that definition explicitly. This distinction may be important for older veterans (e.g., Vietnam era or earlier), since tax exclusion, imminent danger pay or hostile fire pay may not have existed back then (or these terms may not have existed or been used).

We were particularly interested in the responses of female veterans. Of the four female veterans, only one of them had served in a combat or war zone. However, they indicated that they understood this question to be pertaining to combat zone and not combat fighting. They also noted that the definition helped to clarify this concept. Following these results, recommendation was made to implement Version 2 of this question.

Round 2: Findings

The main purpose of Round 2 interviews was to test the new revisions to question S10 (Year of separation), which had mixed results from Round 1 interviews. Another goal of Round 2 was to re-test the wording selections for questions S4 (Presence of disability) and S11a (Combat) to see if they continue to test well. Findings for question S10 will be discussed first.

S10 (Year of separation):

Three versions of this question were tested: the current wording that is used in the survey and 2 proposed alternate wordings. And they were as follows:

Version 1 (current wording): “In what year were you last separated from active duty in the Armed Forces?’

Version 2 (new): “In what year were you LAST separated from active duty?”

Version 3 (wording from Round 1): “What was your LAST year on active duty in the Armed Forces?”

Although the term “separated from active duty” was familiar with veterans, we were concerned that proxies may not be as familiar with this, as it could be military jargon. Thus, we conducted interviews with proxies to see if they shared the same familiarity and preference for it. The proxies in our test said that they were familiar with this language. One participant said, “That’s very familiar to me – that’s all I’ve heard for the last 20 years.” Three out of the four proxies said that they preferred the versions that had the language “separated” in the question. They said it was more clear and precise. One proxy preferred Version 3, which did not have that language, because she felt that the wording in Version 3 put better emphasis on the fact that a year was sought in the response.

Similar to the proxies and the veterans (from Round 1 test), the veterans in this round of testing also preferred the versions that included the word “separated” in the question. One veteran said that it was better especially for veterans who had served multiple times on active duty. We also found that the clause “in the Armed Forces” was not necessary for question comprehension. In summary, it is recommended that Version 2 be implemented in the Supplement for 2009. It uses the language that is familiar to veterans and their spouses, but it is more concise than the current wording.

S4 (Presence of disability): “Thinking about all of your military service, has the Department of Veterans Affairs (VA) or Department of Defense determined that you have a service-connected disability; that is a health condition or impairment caused or made worse by any of your military service?”

This wording was re-tested in Round 2. No problems were observed with this question wording. Both the proxies and the veterans reported that they understood the question as referring to both their time on active duty as well as their time in the reserves. It was recommended (memo from December 19, 2008) to use this wording for the 2009 supplement. However, this wording change may constitute a break in series, which would require significant justification for making any changes. While the inclusion of the introductory phrase aided the comprehension of the question (i.e., that the question refers to all of their military service), the researcher was unable to find compelling evidence that the omission of the phrase would be detrimental to comprehension. Files from 2007 interviews were reviewed and the researcher did not find any major problems with the question, even without the introductory phrase. In addition, Esposito (2003) reported no major problems with this question (with the current wording). However, Esposito (2003) evaluation occurred before the series of questions about the reserves were added and thus, this question did not have the same context as it currently does.

S11a (Combat): “Did you ever serve in a combat or war zone? Persons serving in a combat or war zone usually receive combat zone tax exclusion, Imminent Danger Pay, or Hostile Fire Pay.”

This wording was re-tested in Round 2. The results from the veteran interviews were very similar to those from Round 1. The veterans did not have any difficulty with this question. Similar to Round 1, one veteran commented that the inclusion of the special pay was important for clarification. He explained that veterans could have served in combat or war zones that were not necessarily areas designated as a “contingency area.” For example, veterans could have served in Bosnia and received combat pay, but Bosnia is not a declared war or an official area of contingency.

However, we observed some problems with the proxy interviews. One proxy respondent had difficulty with this question. While she reported that her husband served in Vietnam and was in a hostile area (during time of combat) and was hospitalized as a result, she was unsure of how to answer this question because she did not know whether he received any special pay. Another proxy respondent was able to answer this question (i.e., yes, the veteran served in a combat or war zone (at Pearl Harbor after the attack) but also showed some confusion about the special pay. She knew that the veteran had served 47 months of combat service, but went back and forth on how to answer because she did not know about his pay.

The other two proxy respondents had no difficulties with this question. In fact, one proxy said that the definition about the pay status made it clear for her what was being asked.

The main difference between the proxy respondents who had difficulty and who did not have difficulty was whether the respondent knew the veteran during their time in the Armed Forces. In other words, the wives who were married to the veterans during their time in the Armed Forces understood this question without any difficulty and also knew about the type of pay the veterans received. However, the proxy respondents, who did not know the veterans during their time in the Armed Forces, had difficulty with this question because they have no knowledge of the type of pay the veterans received. Further research is recommended for the future. In particular, we should evaluate ways to screen out proxies who did not know the veteran during their time in the Armed Forces. They may not be able to provide accurate responses. Another possibility is to revise interviewer instructions so that they can provide appropriate prompts if respondents have difficulty answering about special combat pay.

For the 2009 supplement, it is recommended that the wording of S11a be revised so that there is less emphasis on the special pay portion of the question.

4. Recommended Wording for 2009 Veteran’s Supplement

• S4 (Presence of disability): Keep the current version.

“Has the Department of Veterans Affairs (VA) or Department of Defense determined that you have a service-connected disability; that is a health condition or impairment caused or made worse by any of your military service?”

• S10 (Year of separation): Revise to –

“In what year were you LAST separated from active duty?”

• S11a (Combat): Revise to -

“Did you ever serve in a combat or war zone? Persons serving in a combat or war zone often² receive combat zone tax exclusion, Imminent Danger Pay, or Hostile Fire Pay.”

1 Gender refers to the participant who was interviewed.

2 Initially, the phrase “can receive” was recommended to replace “usually receive.” After discussion, the word “often” was substituted for “can.” It was felt that it struck the best balance between making the pay definition helpful for veterans, and also de-emphasizing it for proxies, who may not know about the veteran’s pay status.