Note to Reviewer - CPS Veterans Supplement

April 8, 2020

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Erica Yu Office of Survey Methods Research
SUBJECT:	Submission of Materials for Cognitive Testing of the Current Population Survey Veterans Supplement

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden for this study is 207 hours.

If there are any questions regarding this project, please contact Erica Yu at 202-691-7924.

1. Introduction

The Current Population Survey (CPS) is a monthly household survey of the civilian non-institutionalized U.S. population that provides information on labor force activity in the United States. The CPS provides a rich body of data on the labor force, employment, unemployment, persons not in the labor force, and other demographic and labor force characteristics.

The Veterans Supplement to the CPS (CPS-VS) is used to provide estimates on the demographic and labor force characteristics of veterans ages 17 or older. This supplement is sponsored by the Department of Veterans Affairs (VA) and the Department of Labor Veterans' Employment and Training Service (DOL VETS).

DOL VETS/VA have revised the supplement questionnaire for 2021 and requested the help of the Office of Survey Methods Research (OSMR) to cognitively test the new items to ensure that the questions in the supplement work as intended and that respondents can understand and answer them both for themselves and for other members in their household.

2. Methodology

Due to the coronavirus pandemic, traditional in-person research methods are not available. To collect feedback about the questionnaire, this study uses telephone cognitive interviews and online surveys.

• Individual Cognitive Interviews

Cognitive interviews provide an in-depth understanding of the participant’s thought processes and reactions to the questions. Interviews will be conducted using traditional telephone calls by staff who are experienced in conducting these types of interviews.

Participants will be asked select survey questions from the CPS to provide context and then asked the proposed Veterans supplement questions. After the survey questions, participants will be asked debriefing questions to better understand their responses and reactions to the questions, both when reporting for themselves and when reporting for other household members. The survey questions and full semi-scripted cognitive interview testing protocol is included in Appendix A.

The interviews will be conducted by phone with participants from anywhere in the United States. Observers from BLS, DOL VETS, or VA may attend interviews.

• Online Survey

We will conduct an online survey to collect information from a larger, and also more targeted sample than would be possible using in-person methods alone.

This research design includes four rounds of up to 100 participants each. Participants will be asked select survey questions from the Current Population Survey (CPS) to provide context and then asked a subset of the proposed Veterans supplement questions. After the survey questions, participants will be asked debriefing questions to better understand their responses and reactions to the questions, both when reporting for themselves and when reporting for other household members. This online testing is a type of unmoderated cognitive interview. This sample size takes into account break-offs, incomplete data, and participants who do not follow the task instructions. The survey protocol is included in Appendix B.

The findings from all three collections will be evaluated qualitatively, and used to arrive at conclusions about the effectiveness of the proposed wording. As this is pretesting, we expect that modifications to the question wordings or response options may be made during the course of the study based on initial results. Although the goals of the testing, and overall design, will remain the same, findings from preliminary results may be used to improve the interview questions. Modifications are likely to range from slight changes to question wording to the order of the questions.

3. Participants

Individual Cognitive Interviews

This research design includes up to two rounds of individual cognitive interviews (up to 36 participants each round); total of 72 participants). Individual interviews are planned for late April and May. However, scheduling will depend on recruitment and participant availability.

Up to 72 participants will be recruited for cognitive interviews using several different strategies:

• E-mail advertisement to the Veterans Health Administration’s Veteran Insights Panel (VIP; voluntary online panel comprising 3,429 veterans) in collaboration with and oversight by the VA (Appendix C).

• Advertisements in online local bulletin boards (Appendix D).

• Calls to potential participants in OSMR’s existing participant database

Potential participants will be screened for eligibility using a screening script (Appendix E).

Participants for the cognitive interviews will be selected for characteristics fitting into the below target subgroups, for both veterans and veteran proxy respondents (spouse, parent, or adult child).

• Are or have been married (either before or after their time in the service)

• Are retired from the military

• Have a disability (either connected or not connected to service)

• Have served in the National Guard or Reserves

• Have separated from service fewer than 10 years ago

• Are not employed or consider themselves underemployed

Online Survey

Up to 400 participants will be recruited as a convenience sample from Amazon Mechanical Turk of adults (18 years and older) residing in the U.S. Using a self-report eligibility screening question, we will limit our sample to the following subgroups:

• Have a disability (either connected or not connected to service)

• Have served in the National Guard or Reserves

• Have separated from service fewer than 10 years ago

• Living in a household with a veteran

4. Burden Hours

Total expected burden for this study is 208 hours, as shown in the table below:

	Number Contacted	Time spent Recruiting	Screening Burden	Number of Participants	Session Length	Session Burden	Total Burden
Individual Cognitive Interview	216	5 minutes	18 hours	72	60 minutes	72 hours	90 hours
Online Survey	500	2 minutes	16.7 hours	400	15 minutes	100 hours	117 hours
Overall Total Burden Hours							207 hours

5. Payment to Participants

Interview participants will not receive payment.

Participants from Amazon Mechanical Turk will be paid $2.00 for participating in the study ($800.00 total). This aligns with typical rates offered on the Amazon Mechanical Turk platform for similar tasks.

6. Data Confidentiality

Interview participants will be asked for their verbal consent to participate in the study, outlined in Appendix F.

Once online survey participants are recruited into the study, they will be given a link to the survey, which is hosted by SurveyMonkey. The data collected as part of this study will be stored on SurveyMonkey servers. Using the language shown below, participants will be informed of the voluntary nature of the study and they will not be given a pledge of confidentiality.

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141 (Expiration Date: March 31, 2021). Without this currently approved number, we could not conduct this survey. We estimate that it will take on average 15 minutes to complete this survey. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by SurveyMonkey and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

7. Attachments

Appendix A: Cognitive Interviewing protocol

Appendix B: Online Survey protocol

Appendix C: Email advertisement to VIP panelist

Appendix D: Online bulletin board electronic advertisement

Appendix E: Recruitment and eligibility script

Appendix F: Remote testing verbal consent

5