Justification for Non-Substantive Change Memo

COGNITIVE TESTING OF THE

VETERAN READINESS AND EMPLOYMENT (VR&E)

LONGITUDINAL STUDY SURVEY QUESTIONNAIRE

OMB #2900–0786

A REDESIGN / REVISION

NON-SUBSTANTIVE CHANGE

Purpose: This request seeks approval of non-substantive changes to the approved information collection for the VR&E Longitudinal Study (OMB #2900–0786).

Summary: There are no changes to the ICR or costs to the Government.

Background

The Veterans Benefits Administration (VBA), within the U.S. Department of Veterans Affairs (VA), operates the Veteran Readiness and Employment (VR&E) program that assists Veterans and Service members who have service-connected disabilities and employment barriers to prepare for, obtain, and maintain suitable employment.

Congress passed the Veterans’ Benefits Improvement Act in 2008, which improved and enhanced benefits for Veterans. A section of this legislation required the Department of Veterans Affairs (VA) to conduct a 20-year longitudinal study of Veterans who participated in the program and annually report on a statistically valid sample of Veterans who began participating in the VR&E program in fiscal years 2010, 2012, and 2014. VR&E program participant eventually leave the program (through successful rehabilitation or voluntary/involuntary withdrawal from the program) and the VA would have no other source of information on long-term outcomes, other than to contact them directly to find out about employment and other outcomes, as far as 20 years after program enrollment. The VR&E Longitudinal Study survey is annually administered to the sample of Veterans selected to participate in the study and to collect information on the outcomes of Veterans long after leaving the program.

Overview of Requested Changes

The content of the VR&E Longitudinal Study survey was originally designed at the beginning of the study and first administered in 2012 when the vast majority of participating Veterans were actively participating in the program. The study has been ongoing for 11 years and 93 percent of participating Veterans have left the VR&E program. The VR&E Longitudinal Study survey has been revised to better suit the current participant population with the majority having left the program.

The VA is requesting OMB clearance to conduct cognitive testing on the revised VR&E Longitudinal Study Survey questionnaire. We plan to interview up to 20 VR&E Longitudinal Study participants to understand respondents’ thought processes as they interpret and respond to survey questions. We are requesting approval for the following procedural revisions to the approved VR&E Longitudinal Study in order to recruit for and effectively implement the cognitive testing of the revised questionnaire:

• Modification of the survey invitation letter to clearly distinguish this data collection activity from the annual survey invitation. It will include a description of the cognitive interview procedure, stating that respondents will be asked to complete the questionnaire and provide feedback on the presented questions and response options.

• Modification of the informed consent document to provide a summary of the cognitive interview procedures, risks, benefits, and incentive.

• Offering participants a $50 incentive upon completion of the cognitive interview as a token of appreciation for their time.

We will utilize a cognitive interview protocol (see Appendix E) to test the revised questionnaire. This protocol differs from the approved VR&E Longitudinal Study survey as it includes not only the revised survey questions but also a series of open-ended probes to capture (1) respondents’ understanding of the survey questions, (2) feasibility of response (e.g., respondent ability to recall information), (3) the flow and sequence of the survey topics and items, and (4) reporting accuracy (based on respondents’ descriptions of how they answered the questions). The information collected through these interviews will help improve and refine the revised VR&E Longitudinal Study survey questionnaire.

The questionnaire revisions include various improvements, including question wording, removal of unnecessary questions, and the introduction of new questions to capture previously unavailable information, as specified below:

• Addition of two new questions to capture the career advancement of the respondent: “Were you working at this job when you exited the VR&E program?” and the follow-up question, “If YES, since exiting the program, have you received a promotion in your current job?”

• Additional of two new questions to capture use of VA educational benefits: “During the past 12 months, have you used any VA educational benefits for pursuing education or training?” and the follow-up question, “Which of the following VA educational benefits have you used during the past 12 months?”

• Introduction of a new matrix question to assess the extent to which the respondent believes the VR&E program assisted them in various employment activities (e.g., identifying and applying for jobs, getting hired for a job).

• Modification of a series of Likert scale questions by reducing the 9-point agree/disagree scale to a more streamlined 5-point agree/disagree scale.

• Replacing the phrase “for pay or profit” with “for pay or self-employment” throughout the questionnaire.

• Revision of two questions that capture the number and type of recent visits to medical facilities to clarify that the response should only include visits pertaining to the respondent personally, excluding visits made by family members of dependents.

• Simplification of the question regarding the number of dependents a respondent has ("How many dependents do you currently have?") by eliminating a subsequent question that asks for specific quantities of each dependent type ("Please specify the quantity of each type of dependent, such as spouse, child under 18, etc.").

• Reorganization of the questionnaire by grouping similar questions within the same section to enhance the overall coherence and flow of the questionnaire.

We plan to conduct 20 interviews with individual VR&E Longitudinal Study participants, with each interview estimated to last approximately one hour. Therefore, we anticipate a total burden of 20 hours associated with the cognitive interviews.

Cognitive Interview Procedures

The VA has contracted with Advanced Survey Design (ASD) and its subcontractor Westat to conduct the survey and cognitively test the questionnaire. A Westat statistician will draw a random sample of current VR&E Longitudinal Study participants will be invited to complete a cognitive interview. The statistician will use a stratified sampling approach to select potential interview participants. In the first stage (stratification), the statistician will stratify the pool of current study participants who completed the most recent survey administration into three groups based on their current study participation status: active (actively participating in the VR&E program), rehabilitated (successfully rehabilitated from the VR&E program), and discontinued (voluntarily/involuntarily withdrew from the program). In the second stage (random sampling), the statistician will randomly sample approximately 200 current study participants across the three participation groups.

ASD will send all sampled participants a letter (see Appendix A) via email to invite them to participate in a cognitive interview. The letter includes a description of the research activity, their rights as a participant, information about the incentive, and a toll-free number for interested individuals to call to volunteer to participate. The letter will also include the electronic signature of the VR&E director. Westat will manage incoming calls from the toll-free number and discuss the details of the research activity with interested potential participants (see Appendix B, Telephone script). Incoming callers will receive additional information about the cognitive interview process, an overview of the informed consent statement, and have the opportunity to have their questions answered by an experienced member of the Westat research team. Anyone interested in moving forward, will be scheduled for a cognitive interview and sent a follow-up confirmation email (see Appendix C) that includes the date and time of the interview, the videoconferencing link to access the virtual meeting, and an electronic copy of the full informed consent statement (see Appendix D).

Informed Consent and Interview Procedures

Variations of the informed consent statement (see Appendix D) will be provided to respondents three times during recruitment and participation. First, a summary of informed consent information is provided to interested potential participants during their initial telephone call with the research team. Following the telephone call, the research team will email a full copy of the informed consent statement to each scheduled participant. At the start of each interview, the interviewer will summarize the informed consent statement with the participant and ask for verbal consent from the participant to continue with the interview. The interviewer will also confirm that the participant does not have any further questions about the study or the interview, prior to providing consent. If the participant declines, the interviewer will end the interview and thank them for their time. If the participant agrees, the interview will ask if the participant agrees to have the interview audio and video recorded. The interviewer will explain the purpose of the recording and that we will protect privacy; the interviewer will also explain that the interview can still proceed if the participant does not consent to be recorded.

The cognitive interview protocol (see Appendix E) includes questions about VR&E program participation, recent employment and education experiences, recent visits to medical facilities, and VR&E program satisfaction. When a participant answers specific survey questions, the interviewer will ask a series of predetermined and ad-hoc probes (e.g., “How did you come up with your answer?”) to assess respondent comprehension, recall, and retrieval. At the end of the interview, respondents are provided an opportunity to offer additional feedback or reactions about the survey question that they did not have a chance to share previously. We will conduct interviews virtually via video call using Microsoft Teams or Zoom videoconferencing software. In our experience, it is easier to build rapport with respondents over video call, as the interviewer can respond to nonverbal cues. The target length of these interviews is approximately one hour.

Reporting and Confidentiality

The cognitive interviews will be recorded (audio and video) to allow interviewers to conduct a more thorough analysis than could be achieved simply by listening and taking notes. For each interview conducted, interviewers will summarize their findings based on notes and review of recordings. Westat will prepare a memorandum summarizing the results across all interviews and will include recommendations for instrument revisions. The individual and collective summaries will include a description of what the respondent stated in the interview and may include verbatim quotes. No identifying information will be included in the report and no respondent names will be attributed to any quotes that are included.

Participation in the cognitive interviews is voluntary and choosing not to participate will not affect participation in the VR&E program, longitudinal study, or the services individuals may receive in the future. Personally identifiable information (PII), including names and contract information, will be stored on secure network computers that will only be accessible by the Westat team cleared to work on the project. Interview records and all PII will be destroyed once the final survey questionnaire is approved by VA.

Incentives

We will provide all participants with a $50 electronic gift card. We recognize the time burden on the respondent and this reimbursement expresses our appreciation for their input to improve the study questionnaire. Participants will receive the gift card information via email (see Appendix F) upon completing the cognitive interview.

The following attachments are included for your review regarding this request:

• Appendix A – Participant Recruitment Letter

• Appendix B – Telephone Script

• Appendix C – Interview Confirmation Email Template

• Appendix D – Informed Consent Document

• Appendix E – Cognitive Interview Protocol

• Appendix F – Incentive Email Template

Thank You,

Veterans Benefits Administration

Veteran Readiness and Employment (VR&E) Service