Fy15 Non Response Bias Report - VRE

Voice of the Veteran Line of Business Tracking Study
Vocational Rehabilitation and Employment
Access and Servicing
Fiscal Year 2015 Non-Response Bias Analysis

VETERANS BENEFITS ADMINISTRATION

[FY15 REPORT]

Table of Contents
Executive Summary .................................................................................................................................. 4
Introduction ............................................................................................................................................. 6
Methodology ............................................................................................................................................ 7
2.1 J.D. Power Index Model ................................................................................................................... 7
2.2 Sampling......................................................................................................................................... 11
2.3 Data Collection ............................................................................................................................... 11
Non-Response Bias Analysis................................................................................................................... 12
3.1 Survey Yield .................................................................................................................................... 20
3.2 Missing Data Patterns and Mechanisms ........................................................................................ 26
3.3 Margin of Error .............................................................................................................................. 27
3.3.1 Sampling Distribution ....................................................................................................... 28
3.3.2 Distribution of Overall Satisfaction Index Scores.............................................................. 29
3.3.3 Analysis for Demographic Differences .............................................................................. 30
3.3.4 Data Imputation Analysis for Demographic Differences .................................................. 35
Findings .................................................................................................................................................. 38
Conclusion .............................................................................................................................................. 39
References ............................................................................................................................................. 40
List of Appendices
Appendix A Missing Data Patterns and Mechanisms .................................................................... 42
Appendix B Item Response Rates .................................................................................................. 43
Appendix C Study Overview ........................................................................................................... 48
1.1 Study Background .................................................................................................................. 48
1.2 Methodology ........................................................................................................................ 48
1.3 Data Cleaning ......................................................................................................................... 49
1.4 Order generation and fulfillment process ............................................................................. 50
1.5 Reporting ............................................................................................................................... 51
Sample Plan Overview ................................................................................................................... 52
2.1 Sample Criteria ...................................................................................................................... 52
2.2 Fielding/Sampling Frequency ................................................................................................ 52
2.3 Data Transfer ......................................................................................................................... 53
2.4 Sample Cleaning Rules Glossary ............................................................................................ 54
2.5 Sample Selection.................................................................................................................... 54
2.6 Data Collection....................................................................................................................... 55
Appendix D Approaches to Effects of Non-Response Bias and Improving Response Rates .......... 56

ii

Approaches to Mitigating the Effect of Non-Response Bias .......................................................... 56
1.1 Approach 1: Strategies to Maximize Response Rates ........................................................... 56
1.2 Approach 2: Correcting Unit Non-response Bias with Sample Weighting and Survey Raking57
Strategies to Improve Response Rate ............................................................................................ 58
Appendix E Impact of FAR 8.8 ........................................................................................................ 59
1.1 Impact .................................................................................................................................... 59
Appendix F Survey Questionnaire.................................................................................................. 60
Appendix G List of Acronyms………………………………………………………………..………………………,.………..96

iii

Executive Summary
The Voice of the Veteran (VOV) Line of Business Tracking Satisfaction Research Study was
developed to establish continuous satisfaction measurement and incorporate direct Veteran
feedback in the decision-making process in order to improve the level of service to
Servicemembers, Veterans, and their beneficiaries.
As part of this study, a survey was fielded in Fiscal Year 2015 (FY15) for the Department of
Veterans Affairs (VA), Veterans’ Benefits Administration (VBA) Vocational Rehabilitation and
Employment (VR&E) Access (Access) and VR&E Servicing (Servicing). The survey yielded a
response rate of 9.62% (decrease of 2.27% from FY14) for Access and 13.78% (increase of
4.43% from FY14) for Servicing which was lower than the estimated response rate submitted
with the information collection request (ICR) as well as lower than the Office of Management
and Budget’s standard of 80% (at the overall unit response rate).
OMB’s “Standards and Guidelines for Statistical Surveys,” Section 3.2, Guideline 3.2.9, notes
that a non-response analysis should be conducted for surveys with an overall unit response rate
of less than 80%. Therefore, J.D. Power (JDP) conducted the necessary statistical tests in
accordance with OMB’s guidelines in order to verify the validity of the Access and Servicing
survey results for FY15.
The initial 2015 analyses for these reports were done in consultation with Dr. Don Dillman , a
professor at Washington State University. Dr. Dillman is regarded as a key survey method
expert on non-response bias research and the report conforms to sound statistical research
practices in accordance with OMB standards. The analysis performed also includes an iterative
survey raking procedure to derive sample weightings based on a simultaneous balancing
analysis of the demographic differences.
After adjusting for demographic differences between survey respondents and nonrespondents, the statistical tests performed on the survey responses for the surveys collected,
illustrate that no differences were found in the Overall Satisfaction Index score and Advocacy
rating (likelihood to inform others about VA benefits).
The sample for the Access population was defined as individuals as Veterans who had an initial
meeting with their VR&E counselor and were granted a decision regarding their entitlement
within the past 30 days. These veterans include those who applied and showed up for the initial
appointment and were either found entitled and pursued the program or found not entitled
and did not pursue the program.
The sample for Servicing is defined as Veterans who have entered and been enrolled in one of
the five tracks for at least 60 days. These individuals include Veterans who have been
rehabilitated; Veterans who have reached maximum rehabilitation gain and could not proceed
in the program; and Veterans still pursuing benefits.

4

The analysis performed was done in consultation with Dr. Don Dillman, professor from
Washington State University, who is regarded as a key survey method expert on non-response
research. The analysis also includes an iterative survey raking procedure to derive sample
weightings based on a simultaneous balancing analysis of the demographic differences.
The Overall Satisfaction Index score and Advocacy rating for Access (600 on a 1,000-point scale
and 3.48 on a scale of 1-4 points, respectively) are not impacted in any meaningful way by nonresponse bias. The Overall Satisfaction Index score and Advocacy rating for Servicing (715 and
3.64, respectively) are not impacted in any meaningful way by non-response bias. This analysis
confirms that data collected during Fiscal Year 2015 is valid for use by VBA.

5

Introduction
In an effort to achieve top level customer service, VBA partnered with J.D. Power to conduct
Veteran satisfaction research on its behalf. VBA’s Voice of the Veteran (VOV) Satisfaction
Initiative was established to continuously measure and improve the level of service to
Servicemembers, Veterans, and their beneficiaries.
The intent of this initiative is to:






Reinstate VBA’s customer satisfaction research program in order to incorporate Veteran
feedback into the decision-making process
Identify the critical factors to Veterans’ satisfaction with benefits and services provided
by VBA
Provide continuous feedback to validate effectiveness of new initiatives and process
changes
Provide decision-makers and stakeholders with timely and actionable feedback on a
continuous basis
Identify and document best practices, and act as a vehicle to celebrate successful
interactions and experiences

VBA’s VOV Line of Business Tracking Satisfaction Research Study was developed to continuously
field customer satisfaction survey instruments to provide Veteran and beneficiary feedback on
the following VBA lines of business and benefit programs: Compensation, Pension, Education,
Vocational Rehabilitation and Employment, and Loan Guaranty (including Specially Adapted
Housing). In support of this effort, in FY15, JDP fielded a survey instrument regarding the Access
and Servicing processes on behalf of the VR&E program. The purpose of the Access and
Servicing process surveys was to identify the factors critical to Veteran satisfaction with
enrollment and receipt of benefits issued by VBA and to improve the level of services provided.
The survey instruments for the Servicing and the Access processes were developed in
collaboration with VA’s VR&E Division and in accordance with OMB’s guidelines concerning
statistical collection procedures and methods. After the initial survey instrument was designed,
cognitive labs using the “think aloud” method were conducted to evaluate user experiences
when filling out the survey. Prior to the FY15 fielding of the Servicing and Access process
survey, a benchmark (pilot) study was conducted from October 2012 through January 2013 to
further assess the effectiveness of the methodology and conformance to OMB’s standards. The
survey was fielded in 2014 as well and the 2015 fielding will be the third iteration of the survey
fielding.

6

Methodology
2.1 J.D. Power Index Model
J.D. Power defines customer satisfaction as a measure of how well product or service
experiences fit the expectations of customers. All JDP index models assume a two-tiered
regression model involving factors and attributes. Each customer experience is influenced by
several factors (i.e., first tier), which, in turn, are influenced by several attributes or drivers (i.e.,
second tier). A diagram of the index model follows on the subsequent page.
In order to begin the index model calculation, each set of attributes within a factor are used to
predict the Overall Satisfaction Index score (sub-OSAT) for that factor. An importance weight is
assigned to each attribute, where the weight of “importance” of each attribute is defined as the
ability of that attribute to predict Overall Satisfaction. A multiple regression model is used to
estimate the attribute weights. This model produces the “bottom” level weights and is
computed for each factor separately. The bottom level weights are rescaled so that they add up
to 1 point within each subcategory. As a result, the percentage of total explained variation in
the sub-OSAT that is due to a particular attribute constitutes that attribute’s importance weight
within its respective factor.
Following the calculation of attribute (i.e., bottom level) weights, the factor (i.e., top level)
weights are calculated. Factor scores are calculated by taking the sum of the product of the
attribute rating scores and the attribute importance weights. This model produces the top-level
weights, which are rescaled so that they add up to 1 point. Thus, the percentage of the total
explained variation in the Overall Satisfaction Index score that is due to a particular sub-OSAT
constitutes that factor’s importance weight.
After all factor scores are computed, they are weighted so that some contribute more to
Overall Satisfaction than others, based on the index importance weights. The index score is
subsequently calculated by taking the sum of the product of all of the factor scores and the
factor importance weights. Finally, both the index and factor scores are multiplied by 100 so
that the range of each is 100 (if all attributes were rated 1) to 1,000 (if all attributes were rated
10).
By applying the importance weights derived from the two-tiered modeling approach, JDP
creates a weighted index score that ranges from a low of 100 to a high of 1,000. This index
approach has the benefit of being highly reliable and valid and provides increased ability to
discriminate the performance levels of companies and organizations.

7

VR&E Access and Servicing Process Index Weights
In working with VR&E’s subject matter experts and leadership, the design of its survey
encompasses the factors and attributes as outlined in the following tables. The factors (Benefit
Information, Contact with VA, Benefit Application, and Benefit Entitlement) and attributes (Ease
of Accessing Information, Availability of Information, etc.) represent Access and Servicing Index
Models in FY15. The corresponding weights for each factor and attribute are the weights based
on the above index model calculation. The weights are derived from the relative importance of
each factor or attribute to the respondents.

8

Table 2.0. Access: Index Model Weights

Access Index Model Weights
Benefit Information
Contact with VA
VR&E Benefit Application
Process
VR&E Intake Counselor
Benefit Entitlement

Effective Weight
15.15%
7.71%
18.04%
20.56%
38.54%

Table 2.1. Access: Weights by Attribute
Access Weights by Attribute
Effective Weight
Benefit Information
Ease of accessing information

2.98%

Availability of information

1.79%

Clarity of information

2.65%

Usefulness of information

3.49%

Frequency of information

4.24%

VR&E Benefit Application Process
Ease of completing the application

4.31%

Timeliness of eligibility notification

6.64%

Flexibility of application methods

7.09%

VR&E Intake Counselor
Promptness of scheduling
appointments

4.13%

Courtesy of the app. counselor

4.73%

Knowledge of the app. counselor

2.88%

Counselor’s concern for your needs

4.04%

Timeliness of completing your initial
evaluation

4.78%

Benefit Entitlement
Effectiveness of benefit/service

16.88%

Timeliness of receiving
benefits/services

21.66%

Contact with VA

7.71%

9

Table 2.2. Servicing: Index Model Weights

Servicing Index Model Weights

Benefit Information
Contact with VA
VR&E Service Counselors
Benefit Entitlement

Effective
Weight
17.84%
7.53%
26.51%
48.12%

Table 2.3. Servicing: Weights by Attribute
Servicing Weights by Attribute
Effective
Weight
Benefit Information
Ease of accessing information

3.84%

Availability of information

1.60%

Clarity of information

2.85%

Usefulness of information

4.46%

Frequency of information

5.10%

VR&E Service Counselors
Promptness of scheduling
appointments

6.27%

Courtesy of the counselor

5.29%

Knowledge of the counselor

5.15%

Counselor’s concern for your needs

4.10%

Timeliness of completing your initial
evaluation

5.70%

Benefit Entitlement
Effectiveness of benefit/service

22.25%

Timeliness of receiving
benefits/services

25.87%

Contact with VA

7.53%

10

2.2 Sampling
The Servicing survey was fielded to Veterans who had an initial meeting with their VR&E
counselor and were granted a decision regarding their entitlement within the past 30 days.
These individuals include those who applied and showed up for the initial appointment and
were either found entitled and pursued the program or found not entitled and did not pursue
the program. The Access survey was fielded to Veterans who had entered and been enrolled in
one of the five tracks for at least 60 days. These individuals include Veterans who have been
rehabilitated; Veterans who have reached maximum rehabilitation gain and could not proceed
in the program; and Veterans still pursuing benefits.
J.D. Power mailed approximately 60,000 surveys for Servicing and 14,000 for Access to Veterans
across the nation in FY15. The target number of completed surveys for Servicing was 18,000
and 4,200 for Access. The actual number of completed surveys received for Access was 1,106,
and for Servicing it was 8,643.
The samples used in this study, which were provided by VR&E, were random samples from the
available Veterans. See Appendix D, Sample Plan Overview for further detail on sampling.
Survey Instrument

Methodology

Fielding Frequency

Total Mailouts in
FY15

Access

Mail and Online

Annually

14,000

Mail Only

Monthly

60,000

Servicing

2.3 Data Collection
During the survey fielding period, self-administered online survey returns and self-administered
paper surveys were collected. While verbatim responses are recorded by a live survey
processor, responses from paper surveys are scanned through automated imaging software.
Survey returns undergo quality assurance to validate the accuracy of responses captured.
Respondents from the Access study had the option of completing the survey on paper or online
and received two separate mailings:



1st Mailing: Postcard introducing the study to the respondent, which included an online
survey link and a unique access code login for the online survey
2nd Mailing: Survey Package, which included a cover letter, with the online survey link and
login, a paper survey, and a business reply envelope

11

Respondents from the Servicing study completed the survey on paper and received two
separate mailings:



1st Mailing: Survey Package, which included a cover letter introducing the study to the
respondent, a paper survey, and a business reply envelope
2nd Mailing: Survey Package, which included a cover letter, a paper survey, and a business
reply envelope

Each time the surveys were deployed, the postcards and survey packages were subject to a
proof approval process that utilized three levels of approvals by J.D. Power, Benefits Assistance
Service (BAS), and VA Publications Services Division (VAPSD). After the print vendor mailed the
postcards and survey packages, mail receipts were sent to VBA.
During the survey fielding period, JDP provided a toll-free survey hotline and dedicated e-mail
address to answer survey-related inquiries and to provide assistance to respondents for
completing the surveys. The telephone and e-mail helpdesk was staffed with three JDP
employees who answered inquiries during regular business hours (8:00am-5:00pm PST,
Monday thru Friday). A voice message system was available to receive phone messages so
after-hours calls could be responded to the following business day. An automatically generated
e-mail response was sent to all e-mail inquiries informing respondents that their e-mail was
received and they would receive a response within 24 hours. JDP helpdesk representatives
logged each survey-related inquiry in a password protected spreadsheet documenting the
reason for the inquiry, the resolution provided, and the contact information of each caller. At
the end of each month, a log containing all inquiries was provided to the Contracting Officer
Representative (COR) for review. If non-survey related high-severity benefit inquiries were
received, J.D. Power contacted the COR immediately with the respondent’s contact
information.
Throughout the course of the program, weekly status meetings were held between JDP and
BAS to discuss survey administration. Biweekly status meetings were held between the
Government Printing Office print vendor, JDP, BAS, and VAPSD to discuss the printing and
mailing of the survey materials.

Non-Response Bias Analysis
The purpose of the non-response bias analysis is to ascertain the possible causes of variance in
response rates among different respondent demographics and/or determine if any bias has
been introduced with a low response rate. Given that the Voice of the Veteran VR&E Access
Process Study had an overall unit response rate of 9.62% and the Voice of the Veteran VR&E
Servicing Process Study had an overall unit response rate of 13.78% in Fiscal Year 2015, the
following section examines whether a low response rate or other factors may have caused
respondent bias to occur.

12

The Office of Management and Budget’s Questions and Answers, “When Designing Surveys for
Information Collections,” dated January 2006, and “Standards and Guidelines for Statistical
Surveys,” dated September 2006 (see References) provide guidelines on acceptable survey
design and response rates. OMB guidelines recommend a non-response bias evaluation for
surveys with an overall unit response rate of less than 80%.
In addition to the above referenced documents prepared by OMB, J.D. Power assessed other
source documents that were written and published by the Federal Committee on Statistical
Methodology: “Statistical Policy Working Paper 17, Survey Coverage” (1990) and “Statistical
Policy Working Paper 31, Measuring and Reporting Sources of Error in Surveys” (2001).
While high response rates are always desirable in surveys, JDP finds an 80% response rate is not
achievable for most voluntary, satisfaction-based, survey research studies (Malhotra & Birks,
2007). In particular, survey research studies that do not provide an incentive are subject to not
achieving an 80% response rate. To better illustrate this point, the Dillman Method for survey
fielding was discussed in Dillman, D. A. (2014, pp. 22), detailing the efforts to attain an 80%
response rate.
A survey instrument was fielded to 600 students at the University of Washington, the same
institution that sponsored the study. After five attempts to solicit a response in a closed
university setting, as well as offering a monetary incentive to complete the study, the 80%
response rate was not achieved and instead garnered only a 77% response rate. The JDP team
met with the VA Contracting officer Representative to discuss current trends and realistic
response rates. As noted JDP does not believe that an 80% response rate is achievable and this
concern was shared with the Benefits Assistance Service team.
JDP conducted the following non-response bias analysis to determine if the respondents (i.e.,
those who completed the survey) were different in a meaningful way from the nonrespondents (i.e., those who were sent a survey, but did not complete it). Chi-squared analyses
consist of comparisons between respondents and non-respondents on available demographic
variables, such as gender, age, race, geographical region, war participation (service era), and
military service branch. The U.S. states were converted to standard U.S. census regions
(Midwest, Northeast, South, and West) in order to aggregate the data and enhance regional
comparisons.
J.D. Power research indicates that there is an absence of systematic statistical differences of
respondents’ overall satisfaction on the mail and online survey results. Research does suggest
differences can occur between mixed mode survey methodologies (mail, online, and phone),
but these are primarily related to (a) social desirability and interviewer bias associated with
phone surveys (see Baum, Chandonnet, Fentress, and Rasinowich, 2012, p. 2, for a review), and
(b) that older respondents tend to respond by mail more often than online.

13

The non-response bias analysis was conducted across both mail and online survey collection
modes. However, as a verification check, JDP examined potential differences in mail vs. online
survey responses by utilizing a t-test analysis on the OSAT Index and Advocacy rating, which
serve as measures of Veterans’ overall satisfaction and benefits advocacy. The Overall
Satisfaction Index is defined in the Methodology section of this report. The Advocacy rating is
defined as Veterans’ likelihood to inform others about VA benefits.
Throughout this report, we are conducting statistical analyses to compare survey respondents
and non-respondents. Frequently used statistical tests can include the T-Test, Chi-Square, or
Analyses of Variance (ANOVA). These tests generate relevant t-statistics, Chi-Squares, or F
statistics that are reported. The magnitude of the statistic’s value (either positive or negative)
measures the size of the difference relative to the variation in the data. If the statistic is not
large enough to generate a probability (p-value) less than .05, then it falls below the accepted
standard probability cut-off level that indicates whether a statistical difference is significant. If
a difference is not significant, statisticians regard these results as part of the normal sample
variation that occurs within the same population. Throughout this report, the probability pvalue standard of “must be less than .05 to be significant” is used for all statistics reported.
Table 3a.e. shows there were statistical differences found for VR&E Access between the mail
and online methodologies on Overall Satisfaction but not for Advocacy. Since surveys for VR&E
Servicing were all done by mail, this test was not relevant for Servicing.
Table 3a.e. Access: T-Test Analysis of Mail vs. Online Survey Results
Rating Measure
Overall Satisfaction Index (100 - 1000
range)
Likelihood to inform others about VA
benefits (rating 1 - 4)

Mail

Online

t-statistic

p-value

620

567

3.55

<.0001

3.46

3.51

-0.97

.334

For the Access sample, we found significant differences with the population based on gender
where the Access sample had a higher percentage of females compared to the population:
Table 3b.e. Access: Comparing Gender for Respondents and Non-Respondents
Gender by Respondent Type (%)

Female
Male

Survey
Respondents

NonRespondents

Total

21
79

18
82

18
82

Statistic
Chi-Square

DF
1

Value
4.01

Prob
<.05

14

For the Access sample, the results show significant differences with the population based on
age generation such that a larger number of older Veterans and a fewer number of Generation
Y/Z Veterans completed the survey:
Table 3c.e. Access: Comparing Age Generation for Respondents and Non-Respondents
Age Generation by Respondent Type (%)

Baby-Boomer
(ages 50-68)
Generation X
(ages 37-49)
Generation YZ
(ages 18-36)

Survey
Respondents

NonRespondents

Total

40

23

25

34

30

30

26

47

45

Statistic

DF

Value

Prob

Chi-Square

3

215

<.0001

For the Access sample, significant differences were found with the population based on race.
The differences indicate that the survey respondents were more likely to fall into the Other
category as compared to Asian, Black, and White:
Table 3d.e. Access: Comparing Race for Respondents and Non-Respondents
Race by Respondent Type (%)

Asian
Black
White
Other

Survey
Respondents

NonRespondents

Total

4
27
49
19

6
29
53
13

6
29
52
14

Statistic
Chi-Square

DF
3

Value
30

Prob
<.0001

For the Access survey, no significant differences were found with the population based on
geographical region:
Table 3e.e. Access: Comparing Census Region for Respondents and Non-Respondents
U.S. Census Region by Respondent Type (%)

Midwest
Northeast
South
West

Survey
Respondents

NonRespondents

Total

13
7
56
24

13
7
55
24

13
7
55
24

Statistic
Chi-Square

DF
3

Value
.67

Prob
.881

15

For the Access sample, significant differences were found with the population based on branch
of service. The data results show that the survey respondents were more likely to be in the
Navy and less likely to be in the Army or Marines:
Table 3f.e. Access: Comparing Military Service Branch for Respondents and Non-Respondents
Military Service Branch by Respondent Type (%)

Air Force
Army
Marines
Navy
Other

Survey
Respondents

NonRespondents

Total

14
50
10
17
9

13
53
12
15
6

14
53
12
15
6

Statistic
Chi-Square

DF
4

Value
25

Prob
<.0001

For the Access survey, significant differences were found in war service era, with fewer surveys
returned by OEF/OIF veterans than by other war Veterans:
Table 3g.e. Access: Comparing War Participation in OIF and OEF for Respondents and NonRespondents
OIF and OEF War Service by Respondent Type (%)

All others
OEF/OIF

Survey
Respondents

NonRespondents

Total

57
43

51
49

51
49

Statistic
Chi-Square

DF
1

Value
14

Prob
<.0002

Note: OIF is Operation Iraqi Freedom and OEF is Operation Enduring Freedom.

For the Access sample, significant differences were found with the population based on days of
active service the data indicates that survey respondents were more likely to have either served
1,000 days or less and 4,001 days or more, but not between 1,001 and 4,000 days:
Table 3h.e. Access: Comparing Days of Active Service for Respondents and Non-Respondents
Days of Active Service by Respondent Type (%)

1000 days
or less
1001-2000
days
2001-4000
days
4001 days
or more

Survey
Respondents

NonRespondents

Total

22

19

20

25

32

31

21

23

22

33

26

27

Statistic
Chi-Square

DF
3

Value
37

Prob
<.0001

16

For the Access sample, significant differences were found with the population based on case
status. We found that more surveys were completed by Veterans in employment rehab status
and fewer were completed by those in discontinued status:
Table 3i.e. Access: Comparing Case Status for Respondents and Non-Respondents
Case Status by Respondent Type (%)

Survey
NonRespondents Respondents
Discontinued
Employ Rehab
Other

59
36
5

72
24
3

Total

Statistic
Chi-Square

DF
2

Value
84

Prob
<.0001

71
26
3

For VR&E Servicing, surveys were all conducted by mail (no online methods used).
For the Servicing sample, significant differences were found with gender where the data
reflects that the Servicing sample had a higher percentage of females than males:
Table 3b.s. Servicing: Comparing Gender for Respondents and Non-Respondents
Gender by Respondent Type (%)

Female
Male

Survey
Respondents

NonRespondents

Total

22
78

20
80

20
80

Statistic
Chi-Square

DF
1

Value
7.46

Prob
<.007

For the Servicing sample, significant differences were found with the population based on age
generation where the results show that a larger number of older Veterans and a fewer number
of Generations X and Y/Z Veterans had completed the survey.
Table 3c.s. Servicing: Comparing Age Generation for Respondents and Non-Respondents
Age Generation by Respondent Type (%)

Baby-Boomer
(ages 50-68)
Generation X
(ages 37-49)
Generation YZ
(ages 18-36)

Survey
Respondents

NonRespondents

Total

45

27

30

31

33

33

24

40

38

Statistic

DF

Value

Prob

Chi-Square

3

1154

<.0001

17

For the Servicing sample, significant differences were found with the population based on race.
The differences indicate that survey respondents were less likely to be White or Black, and
more likely to fall into the Other category:
Table 3d.s. Servicing: Comparing Race for Respondents and Non-Respondents
Race by Respondent Type (%)

Asian
Black
White
Other

Survey
Respondents

NonRespondents

Total

5
25
53
17

5
27
54
14

5
27
54
14

Statistic
Chi-Square

DF
3

Value
79

Prob
<.0001

For the Servicing survey, significant differences were found with the population based on
geographical region and the information reflects that survey respondents were less often from
the South and more often from the West:
Table 3e.s. Servicing: Comparing Census Region for Respondents and Non-Respondents
U.S. Census Region by Respondent Type (%)

Midwest
Northeast
South
West

Survey
Respondents

NonRespondents

Total

20
14
39
27

20
14
41
26

20
14
40
26

Statistic
Chi-Square

DF
3

Value
11.72

Prob
<.009

For the Servicing sample, significant differences were found with the population based on
branch of service. The results indicate that the survey respondents were more likely to be in the
Air Force, Navy, and Other services, and were less likely to be in the Army or Marines:
Table 3f.s. Servicing: Comparing Military Service Branch for Respondents and Non-Respondents
Military Service Branch by Respondent Type (%)

Air Force
Army
Marines
Navy
Other

Survey
Respondents

NonRespondents

Total

14
49
11
18
8

13
52
13
17
5

13
52
13
17
6

Statistic
Chi-Square

DF
4

Value
147

Prob
<.0001

18

For the Servicing survey, we found significant differences in war service era with fewer surveys
returned by OEF/OIF veterans than were returned by other war Veterans:
Table 3g.s. Servicing: Comparing War Participation in OIF and OEF for Respondents and NonRespondents
OIF and OEF War Service by Respondent Type (%)

All others
OEF/OIF

Survey
Respondents

NonRespondents

Total

69
31

64
36

65
35

Statistic
Chi-Square

DF
1

Value
59

Prob
<.0001

Note: OIF is Operation Iraqi Freedom and OEF is Operation Enduring Freedom.

For the Servicing sample, significant differences were found with the population based on days
of active service. Thesurvey respondents were more likely to have served 4,001 or more days
and less likely to have served 1,001 to 4,000 days compared to the population:
Table 3h.s. Servicing: Comparing Days of Active Service for Respondents and Non-Respondents
Days of Active Service by Respondent Type (%)

1000 days
or less
1001-2000
days
2001-4000
days
4001 days
or more

Survey
Respondents

NonRespondents

Total

25

25

25

26

32

32

19

22

21

31

21

22

Statistic
Chi-Square

DF
3

Value
386

Prob
<.0001

For the Servicing sample, significant differences were found with the population based on case
status. Fewer surveys were completed by Employ Rehab Veterans and more by Other Veterans:
Table 3i.e. Servicing: Comparing Case Status for Respondents and Non-Respondents
Case Status by Respondent Type (%)

Survey
NonRespondents Respondents
Discontinued
Employ Rehab
Other

2
71
26

3
74
23

Total

Statistic
Chi-Square

DF
2

Value
39

Prob
<.0001

3
74
24

19

3.1 Survey Yield
In accordance with OMB “Standards and Guidelines for Statistical Surveys,” an agency must
appropriately measure, adjust for, report, and analyze unit and item non-response when the
intended response for a targeted population is not met.1 In assessing VR&E’s data in
accordance with Section 3.2, and Guidelines 3.2.1-3.2.3, the unweighted unit response rate was
calculated as the ratio of the number of completed cases to the number of in-scope sample
cases (Ellis, 2000; AAPOR, 2000).
Table 3.1a.e below shows the sample distribution and response rate for the VR&E Access target
population.
Table 3.1a.e. Sample Distribution and Response Rates for VR&E Access Population
Total Access Population FY2015
Total records received
Duplicate records in sample file
Duplicate record history
Invalid address
Invalid values
Blanks
Do not contact
2
Total records available after cleaning
Total records selected
Undeliverable addresses
Total mailed (excludes undeliverable)
Total completed mail surveys
Total completed online surveys
Total completed surveys
3
Total completed surveys with Overall Index Score
4
Total Sample Response Rate
5
Eligible Sample Response Rate

19,484
0
6,013
551
27
0
24
12,869
11,118
441
10,677
703
403
1,106
1,069
9.62%
10.36%

1

As defined by OMB and FCSM, unit non-response occurs when a respondent fails to respond to all required response items
(i.e., fails to fill out or return a data collection instrument); item non-response occurs when a respondent fails to respond to one
or more relevant item(s) on a survey
2
Glossary of sample cleaning rules included in Appendix E.
3
Findings in the report are based on the “Total completed surveys with Overall Index Score” (N=1,069).
4
Response rate calculation per OMB Standards and Guidelines for Statistical Surveys, section 3.2, guideline 3.2.9 (includes
undeliverables as number of noncontacted sample units known to be eligible).
5
Response rate calculation per Council of American Survey Research Organizations (CASRO) (includes number of completed
interviews with reporting units/number of eligible reporting units in sample). The American Association for Public Opinion
Research (APPOR) also uses this method for calculation and cites CASRO (APPOR Standard Definitions, 2008, pp. 34).

20

Table 3.1a.s. below shows the sample distribution and response rate for VR&E Servicing target
population.
Table 3.1a.s. Sample Distribution and Response Rates for VR&E Servicing Population
Total Servicing Population FY2015
Total records received
Duplicate records in sample file
Duplicate record history
Invalid address
Invalid values
Blanks
Do not contact
6
Total records available after cleaning
Total records selected
Undeliverable addresses
Total mailed (excludes undeliverable)
Total completed mail surveys
Total completed online surveys
Total completed surveys
7
Total completed surveys with Overall Index Score
8
Total Sample Response Rate
9
Eligible Sample Response Rate

1,305,948
29,926
594,438
218,885
719
0
39,902
426,078
60,000
4,785
55,215
8,643
N/A
8,643
8,270
13.78%
15.65%

Of the 19,484 total records received from Access, 6,615 records were purged from the sample
due to cleaning rules, such as duplicate records, invalid addresses and values, blanks, and do
not contact opt outs. From the 6,615 records purged, 6,013 records were cleaned out due to
duplicate records across VBA’s other business line surveys (i.e., duplicate record history). Of the
1,305,948 total records received from Servicing, 879,870 records were purged from the sample
due to cleaning rules, such as duplicate records, invalid addresses and values, blanks, and do
not contact opt outs. Also, from the 879,870 records that were purged, there were 594,438
records that were cleaned due to duplicate records across other business lines.

6

Glossary of sample cleaning rules included in Appendix E.
Findings in the report are based on the “Total completed surveys with Overall Index Score” (N=8,270).
8
Response rate calculation per OMB Standards and Guidelines for Statistical Surveys, section 3.2, guideline 3.2.9 (includes
undeliverables as number of noncontacted sample units known to be eligible).
9
Response rate calculation per Council of American Survey Research Organizations (CASRO) (includes number of completed
interviews with reporting units/number of eligible reporting units in sample). The American Association for Public Opinion
Research (APPOR) also uses this method for calculation and cites CASRO (APPOR Standard Definitions, 2008, pp. 34).
7

21

The purpose of this cleaning rule is to prevent respondents from being re-contacted if they
were previously selected to participate in any of VBA’s business line surveys in the past 12
months. The cleaning rule is a JDP and survey research best practice and is intended to promote
proper conduct in market research. From the total records provided, about 67% were removed
from Servicing and about 34% were removed from Access due to this cleaning rule. It is unlikely
that the cleaning rules impacted the unit non-response since JDP was able to secure the
designated number of records for both Servicing and Access.
Table 3.1b.e. Access: Weight/Person for Completed Surveys per Population
Completed Surveys

Access 2015 Population

Weight/Person

1,106

19,484

18

In Table 3.1b.e the 18 in the Weight/Person column means that every survey completed and
returned represents the views of 18 Veterans using VR&E Access benefits.
Table 3.1b.s. Servicing: Weight/Person for Completed Surveys per Population
Completed Surveys

Servicing 2015 Population

Weight/Person

8,643

1,305,948

151

In Table 3.1b.s the 151 in the Weight/Person column means that every survey completed and
returned represents the views of 151 Veterans using VR&E Servicing benefits.
To confirm the sample’s representativeness for both Access and Servicing, a comparison was
conducted among the total records provided and the records available after cleaning. The
intent of this analysis was to determine whether the cleaning rules caused the remaining
sample to vary in a meaningful way from the original sampling frame.
To confirm the sample’s representativeness, a comparison was conducted among the total
records provided and the records available after cleaning. The intent of this analysis was to
determine whether the cleaning rules caused the remaining sample to vary in a meaningful way
from the original sampling frame.
Table 3.1c.e. (Access) and Table 3.1c.s. (Servicing) indicate that characteristics such as gender,
age, and geographical region are similar among the total records provided and the records
available after cleaning. Regional U.S. state comparisons yield differences that are mostly less
than 1 percentage points, with a few exceptions in generation and region where differences
were wider (e.g., for states of FL and TX). Overall, these comparisons suggest the cleaning rules
did not significantly alter the proportion of respondent characteristics provided in the original
sampling frame.

22

Table 3.1c.e Comparing Gender, Generation, and U.S. States to Total Population

Gender
Female
Male
Generation
Boomer
Generation X
Generation Y/Z
U.S. State
AK
AL
AR
AZ
CA
CO
CT
DC
DE
FL
GA
HI
IA
ID
IL
IN
KS
KY
LA
MA
MD
ME
MI
MN
MO
MS
MT
NC
ND
NE
NH
NJ
NM

Total
Population
(%)

Records Available
(%)

% Point
Difference

18.84
81.16

18.42
81.58

-0.42
0.42

22.13
29.69
48.18

24.97
30.38
44.64

2.84
0.69
-3.53

0.62
1.88
1.12
2.36
9.23
2.92
1.08
0.21
0.14
7.68
5.43
1
0.73
0.62
1.2
1.67
0.85
1.72
1.1
0.61
2.59
0.69
2.18
0.93
1.66
0.67
0.39
4.36
0.16
0.41
0.33
1.1
0.89

0.57
1.86
0.96
2.6
9.59
2.77
0.69
0.17
0.14
9.66
6.37
0.77
0.54
0.58
1.02
1.55
0.66
1.21
0.93
0.45
1.86
0.55
2.13
1
1.43
0.44
0.33
4.64
0.13
0.31
0.24
0.93
0.75

-0.05
-0.02
-0.15
0.24
0.36
-0.15
-0.39
-0.03
0
1.98
0.95
-0.23
-0.19
-0.04
-0.18
-0.13
-0.19
-0.51
-0.17
-0.16
-0.73
-0.15
-0.05
0.07
-0.23
-0.23
-0.05
0.28
-0.03
-0.1
-0.09
-0.17
-0.13

23

Table 3.1c.e Comparing Gender, Generation, and U.S. States to Total Population (Continued)

NV
NY
OH
OK
OR
PA
RI
SC
SD
TN
TX
UT
VA
VT
WA
WI
WV
WY

Total
Population
(%)

Records Available
(%)

% Point
Difference

1.3
2.28
2.44
1.69
1.09
1.81
0.43
3.13
0.52
2.58
13.08
0.9
4.35
0.23
3.18
1.35
0.38
0.05

1.2
1.74
2.73
1.47
0.96
1.57
0.28
2.84
0.36
2.36
15.61
0.75
4.45
0.14
3.45
1.34
0.31
0.05

-0.1
-0.53
0.29
-0.21
-0.13
-0.23
-0.15
-0.29
-0.17
-0.22
2.53
-0.15
0.1
-0.09
0.27
-0.01
-0.07
0

Table 3.1c.s. Servicing: Comparing Gender, Generation, and U.S. States to Total Population
Total Population
(%)

Records Available
(%)

% Point
Difference

20.77
79.23

20.98
79.02

0.21
-0.21

30.77
33.15
36.08

31.31
33.51
35.19

0.53
0.36
-0.89

0.64
2.09
1.34
1.66
8.93

0.43
2.10
0.92
1.17
9.88

-0.22
0.02
-0.42
-0.49
0.94

Gender
Female
Male
Generation
Baby-Boomer
Gen X
Gen Y

U.S. State
AK
AL
AR
AZ
CA

24

Table 3.1c.s.Servicing: Comparing Gender, Generation, and U.S. States to Total Population (Continued)
CO

MO

3.01
1.24
0.23
0.21
8.49
4.02
0.75
1.01
0.50
1.61
2.43
0.73
2.22
0.99
1.25
2.85
0.58
3.09
1.10
1.70

3.23
0.84
0.18
0.09
13.85
5.21
0.49
0.64
0.28
1.04
2.26
0.44
1.79
0.61
0.85
2.07
0.32
3.46
0.72
1.14

0.22
-0.40
-0.05
-0.13
5.36
1.20
-0.25
-0.38
-0.21
-0.57
-0.18
-0.29
-0.44
-0.38
-0.40
-0.78
-0.26
0.36
-0.39
-0.57

MS

1.01

0.69

-0.32

0.50
3.31
0.22
0.56
0.44
1.49
0.94
0.82
2.77
3.09
1.84
1.54
2.09
0.26
2.48
0.51
2.11
11.65
0.89
3.13

0.29
3.60
0.10
0.34
0.22
0.92
0.66
0.46
1.70
3.51
1.34
1.09
1.35
0.14
2.06
0.27
1.67
18.02
0.56
2.56

-0.21
0.29
-0.11
-0.22
-0.23
-0.57
-0.28
-0.36
-1.07
0.42
-0.50
-0.44
-0.74
-0.13
-0.42
-0.24
-0.45
6.36
-0.33
-0.57

CT
DC
DE
FL
GA
HI
IA
ID
IL
IN
KS
KY
LA
MA
MD
ME
MI
MN

MT
NC
ND
NE
NH
NJ
NM
NV
NY
OH
OK
OR
PA
RI
SC
SD
TN
TX
UT
VA

25

Table 3.1c.s.Servicing: Comparing Gender, Generation, and U.S. States to Total Population (Continued)
VT

0.25
2.82
1.08
0.57
0.10

WA
WI
WV
WY

0.10
2.81
0.71
0.36
0.09

-0.16
-0.01
-0.37
-0.20
-0.01

3.2 Missing Data Patterns and Mechanisms
In accordance with the OMB “Standards and Guidelines for Statistical Surveys,” Guidelines 3.2.9
and 3.2.11, an investigation of missing data patterns was performed on the 1,677 total surveys
received for Access and the 5,898 total surveys received for Servicing. In order to assess the
distribution of missing data, a procedure was performed to process missing values involving
iterative multiple imputation chains using expectation–maximization (MCMC) algorithms and
divide these into distribution interval groupings (Pierchala, Carl E. (2001)). This was done on the
key measures of the Overall Satisfaction Index score (see Appendix A for calculation) and
Advocacy ratings related to Veterans’ likelihood to recommend VA benefits.
As shown in Tables 3.2.e. and 3.2.s. for Access and Servicing, respectively, there were no
indications of unusual patterns for missing data. For more discussion of missing data
mechanisms (MCAR, MAR, and MNAR), please see Appendix A.
Table 3.2.e. Access: Missing Data Patterns in Satisfaction and Advocacy (0 = missing, 1 = data)
Group Means

Group
1
2
3
4

Overall
Satisfaction

Likelihood
to inform
others

Freq

Percent

OSAT
Index

Age

% Male

0
0
1
1

0
1
0
1

45
16
8
1000

4%
1%
1%
94%

655
591
535
598

50
49
51
46

76%
75%
88%
80%

Table 3.2.s. Servicing: Missing Data Patterns in Satisfaction and Advocacy (0 = missing, 1 = data)
Group Means

Group
1
2
3
4

Overall
Satisfaction

Likelihood
to inform
others

Freq

Percent

OSAT
Index

Age

% Male

0
0
1
1

0
1
0
1

151
72
384
7254

2%
.9%
5%
92%

725
686
716
717

48
47
51
47

79%
68%
85%
78%

26

3.3 Margin of Error
The margin of error expresses the maximum expected difference between the true population
parameter and a sample estimate of that parameter. It is often used to indicate the accuracy of
survey results. The larger the margin of error around an estimated value, the less accurate the
estimated value will be. Larger samples are more likely to yield results close to the true
population quantity and thus have smaller margins of error than smaller samples.
Based on a sample of 1,106 Veterans, the Overall Satisfaction Index score in the Access study is
600 with a margin of error of 14 index points at the 95% confidence level. This indicates that if
the survey were repeated many times with different samples, the true mean Overall
Satisfaction Index score would fall within 14 index points 95% of the time.
Table 3.3.e. below demonstrates relative decreases in margin of error as the study sample size
increases. A 20% response rate (2,135 completes) would be associated with a margin of error of
10 index points, similar to the margin of error for a 30% response rate (3,203 completes).
Results from this analysis indicate the Overall Satisfaction Index (OSAT) score calculated from
the Access study is an accurate measurement of the true population mean.
Table 3.3.e. Access: Margin of Error for Larger Sample Sizes
Sample

Response
Rate

Completes
(N)

OSAT
(mean)

Standard
Deviation

Standard
Error

Margin of error
(95% confidence
interval)

10,677
10,677
10,677
10,677
10,677
10,677
10,677

10.36%
20%
30%
40%
50%
60%
80%

1,106
2,135
3,203
4,271
5,339
6,406
8,542

600
600
600
600
600
600
600

233
233
233
233
233
233
233

7.0
5.0
4.1
3.6
3.2
2.9
2.5

14
10
8
7
6
6
5

Based on a sample of 8,643 Veterans, the Overall Satisfaction Index score in the Servicing study
is 715 with a margin of error of 5 index points at the 95% confidence level. This indicates that if
the survey were repeated many times with different samples, the true mean Overall
Satisfaction Index score would fall within 6 index points 95% of the time.
Table 3.3.s. below demonstrates relative decreases in margin of error as the study sample size
increases. A 20% response rate (11,043 completes) would be associated with a margin of error
of 4 index points, similar to the margin of error for a 30% response rate (16,565 completes).
Results from this analysis indicate the Overall Satisfaction Index (OSAT) score calculated from
the Servicing study is an accurate measurement of the true population mean.

27

Table 3.3.s. Servicing: Margin of Error for Larger Sample Sizes
Sample

Response
Rate

Completes
(N)

OSAT
(mean)

Standard
Deviation

Standard
Error

Margin of error
(95% confidence
interval)

55,215
55,215
55,215
55,215
55,215
55,215
55,215

15.65%
20%
30%
40%
50%
60%
80%

8,643
11,043
16,565
22,086
27,608
33,129
44,172

715
715
715
715
715
715
715

229
229
229
229
229
229
229

2.5
2.2
1.8
1.5
1.4
1.3
1.1

5
4
3
3
3
2
2

In the margin of error analysis noted on the previous page and in subsequent analyses included
in this report, the Overall Satisfaction Index score is the main dependent variable and is the
basis for the analysis. The Overall Satisfaction Index score is the survey metric that VBA utilizes
to measure customer satisfaction and benchmark performance against other industries. It is the
primary measurement in all reports. The Overall Satisfaction Index model encompasses all
aspects of the customer experience,10 and can therefore be used as a reliable indicator for the
presence or absence of respondent bias in the survey results as a whole. For these reasons, the
Overall Satisfaction Index score is used as the main dependent variable in the margin of error
analysis and subsequent t-test analyses included in this report.

3.3.1 Sampling Distribution
Respondent characteristics such as gender and age were compared to that of the total sample
to determine whether respondents and non-respondents differed on key variables of interest.
Compared to the population of all eligible respondents (Access 11,118), survey respondents
demonstrate the same gender characteristics. For Access, Table 3.3.1.e. below illustrates that
20% of survey respondents were female and 80% were male, similar to the total sample
population. The distribution of age shows that survey respondents tend to be older.

10

Explanation of J.D. Power Index Model Calculation included in Methodology.

28

Table 3.3.1.e. Access: Comparing Gender and Age of Survey Respondents to Total Sample

Gender
Female
Male
Age Generation
Baby Boomer
Generation X
Generation Y/Z

Respondents
(%)

Sample
Size (N)

Total
Sample (%)

Sample
Size (N)

% Point
Difference

20
80

226
880

18
82

2046
9072

-2
2

41
33
26

457
365
284

25
30
45

2770
3380
4968

-16
-3
19

Compared to the population of all eligible respondents (Servicing 60,000), survey respondents
demonstrate the same gender characteristics. For Servicing, Table 3.3.1.s. below illustrates that
21% of survey respondents were female and 79% were male, similar to the total sample
population. The distribution of age shows that survey respondents tend to be older.
Table 3.3.1.s. Servicing: Comparing Gender and Age of Survey Respondents to the Total
Sample

Gender
Female
Male
Age Generation
Baby Boomer
Generation X
Generation Y/Z

Respondents
(%)

Sample
Size (N)

Total
Sample (%)

Sample
Size (N)

% Point
Difference

21
79

1,763
6,455

20
80

12,251
47,749

-1
1

46
30
24

3766
2505
1947

30
33
37

17,907
19,645
22,448

-16
2
14

3.3.2 Distribution of Overall Satisfaction Index Scores
Following the comparison of sampling distributions, a comparison of Overall Satisfaction scores
was conducted to determine whether differences in age and gender among respondents
correlate with differences in the Overall Satisfaction Index.
For Access, Table 3.3.2.e. below indicates differences in Overall Satisfaction scores are notable
between gender groups. On average, females tend to rate their experience higher than males
(607 vs. 598, respectively). Comparing age groups shows that Generation Y/Z had much higher
satisfaction than Generation X and Boomers.

29

Table 3.3.2.e. Access: Overall Satisfaction Scores for Gender and Age Groups
Characteristics
Gender
Female
Male
Age Generation
Baby Boomer
Generation X
Generation Y/Z

OSAT (mean)

Standard Deviation

Sample Size (N)

607
598

227
234

221
848

569
610
636

234
226
234

432
360
277

For Servicing, Table 3.3.2.s. below indicates differences in Overall Satisfaction scores are
notable between gender groups. On average, females tend to rate their experience higher than
males (724 vs. 715, respectively). Comparing age groups shows that Generation X had much
higher satisfaction than Generation Y/Z.
Table 3.3.2.s. Servicing: Overall Satisfaction Scores for Gender and Age Groups
Characteristics
Gender
Female
Male
Age Generation
Baby Boomer
Generation X
Generation Y/Z

OSAT (mean)

Standard Deviation

Sample Size (N)

724
715

223
230

1,697
6,164

713
728
709

228
225
235

3,528
2,435
1,898

3.3.3 Analysis for Demographic Differences
T-test analyses were conducted to determine whether differences in demographic groups
produced statistical differences in Overall Satisfaction (OSAT) scores. T-tests are typically used
to determine whether or not the difference between two groups’ averages most likely reflect a
meaningful difference in the population from which the groups were sampled.
For Access, gender differences were not statistically significantly different, whereas war
participation was statistically significant where we found that OEF/OIF respondents had lower
overall satisfaction:

30

Table 3.3.3a.e. Access: T-Test Analysis for Pairs of Characteristics in Veterans’ Satisfaction

Characteristics
Gender
Female vs. Male
War Participation
OEF/OIF vs. All Others

T-Test Statistic

Statistical Difference
(95% confidence level)

.50

No

-2.26

Yes

For Servicing, the differences for gender were not significant, but war participation was
statistically significant and the data indicates that OEF/OIF respondents had lower overall
satisfaction:
Table 3.3.3a.s. Servicing: T-Test Analysis for Pairs of Characteristics in Veterans’ Satisfaction
Characteristics
Gender
Female vs. Male
War Participation
OEF/OIF vs. All Others

T-Test Statistic

Statistical Difference
(95% confidence level)

1.46

No

-2.19

Yes

Analyses of Variance (ANOVA) were conducted to determine whether differences in
demographic groups produced statistical differences in Overall Satisfaction scores. ANOVAs are
typically used to determine whether or not the difference between three or more groups’
averages most likely reflect a meaningful difference in the population from which the groups
were sampled.
For Access, differences in Overall Satisfaction by age generation were significant (F = 7.68, pvalue < .0005) and the data reflects that Generation Y/Z had the highest satisfaction:
Table 3.3.3b.e. Access: Overall Satisfaction for Age Generation
Generation

OSAT (mean)

Sample Size (N)

Baby Boomer
Generation X
Generation Y/Z

569
610
636

432
360
277

31

For Access, differences in Overall Satisfaction by region were not significant (F = .98, p-value
.399):
Table 3.3.3c.e. Access: Overall Satisfaction by Region
Regions

OSAT (mean)

Sample Size (N)

Midwest
Northeast
South
West

606
558
601
608

134
80
598
257

For Access, racial differences in Overall Satisfaction were not significant (F = 1.71, p-value = .16):
Table 3.3.3d.e. Access: Overall Satisfaction for Race
Race

OSAT (mean)

Sample Size (N)

Asian
Black
Other
White

612
600
568
611

46
294
201
528

For Access, differences in Overall Satisfaction by branch of service were not significant (F =
1.87, p-value = . 11):
Table 3.3.3e.e. Access: Overall Satisfaction for Military Service Branch
Military Service
Branches

OSAT (mean)

Sample Size (N)

Air Force
Army
Marines
Navy
Other

627
605
591
598
547

149
539
104
178
99

For Access, differences in Overall Satisfaction by days of active service were significant (F =
2.69, p-value < .05). The data indicates that the respondents with active service of 1001-2000
days or more than 4001 days had higher satisfaction levels:

32

Table 3.3.3f.e. Access: Overall Satisfaction for Days of Active Service
Days of Active
Service
1000 days or less
1001-2000 days
2001-4000 days
4001 days or more

OSAT (mean)
562
613
605
612

Sample Size (N)
231
265
221
352

For Access, differences in Overall Satisfaction by case status were significant (F = 146, p-value <
.0001) such that respondents with Employ Rehab status had the highest satisfaction:
Table 3.3.3g.e. Access: Overall Satisfaction for Case Status
Case Status
Discontinued
Employ Rehab
Other

OSAT (mean)
511
736
671

Sample Size (N)
629
388
52

For Servicing, age generation differences in Overall Satisfaction were significant (F = 4.26, pvalue < .02) such that Generation X respondents had the highest satisfaction:
Table 3.3.3b.s. Servicing: Overall Satisfaction for Generation
Generation

OSAT (mean)

Sample Size (N)

Baby Boomer
Generation X
Generation Y/Z

713
728
709

3,528
2,435
1,898

For Servicing, region differences in Overall Satisfaction were significant (F = 15.41, p-value <
.0001) such that respondents from the Midwest and Northeast had the highest satisfaction:
Table 3.3.3c.s. Servicing: Overall Satisfaction by Region
Regions

OSAT (mean)

Sample Size (N)

Midwest
Northeast
South
West

740
741
701
708

1,599
1,133
3,032
2,097

For Servicing, racial differences in Overall Satisfaction were significant (F = 3.65, p-value < .02)
such that White respondents had the highest satisfaction:

33

Table 3.3.3d.s. Servicing: Overall Satisfaction for Race
Race

OSAT (mean)

Sample Size (N)

Asian
Black
Other
White

716
710
703
724

366
1,940
1,357
4,198

For Servicing, differences in Overall Satisfaction by branch of service were significant (F = 3.08,
p-value < .0002) such that Air Force respondents had the highest satisfaction:
Table 3.3.3e.s. Servicing: Overall Satisfaction for Military Service Branch
Military Service

OSAT (mean)

Sample Size (N)

Air Force
Army
Marines
Navy
Other

729
719
713
714
691

1,099
3,877
855
1,377
653

For Servicing, differences in Overall Satisfaction by days of active service were significant (F =
24.49, p-value <.0001) such that respondents with 4,001 days or more of active service had the
highest satisfaction:
Table 3.3.3h.s. Servicing: Overall Satisfaction for Days of Active Service
Days of Active
Service
1,000 days or less
1,001-2,000 days
2,001-4,000 days
4,001 days or more

OSAT (mean)

Sample Size (N)

699
698
713
749

1,932
2,039
1,487
2,403

For Servicing, differences in Overall Satisfaction by case status were significant (F = 23.39, pvalue < .0001) such that respondents with Employ Rehab status had the highest satisfaction:
Table 3.3.3g.e. Access: Overall Satisfaction for Case Status
Case Status
Discontinued
Employ Rehab
Other

OSAT (mean)
633
726
699

Sample Size (N)
182
5,600
2,079

34

3.3.4 Data Imputation Analysis for Demographic Differences
A pairwise comparison t-test analysis was done to evaluate whether data imputation for
missing values across significant demographic differences shown in section 3.3.3 would impact
Overall Satisfaction Index scores. This analysis included survey raking across demographic
differences as one level of comparison.
These results (Tables 3.3.4a.e. and 3.3.4a.s.) show that there were no significant differences
between the non-imputed mean and the imputed mean of the satisfaction index across
demographics, sample sizes, or survey raked values. We want to highlight that after statistical
adjustment for the differences found between respondents and non-respondents reported
earlier, there were no differences in overall satisfaction levels. These results support the
conclusion that the survey’s findings for Veterans’ Overall Satisfaction scores are accurate.
Table 3.3.4a.e. Access: Comparison of Imputed vs. Non-Imputed on Veterans’ Satisfaction
T-Tests on Imputed vs. Non-Imputed raked for Demographic Differences

Overall Satisfaction Index
(100 - 1000 range)
Imputed demographics
(1,069 final sample size)
Imputed survey-raked demographics
(1,069 final sample size)
Imputed survey-raked demographics
(1,106 total respondents)

mean
(imputed)

mean (nonimputed)

t-statistic

p-value

587.70

600.17

1.22

0.22

599.84

609.33

0.93

0.35

599.61

610.64

1.09

0.27

Note: Non-imputed is based on the 1,069 final cleaned sample size used in this report.

Table 3.3.4a.s. Servicing: Comparison of Imputed vs. Non-Imputed on Veterans’ Satisfaction
T-Tests on Imputed vs. Non-Imputed raked for Demographic Differences

Overall Satisfaction Index
(100 - 1000 range)
Imputed demographics
(7,861 final sample size)
Imputed survey-raked demographics
(7,861 final sample size)
Imputed survey-raked demographics
(8,218 total respondents)

mean
(imputed)

mean (nonimputed)

t-statistic

p-value

716.56

716.62

0.02

0.99

715.61

715.53

-0.02

0.98

715.46

715.63

0.05

0.96

Note: Non-imputed is based on the 7,861 final cleaned sample size used in this report.

35

Survey Raking for Sample Weights to Adjust for Differences and Compare Overall Satisfaction
and Advocacy Ratings
The procedure known as raking adjusts a set of data so that its marginal totals match specified
control totals on a specified set of variables. The term “raking” suggests an analogy with the
process of smoothing the soil in a garden plot by alternately working it back and forth with a
rake in two perpendicular directions (Izrael and Battaglia (2004)).
Survey raking is an iterative sample-balancing algorithm-based technique that provides sample
weighting convergence across multiple variables and multiple categories (see Battaglia, Izrael,
Hoaglin, and Frankel (2009)).
In keeping with OMB “Standards and Guidelines for Statistical Surveys,” Guidelines 3.2.12 and
3.2.13, J.D. Power selected the best statistical method to simultaneously adjust for multiple
differences between groups by applying a survey raking procedure (see Anderson, L., and R.D.
Fricker, Jr. (2015)).
The JDP raking procedure is a proprietary improved version based on the excellent methods
initially developed by Izrael and Battaglia (2000, 2004) and Battaglia, Izrael, Hoaglin, and
Frankel (2004). JDP raking improvements are primarily related to better handling of low cell
values during iterative convergence processing. For this analysis, 50 iterations were set
(although less were needed) to converge on the best sample weights (.2 estimation margin) to
simultaneously adjust for non-response bias in age, race, region, and war (service era)
demographic categories. For additional background about survey raking methodologies, see
Wallace and Rust (1996).
The estimated population distributions are used as convergence targets. In this case, the data
set of all eligible respondents for Access (14,000) and Servicing (60,000) were used as the
estimated population to derive sample weightings for the Access survey respondents (1,069)
and the Servicing survey respondents (7,861).
In accordance with OMB “Standards and Guidelines for Statistical Surveys,” Guideline 3.2.13, a
series of t-tests were conducted to determine whether non-response bias in demographic areas
produced statistical differences in Overall Satisfaction scores and Advocacy ratings. Typically, ttests are used to determine whether differences between two groups’ averages and variances
reflect a meaningful difference in the population. The sample weightings derived from the
survey raking procedure were included in the t-tests to equalize the survey respondent
differences with non-respondents.
For Access, there were no significant differences in Overall Satisfaction or Advocacy levels when
the data was adjusted for demographic differences between survey respondents and nonrespondents. These results support the conclusion that the survey’s findings for Veterans’
Overall Satisfaction scores are accurate.

36

Table 3.3.4b.e. Access: Overall Satisfaction and Advocacy for Survey Respondents Unweighted
and Weighted
Analysis of Survey Respondent Scores with Weighted Adjustment for Non-Response Bias
Mean
Standard
Standard
Mean
tpRating Measure
(Unweight
Deviation
Deviation
(Weighted)
statistic value
ed)
(Unweighted)
(Weighted)
Overall
Satisfaction Index
(100 - 1000 range)
Likelihood to
inform others
about VA benefits
(rating 1 - 4)

600

609

233

233

-0.91

0.36

3.48

3.51

0.77

0.76

-0.71

0.48

For Servicing, there were no significant differences in Overall Satisfaction or Advocacy levels
when the data was adjusted for demographic differences between survey respondents and
non-respondents. These results support the conclusion that the survey’s findings for Veterans’
Overall Satisfaction scores are accurate.
Table 3.3.4b.s. Servicing: Overall Satisfaction and Advocacy for Survey Respondents
Unweighted and Weighted
Analysis of Survey Respondent Scores with Weighted Adjustment for Non-Response Bias
Mean
Standard
Standard
Mean
tpRating Measure
(Unweight
Deviation
Deviation
(Weighted)
statistic value
ed)
(Unweighted)
(Weighted)
Overall
Satisfaction Index
(100 - 1000 range)
Likelihood to
inform others
about VA benefits
(rating 1 - 4)

717

716

229

230

0.30

0.77

3.64

3.64

0.63

0.62

-0.83

0.41

37

Findings
Results from the non-response bias analysis indicate that the Overall Satisfaction Index score
and the Advocacy rating from the VR&E Access and Servicing studies reflect the experience of
all Veterans who applied, were found entitled and pursued the program or found not entitled
and did not pursue the program, have been rehabilitated, reached maximum rehabilitation
gain, and Veterans still pursuing benefits.

Sample Cleaning: Initial comparisons on age, gender, and geographical characteristics
between the total records provided and the records available after cleaning (see Survey Yield,
Section 3.1) suggest the sample utilized in the study exhibits similar characteristics as the total
sample. Additional comparisons (see Margin of Error and Sampling Distribution, Section 3.3,)
suggest the sample cleaning rules did not impact the sample’s representativeness and, thus, the
results are conclusive.

Non-Response Bias Analysis: Results from the non-response bias analysis did show group
differences for age, gender, region, race, military branch, days of service, and war participation
between survey respondents and non-respondents. After correcting for these differences using
a recommended sample-balancing survey raking method to derive sample weights (see Margin
of Error, Section 3.3.4 Data Imputation Analysis for Demographic Variables), there were no
differences found in Veterans’ Overall Satisfaction and Advocacy (likelihood to inform others
about VA benefits) between weighted and unweighted survey respondents.

Item Response Rate Calculations: Results from the survey item response rate calculations
show high item response rates, with none falling below OMB guidelines (see Appendix B for
Item Response Rates). According to OMB Guideline 3.2.10, given that neither study had a
response rate lower than 70%, a non-response bias analysis was not necessary at the item level.
The research and approach taken by J.D. Power are in accordance with sound market research
and current best practices from the American Association for Public Opinion Research (AAPOR)
regarding response rate recommendations: “Results that show the least bias have turned out,
in some cases, to come from surveys with less than optimal response rates. Experimental
comparisons have also revealed few significant differences between estimates from surveys
with low response rates and short field periods and surveys with high response rates and long
field periods.” See AAPOR “Response Rates – An Overview” (2015) and Special Issue of Public
Opinion Quarterly "Nonresponse Bias in Household Surveys" (Singer, 2006).

38

Conclusion
The Overall Satisfaction Index score and Advocacy rating (likelihood to inform others about VA
benefits) are not impacted in any meaningful way by non-response bias. This analysis confirms
that the data collected during Fiscal Year 2015 is valid.
The FY15 Voice of the Veteran Line of Business Tracking Satisfaction Study data for the VR&E
Access and Servicing surveys can be used to infer reliable Overall Satisfaction scores and
Advocacy ratings. The Overall Satisfaction Index score reflects the experience of all Veterans
who applied, were found entitled and pursued the program or found not entitled and did not
pursue the program, have been rehabilitated; reached maximum rehabilitation gain, and
Veterans still pursuing benefits.
The sample utilized in the study exhibits similar characteristics for age, gender, and geography
as the total sample provided. This indicates the sample cleaning rules did not impact the
sample’s representativeness.
While the results from the non-response bias analysis did show group differences on
demographic characteristics between survey respondents and non-respondents, there were no
differences found in Veterans’ Overall Satisfaction scores and Advocacy ratings between
weighted and unweighted survey respondents. This was evaluated after correcting for these
differences using a recommended sample-balancing survey raking method to derive sample
weights. JDP conducted all necessary statistical tests in accordance with OMB standards.
J.D. Power certifies the results contained within this report.

39

References
Anderson, L., and R.D. Fricker, Jr. (2015). “Raking: An Important and Often Overlooked Survey Analysis
Tool.” Phalanxhttp://faculty.nps.edu/rdfricke/docs/Analysis%20process_v4.pdf
American Association for Public Opinion Research (2008). Standard Definitions: Final Disposition of
Case Codes and Outcome Rates for Surveys. Ann Arbor, Michigan: AAPOR.
(http://www.aapor.org/AAPORKentico/AAPOR_Main/media/MainSiteFiles/Standard_Definitions_07
_08_Final.pdf).
American Association for Public Opinion Research (2015). “Response Rates – An Overview.”
http://www.aapor.org/AAPORKentico/Education-Resources/For-Researchers/Poll-SurveyFAQ/Response-Rates-An-Overview.aspx
Battaglia, Michael P., Izrael, David, Hoaglin, David C., and Frankel, Martin R. (2004). “To Rake or Not To
Rake Is Not the Question Anymore with the Enhanced Raking Macro.” Proceedings of the 29th Annual
SAS Users Group International Conference, Paper 207.
Battaglia, Michael P., Izrael, David, Hoaglin, David C., and Frankel, Martin R. (2009). “Practical
Considerations in Raking Survey Data.” Survey Practice, Vol 2, No. 5.
Baum, Herbert M., Ph.D.; Chandonnet, Anna M.A.; Fentress, Jack M.S., M.B.A.; and Rasinowich, Colleen,
B.A. (2012). “Mixed-Mode Methods for Conducting Survey Research.” Data Recognition Corporation.
http://www.datarecognitioncorp.com/survey-services/Documents/Mixed-Mode-Methods-forConducting-Survey-Research.pdf
Dillman, D. A. and JDP (2015). “Conference call discussion on non-response bias, avoidance methods, and
post-hoc sample weighting.” Conference call between Dr. Dillman and JDP (Greg Truex, Jay Meyers,
Ph.D., Lee Quintanar, Ph.D.), May 20, 2015 (2 p.m. PDT).
Dillman, D. A. (2014). Internet, Phone, Mail and Mixed-Mode Surveys: The Tailored Design Method.
Fourth Edition. John Wiley & Sons, Inc: New York.
Ellis, J. M. (2000). “Estimating the Number of Eligible Respondents for a Telephone Survey of LowIncidence Households.” Paper presented at the annual meeting of the American Association for
Public Opinion Research, Portland, OR, May 21.
Federal Committee on Statistical Methodology’s “Statistical Policy Working Paper 31, Measuring and
Reporting Sources of Error in Surveys.” (2001). Washington, D.C.
Izrael, David, Hoaglin, David C., and Battaglia, Michael P. (2000). “A SAS Macro for Balancing a Weighted
Sample.” Proceedings of the Twenty-Fifth Annual SAS Users Group International Conference, Paper
275.
Izrael, David, Hoaglin, David C., and Battaglia, Michael P. (2004). “Tips and Tricks for Raking Survey Data
(a.k.a. Sample Balancing).” Proceedings of the 2004 American Association for Public Opinion
Research (AAPOR) Conference.

40

Malhotra, N.K, and Birks, D.F. (2007). Marketing Research: An Applied Approach. Third edition. Prentice
Hall/Financial Times: England.
Pierchala, Carl E. (2001). “PROC MI® as the Basis for a Macro for the Study of Patterns of Missing Data.”
Northeast SAS Users Group. http://www.lexjansen.com/nesug/nesug03/st/st009.pdf
Singer, E. (2006). Special Issue: Nonresponse Bias in Household Surveys. Public Opinion Quarterly, Vol
70, Issue 5.
U.S. Office of Management and Budget. (1990). "Survey Coverage." Statistical Policy Working Paper 17.
Washington, D.C.
U.S. Office of Management and Budget Publication. (January 2006). “When Designing Surveys for
Information Collections.” The Office of Management and Budget, 725 17th Street, NW. Washington,
D.C. 20503 USA
U.S. Office of Management and Budget Publication. (September 2006). “Standards and Guidelines for
Statistical Surveys.” The Office of Management and Budget, 725 17th Street, NW. Washington, D.C.
20503 USA
U.S. Office of Management and Budget Publication. (2008). VBA VR&E OMB - Part B Supporting
statement for “Collections of Information Employing Statistical Methods.” Washington, D.C.
Vogt, W. Paul, Vogt, Elaine R., Gardner, Dianne C., and Haeffele, Lynne M. (2014). Selecting the Right
Analyses for Your Data - Quantitative, Qualitative, and Mixed Method. Guilford Press. New York, NY.
Wallace, Leslie and Rust, Keith (1996). A Comparison of Raking and Poststratification Using 1994 NAEP
Data. Leslie Wallace, West Inc. 584-589.

41

Appendix A
Missing Data Patterns and Mechanisms
An excellent discussion of missing data patterns, mechanisms, and research analysis methods is
provided in Vogt, W. Paul, Vogt, Elaine R., Gardner, Dianne C., and Haeffele, Lynne M. (2014).
An overview of the missing data types and issues is described below.
Understanding the reasons data is missing can help when analyzing the remaining data. If
values are missing at random, the data sample may still be representative of the population;
however, if the values are missing systematically, analysis may be harder.






Missing completely at random. Values in a data set are missing completely at random
(MCAR) if the events that lead to any particular data item being missing are independent
both of observable variables and of unobservable parameters of interest, and occur entirely
at random. When data are MCAR, the analyses performed on the data are unbiased;
however, data are rarely MCAR.
Missing at random. Missing at random (MAR) is an alternative and occurs when the
missingness is related to a particular variable, but is not related to the value of the variable
that has missing data. An example of this is accidentally omitting an answer on a
questionnaire.
Missing not at random. Missing not at random (MNAR) is data that is missing for a specific
reason (i.e., the value of the variable that is missing is related to the reason it is missing). An
example of this is if a certain question on a questionnaire tends to be skipped deliberately
by participants with certain characteristics. Graphical models can be used to describe the
missing data mechanism in detail.

While it is clear that MNAR can introduce statistical bias, there is no definitive test (see Vogt et
al. (2014)). It is also clear that MCAR is rarely evident in research data and most tests of it will
fail. However, MAR is fully acceptable for valid statistical analyses (Vogt et al., 2014). MAR is
essentially “missing partially at random,” whereby the intra-group missingness remains random
despite some differences between group tendencies. Graphical data representations are the
typical tool used in assessment as described above and in Pierchala, Carl E. (2001).
See Section 3.2 Missing Data Patterns and Mechanisms for findings specific to VR&E’s data.

42

Appendix B
Item Response Rates
In accordance with OMB “Standards and Guidelines for Statistical Surveys,” Section 3.2,
Guidelines 3.2.6-3.2.7, the item response rate was calculated as the ratio of the number of
respondents for whom an in-scope response was obtained to the number of respondents who
were asked to answer that item. The number asked to answer an item is the number of unitlevel respondents minus the number of respondents with a valid skip pattern. In addition to
item response rate, total item response rate was calculated as the product of the overall unit
response rate and the item response rate for each item. The purpose of these calculations is to
assess the item non-response, which occurs when one or more survey items are left blank in an
otherwise completed questionnaire. Tables B1.e. and B1.s. display the item and total item
response rates for these surveys.
The OMB “Standards and Guidelines for Statistical Surveys,” Guideline 3.2.10 states that an
item non-response analysis should be conducted for items with an item response rate of less
than 70%. Since none of the survey item response rates fall below 70%, for Access or Servicing,
an item-level analysis of non-response bias was not necessary. The Access item response rates
range from 84% to 100% with a 96% average, while Servicing response rates range from 82% to
100% with a 97% average.
Table B1.e. Access: Item and Total Item Response Rate11

Question
Number
1
2
3
4
5
6
7a
7b
7c
7d
7e
7f
8

Item
Response
Rate
95%
99%
99%
94%
95%
97%
99%
99%
99%
98%
96%
98%
99%

Unit Response Rate
9%
9%
9%
9%
9%
9%
10%
9%
9%
9%
9%
9%
10%

11

Open capture question for additional comments about your experience and email opt in questions display “N/A” and were
not included in item and total item response rate calculations

43

Table B1.e. Access: Item and Total Item Response Rate
(Continued)
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34a
34b
34c
34d
35a
35b
35c
35d
35e
35f
36
37
38a
38b
38c
38d
39
40
41

96%
98%
95%
99%
99%
97%
94%
97%
96%
98%
94%
91%
84%
91%
94%
99%
96%
94%
92%
91%
88%
99%
N/A
N/A
N/A
90%
90%
90%
89%
97%
97%
97%
98%
97%
98%
N/A
87%
94%
93%
92%
86%
99%
94%
95%

9%
9%
9%
9%
10%
9%
9%
9%
9%
9%
9%
9%
8%
9%
9%
9%
9%
9%
9%
9%
8%
10%
N/A
N/A
N/A
9%
9%
9%
9%
9%
9%
9%
9%
9%
9%
N/A
8%
9%
9%
9%
8%
9%
9%
9%

44

Table B1.e. Access: Item and Total Item Response Rate
(Continued)
42
43
44
45
46
47
48
49a.
49b.
49c.
49d.
49e.
49f.
49g.
49h.
49i.
49j.
49k.
49l.
49m.
49n.
49o.
49p.
49q.
49r.
49s.
50
50a.
50b.
50c.
50d.
51

N/A
N/A
96%
98%
97%
96%
98%
100%
100%
100%
100%
100%
100%
100%
96%
N/A
100%
100%
100%
100%
100%
100%
100%
100%
100%
100%
100%
100%
100%
100%
100%
N/A

N/A
N/A
9%
9%
9%
9%
9%
10%
10%
10%
10%
10%
10%
10%
9%
N/A
10%
10%
10%
10%
10%
10%
10%
10%
10%
10%
10%
10%
10%
10%
10%
N/A

45

Table B1.s. Servicing: Item and Total Item Response Rate12
Question
Number
1
2
3
4
5a
5b
5c
5d
5e
5f
6
7
8
9
10
11
12
13
14
15
16
17
18
19a
19b
19c
19d
19e
19f
20
21
22
23
24
25
26a

Item Response
Rate
92%
99%
98%
98%
98%
98%
98%
97%
97%
97%
100%
94%
98%
90%
99%
95%
96%
100%
100%
100%
98%
100%
98%
98%
97%
97%
97%
95%
97%
N/A
99%
95%
97%
96%
96%
95%

Unit Response
Rate
13%
14%
14%
13%
13%
14%
13%
13%
13%
13%
14%
13%
13%
12%
14%
13%
13%
14%
14%
14%
14%
14%
14%
14%
13%
13%
13%
13%
13%
N/A
14%
13%
13%
13%
13%
13%

12

Open capture questions for additional comments about your experience and items unclear in letter and e-mail opt in
questions display “N/A” and were not included in item and total item response rate calculations

46

Table B1.s. Servicing: Item and Total Item Response Rate
(Continued)
26b
26c
26d
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

82%
92%
93%
97%
97%
93%
96%
99%
98%
99%
96%
100%
97%
99%
97%
98%
99%
99%
94%
N/A
N/A
N/A

11%
13%
13%
13%
13%
13%
13%
14%
14%
14%
13%
14%
13%
14%
13%
14%
14%
14%
13%
N/A
N/A
N/A

In the item response rate calculations above, J.D. Power considered blanks as non-response for
mail returns and “don’t know” selections in addition to blanks as non-response for online
returns. “Don’t know” selections are included as non-response for online returns since
respondents are forced to select a response in the online survey.
Similarly, “N/A” responses were also included as non-response for rating questions in online
returns. For respondents taking the survey online, the respondent must answer each question
before proceeding to the next question in the survey, “Not Applicable” or “N/A” could either
mean that the respondent was answering “N/A” to the question or did not wish to answer it.
Therefore, this response option was included as non-response.

47

Appendix C
Study Overview
1.1 Study Background
The Voice of the Veteran Satisfaction Initiative tracks Veteran satisfaction with the benefits and
services received from VBA. The VOV Tracking Satisfaction Research Study is ongoing survey
research tracking Veteran satisfaction with VBA’s lines of business: Compensation, Pension,
Education, Vocational Rehabilitation & Employment (VR&E), and Loan Guaranty (LGY).
As part of Executive Order 13571 Streamlining Service Delivery and Improving Customer
Service, agencies that provide significant services directly to the public to identify and survey
customers, establish service standards and track performance against those standards, and
benchmark customer service against the best in businessThis This program enables VBA to
understand what is important to Veterans relative to benefits received and services provided.
This program provides timely and actionable Veteran feedback on how well VBA is providing
services. Insights from this program identify opportunities for improvement and measure the
impact of improvement initiatives as well as continuously measure performance outcomes.
The VR&E survey instrument is regarding Veteran satisfaction with enrollment and receipt of
benefits issued by VBA. In FY15, fielding occurred continuously on a monthly basis for Access
and annually for Servicing. Surveys remained open in field until the end of each quarter. The
returns from surveys received after fielding closed at the end of a quarter were counted in the
next quarter’s number of returns.
Total Mailouts
Per Year

Target
Number of
Completes

Survey

Methodology

Fielding
Frequency

Access

Mail and Online

Annually

14,000

4,200

Mail Only

Monthly

60,000

18,000

Servicing

1.2 Methodology
Respondents from the Access study had the option of completing the survey on paper or
online, and received two separate mailings. In the first mailing, a postcard was sent introducing
the study, which included an online survey link and a unique access code login for the online
survey. In the second mailing, the respondent was sent a Survey Package, which included a
cover letter with the online survey link and login, a paper survey, and a business reply envelope.
The second survey was cleaned to exclude anyone who completed the survey at least one week
prior to the mailing.
48

Respondents from the Servicing study only had the option to complete the survey on paper,
and received two separate mailings. The first Survey Package mailed included a cover letter
introducing the study, a paper survey, and a business reply envelope. The second Survey
Package mailed included a cover letter, a paper survey, and a business reply envelope. The
second survey was cleaned to exclude anyone who completed the survey at least one week
prior to the mailing.
Sample Population Definition
The targeted populations were identified by VR&E. For Servicing, the target population is
defined as Veterans who have entered and been enrolled in one of the five tracks for at least 60
days. These individuals include Veterans who have been rehabilitated; Veterans who have
reached maximum rehabilitation gain and could not proceed in the program; and Veterans still
pursuing benefits.
For Access, the target population is defined as Veterans who had an initial meeting with their
VR&E counselor and were granted a decision regarding their entitlement within the past 30
days. These individuals include those who applied and showed up for the initial appointment
and were either found entitled and pursued the program or found not entitled and did not
pursue the program.
Sample File Generation









VR&E generates the sample files based upon the sampling definitions and submits sample
files directly to BAS.
BAS receives the sample files and sends to VADIR for processing.
VADIR processes sample files (to remove SSN and append demographics/EDIPI) and returns
them to BAS.
BAS transfers sample files (via EDX platform) to JDP and notifies JDP via email that sample
files are ready for deployment.
JDP cleans the sample file and selects the sample.
Sample is transferred to Government Printing Office (GPO) print vendor (via EDX platform)
for printing and mailing of the postcards and survey packages.

Sample is transferred in accordance with the following schedule:

VOV_LOB
Tracking_Production Schedule_10.06.15.pdf

1.3 Data Cleaning
JDP processed the sample according to the following cleaning rules:
1. De-duplicate records within each business line and across surveys based on the unique
identifier (EDI_PI or VA_ID) for each record. Note: EDIPI is Electronic Data Interchange
Personal Identifier.

49

2.
3.

4.
5.
6.

a) Exception: For Pension Access (v1) and Pension Servicing (v8), de-duplicate records
based on EDI_PI and Claim Number.
b) When each new sample file is received, JDP cleans it against all sample selected from
every sample batch that has been delivered 12 months prior to ensure a respondent
does not receive a VA line of business survey more than once in a 12 month-period. In
the case of duplicates occurring within the same sample month, priority is assigned to
business lines with the lowest number of sample records.
Clean out records present on the JDP Do Not Contact list and clean against the National
Change of Address (NCOA) list.
Clean out any respondents who do not have any EDI_PI or VA_ID included in their sample
record.
a) Exception: For Pension Access (v1) and Pension Servicing (v8), clean out records with
blank EDI_PI and Claim Number.
Clean out any respondents not specified as a dependent/spouse who have a date of death
(DOD) in their sample record.
Clean out any respondents who do not have any address included in their sample record.
Assign and maintain unique sampling identifiers to each sample record in order to track
history of sampling. Exclude records that have been sampled in the past 12 months to
ensure no respondent is mailed surveys more than once in a 12-month time frame. This rule
may not apply to those who completed a survey.

1.4 Order Generation and Fulfillment Process
Federal Acquisition Regulations (FAR 8.8) mandate government agencies to solicit all printing
requirements through the Government Printing Office. GPO utilizes print vendors to fulfill
orders. A Data Transfer Agreement (DTA) must be in place with the print vendor and contractor
before BAS can obligate funds or transfer sample files to the print vendor and contractor.
Prior to mailing the postcards and mail surveys, print orders must be generated for each survey.
The entire process can take 2-4 weeks from inception of the print order to the mailing of the
Survey Package or postcard. Below are the steps involved in order generation and order
fulfillment.
Order generation






After sample is received by JDP, the sample files are cleaned and selected. Then Letter Work
Orders (LWOs) are created to provide the print vendor with the necessary information to
match the sample files to the correct survey instrument. (1 day)
JDP creates the print order and sends over to BAS Contractor Officer’s Representative
(COR). (Same day as above step)
The COR reviews, authorizes, and submits the print order. (1 day)
The BAS Publication Officer and/or COR submits the orders to the VA Publications Services
Division (VAPSD). (Same day as above step)
The order is issued a control number by a VBA Management Analyst, Publications. (Variable
timing)
50





Once the control number is assigned, the order goes to the VA Publication Services Division
liaison to forward to GPO Contracting Officer. (Variable timing) Note: the amount of time an
order is with VAPSD varies from 3 days to 20 days.
The GPO Contracting Officer sends the printing and mailing order to the print vendor.

Order fulfillment





Once the order is placed, the GPO print vendor is allotted 9 business days to fulfill the order
(2 days to generate proofs, 2 days for proof review/correction, and 5 days to print and
mail).
Upon receipt of the proofs from print vendor, JDP reviews and approves; then BAS reviews
and approves; then VAPSD reviews and approves.
After the orders have been mailed, the print vendor provides the mail receipts to
contractor, BAS, and VAPSD.
Upon order completion, VAPSD provides actual costs to BAS.

1.5 Reporting
Reporting occurs four times yearly for the Servicing Process survey.
On a quarterly basis, the following deliverables are provided:





Scorecard
Data matrices
Data is loaded to the VOV reporting site
Open-ended comments (verbatims)

On a semiannual (twice yearly) basis, the following deliverable is provided:


Data and Analysis Presentation

Reporting occurs once annually for the Access Process survey.
On an annual basis, the following deliverables are provided:






Scorecard
Data matrices
Data is loaded to the VOV reporting site
Open-ended comments (verbatims)
Data and analysis presentation

51

Sample Plan Overview
2.1 Sample Criteria
VBA was responsible for providing sample to JDP that met the following sampling criteria:
Sample Population

Inclusion Criteria

Frequency of Data Request

Access Survey

For Access, the target population is
defined as Veterans who had an
initial meeting with their VR&E
counselor and were granted a
decision regarding their entitlement
within the past 30 days. These
individuals include those who
applied and showed up for the
initial appointment and were either
found entitled and pursued the
program or found not entitled and
did not pursue the program.

Annually

Servicing Survey

For Servicing, the target population
is defined as Veterans who have
entered and been enrolled in one of
the five tracks for at least 60 days.
These individuals include Veterans
who have been rehabilitated;
Veterans who have reached
maximum rehabilitation gain and
could not proceed in the program;
and Veterans still pursuing benefits.

Monthly

2.2 Fielding/Sampling Frequency
Survey
Instrument
Access
Survey
Servicing
Survey

Methodology

Total
Survey
Instruments

Targeted
Number of
Completes

Number of
Postcards
(eSurvey)

Number
of Mail
Packages

Fielding
Frequency

Mail and Online

14,000

4,200

14,000

14,000

Annually

Mail Only

60,000

18,000

N/A

60,000

Monthly

52

2.3 Data Transfer
The sample was posted by BAS once a month within the sampling folder on the VOV EDX site.
Sample should be provided in a file layout consistent with the file layout provided for the study
as outlined below.
VR&E File Layout
ADDRESS_1
ADDRESS_2
ADDRESS_LINE_1
ADDRESS_LINE_2
AGE
BAH Rate
BRANCH_1
BRANCH2
BRANCH3
CASE_STATUS_CODE
CITY
Claim_Number
Date_of_Birth
DIAG_CODE
ELGBTY_TRMNTN_DT
Email_Address
EOD_1
EOD_2
EOD_3
First_Name
FIRST_NOTICE_OF_DEATH
GENDER

Last_Name
MILITARY_RANK
PHONE_NUMBER
PHONE_NUMBER_2
POSTAL_CODE
PRCNT_NBR
RAD_1
RAD_2
RAD_3
SEH_STATUS
SERVICE_ERA_1
SERVICE_ERA_2
SERVICE_ERA_3

53

VR&E File Layout (Continued.)
SERVICE_PERIOD_MONTH
SSN_NBR
STATE
STN_NBR
VETERAN_DOB
Zip
Zip_Code

2.4 Sample Cleaning Rules Glossary
Duplicate records in sample file – The record is cleaned out if there is more than one record
within the same sample file for the same respondent
Duplicate record history – The record is cleaned out if the record has been selected within the
past 12 months for any of VBA’s business line surveys (i.e., Compensation, Pension, Education,
Home Loan Guaranty, and Vocational Rehabilitation), regardless of whether the respondent
completed the survey
Invalid address – The record is cleaned out if JDP’s address verification software indicates an
invalid address code
Invalid values – The record is cleaned out if the “VA_ID” field is blank
Blanks – The record is cleaned out if the “Name” field corresponding to the record is blank
Do not contact – The record is cleaned out if the individual is listed on JDP’s Do Not Contact list

2.5 Sample Selection
JDP selected sample records following the completion of the sample cleaning process. The
following guidelines are referenced when selecting sample:
1. Total Sampling Targets: The table below summarizes the total sampling target per an RO per
a fielding period. The “Sampling Target per RO” column indicates the minimum number of
sample records that should be selected per an RO for each survey. If this minimum target
number cannot be reached for a particular RO, sample from a different RO will be selected
to make up the difference.

Access
Survey
Servicing
Survey

Frequency

Total
Sampling
Target

Sampling
Target Per
Time Period

Sampling
Target Per
RO

Number of
ROs

Annually

14,000

4,200

241

58

Monthly

60,000

18,000

86

58

54

2. The same record cannot be selected for multiple surveys during the same wave.
Respondents who have completed a survey within the past 12 months cannot be selected.
Survey priority is based on the number of records in each sample file. The survey with the
smallest number of records is given first priority.
3. Following sample selection, the JDP project teams receives an automated report confirming
the number of records selected for each survey version. The JDP project team verifies that
the sample selection quantities reflect the sample targets and approves the sample file for
fielding.

2.6 Data Collection
During the survey fielding period, both online survey returns and paper surveys are collected as
they are received and posted on a secure EDX site. Responses from paper surveys are scanned
through automated imaging software while verbatim responses are recorded by a live survey
processor. Survey returns must have all pages intact in order to be processed and counted as a
return. Surveys with missing pages are counted as unusable. Returns are also considered
unusable if there is an indication that the individual completing the survey is not the individual
selected from the sample file (i.e., the respondent name and/or address on the survey is
replaced with a different name and/or address). During each day of fielding, a subset of survey
returns undergoes quality assurance to validate the accuracy of responses captured. If duplicate
surveys are returned (as identified by the unique sampling identifier assigned to each sample
record), the original survey return is processed while the duplicate survey is removed. In the
case of duplicate survey returns from mixed methodology surveys, the date the survey was
received is used to identify the original return while the subsequent return is removed postfielding.

55

Appendix D
Approaches to Mitigating the Effect of NonResponse Bias and Strategies to Improve the
Response Rate
The following section outlines two approaches used in FY15 to mitigate the potential of nonresponse bias. As mentioned earlier in the report, J.D. Power affirms that while high response
rates are always desirable in surveys, an 80% response rate is typically not achievable for a
voluntary, customer satisfaction survey instrument (Malhotra & Birks, 2007), particularly those
that do not provide an incentive (not recommended for this program). To illustrate this point,
the Dillman Method for survey fielding (Dillman, D. A. (2014)) discusses a survey instrument
that was fielded to 600 students at the University of Washington. After five attempts to solicit a
response, as well as offering a monetary incentive to complete the survey, only a 77% response
rate was generated.
The first approach to minimize non-response occurs before and during data collection and
involves introducing measures to maximize survey response rates. The second approach is to
make statistical adjustments after the data is collected.

1.1 Approach 1: Strategies to Maximize Response Rates
Prior to and during fielding the VR&E survey, JDP implemented the following measures to
reduce the chances of non-response:






Respondents were provided with the promise of confidentiality on the survey cover letter
and postcard and assured that their survey responses would not impact their current or
future eligibility for benefits.
Following the first mailing, non-respondents were sent an additional survey mailing.
Respondents were provided with a toll-free telephone number and dedicated email address
to contact JDP about survey-related inquiries (e.g., how to interpret questions and response
items, the purpose of the survey, how to get another copy of the survey if their copy has
been lost/damaged, etc.). Telephone calls and emails are responded to within 24 hours and
answered during regular business hours (8:00 a.m. - 5:00 p.m. PT).
JDP ensured the Web-based surveys were accessible to people with disabilities by
maintaining 508 compliant standards. These standards include:
 Keyboard navigation rather than mouse or other pointing devices
 Customization options for color, size, and style of text displayed
 Compatibility with screen-readers to translate items displayed on the survey in audible
output and/or Braille displays
 Customer support and technical support through JDP Help Desk toll-free phone number
and email address
 Exclusion of non-text elements, image maps, animation, flashing or blinking text.
56




The survey fielding period was extended to offer opportunities to respond for subgroups
having a propensity to respond late (e.g., males, young, full-time employed).
The survey was developed and reviewed in order to enhance respondent understanding of
the survey materials and to improve the relevancy of the data collected:
 Prior to fielding the benchmark study, a series of cognitive labs was conducted with test
users to ensure the survey questions were easily understood and correctly interpreted.
Revisions were made to the survey based on test user feedback (as per OMB Guideline
1.4.1).
 After the benchmark study and prior to fielding the first year of the tracking study, VR&E
Service and JDP conducted a review of the survey instruments and modified the surveys
to improve the relevancy of data collected (as per OMB Guideline 1.4.2).

1.2 Approach 2: Correcting Unit Non-Response Bias with Sample Weighting and
Survey Raking
As stated above, the two approaches to tackling non-response bias include implementing
measures to maximize response rates during the fielding period and making post hoc statistical
adjustments to the survey results afterwards. The following section discusses the statistical
adjustments approach, which includes weighting the data or imputing scores to correct the
amount of non-response bias. An example of this approach would be the survey raking
procedure described earlier in this report. See the associated references in the Survey Raking
Procedure for Sample Weightings section for more information.
The procedure known as raking adjusts a set of data so that its marginal totals match specified
control totals on a specified set of variables. The term “raking” suggests an analogy with the
process of smoothing the soil in a garden plot by alternately working it back and forth with a
rake in two perpendicular directions (Izrael and Battaglia (2004)).
If non-response bias was identified in the survey data, the non-response bias could be
corrected mathematically with a post-stratification survey weight. JDP would weigh the survey
data based on certain demographics (such as age, gender, region, etc.) of the total sample so
that the weighted survey data would conform more to the demographics of the total sample.
The implicit assumption in this approach is that the distributions of characteristics of the nonrespondents within an adjustment class (such as an age group) are the same, on average, as
those of the respondents within the same adjustment class.
See Appendix B for the item response rate for each question in the survey. If the item response
rate was not lower than 70%, as per OMB standards, the imputation of data is not necessary.
In the case that a particular item-level response was less than 70%, JDP would recommend
conducting additional analysis to determine the potential for other factors (i.e., missing or skip
patterns in the survey instrument) to be the cause of non-response.

57

Strategies to Improve Response Rate
In addition to the strategies listed above, JDP recommends the following strategies to improve
response rates going forward:





Issue ongoing public communications (e.g., press releases, information posted on the VA
website) to spread awareness and confirm the legitimacy of the VA VR&E Study.
Educate VA employees and VSOs about the survey to encourage participation. Provide a list
of frequently asked questions and answers to VSOs and VA employees to enable them to
answer Veterans’ questions regarding the survey.
Send email invitations to Veterans rather than mailing postcards to make it easier for
Veterans to complete the survey online.
Reduce the length of the survey to improve respondents’ willingness to respond.
 Reduce overall number of questions and number of response options for each question



To encourage participation, increase the number of contacts to respondents with additional
reminders about the survey.
 Provide respondents with an additional paper survey questionnaire








Reduce the frequency of mailings to reduce the opportunities for delays and errors in the
GPO Print process.
Revise the cover letter and postcard to express the importance of participation in the
survey.
Provide sample from the 30-day period immediately prior to the mailing rather than sample
from 90 days prior in order to improve the recency of their experience with the VR&E
benefit, which improves both participation and recollection.
Change the location of the sequence number to directly follow the survey link on the
postcard and cover letter.
Transfer the responsibility of sample file generation from VR&E to PA&I. A data pull from
PA&I will increase consistency.
To increase participation, enhance formatting on postcard and cover letter to include color
print to make materials more readable.

58

Appendix E
Impact of FAR 8.8
Federal Acquisition Regulation (FAR) 8.8 requires that printing must be conducted through the
Government Printing Office (GPO). The following section outlines limiting factors of the VOV
Line of Business Tracking Satisfaction Research Study that occurred as a result of the FAR
requirement.
Through the utilization of the GPO Print Vendor, the following occurred in FY15:
o Quality issues included:
 Survey instruments were printed and mailed:
 Utilizing the sample population from one survey, but receiving a different
survey (e.g., potential respondents from the pool of one business line
received the survey for a different business line)
 Using a version of the instrument that was outdated; this version did not
contain the current questions or responses that were being fielded
 Mixing content between survey versions
 Using shells from one survey printed with a different survey
o Ongoing timeliness delays occurred with each set of orders placed, as the order
fulfillment process took a minimum of 2-4 weeks

1.1

Impact

The project experienced ongoing delays in the printing and mailing of postcards and survey
packets for VBA’s lines of business. The delays affected the critical processes required to
execute the VOV Program to its fullest potential.
A multitude of quality issues were experienced throughout FY15 that negatively impacted the
VOV Program response rates. The issues that occurred impacted access to the online survey;
readability of mail materials; level of effort required by respondents to take the survey;
relevancy of survey; and the diminishment of brands (VA/JDP) associated with poor quality
materials.

59

Appendix F
NOTE: Questionnaire is not shown in the formatted version that respondents used to fill out
survey.

Survey Questionnaires
[DO NOT DISPLAY/IDENTIFY SECTION HEADERS. DISPLAY SINGLE QUESTION PER PAGE.]
[RESPONSE CODES APPEAR IN BRACKETS AT THE END OF EACH RESPONSE FOR SINGLE
RESPONSES AND IN THE PROGRAMMING INSTRUCTIONS FOR MULTIPLE RESPONSES.]

Servicing Questionnaire
Benefit Information
1. How did you FIRST learn about the VR&E benefit programs? (Mark only one) If
you are unsure, please indicate the first way you remember learning about the
VR&E benefit programs. [RADIO BUTTONS. SINGLE RESPONSE.]
a. VA website [1]
b. eBenefits.va.gov [3]
c. Veterans Employment Center in eBenefits [2]
d. Social media websites (e.g., Facebook, Twitter, etc.) [11]
e. Internet (excluding VA and social media sites) [14]
f. Mail (from VA) [4]
g. VA phone number (800-827-1000) [5]
h. In person at a Regional Office [10]
i. VA medical center [8]
j. VA Vet Center [9]
k. Visit from a VA employee [12]
l. Transition Assistance Program/Disabled Transition Assistance Program
briefings [6]
m. Veterans Service Organizations(e.g., Amer. Legion, DAV, VFW, PVA,
MOPH, etc.) (Specify)
[TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
n. Other Veterans/Servicemembers
o. Friends or family [15]
p. Information came with notification/ratings letter [16]
q. Other publications (e.g., Army Times, local newspapers, etc.)
r. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.] [97]
s. Don’t know or not sure [99]
2. What method(s) do you MOST FREQUENTLY use to obtain general information
about VA’s VR&E benefits or services? (Mark all that apply) [CHECK BOXES.
60

MULTIPLE RESPONSE. MULTIPLE RESPONSE. CODE EACH RESPONSE
AS 0 IF UNCHECKED OR 1 IF CHECKED]
a. VA website
b. eBenefits.va.gov
d. Veterans Employment Center in eBenefits Social media websites (e.g.,
Facebook, Twitter, etc.)
e. Other websites (excluding VA or social media sites)
f. Phone
g. Mail
h. Email
i. In person at a Regional Office
j. VA medical center
k. VA Vet Center
l. Veterans Service Organizations (e.g., Amer. Legion, DAV, VFW, PVA,
MOPH, etc.) (Specify)
[TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
m. Disabled Veterans’ Outreach Program
n. Friends or family
o. Other Servicemembers
p. Other publications (e.g., Army Times, local newspapers, etc.)
q. School
r. VR&E Office
s. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
t. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE]
u. None of the above [MUTUALLY EXCLUSIVE RESPONSE]
3. How frequently would you like to receive communications (e.g., emails, letters,
newsletters, etc.) from VA about VR&E benefits or services? (Mark only one)
[RADIO BUTTONS. SINGLE RESPONSE.]
a. Weekly
b. Monthly
c. Quarterly (every 3 months)
d. Semiannually (twice per year)
e. Annually (once per year)
f. Never
g. Don’t know or not sure
4. How would you like to receive information from VA about VR&E benefits or
services? (Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE.
CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED]
a. Phone
b. Mail
c. Email
d. VA website
e. Social media websites (e.g., Facebook, Twitter, etc.)
f. In person at a Regional Office
61

g. Veterans Service Organizations (e.g., Amer. Legion, DAV, VFW, PVA,
MOPH, etc.) (Specify)
[TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
h. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
i. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE]
The following question asks you to rate various aspects of your experience with
Vocational Rehabilitation and Employment, using a scale of 1 to 10, where 1 is
Unacceptable, 10 is Outstanding, and 5 is Average. [SHOW ON SAME PAGE AS THE
QUESTION THAT FOLLOWS]
5. When thinking about your most frequently used methods of communication,
please rate your experience in obtaining information about your VR&E benefit
on the following items: (Mark only one per row) [SHOW RESPONSES IN GRID
WITH 10-POINT SCALE IN COLUMNS AND ATTRIBUTES/RESPONSES IN
ROWS (SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC
DETAILS OF LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS,
ALTERNATE SHADES IN ROWS. SINGLE RESPONSE PER ROW.
RANDOMIZE ALL ATTRIBUTES EXCEPT THE LAST ONE.]
a. Ease of accessing information [ALLOW N/A RESPONSE] [1-10, N/A=99]
b. Availability of information [ALLOW N/A RESPONSE] [1-10, N/A=99]
c. Clarity of information [ALLOW N/A RESPONSE] [1-10, N/A=99]
d. Usefulness of information [ALLOW N/A RESPONSE] [1-10, N/A=99]
e. Frequency of information provided by VA [ALLOW N/A RESPONSE] [110, N/A=99]
f. Overall rating of information

Contact with VA
6. During the past 6 months, did you contact anyone from VA about your VR&E
benefit, excluding any contacts with your Vocational Rehabilitation and
Employment Counselor? (Mark only one) [RADIO BUTTONS. SINGLE
RESPONSE.]
a. Yes [1]
b. No [0]
(Ask Q7-Q12 if Q6 is yes, otherwise go to Q13)
7. Which of the following best describes the reason for your most recent contact?
(Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. Resolve a problem [1]
b. Ask a question [2]
c. Request a change to your records/provide information [3]

62

8. Can you briefly describe the nature of your most recent contact? (Mark all that
apply) [CHECK BOXES. MULTIPLE RESPONSE. MULTIPLE RESPONSE.
CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED]
a. Receive help regarding a paperwork issue
b. Receive help regarding a medical issue
c. Receive help regarding a training issue
d. Receive help regarding an employment issue
e. Change your address or direct deposit information
f. Report the death of an individual who received VA benefits
g. Report a problem with counselor/case manager
h. Report a problem with a VA customer service representative
i. Ask a general question
j. Obtain information about submitting/re-opening a claim
k. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
9. Thinking about your most recent contact, how did you contact VA? (Mark only
one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. Phone [1]
b. Online Chat
c. Website [6]
d. Email [7]
e. Mail [9]
f. In person [3]

10. Was your most recent issue resolved? (Mark only one) [RADIO BUTTONS.
SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
(Ask Q11 if Q10 is No, otherwise go to Q12)
11. Why wasn’t your most recent issue resolved? [CHECK BOXES. MULTIPLE
RESPONSE. MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0 IF
UNCHECKED OR 1 IF CHECKED]
a. Did not receive all of the information required
b. Received incorrect information
c. Was referred to the incorrect office/person
d. Waiting for follow-up from VA
e. Other (Specify) ____________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
f. Don't know or not sure [MUTUALLY EXCLUSIVE RESPONSE]

12. Thinking of your most recent contact with the VA, how would you rate your
overall customer service experience with the VA or VA representatives using a
scale of 1 to 10 where 1 is Unacceptable, 10 is Outstanding, and 5 is Average?
63

[SHOW RESPONSES IN GRID WITH 10-POINT SCALE IN COLUMNS AND
SINGLE ROW (SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR
SPECIFIC DETAILS OF LAYOUT). EVENLY SPACED RADIO
BUTTONS/COLUMNS, SINGLE RESPONSE PER ROW.][1-10]

Benefit Entitlement
13. Does/did your rehabilitation plan include an education or training phase? (Mark
only one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
c. Don’t know or not sure [99]

(Ask Q14-15 if Q13 is yes, otherwise go to Q16)
14. Did the same counselor who developed your rehabilitation plan also provide
case management sessions during the education and training phase? (Mark
only one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
c. Don’t know or not sure [99]
d. Not applicable [96]
15. Were you given a time frame from VA for completing the education/training
phase of your rehabilitation plan? (Mark only one) [RADIO BUTTONS. SINGLE
RESPONSE.]
a. Yes [1]
b. No [0]
c. Don’t know or not sure [99]
16. How many times in the past 6 months has a counseling appointment been
cancelled or rescheduled by your counselor? (Open Capture)
a. Never been cancelled or rescheduled [CHECK BOX; MUTUALLY
EXCLUSIVE]
b. Number of times (1-99)___________ [CHECK BOX; MUTUALLY
EXCLUSIVE]
c. Don’t know or not sure [CHECK BOX; MUTUALLY EXCLUSIVE] [CODE
RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED]

(If your counseling appointment has been cancelled or rescheduled by your counselor
1 or more times, please answer Q17. Otherwise, please skip to Q18.)
17. If your counseling appointment was cancelled or rescheduled at least once,
were you scheduled for a new appointment without having to ask? (Mark only
one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
64

c. Don’t know or not sure [99]

18. Which of the following types of counseling or referrals has your counselor
provided? (Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE.
CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED]
a. Education/Training enrollment assistance
b. Career counseling
c. Personal counseling
d. Financial counseling
e. Problem-solving techniques
f. Referrals to potential employers (e.g., government, private, etc.)
g. Referrals to employment agencies or job banks
h. Referrals to health providers (e.g., medical, dental, optical)
i. Referrals to other counseling programs
j. Referrals to Veterans Service Organizations (e.g., American Legion)
k. None of the above [MUTUALLY EXCLUSIVE RESPONSE]
The following question asks you to rate various aspects of your experience with
Vocational Rehabilitation and Employment (VR&E), using a scale of 1 to 10 where 1 is
Unacceptable, 10 is Outstanding, and 5 is Average. [SHOW ON SAME PAGE AS THE
QUESTION THAT FOLLOWS]
Please answer the following question based on your best ability to recall your experience with
your VR&E counselor(s). [SHOW ON SAME PAGE AS THE QUESTION THAT FOLLOWS]

19. Please rate your experience with VR&E counselors on the following items:
(Mark only one per row) [SHOW RESPONSES IN GRID WITH 10-POINT
SCALE IN COLUMNS AND ATTRIBUTES/RESPONSES IN ROWS (SEE
JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC DETAILS OF
LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS, ALTERNATE
SHADES IN ROWS. SINGLE RESPONSE PER ROW. RANDOMIZE ALL
ATTRIBUTES EXCEPT THE LAST ONE.]
a. Promptness of scheduling appointments or returning calls [ALLOW N/A
RESPONSE] [1-10, N/A=99]
b. Courtesy of the counselor [ALLOW N/A RESPONSE] [1-10, N/A=99]
c. Knowledge of the counselor [ALLOW N/A RESPONSE] [1-10, N/A=99]
d. Counselor’s concern for your needs [ALLOW N/A RESPONSE] [1-10,
N/A=99]
e. Timeliness of completing your initial evaluation [ALLOW N/A
RESPONSE] [1-10, N/A=99]
f. Overall counselor experience
20. Why did you give your overall experience with your counselor that rating? (Open
Capture) [OPEN-END. TEXT BOX. 1000 CHARACTERS MAX. ALLOW NO
COMMENT, MUTUALLY EXCLUSIVE CHECK BOX. CODE NO COMMENT
AS 0 IF UNCHECKED AND 1 IF CHECKED]
65

21. Which of the following benefits did you or will you receive as part of your
rehabilitation plan? (Mark all that apply) [CHECK BOXES. MULTIPLE
RESPONSE. CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF
CHECKED]
a. Tuition
b. Subsistence allowance
c. Books/Supplies
d. Computer equipment/software
e. Health services (e.g., medical, dental, optical)
f. Tutoring
g. Independent Living services
h. Employment services (e.g., resume preparation, interview skills, obtaining
licenses/certifications, etc.)
i. None of the above [MUTUALLY EXCLUSIVE RESPONSE]
22. Which of the following types of employment services did/will you receive as part
of your rehabilitation plan? (Mark all that apply) [CHECK BOXES. MULTIPLE
RESPONSE. CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF
CHECKED]
a. Resume preparation
b. Interview skills
c. Obtaining licenses/certifications
d. Job hunting strategies
e. Information interview with potential employers
f. Job placement assistance
g. None of the above [MUTUALLY EXCLUSIVE RESPONSE]
23. Were you given a time frame from VA for completing your VR&E rehabilitation
plan? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
c. Don’t know or not sure [99]
(Ask Q24 if Q23 is yes, otherwise go to Q25)
24. How long was/is the time frame for completing your VR&E rehabilitation plan?
(Open Capture) Please respond using any or all of the following categories
a. Months (0-99 months) _____________ [NUMERIC TEXT BOX;
ACCEPTABLE RANGE [0-99]]
b. Years (0-99 years) _________ [NUMERIC TEXT BOX; ACCEPTABLE
RANGE [0-99]]
c. Don’t know or not sure [CHECK BOX; MUTUALLY EXCLUSIVE] [CODE
RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED]
25. Were the amount of services you received as part of your VR&E program more
than, less than, or what you expected? (Mark only one) [RADIO BUTTONS.
SINGLE RESPONSE.]
a. Less than [1]
b. What I expected [2]
66

c. More than [3]

The following question asks you to rate various aspects of your experience with
Vocational Rehabilitation and Employment, using a scale of 1 to 10 where 1 is
Unacceptable, 10 is Outstanding, and 5 is Average. [SHOW ON SAME PAGE AS THE
QUESTION THAT FOLLOWS]
26. Please rate your VR&E benefit (e.g., training and counseling) on the following
items: (Mark only one per row) [SHOW RESPONSES IN GRID WITH 10-POINT
SCALE IN COLUMNS AND ATTRIBUTES/RESPONSES IN ROWS (SEE JDP
CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC DETAILS OF LAYOUT).
EVENLY SPACED RADIO BUTTONS/COLUMNS, ALTERNATE SHADES IN
ROWS. SINGLE RESPONSE PER ROW. RANDOMIZE ALL ATTRIBUTES
EXCEPT THE LAST ONE.]
a. Amount of benefits [ALLOW N/A RESPONSE] [1-10, N/A=99]
b. Effectiveness of benefit/service in preparing and obtaining suitable
employment [ALLOW N/A RESPONSE] [1-10, N/A=99]
c. Timeliness of receiving benefit payment [ALLOW N/A RESPONSE] [1-10,
N/A=99]
d. Overall rating of benefit payment
Overall Experience with Benefit Program
27. Thinking about ALL aspects of your experience with Vocational Rehabilitation
and Employment benefits, please rate VA overall, using a scale of 1 to 10
where 1 is Unacceptable, 10 is Outstanding, and 5 is Average. (Mark only one)
[SHOW RESPONSES IN GRID WITH 10-POINT SCALE IN COLUMNS AND
SINGLE ROW (SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR
SPECIFIC DETAILS OF LAYOUT). EVENLY SPACED RADIO
BUTTONS/COLUMNS, SINGLE RESPONSE PER ROW.][1-10]

Overall Experience with VA
28. Taking into consideration all of the non-medical benefits (e.g., education,
compensation, pension, home loan guaranty, vocational rehabilitation and
employment, insurance, etc.) you have applied for or currently receive, please
rate your experience with VA overall, using a scale of 1 to 10 where 1 is
Unacceptable, 10 is Outstanding, and 5 is Average. (Mark only one) [SHOW
RESPONSES IN GRID WITH 10-POINT SCALE IN COLUMNS AND SINGLE
ROW (SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC
DETAILS OF LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS,
SINGLE RESPONSE PER ROW.][1-10]

67

29. How likely are you to inform other Veterans about your experience with VA
benefits or services? (Mark only one) [RADIO BUTTONS. SINGLE
RESPONSE.]
a. Definitely will not [1]
b. Probably will not [2]
c. Probably will [3]
d. Definitely will [4]
About You
30. What is your current status in the Vocational Rehabilitation and Employment
program? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. Completed program successfully [1]
b. Currently participating in program [2]
c. VA initiated interruption in program [3]
d. VA initiated discontinuation (withdrawal) from program [4]
e. Voluntary interruption in program [5]
f. Voluntary discontinuation (withdrawal) from program [6]
g. Prefer not to answer [98]
(Ask Q31 if Q30 is voluntary interruption or withdrawal, otherwise go to Q32)
31. Why did you interrupt or withdraw from your rehabilitation program? (Mark all that
apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE
AS 0 IF UNCHECKED OR 1 IF CHECKED]
a. Medical difficulties
b. Financial difficulties
c. Family responsibilities
d. Found a job prior to program completion
e. Transportation difficulties
f. Program did not meet needs
g. Program requirements were too difficult
h. VA initiated interruption/discontinuation (withdrawal)
i. Problems with counselor
j. Lost interest
k. To pursue another education benefit (CH33, State Vocational
Rehabilitation, etc.)
l. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
m. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE]
32. Do you plan to complete your rehabilitation program now or in the future? (Mark
only one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
c. Don’t know or not sure [99]
d. Prefer not to answer [98]

68

33. At any point during the VR&E program, did you register for the Veterans
Employment Center in eBenefits? (Mark only one) [RADIO BUTTONS. SINGLE
RESPONSE.]
a. Yes [1]
b. No [0]
c. Don’t know or not sure [99]
(Ask Q34 if Q33 is No, otherwise go to Q35)
34. Why didn’t you register for the Veterans Employment Center in eBenefits? (Mark
all that apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH
RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED]
a. Not aware of the Veterans Employment Center
b. Opted not to use the Veterans Employment Center
c. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
d. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE]

35. Are you currently enrolled in a 2-year college (e.g., community college), 4- year
college (e.g., university), postgraduate program, technical or trade school, flight
school or on-the-job training program? [RADIO BUTTONS. SINGLE
RESPONSE.]
a. Yes [1]
b. No [0]
36. Are you a … [RADIO BUTTONS. SINGLE RESPONSE.]
a. Part-time student [1]
b. Full-time student [2]
c. Not currently enrolled [3]
d. Don’t know or not sure [99]
(Ask Q37-39p if Q36 is a or b, otherwise go to Q40)
37.
What is the format of the program you are enrolled in? (Mark only one) [RADIO
BUTTONS. SINGLE RESPONSE.]
a. Traditional (classes in classroom/school facility) [1]
b. Online (classes on the Internet) [2]
c. Mixed (classroom and online) [3]
38.
What type of degree/training program are you currently pursuing? (Mark only
one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. On-the-job training or apprenticeship [1]
b. Certificate/License [2]
c. Associate degree [3]
d. Bachelor’s degree [4]
e. Master’s degree [5]
f. Doctorate [6]
69

39.
What type of academic institution or training facility are you enrolled in? (Mark
only one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. 2-year college (e.g., community college) [1]
b. 4-year college (e.g., university) [2]
c. Postgraduate program [3]
d. Technical or trade school [4]
e. Flight school [5]
f. Job training site [6]
g. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.] [97]

(Ask Q39a if enrolled in a 2-year college in Q39, otherwise go to Q39b)
39a..(Online only) Do you plan on attending a 4-year college in the future? (Mark only
one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
c. Prefer not to state [98]
39b.
(Online only) Prior to the current program, what was the last year of school you
completed? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. High school graduate or equivalent [1]
b. Trade/Technical school [2]
c. Some college (2-year program) [3]
d. Some college (4-year program) [4]
e. 2-year college degree [5]
f. 4-year college degree [6]
g. Some graduate courses [7]
h. Advanced degree [8]
i. Prefer not to answer [98]

39c.
(Online only) Why did you select your current school/training facility? (Mark all
that apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0
IF UNCHECKED OR 1 IF CHECKED]
a. Lower tuition/program costs
b. Good counselors
c. Convenient location
d. Easy initial application process
e. Convenient course/program enrollment process
f. Variety of course/training offerings
g. Variety of available student support
h. School specialization in subject of interest
i. Reputation of school/training facility
j. Reputation of instructors
k. Past experience
l. Recommendation from friends/relatives
70

m.
n.
o.
p.

Availability of online classes
Flexibility of course/training scheduling
Financial aid
Other (Specify) _____________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]

39d.
(Online only) When did you first enter into your current degree/training
program? (Open Capture)
a. Please enter the month and year: mm _____ yy _______ [TWO
NUMERICTEXT BOXES; ONE FOR MONTHS [ACCEPTABLE RANGE
1-12) AND ONE FOR TWO-DIGIT YEAR (ACCEPTABLE RANGE 0099)]
b. Prefer not to answer [CHECK BOX. MUTUALLY EXCLUSIVE
RESPONSE.] [CODE RESPONSE AS 0 IF UNCHECKED AND 1 IF
CHECKED]
39e.
(Online only) How many years have you completed in your current
degree/training program? (Open Capture) If you have completed less than 1 year, enter
0.
a. Number of years _________ [NUMERIC TEXT BOX. ACCEPTABLE
RANGE 0-99]
b. Prefer not to answer [CHECK BOX. MUTUALLY EXCLUSIVE
RESPONSE.] [CODE RESPONSE AS 0 IF UNCHECKED AND 1 IF
CHECKED]

39f.
(Online only) Why did you select your current degree/training program? (Mark
all that apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESONSE AS
0 IF UNCHECKED AND 1 IF CHECKED]
a. Preparation for career
b. Salary/Wages in associated careers
c. Status/Esteem associated with type of degree/program
d. Personal growth/development
e. Interested in subject matter
f. Number of course requirements
g. Preparation for advanced degree
h. Ease of completion requirements
i. Reputation of instructors
j. Recommendation from friends/relatives
k. Availability of online classes
l. Flexibility of course/training scheduling
m. Other (Specify) _____________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]

39g..
(Online only) Have you ever taken any time off from your current
degree/training program? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.]
71

a. Yes [1]
b. No [0]
c. Prefer not to answer [98]
(Ask Q39h-Q39iif Q39g is yes, otherwise go to Q39j)
39h.
(Online only) How much time have you taken off from your current
degree/training program? (Open Capture) Please respond using any or all of the
following categories.
a. Days (0-99 days) __________ [NUMERIC TEXT BOX. ACCEPTABLE
RANGE 0-99.]
b. Months (0-99 months) _________ [NUMERIC TEXT BOX.
ACCEPTABLE RANGE 0-99.]
c. Years (0-99 years) _________ [NUMERIC TEXT BOX. ACCEPTABLE
RANGE 0-99.]
c. Don’t know or not sure [CHECK BOX. MUTUALLY EXCLUSIVE
RESPONSE.] [CODE RESPONSE AS 0 IF UNCHECKED AND 1 IF
CHECKED]

39i.
(Online only) Why did you take time off? (Open Capture) [OPEN-END. TEXT
BOX. 1000 CHARACTER MAX. ALLOW NO COMMENT, MUTUALLY EXCLUSIVE
CHECK BOX. CODE NO COMMENT AS 0 IF UNCHECKED AND 1 IF CHECKED]
______________________________________________________________________
________________________________________________
39j.
(Online only) Have you been called to active duty at any point during your
current degree/training program? (Mark only one) [RADIO BUTTONS. SINGLE
RESPONSE.]
a. Yes [1]
b. No [0]
c. Prefer not to answer [98]
(Ask Q39k if Q39j is yes, otherwise go to Q39l)
39k.
(Online only) How long was your call to active duty? (Open Capture)
a. Months (0-99 months) _________ [NUMERIC TEXT BOX.
ACCEPTABLE RANGE 0-99.]
b. Don’t know or not sure [CHECK BOX. MUTUALLY EXCLUSIVE
RESPONSE.] [CODE RESPONSE AS 0 IF UNCHECKED AND 1 IF
CHECKED]
39l.
(Online only) Have you ever been on academic probation or had less than
satisfactory standing with your school/training program? (Mark only one) [RADIO
BUTTONS. SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
c. Prefer not to answer [98]

72

39m.
(Online only) Do you plan to obtain a degree or completion certificate in your
current field of study/training? (Mark only one) [RADIO BUTTONS. SINGLE
RESPONSE.]
a. Yes, from the degree/training program at my current school/facility [1]
b. Yes, from a degree/training program at another school/facility [2]
c. No [0]
d. Prefer not to answer [98]
(Ask Q39n if Q39m is yes, otherwise go to Q39o)
39n.
(Online only) When do you expect to complete or graduate with a degree or
completion certificate in your current field of study/training? (Open Capture)
a. Please enter the month and year: mm _____ yy _______ [TWO
NUMERICTEXT BOXES; ONE FOR MONTHS [ACCEPTABLE RANGE
1-12) AND ONE FOR TWO-DIGIT YEAR (ACCEPTABLE RANGE 1299)]
b. Prefer not to answer [CHECK BOX. MUTUALLY EXCLUSIVE
RESPONSE.] [CODE RESPONSE AS 0 IF UNCHECKED AND 1 IF
CHECKED]
39o.
(Online only) Do you plan to continue your enrollment as a full-time student until
you complete or graduate your degree/training program? (Mark only one) [RADIO
BUTTONS. SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
c. Prefer not to answer [98]
39p.
(Online only) Which of the following services are available from your current
school/training facility? (Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE.
CODE EACH RESPONSE AS 0 IF UNCHECKED AND 1 IF CHECKED]
a. Academic counseling
b. Tutoring
c. Financial counseling
d. Dependent care services (e.g., babysitting, elder care)
e. Employment counseling
f. Financial aid
g. Technology assistance (e.g., internet access, computer, etc.)
h. Other (Specify) _____________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
i. Don’t know [MUTUALLY EXCLUSIVE RESPONSE]

39q. (Online only) What concerns, if any, do you have about achieving your
educational goals? (Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE.
CODE EACH RESPONSE AS 0 IF UNCHECKED AND 1 IF CHECKED]
a. Academic requirements
b. Difficulty of subject matter
c. Financial requirements
d. Family obligations
73

e.
f.
g.
h.
i.
j.

Employment obligations
Course scheduling
Time commitment (i.e., amount of time required)
Availability of technology (e.g., access to internet/computer)
Other (Specify) _____________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
Do not have concerns [MUTUALLY EXCLUSIVE RESPONSE]

39r.
(Online only) Which of the following services would you like or expect in order
to achieve your educational goals? (Mark all that apply) [CHECK BOXES. MULTIPLE
RESPONSE. CODE EACH RESPONSE AS 0 IF UNCHECKED AND 1 IF CHECKED]
a. Academic counseling
b. Tutoring
c. Financial counseling
d. Dependent care services (e.g., babysitting, elder care)
e. Employment counseling
f. Financial aid
g. Technology assistance (e.g., internet access, computer, etc.)
h. Other (Specify) _____________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
i. Don’t know [MUTUALLY EXCLUSIVE RESPONSE]

39s.
(Online only) What are your personal career goals? (Mark all that apply)
[CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0 IF
UNCHECKED AND 1 IF CHECKED]
a. Obtain financial security
b. Achieve work-life balance
c. Become an independent business owner
d. Become a manager
e. Become an executive
f. Work internationally
g. Contribute to society
h. Work in a specialized field (e.g., technology, medicine, etc.)
i. Other (Specify) _____________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]

40. Are you currently employed? (Mark only one) [RADIO BUTTONS. SINGLE
RESPONSE.]
a. Yes [1]
b. No [0]
c. Prefer not to answer[98]

(Ask Q41-Q42b if Q40 is Yes, otherwise go to Q42d)
74

41. Which of the following were the three most important resources in obtaining your
current job? (Mark top three) [CHECK BOX; MULTIPLE RESPONSE; ONLY ACCEPT
3 RESPONSES; CODE EACH RESPONSE AS 0 IF UNCHECKED AND 1 IF
CHECKED]
a. VR&E Counselor/Contract Counselor
b. Employment Coordinator
c. Veterans Employment Center in eBenefits
d. Newspaper
e. Online job site
f. Recommendations of friends/family
g. School
h. Other (Specify) _____________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
i. None of the above [MUTUALLY EXCLUSIVE RESPONSE]
42. Relative to when you began to receive Vocational Rehabilitation and Employment
services, when did you obtain employment? (Mark only one) [RADIO BUTTONS.
SINGLE RESPONSE.]
a. Prior to program completion [1]
b. After program completion [2]
c. Don’t know or not sure [99]
42a.
(Online only) How many hours do you currently work in a typical week? (Open
Capture)
a. Hours (0-40 hours) _________ [NUMERIC TEXT BOX. ACCEPTABLE
RANGE 0-40.]
b. Don’t know or not sure [CHECK BOX. MUTUALLY EXCLUSIVE
RESPONSE.] [CODE RESPONE AS 0 IF UNCHECKED AND 1 IF
CHECKED]
42b.
(Online only) Are you currently employed in a field related to your current
degree/training program? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
c. Prefer not to answer [98]

42c.
(Online only) Are you pursuing employment in your current field of study? (Mark
only one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
c. Prefer not to answer [98]

(Ask Q42d if Q42c is yes, otherwise go to Q43)
42d.
(Online only) Upon completion of your current degree/training program, what
will be your primary method of obtaining employment information? [RADIO BUTTONS.
SINGLE RESPONSE.]
75

a.
b.
c.
d.
e.
f.
g.
h.
i.
j.

VA counselor [1]
Recommendations of friends/family [2]
Student career/employment center [3]
Local or state job services [4]
Federal job services [5]
Newspaper [6]
Online job site [7]
Private employment agency [8]
Other (Specify) _____________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.] [97]
Don’t know [99]

43. Do you have any other comments or concerns about your experience? (Open
Capture) [OPEN-END. TEXT BOX. 1000 CHARACTER MAX. ALLOW NO
COMMENT, MUTUALLY EXCLUSIVE CHECK BOX. CODE NO COMMENT AS 0 IF
UNCHECKED AND 1 IF CHECKED]
____________________________________________________
As a reminder, your responses will be kept completely confidential and your email
address will not be sent to VA with any responses on this survey. [SHOW ON THE
SAME PAGE AS THE QUESTION THAT FOLLOWS]
44. Would you like to provide an email address so VA can contact you with general
information about VA benefits and services? (Mark only one) [RADIO BUTTONS.
SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
c. I do not have an e-mail address [96]
d. Prefer not to answer [98]
(Ask Q45 if Yes in Q44)
45. Please enter your preferred email address where you would like to be contacted:
(Open Capture)
a. Email: [OPEN CAPTURE. 100 CHARACTER MAX.]

76

Access Questionnaire
Benefit Information
1. How did you FIRST learn about the VR&E benefit programs? (Mark only one) if
you are unsure, please indicate the first way you remember learning about the
VR&E benefit programs . [RADIO BUTTONS. SINGLE RESPONSE.]
a. VA website [1]
b. eBenefits.va.gov [2]
c. Veterans Employment Center in eBenefits [3]
d. Social media websites (e.g., Facebook, Twitter, etc.) [11]
e. Internet (excluding VA and social media sites) [14]
f. Mail (from VA) [4]
g. VA phone number (800-827-1000) [5]
h. In person at a Regional Office [10]
i. VA medical center [8]
j. VA Vet Center [9]
k. Visit from a VA employee [12]
l. Transition Assistance Program/Disabled Transition Assistance Program
briefings [6]
m. Veterans Service Organizations (e.g., Amer. Legion, DAV, VFW, PVA,
MOPH, etc.) (Specify)
[TEXTBOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX. [7]
n. Other Veterans/Servicemembers [13]
o. Friends or family [15]
p. Information came with notification/ratings letter [16]
q. Other publications (e.g., Army Times, local newspapers, etc.)
r. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.] [97]
s. Don’t know or not sure [99]
2. What method(s) do you MOST FREQUENTLY use to obtain general information
about VA’s Vocational Rehabilitation and Employment (VR&E) benefits or
services? (Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE.
CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED]
a. VA website
b. eBenefits.va.gov
c. Veterans Employment Center in eBenefits
d. Social media websites (e.g., Facebook, Twitter, etc.)
e. Other websites (excluding VA or social media sites)
f. Phone
g. Mail
h. Email
i. In person at a Regional Office
j. VA medical center
k. VA Vet Center

77

l. Veterans Service Organizations( e.g., Amer. Legion, DAV, VFW, PVA,
MOPH, etc.) (Specify)
[TEXTBOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX. [7]
m. Disabled Veterans’ Outreach Program
n. Friends or family
o. Other Veterans/Servicemembers
p. School
q. Other publications (e.g., Army Times, local newspapers, etc.)
r. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
s. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE]
t. None of the above [MUTUALLY EXCLUSIVE RESPONSE]

3. How did you receive information about the application process for your most
recent Vocational Rehabilitation and Employment benefit application? (Mark all
that apply) [CHECK BOXES, MULTIPLE RESPONSE. CODE EACH
RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED]
a. Transition Assistance Program
b. Disabled Transition Assistance Program briefings
c. Integrated Disability Evaluation System
d. Phone
e. Mail
f. Email
g. Pamphlets/Brochures
h. VA website
i. VA medical center
j. VA Vet Center
k. In person at a Regional Office
l. Veterans Service Organizations (e.g., Amer. Legion, DAV, VFW, PVA,
MOPH, etc.) (Specify)
[TEXTBOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX. [7]
m. Disabled Veterans’ Outreach Program
n. IRIS (Inquiry Routing & Information System)
o. Compensation briefing
p. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
q. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE]
r. Did not receive information about application process [MUTUALLY
EXCLUSIVE RESPONSE]

4. How did you receive the Vocational Rehabilitation and Employment benefit
application? (Mark only one) if you are unsure, please indicate the first way you
remember learning about the VR&E benefit programs. [RADIO BUTTONS.
SINGLE RESPONSE.]
a. Transition Assistance Program [1]
b. Disabled Transition Assistance Program briefings [2]
78

c.
d.
e.
f.
g.
h.
i.
j.
k.
l.

m.
n.
o.
p.
q.
r.

Integrated Disability Evaluation System [3]
Phone [4]
Mail [5]
Email [6]
Pamphlets/brochures [7]
VA website [8]
VA medical center [9]
VA Vet Center [10]
In person at a Regional Office [11]
Veterans Service Organizations( e.g., Amer. Legion, DAV, VFW, PVA,
MOPH, etc.) (Specify)
[TEXTBOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX. [7]
Disabled Veterans’ Outreach Program [13]
IRIS (Inquiry Routing & Information System) [14]
Compensation briefing [15]
Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.] [97]
Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE] [99]
Did not receive information about application process [MUTUALLY
EXCLUSIVE RESPONSE] [98]

5. How frequently would you like to receive communications (e.g., emails, letters,
newsletters, etc.) from VA about VR&E benefits or services? (Mark only one)
[RADIO BUTTONS, SINGLE RESPONSE]
a. Weekly [1]
b. Monthly [2]
c. Quarterly (every 3 months) [3]
d. Semiannually (twice per year) [4]
e. Annually (once per year) [5]
f. Never [6]
g. Don’t know or not sure [99]

6. How would you like to receive information from VA about applying for VR&E
benefits or services? (Mark all that apply) [CHECK BOXES, MULTIPLE
RESPONSE. CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF
CHECKED]
a. Phone
b. Mail
c. Email
d. VA website
e. Social media websites (e.g., Facebook, Twitter, etc.)
f. In person at a Regional Office
g. Veterans Service Organizations (e.g., Amer. Legion,DAV, VFW, PVA,
MOPH, etc.).) (Specify)
[TEXTBOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX. [7]
79

h. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
i. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE]
The following question asks you to rate various aspects of your experience with
Vocational Rehabilitation and Employment, using a scale of 1 to 10 where 1 is
Unacceptable, 10 is Outstanding, and 5 is Average. [SHOW ON SAME PAGE AS THE
QUESTION THAT FOLLOWS]
7. When thinking about your most frequently used methods of communication,
please rate your experience in obtaining information about your VR&E benefit
application on the following items: (Mark only one per row) [SHOW
RESPONSES IN GRID WITH 10-POINT SCALE IN COLUMNS AND
ATTRIBUTES/RESPONSES IN ROWS (SEE JDPA CONVENTIONS
DOCUMENT PG. 1 FOR SPECIFIC DETAILS OF LAYOUT). EVENLY SPACED
RADIO BUTTONS/COLUMNS, ALTERNATE SHADES IN ROWS. SINGLE
RESPONSE PER ROW. RANDOMIZE ALL ATTRIBUTES EXCEPT THE LAST
ONE.]
a. Ease of accessing information [ALLOW N/A RESPONSE] [1-10, N/A=99]
b. Availability of information [ALLOW N/A RESPONSE] [1-10, N/A=99]
c. Clarity of information [ALLOW N/A RESPONSE] [1-10, N/A=99]
d. Usefulness of information [ALLOW N/A RESPONSE] [1-10, N/A=99]
e. Frequency of information provided by VA [ALLOW N/A RESPONSE] [110, N/A=99]
f. Overall rating of information
Contact with VA
8. During the past 6 months, did you contact anyone from VA about the VR&E
benefit application process (excluding any contacts with your Vocational
Rehabilitation and Employment counselor)? (Mark only one) [RADIO
BUTTONS. SINGLE RESPONSE]
a. Yes [1]
b. No [0]

(Ask Q9-14 if Q8 is yes, otherwise go to Q15)
9. Which of the following best describes the reason for your most recent contact?
(Mark only one) [RADIO BUTTONS. SINGLE RESPONSE]
a. Resolve a problem [1]
b. Ask a question [2]
c. Request a change to your records/provide information [3]

10. Can you briefly describe the nature of your most recent contact? (Mark all that
apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE
AS 0 IF UNCHECKED OR 1 IF CHECKED]
a. Questions about the application form
80

b. Receive help regarding a paperwork issue
c. Receive help regarding a medical issue
d. Receive help regarding a training issue
e. Receive help regarding an employment issue
f. Change your address or direct deposit information
g. Report the death of an individual who received VA benefits
h. Report a problem with counselor/case manager
i. Report a problem with a VA customer service representative
j. Ask a general question
k. Obtain information about submitting/re-opening a claim
l. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
11. Thinking about your most recent contact, how did you contact VA? (Mark only
one) [RADIO BUTTONS. SINGLE RESPONSE]
a. Phone [1]
c. Online Chat Website [6]
d. Email [7]
e. Mail [9]
f. In person [3]
12. Was your most recent issue resolved? (Mark only one) [RADIO BUTTONS.
SINGLE RESPONSE]
a.
Yes [1]
b.
No [0]
(Ask Q13 if Q12 is No, otherwise go to Q14)
13. Why wasn’t your most recent issue resolved? [CHECK BOXES. MULTIPLE
RESPONSE. CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF
CHECKE ]
a.
Did not receive all of the information required
b.
Received incorrect information
c.
Was referred to the incorrect office/person
d.
Waiting for follow-up from VA
e.
Other (Specify) ____________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
f.
Don't know or not sure [MUTUALLY EXCLUSIVE RESPONSE]

14. Thinking of your most recent contact with VA, how would you rate your overall
customer service experience with VA or VA representatives using a scale of 1
to 10 where 1 is Unacceptable, 10 is Outstanding, and 5 is Average? [SHOW
RESPONSES IN GRID WITH 10-POINT SCALE IN COLUMNS AND SINGLE
ROW (SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC
DETAILS OF LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS,
SINGLE RESPONSE PER ROW.] [1-10]
81

Benefit Eligibility and Application
15. What is the primary reason you applied/will apply for the VR&E program? (Mark
only one) [RADIO BUTTONS. SINGLE RESPONSE]
a. Get any job [1]
b. Get a better job [2]
c. Further my education [3]
d. Get training for a new job [4]
e. Get a job that accommodates my disability [5]
f. Improve job-seeking skills [6]
g. Career counseling [7]
h. Other (Specify) __________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.] [97]
16. Thinking about your most recent VR&E benefit application, what method did you
use to apply for your benefit? (Mark only one) [RADIO BUTTONS. SINGLE
RESPONSE]
a. Veterans Online Application/eBenefits [1]
b. Mail [2]
c. In person at a Regional Office [3]
d. In person at a Veterans Service Organization( e.g., Amer. Legion,DAV,
VFW, PVA, MOPH, etc.)) [4]
e. Veteran Employment Center in eBenefits [5]
f. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.] [97]
g. Do not remember filling out an application (SKIP TO Q31) [96]
h. Don’t know or not sure [99]

17. Did you receive a letter confirming the receipt of your application with information
about your initial meeting with your counselor?
a. Yes
b. No
c. Don’t know or not sure
(Ask Q18 if Q17 is Yes, otherwise go to Q19)
18. Thinking about the letter, was it clear and easy to understand?
a. Not at all clear
b. Somewhat clear
c. Completely clear
d. Don’t know or not sure
e. I did not read the letter
19. Were you updated on the status of your VR&E benefit application without having
to ask? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.]
82

a. Yes [1]
b. No [0]
c. Don’t know or not sure [99]
20. During the application process, did you have to provide the same information or
documentation more than once? (Mark only one) [RADIO BUTTONS. SINGLE
RESPONSE]
a. Yes [1]
b. No [0]
c. Don’t know or not sure [99]
(Ask Q21-Q22 if Q20 is Yes, otherwise go to Q23)
21. How many times did you have to provide the same information? (Open Capture)
a. Number of times (0-99)_____________ [NUMERIC TEXT BOX.
ACCEPTABLE RANGE 0-99]
b. Don’t know or not sure [CHECK BOX. MUTUALLY EXCLUSIVE
RESPONSE] [CODE AS 0 IF UNCHECKED OR 1 IF CHECKED]
22. What information did you have to provide more than once? (Mark all that apply)
[CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0 IF
UNCHECKED OR 1 IF CHECKED]
a. Discharge papers (DD214)
b. Documentation of education/training completed
c. Private medical records
d. Resume
e. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
f. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE]
23. During your initial evaluation appointment, did the counselor have you participate
in any testing? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
c. Don’t know or not sure [99]
(Ask Q24 if Q23 is Yes, otherwise go to Q25)
24. Did the counselor explain the following…? (Mark all that apply) [CHECK
BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0 IF
UNCHECKED OR 1 IF CHECKED]
a. Purpose of the test
b. Results of the test
c. Next steps in the process
d. None of the above [MUTUALLY EXCLUSIVE RESPONSE]
e. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE]
25. How many appointments did you have with a counselor before an entitlement
decision was made? (Open Capture)
83

a. Number of appointments (0-99)____________ [NUMERIC TEXT BOX.
ACCEPTABLE RANGE 0-99.]
b. Don’t know or not sure [CHECK BOX. MUTUALLY EXCLUSIVE
RESPONSE.] [CODE RESPONSE AS 0 IF UNCHECKED OR 1 IF
CHECKED]
(Ask Q26 if Q25 is 2 or more, otherwise go to Q27)
26. Why was it necessary for you to have more than one appointment? (Mark all that
apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE
AS 0 IF UNCHECKED OR 1 IF CHECKED]
a. To provide additional paperwork/documentation (e.g., medical documents)
b. Additional tests
c. To follow up with questions/concerns
d. Initial appointment took too long
e. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
f. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE.]

27. Was the counselor during the planning phase of your program the same
counselor who conducted your initial evaluation? (Mark only one) [RADIO
BUTTONS. SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
c. Don’t know or not sure [99]
28. Did your counselor provide you with information about the Veterans Employment
Center in eBenefits? [RADIO BUTTONS. SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
c. Don’t know or not sure [99]
29. Did you register for the Veterans Employment Center in eBenefits? [RADIO
BUTTONS. SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
c. Don’t know or not sure [99]
(Ask Q30 if Q29 is No, go to Q31 if Yes )
30. Why didn’t you register for the Veterans Employment Center in eBenefits?
[CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0 IF
UNCHECKED OR 1 IF CHECKED]
a. Not aware of the Veterans Employment Center
b. Opted not to use the Veterans Employment Center
c. Other (Specify)___________________________ [TEXT BOX, FORCE TEXT
IF RESPONSE IS SELECTED, 50 CHARACTER MAX.]
d. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE.]
84

(Paper Questionnaire Only: Ask Q31-Q34 if started the rehabilitation program/plan
selection and found entitled, otherwise go to Q35)
31. Did your final rehabilitation plan include your original vocational training choice?
(Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
c. Don’t know or not sure [99]
d. Not Applicable, have not started plan selection and been found
entitled(Online Only Response, if selected, go to Q34) [96]
(Ask Q32 if Q31 is No or Don’t know, otherwise go to Q33)
32. Why didn’t your final rehabilitation plan include your original vocational training
option? (Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE
EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED]
a. Missing documentation
b. Poor labor market
c. Medical reasons
d. Another vocational option suited my needs better
e. Other (Specify)________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
f. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE.]

(Paper Questionnaire Only: Ask Q33 if started one of the five tracks, otherwise go to
Q34)
33. From the time you signed your rehabilitation plan, how long did it take before you
started your program of vocational rehabilitation (e.g., one of the five
rehabilitation program options)? (Mark only one) [RADIO BUTTONS. SINGLE
RESPONSE.]
a. Less than one month [1]
b. 1-3 months [2]
c. 4-6 months [3]
d. More than 6 months [4]
e. Don’t know or not sure [99]
f. Not applicable, have not started program of vocational rehabilitation
(Online Only Response) [96]
The following questions ask you to rate various aspects of your experience with
Vocational Rehabilitation and Employment, using a scale of 1 to 10 where 1 is
Unacceptable, 10 is Outstanding, and 5 is Average. [SHOW ON SAME PAGE AS THE
QUESTION THAT FOLLOWS]
34. Please rate your experience with the VR&E benefit application process on the
following items: (Mark only one per row) [SHOW RESPONSES IN GRID WITH
10-POINT SCALE IN COLUMNS AND ATTRIBUTES/RESPONSES IN ROWS
(SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC DETAILS OF
85

LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS, ALTERNATE
SHADES IN ROWS. SINGLE RESPONSE PER ROW. RANDOMIZE ALL
ATTRIBUTES EXCEPT THE LAST ONE.]
a. Ease of completing the application [ALLOW N/A RESPONSE] [1-10,
N/A=99]
b. Timeliness of eligibility/entitlement notification [ALLOW N/A RESPONSE]
[1-10, N/A=99]
c. Flexibility of application methods [ALLOW N/A RESPONSE] [1-10,
N/A=99]
d. Overall rating of application process
35. Using the same 1 to 10 scale, where 1 is Unacceptable, 10 is Outstanding, and 5
is Average, please rate your experience with Vocational Rehabilitation and
Employment counselors during the initial evaluation of your benefit application on
the following items: (Mark only one per row) [SHOW RESPONSES IN GRID
WITH 10-POINT SCALE IN COLUMNS AND ATTRIBUTES/RESPONSES IN
ROWS (SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC
DETAILS OF LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS,
ALTERNATE SHADES IN ROWS. SINGLE RESPONSE PER ROW.
RANDOMIZE ALL ATTRIBUTES EXCEPT THE LAST ONE.]
a. Promptness of scheduling appointments or returning calls [ALLOW N/A
RESPONSE] [1-10, N/A=99]
b. Courtesy of the counselor [ALLOW N/A RESPONSE] [1-10, N/A=99]
c. Knowledge of the counselor [ALLOW N/A RESPONSE] [1-10, N/A=99]
d. Counselor’s concern for your needs [ALLOW N/A RESPONSE] [1-10,
N/A=99]
e. Timeliness of completing your initial evaluation [ALLOW N/A
RESPONSE] [1-10, N/A=99]
f. Overall counselor experience
36. Why did you give your overall experience with your counselor that rating? (Open
Capture) [OPEN CAPTURE. 1000 CHARACTER MAX. ALLOW NO
COMMENT, MUTUALLY EXCLUSIVE CHECK BOX. CODE NO COMMENT AS
0 IF UNCHECKED AND 1 IF CHECKED]
37. If you were previously found not to be entitled to VR&E benefits, why were you
found not entitled? (Mark all that apply) [CHECK BOXES. MULTIPLE
RESPONSE. CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF
CHECKED]
a. Did not meet eligibility requirements
b. Found suitable employment
c. Exceeded 12-year eligibility period
d. Disability rate less than 20%
e. No remaining entitlement—used 48 months
f. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
g. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE]
h. Not applicable [MUTUALLY EXCLUSIVE RESPONSE]
86

Benefit Entitlement
As a reminder, your responses will be kept completely confidential and will not affect
any current or future benefits you may receive. [SHOW ON THE SAME PAGE AS THE
QUESTION THAT FOLLOWS]
The following question asks you to rate various aspects of your experience with
Vocational Rehabilitation and Employment using a scale of 1 to 10 where 1 is
Unacceptable, 10 is Outstanding, and 5 is Average. [SHOW ON THE SAME PAGE AS
THE QUESTION THAT FOLLOWS]

38. Please rate your Vocational Rehabilitation and Employment benefit on the
following items: (Mark only one per row) [SHOW RESPONSES IN GRID WITH
10-POINT SCALE IN COLUMNS AND ATTRIBUTES/RESPONSES IN ROWS
(SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC DETAILS OF
LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS, ALTERNATE
SHADES IN ROWS. SINGLE RESPONSE PER ROW. RANDOMIZE ALL
ATTRIBUTES EXCEPT THE LAST ONE.]
a. Amount of benefits received [ALLOW N/A RESPONSE] [1-10, N/A=99]
b. Effectiveness of benefit/service in preparing and obtaining suitable
employment [ALLOW N/A RESPONSE] [1-10, N/A=99]
c. Timeliness of receiving benefit payment [ALLOW N/A RESPONSE] [1-10,
N/A=99]
d. Overall rating of benefit payment

Overall Application Experience
39. Thinking about ALL aspects of your experience applying for Vocational
Rehabilitation and Employment benefits, please rate VA Vocational
Rehabilitation and Employment overall, using a scale of 1 to 10 where 1 is
Unacceptable, 10 is Outstanding, and 5 is Average. (Mark only one) [SHOW
RESPONSES IN GRID WITH 10-POINT SCALE IN COLUMNS AND SINGLE
ROW (SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC
DETAILS OF LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS,
SINGLE RESPONSE PER ROW.] [1-10]

Overall Experience with VA
40. Taking into consideration all of the non-medical benefits (e.g., education,
compensation, pension, home loan guaranty, vocational rehabilitation and
employment, insurance, etc.) you have applied for or currently receive, please
rate your experience with VA overall, using a scale of 1 to 10 where 1 is
Unacceptable, 10 is Outstanding, and 5 is Average. (Mark only one) [SHOW
87

RESPONSES IN GRID WITH 10-POINT SCALE IN COLUMNS AND SINGLE
ROW (SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC
DETAILS OF LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS,
SINGLE RESPONSE PER ROW.] [1-10]
41. How likely are you to inform other Veterans about your experience with VA
benefits or services? (Mark only one) [RADIO BUTTONS. SINGLE
RESPONSE.]
a. Definitely will not [1]
b. Probably will not [2]
c. Probably will [3]
d. Definitely will [4]
As a reminder, your responses will be kept completely confidential and your e-mail
address will not be sent to VA with any responses on this survey. [SHOW ON THE
SAME PAGE AS THE QUESTION THAT FOLLOWS.]
42. Would you like to provide an e-mail address so VA can contact you with general
information about VA benefits and services? (Mark only one) [RADIO
BUTTONS. SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
c. I do not have an email address [96]
d. Prefer not to answer [98]
(Ask Q44 if Yes in Q43)
43. Please enter your preferred email address where you would like to be contacted:
(Open Capture)
a. Email: [TEXT BOX. 100 CHARACTER MAX.]
About You
44. Are you currently enrolled in a 2- year college (e.g., community college), 4-year
college (e.g., university), postgraduate program, technical or trade school, flight
school or on-the-job training program? [RADIO BUTTONS. SINGLE
RESPONSE.]
a. Yes [1]
b. No [0]
(Ask Q45-49p if Q44 is yes, otherwise go to Q49q)
45. Are you a … [RADIO BUTTONS. SINGLE RESPONSE.]
a. Part-time student [1]
a. Full-time student [2]
b. Not currently enrolled [3]
c. Don’t know or not sure [99]

88

46. What is the format of the program you are enrolled in? (Mark only one) [RADIO
BUTTONS. SINGLE RESPONSE.]
a. Traditional (classes in classroom/school facility) [1]
b. Online (classes on the Internet) [2]
c. Mixed (classroom and online) [3]
47. What type of degree/training program are you currently pursuing? (Mark only
one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. On-the-job training or apprenticeship [1]
b. Certificate/License [2]
c. Associate degree [3]
d. Bachelor’s degree [4]
e. Master’s degree [5]
f. Doctorate [6]
48. What type of academic institution or training facility are you enrolled in? (Mark
only one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. 2-year college (e.g., community college) [1]
b. 4-year college (e.g., university) [2]
c. Postgraduate program [3]
d. Technical or trade school [4]
e. Flight school [5]
f. Job training site [6]
g. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.] [97]

(Ask Q49a if enrolled in a 2-year college in Q49, otherwise go to Q49b)
49a. (Online only) Do you plan on attending a 4-year college in the future? (Mark
only one) [RADIO BUTTONS. SINGLE RESPONSE.]
d. Yes [1]
e. No [0]
f. Prefer not to state [98]
49b. (Online only) Prior to the current program, what was the last year of school you
completed? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. High school graduate or equivalent [1]
b. Trade/Technical school [2]
c. Some college (2-year program) [3]
d. Some college (4-year program) [4]
e. 2-year college degree [5]
f. 4-year college degree [6]
g. Some graduate courses [7]
h. Advanced degree [8]
i. Prefer not to answer [98]

89

49c. (Online only) Why did you select your current school/training facility? (Mark all
that apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH
RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED]
q. Lower tuition/program costs
r. Good counselors
s. Convenient location
t. Easy initial application process
u. Convenient course/program enrollment process
v. Variety of course/training offerings
w. Variety of available student support
x. School specialization in subject of interest
y. Reputation of school/training facility
z. Reputation of instructors
aa. Past experience
bb. Recommendation from friends/relatives
cc. Availability of online classes
dd. Flexibility of course/training scheduling
ee. Financial aid
ff. Other (Specify) _____________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
49d. (Online only) When did you first enter into your current degree/training
program? (Open Capture)
c. Please enter the month and year: mm _____ yy _______ [TWO
NUMERICTEXT BOXES; ONE FOR MONTHS [ACCEPTABLE RANGE
1-12) AND ONE FOR TWO-DIGIT YEAR (ACCEPTABLE RANGE 0099)]
d. Prefer not to answer [CHECK BOX. MUTUALLY EXCLUSIVE
RESPONSE.] [CODE RESPONSE AS 0 IF UNCHECKED OR 1 IF
CHECKED]

49e. (Online only) How many years have you completed in your current
degree/training program? (Open Capture) If you have completed less than 1
year, enter 0.
d. Number of years _________ [NUMERIC TEXT BOX. ACCEPTABLE
RANGE 0-99]
b. Prefer not to answer [CHECK BOX. MUTUALLY EXCLUSIVE
RESPONSE.] [CODE RESPONSE AS 0 IF UNCHECKED OR 1 IF
CHECKED]

49f. (Online only) Why did you select your current degree/training program? (Mark
all that apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH
RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED]
n. Preparation for career
o. Salary/Wages in associated careers
90

p.
q.
r.
s.
t.
u.
v.
w.
x.
y.
z.

Status/Esteem associated with type of degree/program
Personal growth/development
Interested in subject matter
Number of course requirements
Preparation for advanced degree
Ease of completion requirements
Reputation of instructors
Recommendation from friends/relatives
Availability of online classes
Flexibility of course/training scheduling
Other (Specify) _____________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]

49g. (Online only) Have you ever taken any time off from your current
degree/training program? (Mark only one) [RADIO BUTTONS. SINGLE
RESPONSE.]
d. Yes [1]
e. No [0]
f. Prefer not to answer [98]
(Ask Q49h-49i if Q49g is yes, otherwise go to Q49j)
49h. (Online only) How much time have you taken off from your current
degree/training program? (Open Capture) Please respond using any or all of
the following categories.
d. Days (0-99 days) __________ [NUMERIC TEXT BOX. ACCEPTABLE
RANGE 0-99.]
e. Months (0-99 months) _________ [NUMERIC TEXT BOX.
ACCEPTABLE RANGE 0-99.]
f. Years (0-99 years) _________ [NUMERIC TEXT BOX. ACCEPTABLE
RANGE 0-99.]
d. Don’t know or not sure [CHECK BOX. MUTUALLY EXCLUSIVE
RESPONSE.] [CODE RESPONSE AS 0 IF UNCHECKED OR 1 IF
CHECKED]

49i. (Online only) Why did you take time off? (Open Capture) [OPEN-END. TEXT
BOX. 1000 CHARACTER MAX. ALLOW NO COMMENT, MUTUALLY
EXCLUSIVE CHECK BOX. CODE NO COMMENT AS 0 IF UNCHECKED
AND 1 IF CHECKED ]
________________________________________________________________
______________________________________________________
49j. (Online only) Have you been called to active duty at any point during your
current degree/training program? (Mark only one) [RADIO BUTTONS. SINGLE
RESPONSE.]
d. Yes [1]
91

e. No [0]
f. Prefer not to answer [98]
(Ask Q49k if Q49j is yes, otherwise go to Q49l)
49k. (Online only) How long was your call to active duty? (Open Capture)
c. Months (0-99 months) _________ [NUMERIC TEXT BOX.
ACCEPTABLE RANGE 0-99.]
b. Don’t know or not sure [CHECK BOX. MUTUALLY EXCLUSIVE
RESPONSE.] [CODE RESPONSE AS 0 IF UNCHECKED OR 1 IF
CHECKED]

49l. (Online only) Have you ever been on academic probation or had less than
satisfactory standing with your school/training program? (Mark only one)
[RADIO BUTTONS. SINGLE RESPONSE.]
d. Yes [1]
e. No [0]
f. Prefer not to answer [98]

49m.(Online only) Do you plan to obtain a degree or completion certificate in your
current field of study/training? (Mark only one) [RADIO BUTTONS. SINGLE
RESPONSE.]
e. Yes, from the degree/training program at my current school/facility [1]
f. Yes, from a degree/training program at another school/facility [2]
g. No [3]
h. Prefer not to answer [98]
(Ask Q49n-49o if Q49m is yes, otherwise go to Q49p)
49n. (Online only) When do you expect to complete or graduate with a degree or
completion certificate in your current field of study/training? (Open Capture)
c. Please enter the month and year: mm _____ yy _______ [TWO
NUMERICTEXT BOXES; ONE FOR MONTHS [ACCEPTABLE RANGE
1-12) AND ONE FOR TWO-DIGIT YEAR (ACCEPTABLE RANGE 1299)]
b. Prefer not to answer [CHECK BOX. MUTUALLY EXCLUSIVE
RESPONSE.] [CODE RESPONSE AS 0 IF UNCHECKED OR 1 IF
CHECKED]

49o. (Online only) Do you plan to continue your enrollment as a full-time student until
you complete or graduate your degree/training program? (Mark only one)
[RADIO BUTTONS. SINGLE RESPONSE.]
d. Yes [1]
e. No [0]
f. Prefer not to answer [98]
92

49p. (Online only) Which of the following services are available from your current
school/training facility? (Mark all that apply) [CHECK BOXES. MULTIPLE
RESPONSE. CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF
CHECKED]
j. Academic counseling
k. Tutoring
l. Financial counseling
m. Dependent care services (e.g., babysitting, elder care)
n. Employment counseling
o. Financial aid
p. Technology assistance (e.g., internet access, computer, etc.)
q. Other (Specify) _____________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
r. Don’t know [MUTUALLY EXCLUSIVE RESPONSE]
49q. (Online only) What concerns, if any, do you have about achieving your
educational goals? (Mark all that apply) [CHECK BOXES. MULTIPLE
RESPONSE. CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF
CHECKED]
k. Academic requirements
l. Difficulty of subject matter
m. Financial requirements
n. Family obligations
o. Employment obligations
p. Course scheduling
q. Time commitment (i.e., amount of time required)
r. Availability of technology (e.g., access to internet/computer)
s. Other (Specify) _____________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
t. Do not have concerns [MUTUALLY EXCLUSIVE RESPONSE]
49r. (Online only) Which of the following services would you like or expect in order
to achieve your educational goals? (Mark all that apply) [CHECK BOXES.
MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0 IF UNCHECKED
OR 1 IF CHECKED]
j. Academic counseling
k. Tutoring
l. Financial counseling
m. Dependent care services (e.g., babysitting, elder care)
n. Employment counseling
o. Financial aid
p. Technology assistance (e.g., internet access, computer, etc.)
q. Other (Specify) _____________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
r. Don’t know [MUTUALLY EXCLUSIVE RESPONSE]

93

49s. (Online only) What are your personal career goals? (Mark all that apply)
[CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0 IF
UNCHECKED OR 1 IF CHECKED]
j. Obtain financial security
k. Achieve work-life balance
l. Become an independent business owner
m. Become a manager
n. Become an executive
o. Work internationally
p. Contribute to society
q. Work in a specialized field (e.g., technology, medicine, etc.)
r. Other (Specify) _____________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]

50. Are you currently employed? (Mark only one) [RADIO BUTTONS. SINGLE
RESPONSE.]
a. Yes [1]
b. No [0]
c. Prefer not to state [98]
(Ask Q50a-50b if currently employed, otherwise go to Q50c)
50a. (Online only) How many hours do you currently work in a typical week? (Open
Capture)
c. Hours (0-40 hours) _________ [NUMERIC TEXT BOX. ACCEPTABLE
RANGE 0-40.]
d. Don’t know or not sure [CHECK BOX. MUTUALLY EXCLUSIVE
RESPONSE.] [CODE RESPONSE AS 0 IF UNCHECKED OR 1 IF
CHECKED]
50b. (Online only) Are you currently employed in a field related to your current
degree/training program? (Mark only one) [RADIO BUTTONS. SINGLE
RESPONSE.]
d. Yes [1]
e. No [0]
f. Prefer not to answer [98]
50c. (Online only) Are you pursuing employment in your current field of study? (Mark
only one) [RADIO BUTTONS. SINGLE RESPONSE.]
d. Yes [1]
e. No [0]
f. Prefer not to answer [98]
(Ask Q50d if Q50c is yes, otherwise go to Q51)
50d. (Online only) Upon completion of your current degree/training program, what
will be your primary method of obtaining employment information?(Mark only
one) [RADIO BUTTONS. SINGLE RESPONSE.]
94

k.
l.
m.
n.
o.
p.
q.
r.
s.

VA counselor [1]
Recommendations of friends/family [2]
Student career/employment center [3]
Local or state job services [4]
Federal job services [5]
Newspaper [6]
Online job site [7]
Private employment agency [8]
Other (Specify) _____________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.] [97]
t. Don’t know [99]

51. Do you have any other comments or concerns about your experience? (Open
Capture) [OPEN-END. TEXT BOX. 1000 CHARACTER MAX. ALLOW NO COMMENT,
MUTUALLY EXCLUSIVE CHECK BOX. CODE NO COMMENT AS 0 IF UNCHECKED
AND 1 IF CHECKED]

95

Appendix G
List of Acronyms
AAPOR
ANOVA
BAS
BPA
BRE
CAPS
COR
DTA
EDIPI
EDX
FAR
FY
GPO
ICR
JDP
LGY
LWO
MAR
MCAR
MCMC
MNAR
NPC
OIF
OEF
OMB
OSAT
RO
SSN
US
USA
VA
VADIR
VAPSD
VBA
VOV
VR&E
VSO

American Association for Public Opinion Research
Analysis of Variance
Benefits Assistance Service
Blanket Purchase Agreement
Business Reply Envelope
Centralized Account Processing System
Contracting Officer’s Representative
Data Transfer Agreement
Electronic Data Interchange Personal Identifier
Enterprise Data Exchange
Federal Acquisition Regulations
Fiscal Year
Government Printing Office
Information Collection Request
J.D. Power
Loan Guaranty Service
Letter Work Order
Missing At Random
Missing Completely At Random
Markov chain Monte Carlo algorithm
Missing Not At Random
NPC, Inc. Integrated Print and Digital Solutions
Operation Iraqi Freedom
Operation Enduring Freedom
Office of Management and Budget
Overall Satisfaction Index
Regional Office
Social Security Number
United States
United States of America
Department of Veterans Affairs
VA DoD Identity Repository
VA Publications Services Division
Veterans Benefits Administration
Voice of the Veteran
Vocational Rehabilitation and Employment Service
Veterans Service Organizations

96