Download:
pdf |
pdfVoice of the Veteran Line of Business Tracking Study
Vocational Rehabilitation and Employment
Non-Participant
Fiscal Year 2015 Non-Response Bias Analysis
VETERANS BENEFITS ADMINISTRATION
[FY15 REPORT]
Table of Contents
Executive Summary .................................................................................................................................. 4
Introduction ............................................................................................................................................. 5
Methodology ............................................................................................................................................ 6
2.1 J.D. Power Index Model ................................................................................................................... 6
2.2 Sampling........................................................................................................................................... 9
2.3 Data Collection ................................................................................................................................. 9
Non-Response Bias Analysis................................................................................................................... 10
3.1 Survey Yield .................................................................................................................................... 14
3.2 Missing Data Patterns and Mechanisms ........................................................................................ 18
3.3 Margin of Error .............................................................................................................................. 18
3.3.1 Sampling Distribution ....................................................................................................... 19
3.3.2 Distribution of Overall Satisfaction Index Scores.............................................................. 20
3.3.3 Analysis for Demographic Differences .............................................................................. 20
3.3.4 Data Imputation Analysis for Demographic Differences .................................................. 23
Findings .................................................................................................................................................. 25
Conclusion .............................................................................................................................................. 26
References ............................................................................................................................................. 27
List of Appendices
Appendix A Missing Data Patterns and Mechanisms .................................................................... 29
Appendix B Item Response Rates .................................................................................................. 30
Appendix C Study Overview ........................................................................................................... 33
1.1 Study Background .................................................................................................................. 33
1.2 Methodology ......................................................................................................................... 33
1.3 Data Cleaning ......................................................................................................................... 34
1.4 Order generation and fulfillment process ............................................................................. 35
1.5 Reporting ............................................................................................................................... 35
Sample Plan Overview ................................................................................................................... 37
2.1 Sample Criteria ...................................................................................................................... 37
2.2 Fielding/Sampling Frequency ................................................................................................ 37
2.3 Data Transfer ......................................................................................................................... 37
2.4 Sample Cleaning Rules Glossary ............................................................................................ 38
2.5 Sample Selection.................................................................................................................... 39
2.6 Data Collection....................................................................................................................... 40
Appendix D Approaches to Effects of Non-Response Bias and Improving Response Rates .......... 41
1.1 Approach 1: Strategies to Maximize Response Rates ........................................................... 41
ii
1.2 Approach 2: Correcting Unit Non-response Bias with Sample Weighting and Survey Raking42
Strategies to Improve Response Rate ............................................................................................ 43
Appendix E Impact of FAR 8.8 ........................................................................................................ 44
1.1 Impact .................................................................................................................................... 44
Appendix F Survey Questionnaire.................................................................................................. 45
Appendix G List of Acronyms………………………………………………………………………………………..………….58
iii
Executive Summary
The Voice of the Veteran (VOV) Line of Business Tracking Satisfaction Research Study was
developed to establish continuous satisfaction measurement and incorporate direct Veteran
feedback in the decision making process in order to improve the level of service to
Servicemembers, Veterans, and their beneficiaries.
As part of this study, a survey was fielded in Fiscal Year 2015 (FY15) for the Department of
Veterans Affairs (VA), Veterans’ Benefits Administration (VBA) Vocational Rehabilitation and
Employment Service program (VR&E) Non-Participant. This survey is fielded annually on behalf
of the VR&E Service Program. The survey yielded a response rate of 5.60% (decrease of 0.16%
from FY14), which was lower than the estimated response rate submitted with the information
collection request (ICR) as well as lower than the Office of Management and Budget’s standard
of 80% (at the overall unit response rate).
OMB’s “Standards and Guidelines for Statistical Surveys,” Section 3.2, Guideline 3.2.9, notes
that a non-response analysis should be conducted for surveys with an overall unit response rate
of less than 80%. Therefore, J.D. Power (JDP) conducted the necessary statistical tests in
accordance with OMB’s guidelines in order to verify the validity of VR&E’s survey results for
FY15.
The initial 2015 analyses for these reports were done in consultation with Dr. Don Dillman, a
professor at Washington State University. Dr. Dillman is regarded as a key survey method
expert on non-response bias research and the report conforms to sound statistical research
practices in accordance with OMB standards. The analysis preformed also includes an iterative
survey raking procedure to derive sample weightings based on a simultaneous balancing
analysis of the demographic differences.
The statistical tests performed on the survey illustrate that no differences were found in the
Overall Satisfaction Index Score and Advocacy ratings (likelihood to reapply for the program) for
VR&E in FY15 after adjusting for non-response bias in age, gender, race, military branch, days of
service, war participation, and case status.
The VR&E survey is fielded to Veterans who dropped out of the program prior to completing a
rehabilitation plan. These individuals include applicants who never attended the initial meeting
with a counselor; applicants who were entitled to the program but did not pursue; and
applicants who started, but did not complete rehabilitation (i.e., negative closures).
The Overall Satisfaction Index score (582) and advocacy ratings likelihood to reapply for
program (2.63, rating 1-4) are not impacted in any meaningful way by non-response bias.
This analysis confirms that the data collected during Fiscal Year 2015 is valid for use by VBA.
4
Introduction
In an effort to achieve top level customer service, VBA partnered with J.D. Power to conduct
Veteran satisfaction research on its behalf. VBA’s Voice of the Veteran (VOV) Satisfaction
Initiative was established to continuously measure and improve the level of service to Service
members, Veterans, and their beneficiaries.
The intent of this initiative is to:
Reinstate VBA’s customer satisfaction research program in order to incorporate Veteran
feedback into the decision-making process,
Identify the critical factors to Veterans’ satisfaction with benefits and services provided
by VBA,
Provide continuous feedback to validate effectiveness of new initiatives and process
changes,
Provide decision-makers and stakeholders with timely and actionable feedback on a
continuous basis, and;
Identify and document best practices, and act as a vehicle to celebrate successful
interactions and experiences.
VBA’s VOV Line of Business Tracking Satisfaction Research Study was developed to continuously
field customer satisfaction survey instruments to provide Veteran and beneficiary feedback on
the following VBA lines of business and benefit programs: Compensation, Pension, Education,
Vocational Rehabilitation and Employment, and Loan Guaranty (including Specially Adapted
Housing). In support of this effort, in FY15, JDP fielded a survey instrument regarding the NonParticipant process on behalf of VR&E. The purpose of the VR&E Non-Participant study is to
identify factors that may have led Veterans to discontinue the vocational rehabilitation
program and to determine ways to improve the level of services provided.
The survey instrument for the VR&E Non-Participant study was developed in collaboration with
VR&E and in accordance with OMB’s guidelines concerning statistical collection procedures and
methods. After the initial survey instrument was designed, cognitive labs using the “think
aloud” method were conducted to evaluate user experience when filling out the survey. Prior
to the FY15 fielding of the VRE Non-Participant survey, a Benchmark (pilot) study was
conducted from October 2012 through January 2013 to further assess the effectiveness of the
methodology and conformance to OMB’s standards. This was also fielded in 2014 and the 2015
fielding will be the third iteration of the survey fielding.
5
Methodology
2.1 J.D. Power Index Model
J.D. Power defines customer satisfaction as a measure of how well product or service
experiences fit the expectations of customers. All JDP index models assume a two-tiered
regression model involving factors and attributes. Each customer experience is influenced by
several factors (i.e. first tier), which in turn, are influenced by several attributes or drivers (i.e.
second tier). A diagram of the index model follows on the subsequent page.
In order to begin the index model calculation, each set of attributes within a factor are used to
predict the Overall Satisfaction rating (sub-OSAT) for that factor. An importance weight is
assigned to each attribute, where the weight of “importance” of each attribute is defined as the
ability of that attribute to predict Overall Satisfaction. A multiple regression model is used to
estimate the attribute weights. This model produces the “bottom” level weights and is
computed for each factor separately. The bottom level weights are rescaled so that they add up
to one within each subcategory. As a result, the percentage of total explained variation in the
sub-OSAT that is due to a particular attribute constitutes that attribute’s importance weight
within its respective factor.
Following the calculation of attribute (i.e. bottom level) weights, the factor (i.e. top-level)
weights are calculated. Factor scores are calculated by taking the sum of the product of the
attribute rating scores and the attribute importance weights. This model produces the “top”
level weights and these weights are rescaled so that they add up to one. Thus, the percentage
of the total explained variation in the Overall Satisfaction rating that is due to a particular subOSAT constitutes that factor’s importance weight.
After all factor scores are computed, they are weighted so that some contribute more to
Overall Satisfaction than others, based on the index importance weights. The index score is
subsequently calculated by taking the sum of the product of all of the factor scores and the
factor importance weights. Finally, both the index and factor scores are multiplied by 100 so
that the range of each is 100 (if all attributes were rated 1) to 1,000 (if all attributes were rated
10).
By applying the importance weights derived from the two-tiered modeling approach, JDP
creates a weighted index score that ranges from a low of 100 to a high of 1,000. This index
approach has the benefit of being highly reliable and valid and provides increased ability to
discriminate the performance levels of companies.
6
VR&E Non-Participant Process Index Weights
In working with VR&E’s subject matter experts and leadership, the design of its survey
encompasses the factors and attributes as outlined in the tables on the next page. The factors
(Benefit Information, Contact with VA, Benefit Application, and Benefit Entitlement) and
attributes (Ease of Accessing Information, Availability of Information, etc.) represent VR&E’s
Non-Participant Index Model in FY15. The corresponding weights for each factor and attribute
are the weights based on the above index model calculation. The weights are derived from the
relative importance of each factor or attribute to the respondents.
7
Table 2.0 Index Model Weights
VR&E Index Model Weights
Table 2.1 Weights by Attribute
VR&E Weights by Attribute
Effective Weight
Effective Weight
Benefit Information
Intake Counselor
Service Counselor
Benefit Entitlement
29.78%
52.94%
3.32%
13.96%
VR&E Benefit Application Process
Ease of completing the application
7.76%
Timeliness of eligibility notification
9.19%
Flexibility of application methods
12.83%
VR&E Intake Counselor
Promptness of scheduling
appointments
Courtesy of the app. counselor
7.13%
Knowledge of the app. counselor
9.11%
Counselor’s concern for your needs
Timeliness of completing your initial
evaluation
18.11%
9.12%
9.48%
VRE Service Counsellors
Promptness of scheduling
appointments
Courtesy of the counselor
0.46%
Knowledge of the counselor
0.53%
Counselor’s concern for your needs
Timeliness of completing your initial
evaluation
1.12%
0.74%
0.47%
VRE Benefit Entitlement
Effectiveness of benefit/service
Timeliness of receiving
benefit/services
5.08%
8.88%
8
2.2 Sampling
The VR&E survey was fielded to Veterans who dropped out of the program prior to completing
a rehabilitation plan. These individuals include applicants who never attended the initial
meeting with a counselor; applicants who were entitled to the program but did not pursue; and
applicants who started, but did not complete rehabilitation (i.e., negative closures).
J.D. Power mailed approximately 5,000 surveys to Veterans across the nation in FY15. The
targets number of completed surveys was 1,500. The actual number of completed surveys
received was 354. The sample used in this study was provided by VR&E and was a random
sample from the available Veterans provided by VR&E.
Survey Instrument
VRE Non-Participant
Methodology
Fielding Frequency
Total Mailouts in
FY15
Mail Only
Annually
5,000
2.3 Data Collection
During the survey fielding period, self-administered paper surveys were collected. While
verbatim responses are recorded by a live survey processor, responses from paper surveys are
scanned through automated imaging software. Survey returns undergo quality assurance to
validate the accuracy of responses captured.
Respondents from each study completed the survey on paper and received two separate
mailings:
1st Mailing: Survey Package, which included a cover letter introducing the study to the
respondent, a paper survey, and a business reply envelope.
2nd Mailing: Survey Package, which included a cover letter, a paper survey, and a business
reply envelope.
Each time the surveys were deployed, the postcards and survey packages were subject to a
proof approval process that utilized three levels of approvals by J.D. Power, Benefits Assistance
Service (BAS), and VA Publications Services Division (VAPSD). After the print vendor mailed the
survey packages, mail receipts were sent to VBA.
During the survey fielding period, JDP provided a toll-free survey hotline and dedicated e-mail
address to answer survey-related inquiries and to provide assistance to respondents for
completing the surveys.
9
The telephone and e-mail helpdesk was staffed by three JDP employees who answered
inquiries during regular business hours (8:00am-5:00pm PST, Monday thru Friday). A voice
message system was available to receive phone messages so after-hours calls could be
responded to the following business day. An automatically generated e-mail response was sent
to all e-mail inquiries informing respondents that their e-mail was received and they would
receive a response within 24 hours. JDP helpdesk representatives logged each survey-related
inquiry in a password protected spreadsheet documenting the reason for the inquiry, the
resolution provided, and the contact information of each caller. At the end of each month, a log
containing all inquiries was provided to the Contracting Officer Representative (COR) for
review. If non-survey related high-severity benefit inquiries were received, J.D. Power
contacted the COR immediately with the respondent’s contact information.
Throughout the course of the program, weekly status meetings were held between JDP and
BAS to discuss survey administration. Biweekly status meetings were held between the
Government Printing Office print vendor, JDP, BAS and VAPSD to discuss the printing and
mailing of the survey materials.
Non-Response Bias Analysis
The purpose of the non-response bias analysis is to ascertain the possible causes of variance in
response rates among different respondent demographics and/or determine if any bias has
been introduced with a low response rate. Given that the Voice of the Veteran VR&E NonParticipant study had an overall unit response rate of 5.60% in Fiscal Year 2015, the following
section examines whether a low response rate or other factors may have caused respondent
bias to occur.
The Office of Management and Budget’s Questions and Answers, “When Designing Surveys for
Information Collections” dated January 2006, and “Standards and Guidelines for Statistical
Surveys” dated September 2006 (see References) provide guidelines on acceptable survey
design and response rates. OMB guidelines recommend a non-response bias evaluation for
surveys with an overall unit response rate of less than 80%.
In addition to the above referenced documents prepared by OMB, J.D. Power assessed other
source documents that were written and published by the Federal Committee on Statistical
Methodology, “Statistical Policy Working Paper 17, Survey Coverage” (1990) and “Statistical
Policy Working Paper 31, Measuring and Reporting Sources of Error in Surveys” (2001).
While high response rates are always desirable in surveys, JDP finds an 80% response rate is not
achievable for most voluntary, satisfaction-based, survey research studies (Malhotra & Birks,
2007). In particular, survey research studies that do not provide an incentive are subject to not
achieving an 80% response rate. To better illustrate this point, the Dillman Method for survey
fielding was discussed in Dillman, D. A. (2014, pp. 22), detailing the efforts to attain an 80%
response rate.
10
A survey instrument was fielded to 600 students at the University of Washington, the same
University that sponsored the study. After 5 attempts to solicit a response in a closed university
setting, as well as offering a monetary incentive to complete the study, they failed to achieve
an 80% response rate garnering only a 77% response rate. The JDP ream met with the VA
Contracting Officer Representative to discuss current trends and realistic response rates. As
noted JDP does not believe that an 80% response rate is achievable and this concern was
shared with the Benefits Assistance Service team.
JDP conducted the following non-response bias analysis to determine if the respondents (i.e.
those who completed the survey) were different in a meaningful way from the nonrespondents (i.e. those who were sent a survey, but did not complete it). Chi-squared analyses
consist of comparisons between respondents and non-respondents on available demographic
variables such as gender, age, race, geographical region, war participation (service era), and
military service branch. The U.S. states were converted to standard USA census regions
(Midwest, Northeast, South, and West) in order to aggregate the data and enhance regional
comparisons.
Throughout this report, we are conducting statistical analyses to compare survey respondents
and non-respondents. Frequently used statistical tests can include the T-Test, Chi-Square, or
Analyses of Variance (ANOVA). These tests generate relevant t-statistics, Chi-Squares, or F
statistics that are reported. The magnitude of the statistic’s value (either positive or negative)
measures the size of the difference relative to the variation in the data. If the statistic is not
large enough to generate a probability (p-value) less than .05, then it falls below the accepted
standard probability cut-off level that indicates whether a statistical difference is significant. If
a difference is not significant, statisticians regard these results as part of the normal sample
variation that occurs within the same population. Throughout this report, the probability pvalue standard of “must be less than .05 to be significant” is used for all statistics reported.
Significant differences were found between the survey respondent and non-respondent
samples on gender (Table 3b) such that there were more female than male respondents:
Table 3b. Comparing Gender for Respondents and Non-Respondents
Gender by Respondent Type (%)
Female
Male
Survey
Respondents
NonRespondents
Total
20
80
15
85
16
84
Statistic
DF
Value
Prob
Chi-Square
1
4.5
.03
11
Significant differences were found with the population based on age generation as shown in
Table 3c, such that a larger number of older Veterans and a fewer number of generation X and
YZ Veterans completed the survey:
Table 3c. Comparing Age Generation for Respondents and Non-Respondents
Age Generation by Respondent Type (%)
Survey
NonRespondents Respondents
Baby & Pre-Boomer
(ages 50-68)
Generation X
(ages 37-49)
Generation YZ
(ages 24-36)
Total
59
32
33
23
27
27
18
41
40
Statistic
DF
Value
Prob
Chi-Square
2
93
< .0001
Significant differences were found between the survey respondent and non-respondent
samples on race (Table 3d), such that there were fewer white and more black and other that
responded to the survey:
Table 3d. Comparing Race for Respondents and Non-Respondents
Race by Respondent Type (%)
White
Asian
Black
Other
Survey
Respondents
NonRespondents
Total
45
6
23
26
55
6
22
17
55
6
22
18
Statistic
DF
Value
Prob
Chi-Square
3
17
.0007
No significant differences were found between the survey respondent and non-respondent
samples on census region (Table 3e).
Table 3e. Comparing Census Region for Respondents and Non-Respondents
U.S. Census Region by Respondent Type (%)
Midwest
Northeast
South
West
Survey
Respondents
NonRespondents
Total
21
18
35
27
20
20
35
24
20
20
35
24
Statistic
DF
Value
Prob
Chi-Square
3
2.09
.55
12
Significant differences were found with the population based on Military Service Branch as
shown in Table 3f, such that smaller proportion of Army and Marine veterans responded to the
survey compared to the population:
Table 3f. Comparing Military Service Branch for Respondents and Non-Respondents
Military Service Branch by Respondent Type (%)
Survey
Respondents
NonRespondents
Total
Air Force
16
11
12
Army
47
53
52
Marines
7
12
12
Navy
14
14
14
Other
16
10
10
Statistic
DF
Value
Prob
Chi-Square
4
23
.0002
Significant differences were found with the population based on War Service as shown in Table
3g, such that a larger number of Veterans from wars before OEF/OIF completed the survey
compared to non-respondents:
Table 3g. Comparing War Participation in OIF and OEF for Respondents and Non-Respondents
War Service by Respondent Type (%)
Survey
NonRespondents Respondents Total
All other
wars
OEF/OIF
75
61
62
25
39
38
Statistic
DF
Value
Prob
Chi-Square
1
24
< .0001
Note: OIF is Operation Iraqi Freedom and OEF is Operation Enduring Freedom.
Significant differences were found with the population based on days of active service as shown
in Table 3h, such that survey respondents were more likely to have served “1,000 days or less”
or “4,001 or more” days and less likely to have served 1,001 to 4,000 days compared to the
population:
13
Table 3h. Comparing Days of Active Service for Respondents and Non-Respondents
Days of Active Service by Respondent Type (%)
1,000 days
or less
1,0012,000 days
2,0014,000 days
4,001 days
or more
Survey
Respondents
NonRespondents
Total
37
30
31
24
31
31
14
22
21
25
17
18
Statistic
Chi-Square
DF
3
Value
24
Prob
<.0001
Significant differences were found with the population based on Case Status. Fewer surveys
were completed by Discontinued Veterans and more by Other Veterans:
Table 3i.e. Access: Comparing Case Status for Respondents and Non-Respondents
Case Status by Respondent Type (%)
Survey
NonRespondents Respondents
Discontinued
Employ Rehab
Other
52
21
27
64
21
15
Total
Statistic
Chi-Square
DF
2
Value
30
Prob
<.0001
63
21
15
3.1 Survey Yield
In accordance with OMB “Standards and Guidelines for Statistical Surveys,” an agency must
appropriately measure, adjust for, report, and analyze unit and item non-response, when the
intended response for a targeted population is not met.1 In assessing the survey data in
accordance with Section 3.2, and Guidelines 3.2.1-3.2.3, the unweighted unit response rate was
calculated as the ratio of the number of completed cases to the number of in-scope sample
cases (Ellis, 2000; AAPOR, 2000).
Table 3.1a below shows the sample distribution and response rate for VR&E Non-Participant
target population:
1
As defined by OMB and FCSM, unit non-response occurs when a respondent fails to respond to all required response items (i.e.,
fails to fill out or return a data collection instrument); item non-response occurs when a respondent fails to respond to one or
more relevant item(s) on a survey.
14
Table 3.1a. Sample Distribution and Response Rates for VR&E Non-Participant population
VR&E Non-Participant Population FY2015
Total records received
Duplicate records in sample file
Duplicate record history
Invalid Address
Invalid Values
Blanks
Do Not Contact
2
Total records available after cleaning
Total records selected
Undeliverable addresses
Total mailed (excludes undeliverable)
Total completed surveys
3
Total completed surveys with Overall Index Score
4
Total Sample Response Rate
5
Eligible Sample Response Rate
33,666
9,546
950
1,457
70
0
293
21,350
5,000
833
4,167
354
280
5.60%
8.50%
Of the 33,666 total records received from VR&E, 12,316 records were purged from the sample
due to cleaning rules such as duplicate records, invalid addresses and values, blanks, and do not
contact opt outs. From the 12,316 records purged, 950 records were cleaned out due to
duplicate records across VBA’s other business line surveys (i.e., duplicate record history). The
purpose of these cleaning rules is to prevent respondents from being re-contacted if they were
previously selected to participate in any of VBA’s business line surveys in the past 12 months.
The cleaning rules are a JDP and survey research best practice and is intended to promote
proper conduct in market research. About 37% of the total records provided by VR&E were
removed from the sample due to this cleaning rule. A high number of records were removed
because they were duplicates in the sample file provided by VR&E. It is unlikely the cleaning
rule impacted the unit non-response since we were able to secure the target number of records
(5,000) for the survey.
2
Glossary of sample cleaning rules included in Appendix E.
Findings in the report are based on the “Total completed surveys with Overall Index Score” (N=280).
4
Response rate calculation per OMB Standards and Guidelines for Statistical Surveys, section 3.2, guideline 3.2.9 (includes
undeliverables as number of non-contacted sample units known to be eligible).
5
Response rate calculation per Council of American Survey Research Organizations (CASRO) (includes number of completed
interviews with reporting units/number of eligible reporting units in sample).The American Association for Public Opinion
Research (AAPOR) also uses this method for calculation and cites CASRO (AAPOR Standard Definitions, 2008, pp. 34)
3
15
Table 3.1b. Weight/Person for Completed Surveys per Population
Completed Surveys
2015 Population
Weight/Person
354
33,666
95
In the Table 3.1b the 95 in the Weight/Person column means that every survey completed and
returned represents the views of 95 Veterans using VR&E benefits. This was calculated by
dividing the number of completed surveys into the population number.
To confirm the sample’s representativeness, a comparison was conducted among the total
records provided (33,666) and the records available after cleaning (21,350). The intent of this
analysis was to determine whether the cleaning rules caused the remaining sample to vary in a
meaningful way from the original sampling frame.
Table 3.1c indicates characteristics such as gender, age, and geographical region are similar
among the total records provided and the records available after cleaning. Regional USA State
comparisons yield differences that are less than 2% point. These comparisons suggest the
cleaning rules did not alter the proportion of respondent characteristics provided in the original
sampling frame.
Table 3.1c. Comparing Gender, Generation, and U.S. States to Total Population
Gender
Female
Male
Generation
Baby & Pre-Boomer
Generation X
Generation YZ
U.S. State
AK
AL
AR
AZ
CA
CO
CT
DC
DE
Total
Population
(%)
Records Available
(%)
% Point
Difference
16.97
83.03
17.01
82.99
0.04
-0.04
31.38
27.54
41.08
32.61
28.01
39.38
1.22
0.47
-1.69
0.49
2.55
1.12
2.49
9.23
2.4
0.76
0.16
0.19
0.48
2.34
1.07
2.57
9.18
2.26
0.73
0.17
0.21
-0.02
-0.21
-0.05
0.08
-0.05
-0.14
-0.03
0.02
0.02
16
Table 3.1c. Comparing Gender, Generation, and U.S. States to Total Population (Continued)
FL
GA
HI
IA
ID
IL
IN
KS
KY
LA
MA
MD
ME
MI
MN
MO
MS
MT
NC
ND
NE
NH
NJ
NM
NV
NY
OH
OK
OR
PA
RI
SC
SD
TN
TX
UT
VA
VT
WA
WI
WV
Total
Population
(%)
Records Available
(%)
% Point
Difference
6.68
4.78
0.79
0.83
0.37
2.03
1.81
0.61
1.39
1.28
1.14
1.84
0.44
2.13
1.21
1.6
0.86
0.5
3.62
0.18
0.72
0.61
1.05
0.76
1.2
2.92
2.83
2.59
1.89
2.14
0.24
2.8
0.27
2.26
14.25
0.56
3.74
0.34
2.62
0.98
0.53
6.69
4.33
0.77
0.9
0.4
2
1.89
0.56
1.36
1.26
1.18
1.84
0.49
2.28
1.27
1.56
0.74
0.52
3.39
0.18
0.68
0.69
1.04
0.78
1.19
2.99
2.86
2.7
1.92
2.12
0.25
2.58
0.36
2.19
15.17
0.59
3.45
0.45
2.85
1.03
0.51
0.01
-0.45
-0.02
0.06
0.03
-0.02
0.07
-0.05
-0.02
-0.02
0.04
0
0.06
0.15
0.07
-0.04
-0.12
0.01
-0.24
0.01
-0.04
0.08
-0.01
0.02
-0.01
0.07
0.03
0.12
0.03
-0.03
0.01
-0.22
0.08
-0.08
0.92
0.02
-0.29
0.11
0.23
0.05
-0.03
17
Table 3.1c. Comparing Gender, Generation, and U.S. States to Total Population (Continued)
Total
Population
(%)
Records Available
(%)
% Point
Difference
0.18
0.2
0.02
WY
3.2 Missing Data Patterns and Mechanisms
In accordance with the OMB “Standards and Guidelines for Statistical Surveys” Guidelines 3.2.9
and 3.2.11, an investigation of missing data patterns was performed on the 354 total surveys
received. In order to assess the distribution of missing data, a procedure was performed to
process missing values involving iterative multiple imputation chains using expectation–
maximization (MCMC) algorithms and divide these into distribution interval groupings,
Pierchala, Carl E. (2001). This was done on the key measures of the overall satisfaction index
(see Appendix A for calculation) and advocacy ratings related to Veterans’ likelihood to
recommend VA benefits.
As shown in Table 3.2, there were no indications of unusual patterns for missing data. For more
discussion of missing data mechanisms (MCAR, MAR, and MNAR), please see Appendix A.
Table 3.2. Missing Data Patterns in Satisfaction and Advocacy Ratings (0 = missing, 1 = data)
Group Means
Group
Overall
Satisfaction
Likelihood
to re-apply
to program
Freq
Percent
OSAT
Index
Age
% Male
1
2
3
4
0
0
1
1
0
1
0
1
13
2
32
233
5%
1%
11%
83%
640
828
573
578
51
44
49
52
92%
100%
91%
78%
3.3 Margin of Error
The margin of error expresses the maximum expected difference between the true population
parameter and a sample estimate of that parameter. It is often used to indicate the accuracy of
survey results. The larger the margin of error around an estimated value, the less accurate the
estimated value will be. Larger samples are more likely to yield results close to the true
population quantity and thus have smaller margins of error than smaller samples.
18
Based on a sample of 354 Veterans, the FY15 Overall Satisfaction Index for the VR&E NonParticipant study is 582 and has a margin of error of 27 index points, on a 1,000 point scale, at
the 95% confidence level. This indicates that if the survey were repeated many times with
different samples, the true mean Overall Satisfaction Index would fall within 27 index points
95% of the time.
Table 3.3 below demonstrates relative decreases in margin of error as the study sample size
increases. A 20% response rate (833 completes) would be associated with a margin of error of
17 index points, similar to the margin of error for a 30% response rate (1,250 completes).
Results from this analysis indicate the Overall Satisfaction Index (OSAT) calculated from the
VR&E Non-Participant study is an accurate measurement of the true population mean, which is
reported on a 1,000 point scale.
Table 3.3. Margin of Error for Larger Sample Sizes
Sample
Response
Rate
Completes
(N)
OSAT
(mean)
Standard
Deviation
Standard
Error
Margin of error
(95% confidence
interval)
4,167
4,167
4,167
4,167
4,167
4,167
4,167
8.50%
20%
30%
40%
50%
60%
80%
354
833
1,250
1,667
2,084
2,500
3,334
582
582
582
582
582
582
582
257
257
257
257
257
257
257
13.7
8.9
7.3
6.3
5.6
5.1
4.5
27
17
14
12
11
10
9
In the margin of error analysis noted on the previous page and in subsequent analyses included
in this report, the Overall Satisfaction Index Score is the main dependent variable and is the
basis for the analysis. The Overall Satisfaction Index score is the survey metric that VBA utilizes
to measure customer satisfaction and benchmark performance against other industries. It is the
primary measurement in all reports. The Overall Satisfaction Index encompasses all aspects of
the customer experience6, and can therefore be used as a reliable indicator for the presence or
absence of respondent bias in the survey results as a whole. For these reasons, the Overall
Satisfaction Index score is used as the main dependent variable in the margin of error analysis
and subsequent t-test analyses included in this report.
3.3.1
Sampling Distribution
Respondent characteristics such as gender and age were compared to that of the total sample
to determine whether respondents and non-responders differed on key variables of interest.
6
Explanation of J.D. Power Index Model Calculation included in Methodology.
19
Compared to the population of all eligible respondents (40,000), the survey respondents
demonstrate the same gender characteristics. Table 3.1.1 below illustrates 20% of survey
respondents were female and 80% were male, mirroring close to the total sample population.
The distribution of age shows that survey respondents tend to be older. Table 3.1.1. Comparing
Gender and Age of Survey Respondents to the Total Sample
Gender
Female
Male
Age Generation
Baby & Pre-Boomer
Generation X
Generation YZ
3.3.2
Respondents
(%)
Sample
Size (N)
Total
Sample (%)
Sample
Size (N)
% Point
Difference
20
80
70
284
16
84
779
4,221
-4
4
56
25
19
199
88
67
34
27
39
1,677
1,349
1,974
-23
2
21
Distribution of Overall Satisfaction Index Scores
Following the comparison of sampling distributions, a comparison of Overall Satisfaction scores
was conducted to determine whether differences in age and gender among respondents
correlate with differences in Overall Satisfaction.
Table 3.3.2 below indicates differences in Overall Satisfaction scores are the most notable
between gender groups. On average, females tend to rate their experience lower than males
(527 vs. 596). Comparing age groups reveals that Baby & Pre-Boomers had the highest overall
satisfaction with Generation YZ much lower.
Table 3.3.2. Overall Satisfaction Scores for Gender and Age Groups
Gender and Age
Gender
Female
Male
Age Generation
Baby & Pre-Boomer
Generation X
Generation YZ
3.3.3
OSAT (mean)
Standard Deviation
Sample Size (N)
527
596
250
257
56
224
609
547
539
255
257
255
164
65
51
Analysis for Demographic Differences
T-test analyses were conducted to determine whether differences in demographic groups
produced statistical differences in Overall Satisfaction scores. T-tests are typically used to
determine whether or not the difference between two groups’ averages most likely reflect a
meaningful difference in the population from which the groups were sampled.
20
Both gender and war participation demonstrated no differences in Overall Satisfaction scores as
shown in Table 3.3.3a:
Table 3.3.3a. T-Test Analysis for Gender and War Service in Veterans’ Overall Satisfaction
Gender and War
Service
T-Test Statistic
p-value
Statistical Difference
(95% confidence level)
-1.83
.07
No
1.07
.69
No
Gender
Female vs. Male
War Participation
OEF/OIF vs. All other wars
Analyses of Variance (ANOVA) were conducted to determine whether differences in
demographic groups produced statistical differences in overall satisfaction scores. ANOVAs are
typically used to determine whether or not the difference between three or more groups’
averages most likely reflect a meaningful difference in the population from which the groups
were sampled.
Differences in overall satisfaction by generation were not significant (F = 2.24, p-value = .11):
Table 3.3.3b. Overall Satisfaction for Generation
Generation
Baby & Pre-Boomer
Gen-X
Gen-YZ
OSAT (mean)
Sample Size (N)
609
547
539
164
65
51
Differences in overall satisfaction by region were not significant (F = 1.81, p-value = .14):
Table 3.3.3c. Overall Satisfaction for Regions
Regions
Midwest
Northeast
South
West
OSAT (mean)
Sample Size (N)
637
610
565
544
58
49
97
76
21
Differences in overall satisfaction by race were not significant (F = 0.23, p-value = .88):
Table 3.3.3d. Overall Satisfaction for Race
Race
OSAT (mean)
Sample Size (N)
Asian
Black
Other
White
561
588
600
572
16
63
74
127
Differences in overall satisfaction by Branch of Service were not significant (F = 1.85, p-value =
.12):
Table 3.3.3e. Overall Satisfaction for Military Service Branches
Military Service
Air Force
Army
Marines
Navy
Other
OSAT (mean)
Sample Size (N)
570
549
671
589
643
44
131
20
39
46
Differences in overall satisfaction by days of active service were not significant (F = 1.61, p-value
= . 19):
Table 3.3.3f. Overall Satisfaction for Days of Active Service
Days of Active
Service
1000 days or less
1001-2000 days
2001-4000 days
4001 days or more
OSAT (mean)
Sample Size (N)
592
569
512
620
104
67
40
69
There were no significant differences in Overall Satisfaction by Case Status (F = 1.74, p-value =
.18):
Table 3.3.3g.e. Access: Overall Satisfaction for Case Status
Case Status
Discontinued
Employ Rehab
Other
OSAT (mean)
562
635
578
Sample Size (N)
145
60
75
22
3.3.4
Data Imputation Analysis for Demographic Differences
A pairwise comparison T-Test analysis was done to evaluate whether data imputation for
missing values across age, race, region (and other significant demographics) for the final
cleaned sample size of 280 and the 354 total survey respondents generated any changes in the
overall satisfaction index score. This analysis also included survey raking across demographic
differences as one level of comparison.
The results below show that there were no significant differences between the non-imputed
mean and the imputed mean of the satisfaction index across demographics, sample sizes, nor
survey raked values. We want to highlight that after statistical adjustment for the differences
found between respondents and non-respondents reported earlier, there were no differences
in overall satisfaction levels. These results support the conclusion that the survey’s findings for
Veterans’ overall satisfaction ratings are accurate.
Table 3.3.4a. T-Tests of Imputed vs. Non-Imputed on Veterans’ Overall Satisfaction Scores
T-Test Analysis on Imputed vs. Non-Imputed for Age, Race, and Region
Overall Satisfaction Index
(100 - 1000 range)
Imputed demographics
(280 final sample size)
Imputed survey-raked demographics
(280 final sample size)
Imputed survey-raked demographics
(354 total respondents)
mean
(imputed)
mean (nonimputed)
t-statistic
p-value
583.93
582.03
-0.09
0.93
562.00
560.47
-0.07
0.95
560.98
562.32
0.07
0.95
Note: Non-imputed is based on the 280 final cleaned sample size used in this report.
Survey Raking for Sample Weights to Adjust for Differences and Compare Overall Satisfaction
and Advocacy Ratings
The procedure known as “raking” adjusts a set of data so that its marginal totals match
specified control totals on a specified set of variables. The term suggests an analogy with the
process of smoothing the soil in a garden plot by alternately working it back and forth with a
rake in two perpendicular directions, Izrael and Battaglia (2004).
Survey raking is an iterative sample-balancing algorithm-based technique that provides sample
weighting convergence across multiple variables and multiple categories; see Battaglia, Izrael,
Hoaglin, and Frankel (2009).
In keeping with OMB “Standards and Guidelines for Statistical Surveys” guidelines 3.2.12 and
3.2.13, JDP selected the best statistical method to simultaneously adjust for multiple
differences between groups by applying a survey raking procedure, see Anderson, L., and R.D.
Fricker, Jr. (2015).
23
The JDP raking procedure is a proprietary improved version based on the excellent methods
initially developed by Izrael and Battaglia (2000, 2004) and Battaglia, Izrael, Hoaglin, and
Frankel (2004). JDP raking improvements are primarily related to better handling of low cell
values during iterative convergence processing. For this analysis, 50 iterations were set
(although less were needed) to converge on the best sample weights (.2 estimation margin) to
simultaneously adjust for non-response bias in age, race, region, and war (service era)
demographic categories. For additional background about survey raking methodologies, see
Wallace and Rust (1996).
The estimated population distributions are used as convergence targets. In this case, the
dataset of all eligible respondents (5,000) was used as the estimated population to derive
sample weightings for the 354 survey respondents.
In accordance with OMB “Standards and Guidelines for Statistical Surveys” Guideline 3.2.13, a
series of t-tests were conducted to determine whether non-response bias in demographic areas
produced statistical differences in overall satisfaction scores and advocacy ratings. Typically, ttests are used to determine whether differences between two groups’ averages and variances
reflect a meaningful difference in the population. The sample weightings derived from the
survey raking procedure were included in the t-tests to equalize the survey respondent
differences with non-respondents.
There were no significant differences in Overall Satisfaction or advocacy levels when the data
was adjusted for demographic differences between survey respondents and non-respondents.
The results below support the conclusion that the survey’s findings for Veterans’ overall
satisfaction ratings are accurate.
Table 3.3.4b. Overall Satisfaction and Advocacy for Respondents Unweighted and Weighted
Analysis of Survey Respondent Scores with Weighted Adjustment for Non-Response Bias
Standard
Standard
Rating
Mean
Mean
tDeviation
Deviation
p-value
Measure
(Unweighted) (Weighted)
statistic
(Unweighted)
(Weighted)
Overall
Satisfaction
Index (100 1000 range)
Likelihood to
reapply for
program
(rating 1 - 4)
582
560
257
262
0.98
0.33
2.63
2.68
0.97
0.94
-0.59
0.55
24
Findings
Results from the non-response bias analysis indicate that the Overall Customer Satisfaction
Index Score and the Advocacy ratings from the VR&E Non-Participant study reflects the
experience of all Veterans who dropped out of the program prior to completing a rehabilitation
plan.
Sample Cleaning: Initial comparisons on age, gender, and geographical characteristics
between the total records provided and the records available after cleaning, suggests the
sample utilized in the study exhibits similar characteristics as the total sample provided by
VR&E. The tests (see Margin of Error and Sampling Distribution, Section 3.3,) suggest the
sample cleaning rules did not impact the sample’s representativeness and the results are
conclusive.
Non-Response Bias Analysis: Results from the non-response bias analysis did show group
differences for age, gender, race, military branch, days of service, war participation, and case
status between survey respondents and non-respondents. After correcting for these
differences using a recommended sample-balancing survey raking method to derive sample
weights (see Margin of Error, Section 3.3.4 Data Imputation Analysis for Demographic
Variables), there were no differences found in Veterans’ overall satisfaction and advocacy
(likelihood inform others about VA benefits) between weighted and unweighted survey
respondents.
Item Response Rate Calculations: Results from the survey item response rate calculations
reveal high item response rates, with none falling below 70% (see Appendix B for Item
Response Rates). According to OMB Guideline 3.2.10, given this high item response rate, a nonresponse bias analysis was not necessary at the item level.
The research and approach taken by JDP are in accordance with sound market research and
current best practices from the American Association for Public Opinion Research (AAPOR)
regarding response rate recommendations: “Results that show the least bias have turned out,
in some cases, to come from surveys with less than optimal response rates. Experimental
comparisons have also revealed few significant differences between estimates from surveys
with low response rates and short field periods and surveys with high response rates and long
field periods.” See AAPOR “Response Rates – An Overview” (2015) and Special Issue of Public
Opinion Quarterly "Nonresponse Bias in Household Surveys" (Singer, 2006).
25
Conclusion
The Overall Satisfaction Index score and advocacy ratings (likelihood to inform others about VA
benefits) are not impacted in any meaningful way by non-response bias. This analysis confirms
that the data collected during Fiscal Year 2015 is valid.
The FY15 Voice of the Veteran Line of Business Tracking Satisfaction Study data for the VR&E
Non-Participant survey can be used to infer reliable overall customer satisfaction scores and
advocacy ratings. The overall customer satisfaction index score reflects the experience of all
Veterans who dropped out of the program prior to completing a rehabilitation plan.
The sample utilized in the study exhibits similar characteristics for age, gender, and geography
as the total sample provided by VR&E Non-Participant. This indicates the sample cleaning rules
did not impact the sample’s representativeness.
While the results from the non-response bias analysis did show group differences on
demographic characteristics between survey respondents and non-respondents, there were no
differences found in Veterans’ overall satisfaction and advocacy ratings between weighted and
unweighted survey respondents. This was evaluated after correcting for these differences
using a recommended sample-balancing survey raking method to derive sample weights. JDP
conducted all necessary statistical tests in accordance with OMB standards.
J.D. Power certifies the results contained within this report.
26
References
Anderson, L., and R.D. Fricker, Jr. (2015). Raking: An Important and Often Overlooked Survey Analysis
Tool, Phalanx, to appear (September 2015). Preprint available at:
http://faculty.nps.edu/rdfricke/docs/Analysis%20process_v4.pdf
American Association for Public Opinion Research (2008). Standard Definitions: Final Disposition of
Case Codes and Outcome Rates for Surveys. Ann Arbor, Michigan: AAPOR.
(http://www.aapor.org/AAPORKentico/AAPOR_Main/media/MainSiteFiles/Standard_Definitions_07
_08_Final.pdf).
American Association for Public Opinion Research (2015). “Response Rates – An Overview”
http://www.aapor.org/AAPORKentico/Education-Resources/For-Researchers/Poll-SurveyFAQ/Response-Rates-An-Overview.aspx
Battaglia, Michael P., Izrael, David, Hoaglin, David C., and Frankel, Martin R. (2004), “To Rake or Not To
Rake Is Not the Question Anymore with the Enhanced Raking Macro.” Proceedings of the 29th Annual
SAS Users Group International Conference, Paper 207.
Battaglia, Michael P., Izrael, David, Hoaglin, David C., and Frankel, Martin R. (2009). Practical
Considerations in Raking Survey Data. Survey Practice, Vol 2, No. 5.
Baum, Herbert M., Ph.D.; Chandonnet, Anna M.A.; Fentress, Jack M.S., M.B.A.; and Rasinowich, Colleen,
B.A. (2012). “Mixed-Mode Methods for Conducting Survey Research”. Data Recognition
Corporation. http://www.datarecognitioncorp.com/survey-services/Documents/Mixed-ModeMethods-for-Conducting-Survey-Research.pdf
Dillman, D. A. and JDP (2015), “Conference call discussion on non-response bias, avoidance methods, and
post-hoc sample weighting.” Conference call between Dr. Dillman and JDP (Greg Truex, Jay Meyers,
PhD, Lee Quintanar, PhD), May 20, 2015 (2pm PDT).
Dillman, D. A. (2014). Internet, Phone, Mail and Mixed-Mode Surveys: The Tailored Design Method.
Fourth Edition. John Wiley & Sons, Inc: New York.
Ellis, J. M. (2000). Estimating the Number of Eligible Respondents for a Telephone Survey of LowIncidence Households. Paper presented at the annual meeting of the American Association for Public
Opinion Research, Portland OR, May 21.
Federal Committee on Statistical Methodology’s Statistical Policy Working Paper 31, Measuring and
Reporting Sources of Error in Surveys (2001). Washington, D.C.
Izrael, David, Hoaglin, David C., and Battaglia, Michael P. (2000), “A SAS Macro for Balancing a Weighted
Sample.” Proceedings of the Twenty-Fifth Annual SAS Users Group International Conference, Paper
275.
Izrael, David, Hoaglin, David C., and Battaglia, Michael P. (2004), “Tips and Tricks for Raking Survey Data
(a.k.a. Sample Balancing).” Proceedings of the 2004 American Association for Public Opinion
Research (AAPOR) Conference.
27
Malhotra, N.K, and Birks, D.F. (2007). Marketing Research: An Applied Approach, 3rd edition. Prentice
Hall/Financial Times: England.
Pierchala, Carl E. (2001). PROC MI® as the Basis for a Macro for the Study of Patterns of Missing Data.
Northeast SAS Users Group. http://www.lexjansen.com/nesug/nesug03/st/st009.pdf
Singer, E. (2006). Special Issue: Nonresponse Bias in Household Surveys. Public Opinion Quarterly, Vol
70, Issue 5.
U.S. Office of Management and Budget (1990), "Survey Coverage", Statistical Policy Working Paper 17,
Washington, D.C.
U.S. Office of Management and Budget Publication (January 2006). “When Designing Surveys for
Information Collections”. The Office of Management and Budget, 725 17th Street, NW. Washington,
D.C. 20503 USA
U.S. Office of Management and Budget Publication (September 2006). “Standards and Guidelines for
Statistical Surveys”. The Office of Management and Budget, 725 17th Street, NW. Washington, D.C.
20503 USA
U.S. Office of Management and Budget Publication (2008). VBA VR&E OMB - Part B Supporting
statement for “Collections of Information Employing Statistical Methods”. Washington, D.C.
Vogt, W. Paul, Vogt, Elaine R., Gardner, Dianne C., and Haeffele, Lynne M. (2014). Selecting the Right
Analyses for Your Data - Quantitative, Qualitative, and Mixed Method. Guilford Press, New York, NY.
Wallace, Leslie and Rust, Keith (1996). A Comparison of Raking and Poststratification Using 1994 NAEP
Data. Leslie Wallace, West Inc., 584-589.
28
Appendix A
Missing Data Patterns and Mechanisms
An excellent discussion of missing data patterns, mechanisms, and research analysis methods is
provided in Vogt, W. Paul, Vogt, Elaine R., Gardner, Dianne C., and Haeffele, Lynne M. (2014).
An overview of the missing data types and issues is described below:
Understanding the reasons why data is missing can help with analyzing the remaining data. If
values are missing at random, the data sample may still be representative of the population.
But if the values are missing systematically, analysis may be harder.
Missing completely at random. Values in a data set are missing completely at random
(MCAR) if the events that lead to any particular data-item being missing are independent
both of observable variables and of unobservable parameters of interest, and occur entirely
at random. When data are MCAR, the analyses performed on the data are unbiased;
however, data are rarely MCAR.
Missing at random. Missing at random (MAR) is an alternative, and occurs when the
missing value is related to a particular variable, but it is not related to the value of the
variable that has missing data. An example of this is accidentally omitting an answer on a
questionnaire.
Missing not at random. Missing not at random (MNAR) is data that is missing for a specific
reason (i.e. the value of the variable that's missing is related to the reason it's missing). An
example of this is if a certain question on a questionnaire tends to be skipped deliberately
by participants with certain characteristics. Graphical models can be used to describe the
missing data mechanism in detail.
While it is clear that MNAR can introduce statistical bias, there is no definitive test, see Vogt et
al. (2014). It is also clear that MCAR is rarely evident in research data and most tests of it will
fail. However, MAR is fully acceptable for valid statistical analyses (Vogt et. al, 2014). MAR is
essentially “missing partially at random” whereby the intra-group missing value remains
random despite some differences between group tendencies. Graphical data representations
are the typical tool used in assessment as described above and in Pierchala, Carl E. (2001).
See Section 3.2 Missing Data Patterns and Mechanisms for findings specific to VR&E’s data.
29
Appendix B
Item Response Rates
In accordance with OMB “Standards and Guidelines for Statistical Surveys,” Section 3.2,
guidelines 3.2.6-3.2.7, the item response rate was calculated as the ratio of the number of
respondents for whom an in-scope response was obtained to the number of respondent who
were asked to answer that item. The number asked to answer an item is the number of unitlevel respondents minus the number of respondents with a valid skip pattern. In addition to
item response rate, total item response rate was calculated as the product of the overall unit
response rate and the item response rate for each item. The purpose of these calculations is to
assess the item non-response, which occurs when one or more survey items are left blank in an
otherwise completed questionnaire. Table B1 displays the item and total item response rates
for this survey.
The OMB “Standards and Guidelines for Statistical Surveys” (Guideline 3.2.10) states an item
non-response analysis should be conducted for items with an item response rate of less than
70%. Since none of the survey item response rates fall below 70%, an item-level analysis of
non-response bias was not necessary. Results from the item response rate calculation suggest
the item response rate for the VRE Non-Participant study is strong, ranging from 71% to 100%,
with a 90% average. In the item response rate calculation below, JDP considered blanks as nonresponse for mail returns.
Table B1. Comparing Survey Item Response Rates7
Question
Number
1
2
3
4
5
6
7
8
9
10
11
Item
Response
Rate
93%
99%
97%
97%
99%
99%
99%
91%
80%
94%
91%
Unit
Response
Rate
5%
6%
5%
5%
6%
6%
6%
5%
4%
5%
5%
7
E-mail opt in and additional comments about your experience (open capture) questions display “N/A” and were not included in
item and total item response rate calculations
30
Table B1. Comparing Survey Item Response Rates (Continued)
12
13
14
15a
15b
15c
15d
16a
16b
16c
16d
16e
16f
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31a
31b
31c
31d
31e
31f
32
33
34
35a
35b
35c
35d
36
37
38
100%
90%
96%
96%
96%
92%
98%
98%
99%
98%
99%
96%
99%
99%
99%
96%
75%
80%
78%
78%
92%
77%
93%
87%
90%
84%
88%
86%
87%
88%
88%
86%
87%
89%
87%
85%
83%
74%
75%
83%
71%
79%
94%
6%
5%
5%
5%
5%
5%
5%
6%
6%
5%
6%
5%
6%
6%
6%
5%
4%
4%
4%
4%
5%
4%
5%
5%
5%
5%
5%
5%
5%
5%
5%
5%
5%
5%
5%
5%
5%
4%
4%
5%
4%
4%
5%
31
Table B1. Comparing Survey Item Response Rates (Continued)
39
40
41
42
43
44
95%
84%
93%
N/A
N/A
N/A
5%
5%
5%
N/A
N/A
N/A
.
32
Appendix C
Study Overview
1.1 Study Background
The Voice of the Veteran Satisfaction Initiative tracks Veteran satisfaction with the benefits and
services received from VBA. The VOV Tracking Satisfaction Research Study is ongoing survey
research tracking Veteran satisfaction with VBA’s lines of business: Compensation, Pension,
Education, Vocational Rehabilitation & Employment (VR&E), and Loan Guaranty (LGY).
As part of Executive Order 13571 Streamlining Service Delivery and Improving Customer
Service, agencies that provide significant services directly to the public to identify and survey
customers, establish service standards and track performance against those standards, and
benchmark customer service against t hose best practices. This program enables VBA to
understand what is important to Veterans relative to benefits received and services provided.
This program provides timely and actionable Veteran feedback on how well VBA is providing
services. Insights from this program identify opportunities for improvement and measure the
impact of improvement initiatives, as well as continuously measure performance outcomes.
VR&E’s Non-Participant survey instrument was regarding what led Veterans to discontinue their
vocational rehabilitation program.
Survey
Methodology
Fielding
Frequency
Total Mailouts
Per Year
Target
Number of
Completes
Mail Only
Annually
5,000
1,500
VRE Non-Participant
1.2 Methodology
The respondents only had the option of completing a paper survey. Respondents were first
mailed a Survey Package which included a cover letter introducing the study to the respondent,
a paper survey, and a business reply envelope. The second mailing included a cover letter, a
paper survey, and a business reply envelope. The sample for mailings of the survey packet was
cleaned to exclude anyone who completed the survey at least one week prior to the cleaning.
Sample Population Definition
The targeted population was identified by VRE and is defined as Veterans who dropped out of
the program prior to completing a rehabilitation plan. These individuals include applicants who
never attended the initial meeting with a counselor; applicants who were entitled to the
program but did not pursue; and applicants who started, but did not complete rehabilitation
(i.e., negative closures).
33
Sample File Generation
VRE generates the sample files based upon the sampling definition and submits sample files
directly to BAS.
BAS receives the sample files and sends to VADIR for processing.
VADIR processes sample files (to remove SSN and append demographics/EDIPI) and returns
to BAS.
BAS transfers sample files (via EDX platform) to JDP and notifies JDP via email that sample
files are ready for deployment.
JDP cleans the sample file and selects the sample.
Sample is transferred to Government Printing Office (GPO) print vendor (via EDX platform)
for printing and mailing of the postcards and survey packages.
Sample is transferred in accordance with the following schedule:
VOV_LOB
Tracking_Production Schedule_10.06.15.pdf
1.3 Data Cleaning
JDP processed the sample according to the following cleaning rules:
1. De-duplicate records within each business line and across surveys based on the unique
identifier (EDI_PI or VA_ID) for each record. Note: EDIPI is Electronic Data Interchange
Personal Identifier.
a) Exception: For Pension Access (v1) and Pension Servicing (v8), de-duplicate records
based on EDI_PI and Claim Number.
b) When each new sample file is received, JDP cleans it against all sample selected from
every sample batch that has been delivered 12 months prior to ensure a respondent
does not receive a VA line of business survey more than once in a 12 month-period. In
the case of duplicates occurring within the same sample month, priority is assigned to
business lines with the lowest number of sample records.
2. Clean out records present on the JDP Do Not Contact list and clean against the National
Change of Address (NCOA) list.
3. Clean out any respondents who do not have any EDI_PI or VA_ID included in their sample
record.
a) Exception: For Pension Access (v1) and Pension Servicing (v8), clean out records with
blank EDI_PI and Claim Number.
4. Clean out any respondents not specified as a dependent/spouse who have a date of death
(DOD) in their sample record.
5. Clean out any respondents who do not have any address included in their sample record.
6. Assign and maintain unique sampling identifiers to each sample record in order to track
history of sampling. Exclude records that have been sampled in the past 12 months to
ensure no respondent is mailed surveys more than once in a 12-month timeframe. This rule
may not apply to those who completed a survey.
34
1.4 Order generation and fulfillment process
Federal Acquisition Regulations (FAR 8.8) mandate government agencies solicit all printing
requirements through the Government Printing Office. GPO utilizes print vendors to fulfill
orders. A Data Transfer Agreement (DTA) must be in place with print vendor and contractor
before BAS can obligate funds or transfer sample files to the print vendor and contractor.
Prior to mailing the postcards and mail surveys, print orders must be generated for each survey.
The entire process may take up to 2-4 weeks from inception of the print order to the mailing of
the survey package or postcard. Below are the steps involved in order generation and order
fulfillment.
Order generation
After sample is received by JDP, the sample files are cleaned and selected. Then Letter Work
Orders (LWOs) are created to provide the print vendor with the necessary information to
match the sample files to the correct survey instrument. (1 day)
JDP creates the print order and sends over to BAS Contractor Officer’s Representative
(COR). (Same day as above step)
The COR then reviews, authorizes, and submits the print order. (1 day)
The BAS Publication Officer and/or COR submits the orders to the VA Publications Services
Division (VAPSD). (Same day as above step)
The order is issued a control number by a VBA Management Analyst, Publications. (Variable
timing)
Once the control number is assigned, the order goes to VA Publication Services Division
liaison to forward to GPO Contracting Officer. (Variable timing) Note: the amount of time an
order is with VAPSD varies greatly, it could be from 3 days up to 20 days.
The GPO Contracting Officer sends the printing and mailing order to the print vendor.
Order fulfillment
Once the order is placed, the GPO print vendor is allotted 9 business days to fulfill the order
(2 days to generate proofs, 2 days for proof review, corrections, and 5 days to print and
mail).
Upon receipt of the proofs from print vendor, JDP reviews and approves; then BAS reviews
and approves; then VAPSD reviews and approves.
After the orders have been mailed, the print vendor provides the mail receipts to
contractor, BAS and VAPSD.
Upon order completion, VAPSD provides actual costs to BAS.
35
1.5 Reporting
Reporting occurs once per year for the VR&E Non-Participant survey.
On a yearly basis, the following deliverables are provided:
Scorecard
Data Matrices
Data is loaded to the VOV reporting site
Open ended comments (verbatims)
Data and Analysis Presentation
36
Sample Plan Overview
2.1 Sample Criteria
VBA was responsible for providing sample to JDP that meets the following sampling criteria:
Sample Population
Inclusion Criteria
Frequency of Data Request
VRE Non - Participant
The targeted population includes
Veterans who dropped out of the
program prior to completing a
rehabilitation plan. These
individuals include applicants who
never attended the initial meeting
with a counselor; applicants who
were entitled to the program but
did not pursue; and applicants who
started, but did not complete
rehabilitation (i.e., negative
closures).
Annually
2.2 Fielding/Sampling Frequency
Survey
Methodology
Instrument
VRE Non Participant
Total
Targeted
Survey
Number of
Instruments Completes
Mail Only
5,000
1,500
Number of
Postcards
(eSurvey)
Number
of Mail
Packages
Fielding
Frequency
N/A
5,000
Annually
2.3 Data Transfer
The sample was posted by BAS once a month within the sampling folder on the VOV EDX site.
Sample should be provided in a file layout consistent with the file layout provided for the study
as outlined below.
VR&E File Layout
ADDRESS_1
ADDRESS_2
ADDRESS_LINE_1
ADDRESS_LINE_2
AGE
BAH Rate
BRANCH_1
BRANCH2
VR&E File Layout (Continued)
BRANCH3
CASE_STATUS_CODE
37
CITY
Claim_Number
Date_of_Birth
DIAG_CODE
ELGBTY_TRMNTN_DT
Email_Address
EOD_1
EOD_2
EOD_3
First_Name
FIRST_NOTICE_OF_DEATH
GENDER
Last_Name
MILITARY_RANK
PHONE_NUMBER
PHONE_NUMBER_2
POSTAL_CODE
PRCNT_NBR
RAD_1
RAD_2
RAD_3
SEH_STATUS
SERVICE_ERA_1
SERVICE_ERA_2
SERVICE_ERA_3
SERVICE_PERIOD_MONTH
SSN_NBR
STATE
STN_NBR
VETERAN_DOB
Zip
Zip_Code
2.4 Sample Cleaning Rules Glossary
Duplicate records in sample file – the record is cleaned out if there is more than one record
within the same sample file for the same respondent
38
Duplicate record history – the record is cleaned out if the record has been selected within the
past 12 months for any of VBA’s business line surveys (i.e. Compensation, Pension, Education,
Home Loan Guaranty, and Vocational Rehabilitation) regardless of whether the respondent
completed the survey
Invalid address – the record is cleaned out if JDP’s address verification software indicates an
invalid address code
Invalid values – the record is cleaned out if the “VA_ID” field is blank
Blanks – the record is cleaned out if the “Name” field corresponding to the record is blank
Do not contact – the record is cleaned out if the individual is listed on JDP’s Do Not Contact List
2.5 Sample Selection
JDP selected sample records following the completion of the sample cleaning process. The
following guidelines are referenced when selecting sample:
1. Total Sampling Targets: The table below summarizes the total sampling target per an RO per
a fielding period. The “Sampling Target per RO” column indicates the minimum number of
sample records that should be selected per an RO for each survey. If this minimum target
number cannot be reached for a particular RO, sample from a different RO will be selected
to make up the difference.
Survey
Frequency
Total
Sampling
Target
VRE NonParticipant
Annually
5,000
Sampling
Target Per
Time Period
Sampling
Target Per
RO
Number of
ROs
5,000
86
58
2. The same record cannot be selected for multiple surveys during the same wave.
Respondents who have completed a survey within the past 12 months cannot be selected.
Survey priority is based on the number of records in each sample file. The survey with the
smallest number of records is given first priority.
3. Following sample selection, the JDP project teams receives an automated report confirming
the number of records selected for each survey version. The JDP project team verifies that
the sample selection quantities reflect the sample targets and approves the sample file for
fielding.
39
2.6 Data Collection
During the survey fielding period, both online survey returns and paper surveys are collected as
they are received and posted on a secure EDX site. Responses from paper surveys are scanned
through automated imaging software while verbatim responses are recorded by a live survey
processor. Survey returns must have all pages intact in order to be processed and counted as a
return. Surveys with missing pages are counted as unusable. Returns are also considered
unusable, if there is an indication that the individual completing the survey is not the individual
selected from the sample file (i.e. the respondent name and/or address on the survey is
replaced with a different name and/or address). During each day of fielding, a subset of survey
returns undergo quality assurance to validate the accuracy of responses captured. If duplicate
surveys are returned (as identified by the unique sampling identifier assigned to each sample
record), the original survey return is processed while the duplicate survey is removed. In the
case of duplicate survey returns from mixed methodology surveys, the date the survey was
received is used to identify the original return while the subsequent return is removed postfielding.
40
Appendix D
Approaches to Mitigating the Effect of NonResponse Bias and Strategies to Improve the
Response Rate
The following section outlines two approaches used in FY 2015 to mitigate the potential of nonresponse bias. As mentioned earlier in the report, J.D. Power affirms that while high response
rates are always desirable in surveys, an 80% response rate is typically not achievable for a
voluntary, customer-satisfaction survey instrument (Malhotra & Birks, 2007), particularly those
that do not provide an incentive (not recommended for this program). To illustrate this point,
the Dillman Method for survey fielding was discussed in Dillman, D. A. (2014) – a survey
instrument was fielded to 600 students at the University of Washington. After 5 attempts to
solicit a response, as well as offering a monetary incentive to complete the study, a 77%
response rate was reported.
The first approach to minimize non-response occurs before and during data collection and
involves introducing measures to maximize survey response rates. The second approach is to
make statistical adjustments after the data is collected.
1.1 Approach 1: Strategies to Maximize Response Rates
Prior to, and during, fielding the VR&E survey, JDP implemented the following measures to
reduce the chances of non-response:
Respondents were provided with the promise of confidentiality on the survey cover letter
and postcard, and assured that their survey responses would not impact their current or
future eligibility for benefits.
Following the first mailing, non-respondents were sent an additional survey mailing.
Respondents were provided with a toll-free telephone number and dedicated e-mail
address to contact JDP about survey-related inquiries (e.g., how to interpret questions and
response items, the purpose of the survey, how to get another copy of the survey if their
copy has been lost/damaged, etc.). Telephone calls and e-mails are responded to within 24
hours and answered during regular business hours (8:00-5:00pm PT).
JDP ensured the web-based surveys were accessible to people with disabilities by
maintaining 508 compliant standards. These standards include:
Keyboard navigation rather than mouse or other pointing devices
Customization options for color, size, and style of text displayed
Compatibility with screen-readers to translate items displayed on the survey in audible
output and/or Braille displays
Customer support and technical support through JDP Help Desk toll-free phone number
and email address
41
Exclusion of non-text elements, image maps, animation, flashing or blinking text.
The survey fielding period was extended to offer opportunities to respond for subgroups
having a propensity to respond late (e.g., males, young, full-time employed).
The survey was developed and reviewed in order to enhance respondent understanding of
the survey materials and to improve the relevancy of the data collected:
Prior to fielding the Benchmark study, a series of cognitive labs was conducted with test
users to ensure the survey questions were easily understood and correctly interpreted.
Revisions were made to the survey based on test user feedback. (As per OMB Guideline
1.4.1)
After the Benchmark study and prior to fielding the first year of the Tracking study,
VR&E Service and JDP conducted a review of the survey instrument and modified the
survey to improve the relevancy of data collected. (As per OMB Guideline 1.4.2)
1.2 Approach 2: Correcting Unit Non-response Bias with Sample Weighting and
Survey Raking
As stated above, the two approaches to tackling non-response bias include implementing
measures to maximize response rates during the fielding period and making post hoc statistical
adjustments to the survey results afterwards. The following section discusses the statistical
adjustments approach, which include weighting the data or imputing scores to correct the
amount of non-response bias. An example of this approach would be the survey raking
procedure described earlier in this paper. See the associated references in the “Survey Raking
Procedure for Sample Weightings” section for more information.
The procedure known as “raking” adjusts a set of data so that its marginal totals match
specified control totals on a specified set of variables. The term “raking” suggests an analogy
with the process of smoothing the soil in a garden plot by alternately working it back and forth
with a rake in two perpendicular directions, Izrael and Battaglia (2004).
If non-response bias was identified in the survey data, the non-response bias could be
corrected mathematically with a post-stratification survey weight. JDP would weigh the survey
data based on certain demographics (such as age, gender, region, etc.) of the total sample so
that the weighted survey data would conform more to the demographics of the total sample.
The implicit assumption in this approach is the distributions of characteristics of the nonrespondents within an adjustment class (such as an age group) are the same, on average, as
those of the respondents within the same adjustment class.
See Appendix B for the item response rate for each question in the survey. If the item response
rate was not lower than 70%, as per OMB standards, the imputation of data is not necessary.
In the case that a particular item-level response was less than 70%, JDP would recommend
conducting additional analysis to determine the potential for other factors (i.e. missing or skip
patterns in the survey instrument) to be the cause of non-response.
42
Strategies to Improve Response Rate
In addition to the strategies listed above, JDP recommends considering the following strategies
to improve response rates going forward:
Issue ongoing public communications (e.g. press releases, post information on the VA
website) to spread awareness and confirm the legitimacy of the VA VR&E Study.
Educate VA employees and VSOs about the survey to encourage participation. Provide a list
of frequently asked questions and answers to VSOs and VA employees to equip them with
answering Veterans’ questions regarding the survey.
Send e-mail invitations to Veterans rather than mailing postcards to make it easier for
Veterans to complete the survey online.
Reduce the length of the survey to improve respondents’ willingness to respond
Reduce overall number of questions and number of response options for each question.
Increase the number of contacts to respondents with additional reminders about the survey
to encourage participation
Provide respondents with an additional paper survey questionnaire.
Reduce the frequency of mailings to reduce the opportunities for delays and errors in the
GPO Print process.
Revise the cover letter and postcard to express the importance of participation in the
survey.
Provide sample from the 30 day period immediately prior to the mailing rather than sample
from 90 days prior to improve the recency of their experience with the VR&E benefit (which
improves both participation and recollection).
Change location of sequence number to directly follow survey link on postcard and cover
letter.
Alter formatting on postcard and cover letter to include color print to make materials more
readable to increase participation.
Alter the responsibility of sample file generation form VR&E to PA&I. A data pull by PA&I
will increase consistency.
43
Appendix E
Impact of FAR 8.8
Federal Acquisition Regulation (FAR) 8.8 requires that printing must be conducted through the
Government Printing Office (GPO). The following section outlines limiting factors of the VOV
Line of Business Tracking Satisfaction Research Study that occurred as a result of the FAR
requirement.
Through the utilization of the GPO Print Vendor, the following occurred in FY15:
o Quality issues included:
Survey instruments were printed and mailed:
Utilizing the sample population from one survey, but receiving a different
survey (e.g., potential respondents from the pool of one business line
received the survey for a different business line)
Using a version of the instrument that was outdated; this version did not
contain the current questions or responses that were being fielded
Mixing content between survey versions
Using shells from one survey printed with a different survey
o Ongoing timeliness delays occurred with each set of orders placed, as the order
fulfillment process took a minimum of 2-4 weeks
1.1
Impact
The project experienced ongoing delays in the printing and mailing of its postcards and survey
packets for VBA’s lines of business. The delays affected the critical processes required to
execute the VOV Program to its fullest potential.
A multitude of quality issues were experienced throughout FY15 that negatively impacted the
VOV Program response rates. The issues that occurred impacted: access to the online survey;
readability of mail materials; level of effort required by respondents to take the survey;
relevancy of survey; and the diminishment of brands (VA/JDP) associated with poor quality
materials.
44
Appendix F
NOTE: Questionnaire is not shown in the formatted version that respondents used to fill out
survey.
Survey Questionnaire
[DO NOT DISPLAY/IDENTIFY SECTION HEADERS. DISPLAY SINGLE QUESTION PER PAGE.]
[RESPONSE CODES APPEAR IN BRACKETS AT THE END OF EACH RESPONSE FOR SINGLE
RESPONSES AND IN THE PROGRAMMING INSTRUCTIONS FOR MULTIPLE RESPONSES.]
Pre-Application Process
1. How did you FIRST learn about the Vocational Rehabilitation and Employment
(VR&E) benefit program? (Mark only one) If you are unsure, please indicate the first
way you remember learning about the VR&E program. [RADIO BUTTONS. SINGLE
RESPONSE.]
a. VA website [1]
b. eBenefits.va.gov [2]
c. Veterans Employment center in eBenefits [3]
d. Social media websites (e.g., Facebook, Twitter, etc.) [11]
e. Internet (excluding VA and social media sites) [14]
f. Mail (from VA) [4]
g. VA phone number (800-827-1000) [5]
h. VA medical center [8]
i. VA Vet Center [9]
j. In person at a Regional Office [10]
k. Visit from a VA employee [12]
l. Transition Assistance Program/Disabled Transition Assistance Program
briefings [6]
m. Veterans Service Organizations (e.g., Amer. Legion, DAV, VFW, PVA,
MOPH, etc.) (Specify)
[TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
n. Other Veterans/Servicemembers [13]
o. Friends or family [15]
p. Information came with notification/ratings letter [16]
q. Other publications (e.g., Army Times, local newspaper, etc.) [17]
r. Other (Specify) ___________________[TEXT BOX. FORCE TEXT IF
RESPONSE IS SELECTED. 50 CHARACTER MAX.] [97]
s. Don’t know or not sure [99]
2. Thinking about the factors you considered when deciding to apply for benefits, which
of the following describes your reason(s) for applying to the VR&E program? (Mark
all that apply) [CHECK BOXES, MULTIPLE RESPONSE. CODE EACH
RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED]
45
a.
b.
c.
d.
e.
f.
g.
h.
i.
j.
k.
I had a good experience with the VR&E program in the past
A family member or friend recommended the VR&E program
Another Veteran recommended the VR&E program
VA recommended the VR&E program
The program is recommended by an independent source (e.g., Veterans
Service Organizations (e.g., Amer. Legion, DAV, VFW, PVA, MOPH, etc.)
It is easy to find information about the VR&E program
VR&E will assist me in finding and obtaining suitable employment
The VR&E program has a good reputation
The VR&E program offers services I need
VA makes it easy to apply for the VR&E program
Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE]
Reasons for Applying for VR&E Services
3. Which of the following statements BEST describes your plans at the beginning of the
application process? (Mark only one) [RADIO BUTTONS, SINGLE RESPONSE]
a. I was not planning on participating in the rehabilitation process, but
wanted to find out about the rehabilitation services/process and which
services I qualified for [1]
b. I was considering participating in the rehabilitation process if I liked the
services that I qualified for[3]
c. I was considering participating in the rehabilitation process if the process
was not too time-consuming or complicated[4]
d. I definitely planned to participate in the rehabilitation process[5]
e. Other (Specify) _________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.] [97]
f. Don’t know or not sure [99]
4. Were you prompted to apply to the VR&E program for any of the following reasons?
(Mark only one per row) [GRID WITH YES/NO IN COLUMNS AND ATTRIBUTES
IN ROWS. RADIO BUTTONS, SINGLE RESPONSE PER ROW. IF TEXT
ENTERED IN “SPECIFY” BOX, AUTOPUNCH “YES” RESPONSE.] [CODE
RESPONSE AS 0 IF NO IS SELECTED AND 1 IF YES IS SELECTED]
Yes No
Information you received during a Transition Assistance
Program/Disabled Transition Assistance Program briefing
Information you received in a letter from a VA Regional
Office telling you what information you needed to provide
and what VA would do
Change in your life circumstances (e.g., marriage, divorce,
46
loss of job, severity of disability, etc.)
Current employment did not meet your expectations
Recommendation or referral
Other reasons (Specify)
(Ask Q5 if yes to “Change in life circumstances” in Q4, otherwise go to Q6)
5. Which of the following describes the change in your life circumstances? (Mark all
that apply) [CHECK BOXES, MULTIPLE RESPONSE. CODE EACH RESPONSE
AS 0 IF UNCHECKED OR 1 IF CHECKED]
a. Marriage
b. Divorce
c. Death in the family
d. Had children
e. New job
f. Lost job
g. Moved
h. Declared bankruptcy
i. Retirement
j. Severity of disability
k. None of the above [MUTUALLY EXCLUSIVE RESPONSE]
(Ask Q6 if yes to “Current job did not meet expectations in Q4, otherwise go to Q7)
6. In what areas did your current employment not meet your expectations? (Mark all
that apply) [CHECK BOXES, MULTIPLE RESPONSE. CODE EACH RESPONSE
AS 0 IF UNCHECKED OR 1 IF CHECKED]
a. Experienced problems with supervisors
b. Did not utilize my skills/abilities
c. Level of pay
d. Level of responsibility
e. Too many work hours
f. Too few work hours
g. Poor reliability of pay checks
h. Lack of benefits
i. Flexibility of work schedule
j. Job security
k. Other (Specify) __________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
Entitlement Evaluation
7. How soon after you were contacted did you meet with a VR&E representative from
VA in person for your initial evaluation appointment? (Mark only one) [RADIO
BUTTONS. SINGLE RESPONSE]
47
a.
b.
c.
d.
e.
Less than 30 days [1]
31-60 days [2]
More than 60 days [3]
Don’t know or not sure [99]
Did not meet with a VR&E representative [96]
(Ask Q8-Q9 if did not meet with representative in Q7, otherwise go to Q10)
8. Why did you decide not to attend your initial evaluation appointment with VR&E?
(Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH
RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED]
a. I had a poor experience scheduling the initial appointment
b. I had a poor experience with the VR&E representative
c. The VR&E program does not offer the services I need
d. A family member or friend recommended against the VR&E program
e. Another Veteran recommended against the VR&E program
f. Issues related to the application process (too time
consuming/complicated)
g. It is difficult to find information about the VR&E program
h. Concerns about my eligibility for the VR&E program
i. Other (Specify) _________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
j. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE]
9. Did your decision not to attend your initial evaluation appointment involve a change
in any of the following life circumstances occurring after you submitted your
application? (Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE
EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED]
a. Marriage
b. Divorce
c. Death in the family
d. Had children
e. New job
f. Lost job
g. Moved
h. Declared bankruptcy
i. Retirement
j. Severity of disability
k. None of the above [MUTUALLY EXCLUSIVE RESPONSE]
10. Which of the following statements is the most important to you in your decision to
attend the initial evaluation appointment? (Mark only one) [RADIO BUTTONS.
SINGLE RESPONSE]
a. Receiving a call from a VA Representative to schedule your appointment
[1]
b. Change in life circumstances (e.g., marriage, divorce, loss of job, severity
of disability, etc.) [2]
c. Current employment did not meet your expectations [3]
48
d. Recommendation or referral [4]
e. Other (Specify) __________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.] [97]
Entitlement Evaluation Process
(Ask Q11-Q14 if met with a representative in Q7, otherwise go to Q15)
11. During your initial evaluation appointment, did the counselor have you participate in
any testing? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE]
a. Yes [1]
b. No [0]
c. Don’t know or not sure [99]
(Ask Q12 if Q11 is Yes, otherwise go to Q13)
12. Did the counselor explain the following…? (Mark all that apply) [CHECK BOXES.
MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF
CHECKED]
a. Purpose of the test
b. Results of the test
c. Next steps in the process
d. None of the above [MUTUALLY EXCLUSIVE RESPONSE]
e. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE]
13. How many appointments did you have with a counselor before an entitlement
decision was made? (Open Capture)
a. Number of appointments (0-99)____________ [NUMERIC TEXT BOX;
ACCEPT (0-99)]
b. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE] [CODE
AS 0 IF UNCHECKED OR 1 IF CHECKED]
(Ask Q14 if Q13 is 2 or more, otherwise go to Q15)
14. Why was it necessary for you to have more than one appointment? (Mark all that
apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0
IF UNCHECKED OR 1 IF CHECKED]
a. To provide additional paperwork/documentation (e.g., medical documents)
b. Additional tests
c. To follow-up with questions/concerns
d. Initial appointment took too long
e. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
f. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE]
Application and Evaluation Experience
49
The following questions ask you to rate various aspects of your experience with
Vocational Rehabilitation and Employment using a scale of 1 to 10 where 1 is
Unacceptable, 10 is Outstanding, and 5 is Average. [SHOW ON SAME PAGE AS THE
QUESTION THAT FOLLOWS]
15. Please rate your experience with the VR&E benefit application process on the
following items: [SHOW RESPONSES IN GRID WITH 10-POINT SCALE IN
COLUMNS AND ATTRIBUTES/RESPONSES IN ROWS (SEE JDPA
CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC DETAILS OF LAYOUT).
EVENLY SPACED RADIO BUTTONS/COLUMNS, ALTERNATE SHADES IN
ROWS. SINGLE RESPONSE PER ROW. RANDOMIZE ALL ATTRIBUTES
EXCEPT THE LAST ONE.]
a. Ease of completing the application [ALLOW N/A RESPONSE] [1-10,
N/A=99]
b. Timeliness of eligibility notification [ALLOW N/A RESPONSE] [1-10,
N/A=99]
c. Flexibility of application methods [ALLOW N/A RESPONSE] [1-10,
N/A=99]
d. Overall rating of application process
16. Using the same 1 to 10 scale where 1 is Unacceptable, 10 is Outstanding, and 5 is
Average, please rate your experience with Vocational Rehabilitation and
Employment counselors during the initial evaluation of your benefit application on
the following items: [SHOW RESPONSES IN GRID WITH 10-POINT SCALE IN
COLUMNS AND ATTRIBUTES/RESPONSES IN ROWS (SEE JDPA
CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC DETAILS OF LAYOUT).
EVENLY SPACED RADIO BUTTONS/COLUMNS, ALTERNATE SHADES IN
ROWS. SINGLE RESPONSE PER ROW. RANDOMIZE ALL ATTRIBUTES
EXCEPT THE LAST ONE.]
a. Promptness of scheduling appointments or returning calls [ALLOW N/A
RESPONSE] [1-10, N/A=99]
b. Courtesy of the counselor [ALLOW N/A RESPONSE] [1-10, N/A=99]
c. Knowledge of the counselor [ALLOW N/A RESPONSE] [1-10, N/A=99]
d. Counselor’s concern for your needs [ALLOW N/A RESPONSE] [1-10,
N/A=99]
e. Timeliness of completing your initial evaluation [ALLOW N/A
RESPONSE] [1-10, N/A=99]
f. Overall counselor experience
Rehabilitation Program/Plan Selection
17. Did you sign a rehabilitation plan with your counselor? [RADIO BUTTONS. SINGLE
RESPONSE.]
a. Yes [1]
b. No [0]
c. Don’t know or not sure [99] (Skip to Q38)
50
(Ask Q18-Q19 if did not complete a rehabilitation plan in Q17, otherwise go to Q20)
18. Why did you decide not to complete a rehabilitation plan with VR&E? (Mark all that
apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0
IF UNCHECKED OR 1 IF CHECKED]
a. I had a poor experience with the VR&E representative
b. The VR&E program does not offer the services I need
c. I chose to enroll in the GI Bill Program
d. A family member or friend recommended against the VR&E program
e. Another Veteran advised against or recommended that I not use the
VR&E program
f. Issues related to the planning process (too time consuming/complicated)
g. Issues related to transportation
h. Issues related to a medical condition
i. It is difficult to obtain information about the VR&E program
j. Life circumstances
k. Other (Specify) _________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
l. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE]
19. Did your decision not to complete a rehabilitation plan involve a change in any of the
following life circumstances occurring after you received your entitlement decision?
(Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH
RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED]
a. Marriage
b. Divorce
c. Death in the family
d. Had children
e. New job
f. Lost job
g. Moved
h. Declared bankruptcy
i. Retirement
j. Severity of disability
k. None of the above [MUTUALLY EXCLUSIVE RESPONSE]
(Ask Q20-37 if completed a rehabilitation plan in Q17, otherwise go to Q38)
20. Which of the following statements would you say was the most important to you in
your decision to complete the rehabilitation plan process? (Mark only one) [RADIO
BUTTONS. SINGLE RESPONSE.]
a. Access to an assigned VR&E counselor [1]
b. Receiving continuous contact from the same VR&E counselor [2]
c. Change in life circumstances (e.g., marriage, divorce, loss of job, severity
of disability, etc.) [3]
d. Current employment did not meet your expectations [4]
51
e. Recommendation or referral [5]
f. Other (Specify) _________________ [TEXT BOX, FORCE TEXT IF RESPONSE IS
SELECTED, 50 CHARACTER MAX.] [97]
21. Was the counselor during the planning phase of your program the same counselor
who conducted your initial evaluation? (Mark only one) [RADIO BUTTONS. SINGLE
RESPONSE.]
a. Yes [1]
b. No [0]
c. Don’t know or not sure [99]
22. Did your counselor provide you with information about the Veterans Employment
Center in eBenefits? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
c. Don’t know or not sure [99]
23. Did you register for theVeterans Employment Center in eBenefits? (Mark only one)
[RADIO BUTTONS. SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
c. Don’t know or not sure [99]
(Ask Q24 if Q23 is No, otherwise go to Q25)
24. Why didn’t you register for the Veterans Employment Center in eBenefits? (Mark all
that apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE
AS 0 IF UNCHECKED OR 1 IF CHECKED]
a. Not aware of the Veterans Employment Center
b. Opted not to use the Veterans Employment Center
c. Other (Specify:)___________________________ [TEXT BOX, FORCE TEXT
IF RESPONSE IS SELECTED, 50 CHARACTER MAX.]
d. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE]
25. Did your final rehabilitation plan include your original vocational training choice?
(Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
c. Don’t know or not sure [99]
(Ask Q26 if Q25 is No or Don’t know, otherwise go to Q27)
26. Why didn’t your final rehabilitation plan include your original vocational training
option? (Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE
EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED]
a. Poor labor market
b. Medical reasons
c. Another vocational option suited my needs better
52
d. Other (Specify: )________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.]
e. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE]
Rehabilitation Experience
27. From the time you signed your rehabilitation plan, how long did it take before
services were initiated for your plan? (Open Capture) Please respond using any or
all of the following categories
(Web only: IF 0 IS SELECTED FOR DAYS, WEEKS, AND MONTHS, SHOW: Please
select “don’t know or not sure” or “did not begin one of the five rehabilitation tracks”)
a. Days (0-99 days) _________ [NUMERIC TEXT BOX; ACCEPT (0-99)]
b. Weeks (0-99 weeks) ________ [NUMERIC TEXT BOX; ACCEPT (0-99)]
c. Months (0-99 months) __________ [NUMERIC TEXT BOX; ACCEPT (099)]
d. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE] [CODE
AS 0 IF UNCHECKED AND 1 IF CHECKED]
e. Did not begin one of the five rehabilitation tracks [MUTUALLY
EXCLUSIVE RESPONSE] [CODE AS 0 IF UNCHECKED AND 1 IF
CHECKED]
28. Did the same counselor who developed your rehabilitation plan also provide case
management sessions during the education and training phase? (Mark only one)
[RADIO BUTTONS. SINGLE RESPONSE.]
a. Yes [1]
b. No [0]
c. Don’t know or not sure [99]
d. Not applicable [96]
29. Were you given a time frame from VA for completing the education/training phase of
your rehabilitation plan? (Mark only one) [RADIO BUTTONS. SINGLE
RESPONSE.]
a. Yes [1]
b. No [0]
c. Don’t know or not sure [99]
30. Which of the following types of counseling or referrals has your counselor provided?
(Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH
RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED]
a. Education/training enrollment assistance
b. Career counseling
c. Personal counseling
53
d.
e.
f.
g.
h.
i.
j.
k.
Financial counseling
Problem-solving techniques
Referrals to potential employers (e.g., government, private, etc.)
Referrals to employment agencies or job banks
Referrals to health providers (e.g., medical, dental, optical)
Referrals to other counseling programs
Referrals to Veterans Service Organizations (e.g., American Legion)
None of the above [MUTUALLY EXCLUSIVE RESPONSE]
The following question asks you to rate various aspects of your experience with
Vocational Rehabilitation and Employment (VR&E) using a scale of 1 to 10 where 1 is
Unacceptable, 10 is Outstanding, and 5 is Average. [SHOW ON THE SAME PAGE AS
THE QUESTION THAT FOLLOWS]
Please answer the following question based on your best ability to recall your experience with
your VR&E counselor(s). [SHOW ON THE SAME PAGE AS THE QUESTION THAT
FOLLOWS]
31. Please rate your experience with VR&E counselors on the following items: [SHOW
RESPONSES IN GRID WITH 10-POINT SCALE IN COLUMNS AND
ATTRIBUTES/RESPONSES IN ROWS (SEE JDPA CONVENTIONS DOCUMENT
PG. 1 FOR SPECIFIC DETAILS OF LAYOUT). EVENLY SPACED RADIO
BUTTONS/COLUMNS, ALTERNATE SHADES IN ROWS. SINGLE RESPONSE
PER ROW. RANDOMIZE ALL ATTRIBUTES EXCEPT THE LAST ONE.]
a. Promptness of scheduling appointments or returning calls [ALLOW N/A
RESPONSE] [1-10, N/A=99]
b. Courtesy of the counselor [ALLOW N/A RESPONSE] [1-10, N/A=99]
c. Knowledge of the counselor [ALLOW N/A RESPONSE] [1-10, N/A=99]
d. Counselor’s concern for your needs [ALLOW N/A RESPONSE] [1-10,
N/A=99]
e. Timeliness of completing your initial evaluation [ALLOW N/A
RESPONSE] [1-10, N/A=99]
f. Overall counselor experience
32. Which of the following benefits did you receive as part of your rehabilitation plan?
(Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH
RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED]
a. Tuition
b. Subsistence allowance
c. Books/supplies
d.
e.
f.
g.
Computer equipment/software
Health services (e.g., medical, dental, optical)
Tutoring
Independent living services
54
h. Employment services (e.g., resumepreparation, interview skills, obtaining
licenses/certifications, etc.)
i. None of the above [MUTUALLY EXCLUSIVE RESPONSE]
33. Which of the following types of employment services did you receive as part of your
rehabilitation plan? (Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE.
CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED]
a. Resume preparation
b. Interview skills
c. Obtaining licenses/certifications
d. Job hunting strategies
e. Information interview with potential employers
f. Job placement assistance
g. None of the above [MUTUALLY EXCLUSIVE RESPONSE]
34. Were the amount of services you received as part of your VR&E program less than,
more than, or what you expected? (Mark only one) [RADIO BUTTONS. SINGLE
RESPONSE.]
a. Less than [1]
b. What I expected [2]
c. More than [3]
The following question asks you to rate various aspects of your experience with
Vocational Rehabilitation and Employment using a scale of 1 to 10 where 1 is
Unacceptable, 10 is Outstanding, and 5 is Average. [SHOW ON THE SAME PAGE AS
THE QUESTION THAT FOLLOWS]
35. Please rate your VR&E benefit entitlement (e.g., training and counseling) on the
following items: [SHOW RESPONSES IN GRID WITH 10-POINT SCALE IN
COLUMNS AND SINGLE ROW (SEE JDPA CONVENTIONS DOCUMENT PG. 1
FOR SPECIFIC DETAILS OF LAYOUT). EVENLY SPACED RADIO
BUTTONS/COLUMNS, SINGLE RESPONSE PER ROW.]
a. Amount of benefits or services [ALLOW N/A RESPONSE] [1-10, N/A=99]
b. Effectiveness of benefit/service in preparing and obtaining suitable
employment [ALLOW N/A RESPONSE] [1-10, N/A=99]
c. Timeliness of receiving benefit payment [ALLOW N/A RESPONSE] [1-10,
N/A=99]
d. Overall rating of benefit payment/entitlement
36. While we understand there may be many reasons for not completing the plan, what
was the primary reason you did not complete your rehabilitation through the VR&E
program? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.]
a. I had a poor experience developing my rehabilitation plan [1]
b. I had a poor experience with the VR&E representative [2]
c. The VR&E program does not offer the services I need [3]
55
d. Issues related to the program requirements (too time
consuming/complicated) [6]
e. Issues related to transportation [7]
f. Issues related to a medical condition [8]
g. It is difficult to obtain information about the VR&E program [9]
h. Concerns about my eligibility for a specific track within the VR&E program
[10]
i. Other (Specify) _________________ [TEXT BOX, FORCE TEXT IF
RESPONSE IS SELECTED, 50 CHARACTER MAX.] [97]
j. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE] [99]
37. Did your decision not to complete your rehabilitation through the VR&E program
involve a change in any of the following life circumstances? (Mark all that apply)
[CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0 IF
UNCHECKED OR 1 IF CHECKED]
a. Marriage
b. Divorce
c. Death in the family
d. Had children
e. New job
f. Lost job
g. Moved
h. Declared bankruptcy
i. Retirement
j. Severity of disability
k. None of the above [MUTUALLY EXCLUSIVE RESPONSE]
Overall Experience with Benefit Program
38. Thinking about ALL aspects of your experience with Vocational Rehabilitation and
Employment benefits, please rate VA overall, using a scale of 1 to 10 where 1 is
Unacceptable, 10 is Outstanding, and 5 is Average. (Mark only one) [SHOW
RESPONSES IN GRID WITH 10-POINT SCALE IN COLUMNS AND SINGLE ROW
(SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC DETAILS OF
LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS, SINGLE
RESPONSE PER ROW.] [1-10]
Overall Experience with VA
39. Taking into consideration all of the non-medical benefits (e.g., education,
compensation, pension, home loan guaranty, vocational rehabilitation and employment,
insurance, etc.) you have applied for or currently receive, please rate your experience
with VA overall, using a scale of 1 to 10 where 1 is Unacceptable, 10 is Outstanding,
and 5 is Average. (Mark only one) [SHOW RESPONSES IN GRID WITH 10-POINT
SCALE IN COLUMNS AND SINGLE ROW (SEE JDPA CONVENTIONS DOCUMENT
56
PG. 1 FOR SPECIFIC DETAILS OF LAYOUT). EVENLY SPACED RADIO
BUTTONS/COLUMNS, SINGLE RESPONSE PER ROW.] [1-10]
40. How likely are you to reapply for the VR&E program in the future? (Mark only one)
[RADIO BUTTONS. SINGLE RESPONSE.]
a. Definitely will not [1]
b. Probably will not [2]
c. Probably will [3]
d. Definitely will [4]
About You
41. Are you currently employed? (Mark only one) [RADIO BUTTONS. SINGLE
RESPONSE.]
a. Yes [1]
b. No [0]
c. Prefer not to answer [98]
42. Do you have any other comments or concerns about your experience? (Open
Capture) [OPEN-END. TEXT BOX. 1000 CHARACTER MAX. ALLOW NO COMMENT,
MUTUALLY EXCLUSIVE CHECK BOX. CODE NO COMMENT AS 0 IF UNCHECKED
AND 1 IF CHECKED]
As a reminder, your responses will be kept completely confidential and your email
address will not be sent to VA with any responses on this survey. [SHOW ON THE
SAME PAGE AS THE QUESTION THAT FOLLOWS]
43. Would you like to provide an e-mail address so VA can contact you with general
information about VA benefits and services? (Mark only one) [RADIO BUTTONS.
SINGLE RESPONSE.]
e. Yes [1]
f. No [0]
g. I do not have an e-mail address [96]
h. Prefer not to answer [99]
(Ask Q44 if Yes in Q43)
44. Please enter your preferred e-mail address where you would like to be contacted:
(Open Capture)
a. E-mail: [OPEN CAPTURE. 100 CHARACTER MAX.]
57
Appendix G
List of Acronyms
AAPOR
ANOVA
BAS
BPA
BRE
CAPS
COR
DTA
EDIPI
EDX
FAR
FY
GPO
ICR
JDP
LGY
LWO
MAR
MCAR
MCMC
MNAR
NPC
OIF
OEF
OMB
OSAT
RO
SSN
US
USA
VA
VADIR
VAPSD
VBA
VOV
VR&E
VSO
American Association for Public Opinion Research
Analysis of Variance
Benefits Assistance Service
Blanket Purchase Agreement
Business Reply Envelope
Centralized Account Processing System
Contracting Officer’s Representative
Data Transfer Agreement
Electronic Data Interchange Personal Identifier
Enterprise Data Exchange
Federal Acquisition Regulations
Fiscal Year
Government Printing Office
Information Collection Request
J.D. Power
Loan Guaranty Service
Letter Work Order
Missing At Random
Missing Completely At Random
Markov chain Monte Carlo algorithm
Missing Not At Random
NPC, Inc. Integrated Print and Digital Solutions
Operation Iraqi Freedom
Operation Enduring Freedom
Office of Management and Budget
Overall Satisfaction Index
Regional Office
Social Security Number
United States
United States of America
Department of Veterans Affairs
VA DoD Identity Repository
VA Publications Services Division
Veterans Benefits Administration
Voice of the Veteran
Vocational Rehabilitation and Employment Service
Veterans Service Organizations
58
| File Type | application/pdf |
| File Title | Training Catalog, Department of Veterans Affairs |
| Subject | Training Catalog |
| Author | Department of Veterans Affairs, Office of Human Resources and Ad |
| File Modified | 2016-12-19 |
| File Created | 2016-12-19 |