Download:
pdf |
pdf2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
Survey Methodology Final Report
Contract No. GS-23F-8144H
Authors
Tom Bosworth
Michael Hornbostel
Jennifer Kali
October 2017
Prepared for:
Strategic Analysis Services (SAS)
Office of Strategic Planning and Analysis (OSPA)
Veterans Health Administration (VHA)
Department of Veterans Affairs
Prepared by:
Westat
An Employee-Owned Research Corporation®
1600 Research Boulevard
Rockville, Maryland 20850-3129
(301) 251-1500
Table of Contents
Chapter
1
2
3
4
5
Page
Background..........................................................................................................
1
1.1
Overview of the Survey of Enrollees .................................................
1
History of Methodological Changes ................................................................
2017 Sample Design ...........................................................................................
2
5
3.1
3.2
Description of Sampling Frame ..........................................................
Sample Selection....................................................................................
5
5
3.2.1
3.2.2
3.2.3
6
7
8
Development of Sample Targets ........................................
Selecting the Wave 1 Sample ..............................................
Selecting the Wave 2 Sample ..............................................
Questionnaire Design ........................................................................................
11
4.1
4.2
4.3
4.4
2017 Survey Revision Process .............................................................
Survey Sections ......................................................................................
Instrument Design and Programming ...............................................
Survey Communications Materials .....................................................
11
11
13
14
Data Collection ...................................................................................................
15
5.1
5.2
5.3
5.4
5.5
5.6
5.7
Web Survey ............................................................................................
Mail Survey.............................................................................................
Telephone Interviews ...........................................................................
Field Period, Sample Size, and Sample Waves..................................
Survey Communications ......................................................................
U.S. Postal Service Mailings ................................................................
IVR Calls ................................................................................................
15
16
16
17
17
18
18
5.7.1
Survey Communications Schedule .....................................
19
Data Collection Results by Mode .......................................................
19
5.8
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
iii
Contents (continued)
Chapter
6
Page
Information Center ............................................................................................
20
6.1
6.2
6.3
6.4
6.5
6.6
Hours of Operation ..............................................................................
Answering System .................................................................................
Call Volume ...........................................................................................
Emails Received ....................................................................................
Training for Information Center Representatives............................
Distressed Caller Protocol ...................................................................
20
20
21
24
25
26
Processing Multimode Data ..............................................................................
28
7.1
Key Survey Items ..................................................................................
29
8
Survey Statistics ...................................................................................................
31
9
Enhancements and Implementation Goals for Future Surveys ..................
33
9.1
9.2
Enhancement: Reduce Sampled Veteran’s Burden..........................
Goal: Meeting Survey Quotas by Stratum While
Minimizing Overage .............................................................................
Goal: Mail Replacement Survey Requests .........................................
33
Quality Assurance Measures .............................................................................
35
10.1
10.2
10.3
10.4
Development and Testing of the Web Survey .................................
Development and Testing of the Mail Survey ..................................
Survey Management System ................................................................
Quality Control: Mail Procedures .......................................................
35
37
38
39
Weighting .............................................................................................................
41
11.1
11.2
11.3
41
42
44
7
9.3
10
11
Calculation of Base Weights ................................................................
Adjustment for Nonresponse .............................................................
Poststratification Adjustment ..............................................................
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
iv
33
34
Contents (continued)
Appendixes
Page
A
Data Collection Goals and Achievement Rates .............................................
2
B
Web Survey Questionnaire ................................................................................
7
C
Mail Survey Questionnaire ................................................................................
42
D
Postal Communication Letters .........................................................................
55
E
Survey of Enrollees Training Manual ..............................................................
61
F
Distressed Caller Protocol .................................................................................
75
G
Rates by Strata .....................................................................................................
82
H
Rates by Market ..................................................................................................
102
I
Rates by Priority Groups Within Market ........................................................
107
2-1
Wave 1 control and experimental condition group size and
reliance questions ................................................................................................
3
5-1
VISNs and markets where telephone interviews were completed ..............
16
5-2
2017 communications numbers and dates ......................................................
17
5-3
Number of IVR reminder calls .........................................................................
18
5-4
2017 schedule for each survey communication .............................................
19
5-5
Number of completed surveys by mode .........................................................
19
6-1
Call volume by week according to telephone answering system
menu .....................................................................................................................
22
6-2
Reason for calling among those who spoke with a representative ..............
23
6-3
Information Center calls received by time of day (EDT) .............................
23
Tables
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
v
Contents (continued)
Tables
Page
6-4
Number of emails received per week ..............................................................
24
6-5
Reason for emailing the Information Center .................................................
24
6-6
Levels of distress and indicators .......................................................................
27
7-1
Key survey items and associated valid response values ................................
29
11-1
Nonresponse adjustment cell definitions, sizes, and adjustment
factors ...................................................................................................................
44
A-1
Market-level targets ............................................................................................
A-1
A-2
Goals by priority groups within VISNs...........................................................
A-4
B-1
Final list of alternative Medicaid program names by state for 2017
survey ....................................................................................................................
B-33
F-1
CATI distressed protocol ..................................................................................
F-1
F-2
Information center representative distressed protocol .................................
F-4
G-1
Rates by strata .....................................................................................................
G-1
H-1
Rates by market...................................................................................................
H-1
I-1
Rates by priority groups within market ...........................................................
I-1
3-1
Sample selection..................................................................................................
6
6-1
Total number of phone calls and key mailing dates ......................................
22
Figures
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
vi
Background
1.1
1
Overview of the Survey of Enrollees
Since 1999, the U.S. Department of Veterans Affairs’ Survey of Veteran Enrollees’ Health and Use
of Health Care (Survey of Enrollees) has been a primary source of data for understanding who
Veterans enrolled in Veterans Affairs (VA) health care are, what influences their health care choices,
how they perceive their health status, how they have used the VA over the past year, and how they
plan to use it in the future. The Survey of Enrollees is designed to gather information on enrollees
that cannot be obtained in other ways. VA’s authority to provide care to Veterans through the VA
health care system is regulated in part by the Veteran’s Health Care Eligibility Reform Act of 1996
(Public Law 104-262). This law implements a priority-based enrollment system for Veterans and
gives VHA (Veterans Health Administration) the ability to plan to meet the needs of all enrolled
Veterans.
Data gathered from this survey primarily informs the VA Enrollee Health Care Projection Model,
used for projecting enrollment, utilization, and expenditures. In addition, these data may be used for
a variety of strategic analyses at the Veterans Integrated Service Network (VISN) level, the Market
level either within the VISN, or at the National Program Office level.
There have been 15 cycles of the survey (1999, 2000, 2002, 2003, 2005, 2007, 2008, 2010, 2011,
2012, 2013, 2014, 2015, 2016 and 2017) conducted by the Office of the Assistant Deputy Under
Secretary for Health for Policy and Planning. Through 2011, the survey was conducted using a
telephone interview. In 2012, a multi-mode design was introduced, consisting of telephone, mail and
web data collection. The methodology for 2015, 2016, and 2017 employed web and mail data
collection, with emphasis placed on effective strategies for maximizing responses obtained using
web technology. In 2016 and 2017, a small number of Computer Assisted Telephone Interviews
(CATI) were utilized at the end of data collection in order to meet minimum targets in markets.
This report describes the survey approach and methods for the 2017 data collection period, which
covered the period March 29, 2017 through July 17, 2017.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
1
History of Methodological Changes
2
To support the research objectives of the Survey of Enrollees, survey modes, and methods have
been examined and adjusted over the years. Since 2007, various methodological changes and
experiments have been implemented to enhance survey performance and efficiency. Methodological
experiments employed for the 2017 survey, along with a brief history of previous years’ experiments
and changes, are highlighted below. A detailed description of the 2017 methodological experiments
may be found in a separate report, Survey of Enrollees 2017 Methodology Experiments Report.
Survey Modes. Since 2012, the Survey of Enrollees has included experiments using multiple survey
modes (i.e., telephone, mail, and web surveys). The objective was to determine if response rates
could be boosted, or at least maintained, while reducing survey administration costs. Various
combinations of modes have been employed and tested as follows:
Prior to 2012 – Telephone;
From 2012-2014 – Telephone, Web and Mail; and
2015-2017 – Web and Mail. 1
In 2012 through 2014, two self-administered modes (web and mail surveys) were introduced to
supplement telephone surveys.
The 2014 Survey of Enrollees replicated two experiments that were conducted in 2013 to test mode
effects. The experiments tested for mode effects among the telephone-eligible population by
assigning potential respondents to both telephone or mail modes, and testing the effect of multiple
telephone non-responder contacts by sending mail surveys to those non-responders.
Since 2015, the main modes of administration were web and mail. Telephone was planned as a final
attempt to obtain participation from non-responders to the first two modes (sequence of modes was
web-mail-telephone, as compared to telephone-web-mail in 2014). Due to the success of the web
and mail methods, no telephone interviews were necessary in 2015 and an extremely small number
1
To meet the quota of completes by market, in 2017 a small number of surveys were completed by telephone. Overall,
46 surveys or .1 percent of all completed surveys were conducted using the telephone.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
2
were needed in 2016 and 2017 to meet survey goals. The 2015, 2016, and 2017 administrations of
the Survey of Enrollees represented a turning point from the multi-mode sequence used in previous
years. A well-planned series of structured techniques was used to encourage as many Veterans as
possible to respond using the web.
Reliance Questions. An experimental design was developed in 2017 to determine the accuracy of
the reliance questions based on experience from the 2015 and 2016 data collections. The goal of this
experiment was to identify the best way to measure Veterans’ reliance on VA for their health care.
Reliance is measured by asking about the number of VA outpatient visits paid for (fully or partly) by
VA versus outpatient visits that VA did not pay for at all (non-VA.) The Wave 1 sample was
randomly assigned into either a control condition or one of four experimental condition groups. The
control condition group was asked the 2015 survey reliance questions and represented 90 percent of
the Wave 1 sample. The four experimental condition groups were comprised of 2.5 percent of the
Wave 1 sample. Experimental condition group 1 asked the 2016 survey reliance questions.
Experimental condition groups 2, 3, and 4 were asked a variation of the 2015 survey reliance
questions. Table 2-1 demonstrates the size and questions asked of each control and experimental
condition groups. See the 2017 Methodological Experiments Report for a more in-depth description of
this approach and the findings. All Wave 2 respondents received the Wave 1 control condition
version of the questions.
Table 2-1.
Sample size
Percent of
Sample
Reliance
questions
asked
Wave 1 control and experimental condition group size and reliance questions
Control
Condition
82,410
Experimental
Condition 1
2,375
Experimental
Condition 2
2,387
Experimental
Condition 3
2,367
Experimental
Condition 4
2,376
90%
2.5%
2.5%
2.5%
2.5%
2016 survey
reliance
questions
2015 survey
reliance
questions with
VA and non-VA
reversed
2015 survey
reliance
questions using
3 questions
2015 survey
reliance
questions first
asking about
total visits
2015 survey
reliance
questions
Sample Stratification. As in 2015 and 2016, stratification by markets was included in the sample
design for 2017. The number of sampling strata remained at 576, the same as 2016. A subset of
strata was oversampled to achieve the following sample design objectives:
A minimum effective sample size of 315 completed interviews in each market;
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
3
A minimum effective sample size of 597 completed interviews in each of the collapsed
priority groups (1-3, 4-6, and 7-8) in every VISN; and
A minimum of 30 percent of completed interviews among pre-enrollees (enrolled prior
to March 31, 1999).
(Note: The effective sample size for a market or a collapsed priority group within a VISN is the
number of completes adjusted for the design effect resulting from oversampling.)
Data Collection Period. The Survey of Enrollees’ data collection period has fluctuated in recent
survey cycles to allow additional time for mail fulfillment to reach all Veterans and maximize
response rates. In 2010, the data collection period was extended from 10 weeks to 12 weeks. The
2013 data collection period was extended further to 14 weeks. In 2014, the data collection period
was extended to 18 weeks. In 2015, the data collection was completed in 11 weeks. Due to the
nature of the methodological experiments, the 2016 data collection period was 18 weeks. In 2017,
the data collection was completed in 16 weeks.
Sequence of Contacts. All sampled Veterans were given the same sequence of contacts. Every
sampled Veteran was mailed a survey invitation letter with a link to web survey and unique PIN. The
following week every sampled Veteran was also mailed a postcard reminder with a link to the web
survey and unique PIN. Nonrespondents were then mailed a paper survey two weeks later. Oneweek later IVR reminder calls were made to those with a telephone number who received the paper
survey. See section 5 for more information about the survey communication protocol.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
4
2017 Sample Design
3.1
3
Description of Sampling Frame
The sampling frame for VA’s 2017 Survey of Enrollees was selected from the VHA enrollment file
and contained all Veterans enrolled in VA health care as of September 2016. The sample frame
excluded records lacking a valid address and those with addresses outside the U.S. and Puerto Rico.
Those records with missing variables needed for stratification (VISN, market, priority group, and/or
enrollee type) were excluded. Additionally, starting in 2017, enrollees sampled for the 2016 Survey of
Enrollees were excluded from the sample frame (though final weights were computed to account for
the excluded records).
There were 8,180,466 records in the sampling frame. Each record contained variables to be used for
stratification, including VISN, market, priority group, and enrollee type, and variables to be used to
contact sampled persons (name, address, and telephone number). In addition, a variable was added
indicating whether a person was eligible, not eligible, or sampled for 2016 Survey of Enrollees study.
3.2
Sample Selection
A two-phase sampling approach was used to select the survey sample, with the same strata used to
select the samples for both phases. First, Westat determined the target sample sizes for each
sampling stratum and provided them to the VA who, in turn, selected 499,743 Veterans from their
enrollment file. Using this list of Veterans, Westat sampled Veteran enrollees to be contacted for the
study in two waves. Wave 1 was a sample of 96,573 individuals and was selected in February 2017;
Wave 2 was a sample of 34,763 Veterans and was selected in April 2017. Figure 3-1 depicts the
sample selection process.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
5
Figure 3-1.
3.2.1
Sample selection
Development of Sample Targets
Sampling targets were developed to ensure an adequate number of completed interviews for the
following analytic domains:
Individual markets;
Priority groups within VISN, with priorities 1, 2, and 3 being one group; priorities 4, 5,
and 6 being a second group; and priorities 7 and 8 being the third group; and
Enrollee type, with enrollment prior to March 31, 1999 versus enrollment after this
date.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
6
Targets were developed to ensure three goals:
1.
Guarantee an effective sample size of at least 315 completes (90% of the target goal of
350) within each market and for each priority group within a VISN, where the effective
sample size was the number of completes adjusted for the design effect resulting from
oversampling subgroups;
2.
Ensure that at least 30 percent of the completed cases were pre-enrollees; and
3.
Achieve a total sample large enough to ensure 42,000 completed cases.
A complete listing of the sample targets may be found in Appendix A.
3.2.2
Selecting the Wave 1 Sample
Because sample targets were defined for markets, priority groups within VISN and enrollee type, the
sampling strata were the three-dimensional cells based on market, priority group, and enrollee type.
Linear programming methods were used to determine the optimal stratum sample sizes needed to
achieve the desired goals.
The allocation of the sampling targets to the sampling strata provides the number of completes
needed in a given stratum. To account for nonresponse, the number of completes in a sampling
stratum was adjusted by the inverse of the stratum’s response rate for the control (nonexperimental) group of the 2016 study, which used the same protocol as the 2017 study. Response
rates achieved in 2016 were considered a good proxy for expected 2017 response rates, allowing
adjustments to the sample for low performing and high performing strata to ensure that overall
targets were met.
The sample within each sampling strata was selected using systematic sampling from the 499,743
records sampled from the VA enrollment file. The records were sorted by zip code within stratum
to ensure proper geographic representation. Multiplying the desired number of completes for each
sampling stratum by the inverse of the 2016 response rate produced a total sample size of about
145,000 for fielding. This expected sample was scaled down by a constant factor to 96,573 for Wave
1 to allow for the possibility of higher than expected response rates. The Wave 2 sample targeted the
low performing strata.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
7
Beginning with the 2017 study, veterans sampled for the 2016 Survey of Enrollees were ineligible for
the 2017 Survey of Enrollees and excluded from the sampling frame. To account for the excluded
cases, Veterans who were eligible for selection in previous study but not sampled were over
sampled. To achieve this, aggregate counts of Veterans in three subsampling groups were provided
by VA prior to excluding any Veterans: subsampling group one were those sampled for other VA
studies within the same year (N = 165,706), subsampling group two were those eligible for other VA
studies but not sampled (N=8,014,760), and subsampling group three were those not eligible for
other VA studies, i.e., Veterans new to VA (N = 431,002). Veterans in group 1 were not sampled.
Those in group 2 were over-sampled relative to group 3 in order to increase the group 2 sample size,
permitting group 2 to represent both group 1 and group 2.
A question-wording experiment was conducted within Wave 1 only. Four experimental groups of
2,500 Veterans each were selected to receive different versions of the questionnaire to be compared
with the 86,573 remaining Veterans in the control group. Systematic random samples from the
sampled Veterans were selected to assign sampled Veterans to experimental groups. The sampled
Veterans were sorted by the stratification variables in the main survey (market, VISN, priority group,
and type) prior to selecting the treatment group sample.
3.2.3
Selecting the Wave 2 Sample
The Wave 2 sample was selected from the remainder of the VA-provided sample list after the
Wave 1 sample was excluded. Determining the stratum level sample sizes for Wave 2 was more
complicated than for Wave 1. It required estimating the number of completes that would result from
the Wave 1 sample, which was still being fielded, and determining the proper way to distribute the
sample among the sampling strata to meet the sample targets (comprised of overlapping
combinations of sampling strata).
The following six-step process was used to select the Wave 2 sample.
1.
The expected number of Wave 1 completed cases was estimated for each stratum.
2.
The results from Step 1 were used to determine stratum-level shortfalls.
3.
Results from Step 2 were aggregated to the overlapping analytic domains (markets,
priority groups within VISNs and enrollee type) for which target numbers of completed
cases had been specified.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
8
4.
Step 2 stratum-level shortfalls were reallocated across strata in order to minimize the
Wave 2 fielded sample size, subject to satisfying the Step 3 domain of interest level
shortfalls.
5.
Wave 2 fielded sample sizes were determined at the stratum level.
6.
Wave 2 fielded sample sizes were determined for the subsampling groups within each
stratum.
Step 1: Expected Number of Wave 1 Completes. The Wave 1 sample was still in the initial stage
of data collection at the time the Wave 2 sample needed to be drawn. Estimates had to be made
about the number of completes expected from the remaining Wave 1 sample. Estimated completes
were based on the prior years’ experience for each stratum. Estimating in this manner assumes that
strata that are high or low performing in the prior year will continue this performance in the current
data collection cycle.
Step 2: Stratum-Level Shortfalls. The number of completes allocated to a sampling stratum to
select the Wave 1 sample was compared to its expected number of completes computed in Step 1.
The difference between the two numbers was the Wave 1 stratum-level expected shortfall.
Step 3: Domain-Level Shortfalls. The file of stratum-level shortfalls created in Step 2 contained
membership variables for the following types of analytic domains:
Individual markets;
Priority groups within VISNs; and
Veterans who enrolled prior to March 31, 1999, compared to Veterans who enrolled
after this date.
For example, each record in the file of stratum-level shortfalls contained a variable that indicated to
which market the record belonged. For each type of analytic domain, we used the values of the
appropriate domain-membership variables to aggregate stratum-level shortfalls to domain-level
shortfalls.
Step 4: Reallocation of Stratum-Level Shortfalls. To minimize the Wave 2 fielded sample size,
stratum-level shortfalls identified in Step 2 were reallocated across strata. This reallocation was
subject to satisfying the Step 3 domain of interest level shortfalls.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
9
Step 5: Stratum-Level Wave 2 Sample Sizes. Step 4 determined the expected number of needed
completes by sampling stratum. As was done for Wave 1, the fielded number of Wave 2 cases for
each stratum was determined by multiplying the allocated number of completed cases by the inverse
of an estimated response rate. For Wave 2, estimated response rates are based on a combination of
the interim results of Wave 1 and the results of the prior year cycle. At the time of sampling for
Wave 2, the paper survey for Wave 1 had not been mailed yet, so only the web response rates were
available. Therefore, the estimated Wave 2 response rate was a combination of the 2017 web
response rate and the 2016 paper response rate.
Step 6: Wave 2 Sample Sizes for Subsampling Groups with Strata. As was done in Wave 1
within each sampling stratum, Veterans sampled for the 2016 Survey of Enrollees (group 1) were
not sampled and Veterans eligible for sampling in the 2016 Survey of Enrollees and were not
sampled (group 2) were over sampled relative to newly enrolled Veterans who were not enrolled at
the time of the 2016 Survey of Enrollees (group 3).
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
10
Questionnaire Design
4.1
4
2017 Survey Revision Process
Westat and VA jointly modified and revised the 2017 Survey of Enrollees questionnaire to better
support VA’s measurement needs. One change was a methodological experiment Westat suggested
to improve the reliance questions that ask about outpatient health care utilization paid for by VA
and outpatient care utilization not paid for by VA. These data are used to calculate VA reliance,
which is the ratio of number of VA-paid visits to total number of VA and non-VA visits. A detailed
description of the 2017 methodological experiments may be found in a separate report, Survey of
Enrollees 2017 Methodology Experiments Report. Another modification was a skip pattern to allow all
respondents to receive questions about text messaging on cell phones and awareness and use of
VA’s My HealtheVet web site. VA also instructed Westat to remove questions that were no longer
needed for their projection models or policy decisions. The 2017 survey was thus different from
previous-year surveys. While these content changes affect ongoing trend analyses, they also position
the VA to address emerging issues affecting the use of VA health care and the expansion of health
care delivery methods to VA enrollees.
4.2
Survey Sections
Title and Cover Page. The VA retained the survey title from the prior year—Survey of Veteran
Enrollees’ Health and Use of Health Care. Welcome text on the cover briefly described the survey
purpose, stated the expected survey completion time (20 minutes), and assured enrollees not
currently using VA health care services that their participation was important. It also stated that
knowledgeable proxies could answer on behalf of the enrolled Veteran if health problems prevented
the Veteran from taking the survey.
Introduction and Name Verification. This section included statements that participation was
voluntary; VA benefits would not be affected if the enrollee chose not to respond, and results would
be aggregated, and respondents’ identities protected. Two questions verified if the sampled Veteran
was answering the survey or if a person knowledgeable (proxy) about that person’s health care,
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
11
health benefits, and health status was answering on behalf of the Veteran. Proxies were asked an
additional question about their relationship to the Veteran.
Health Benefits. This section opened with a question asking if the Veteran was enrolled in VA
health care. A definition of “enrolled veteran” was included to promote accurate answers and to
reduce “Don’t Know” responses. The remaining questions asked about various types of health care
coverage, including Medicare, Medicaid, TRICARE, and other plans obtained through employers,
family members, or another source. Veterans with “other” health care coverage plans were asked if
their plans included prescription drug coverage. The section also includes a question about longterm care policies covering nursing home care. This section ended with a question asking for the
primary source of information about VA health benefits and eligibility.
Medication Use and Benefits. The questions in this section asked about number of prescription
medications and how many were obtained from the VA.
Your Views about Health Care and Reasons for Using or Not Using VA’s Health Care
System. The first set of questions in this section were part of the methodological experiment
previously discussed. Enrollees were randomly assigned to a control or four experiment groups,
which determined the experimental question ordering, wording, and structure. The experimental set
of questions asked about the number of outpatient visits to VA and non-VA health care facilities for
which the VA paid fully or partially for the visits. The questions in this section also inquired about
ease of getting appointments, the professionalism of the medical staff, and the Veterans’ perceived
involvement in being informed and involved in decisions regarding their care. Veterans who used
health care services other than through the VA were asked about factors when selecting a health
care provider and reasons for using these other health care services. In addition, all Veterans were
asked about how often VA services meets their health care needs and ways to use VA for health care
in the future.
Current Health and Caregiver Assistance. This section was designed to collect more information
about enrollees’ need for caregiver services. Enrollees were asked for a self-evaluation of their health
status and if they require assistance performing select activities of daily living (ADLs), such as
bathing, eating, taking medications properly, or coping with stressful situations. This section closes
with a series of questions on cigarette smoking history. A set of questions asking about caregiver
support and level of dependence on caregivers were removed from the 2017 survey.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
12
Digital Access. This survey section helps the VA better understand how enrollees are using mobile
devices and computers to connect to the Internet. The VA will use the data to identify potential
opportunities for using technology to improve its delivery of care and expand enrollee use of
telehealth. Enrollees who use the Internet, at least occasionally, were asked whether they use it to do
various tasks, where they go online, how often they access the Internet on various types of
computers and mobile devices, and the types of service(s) they use to access the Internet. In 2016,
only enrollees who used the Internet occasionally were asked the remaining questions in this section.
In 2017, a change in skip patterns asked all enrollees the remaining questions in this section,
regardless of using the Internet, if they send or receive text messages on a cell phone and the
remaining questions. Questions asked how willing they would be to do various healthcare-related
activities on at least one of their computers or mobile devices (e.g., fill out VA health-related forms,
communicate securely with their VA health care providers via secure email/text messaging on
mobile devices, and use VA apps to track their health care status). Finally, they were asked if they
were aware of or used VA’s My HealtheVet web site. Those who used the My HealtheVet web site
were asked for what purposes they use it for such as to look for health information or to see my VA
appointments.
About You. In 2016, this section was revised to reduce respondent burden, and began with a
question about the period of military service. Instead of asking for beginning and final end dates of
their active duty service as well as which of nine military periods of service they had served in,
enrollees were asked only about the nine military periods of service. An additional question asked if
the Veteran had ever served in a combat or war zone. The remaining questions were demographic
items, including marital status, total number of dependents and number of dependents under age 18,
employment status, Hispanic ethnicity, race, and total annual household income. Veterans were
asked to indicate their total annual income by choosing among a range of income categories.
Trust in VA. The last question in the survey was to measure Veteran enrollees’ attitudes toward the
statement “I trust VA to fulfill our country’s commitment to Veterans.” This question is asked on a
number of other VA surveys.
4.3
Instrument Design and Programming
To reduce potential measurement error and mode effects across the planned survey modes, Westat
incorporated a mix of basic survey design principles for each specific survey mode, as well as
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
13
programming features of our web and telephone survey application. A few examples of design
principles that were implemented included:
A yes/no response format for each item in check-all-that apply questions;
Use of automated skips in the web and telephone instruments;
Pop-up messages in the web and telephone instruments alerting respondents and
interviewers when answers did not fall within valid response ranges; and
Automatic reformatting of grid items to single-question format when the survey was
viewed on a smartphone.
Prior to the start of data collection, we tested the programmed instruments across various platforms,
browsers, browser versions, and screen resolutions on desktop computers, notebooks, and
smartphones to ensure ease of response and accurate data capture.
Mail Survey. The 52-question mail survey was designed and formatted as a scannable 12-page
black-and-white booklet with an attractive VA logo on the cover. We used simple, clear, and
noticeable instructions to promote accurate navigation. Proxies were reminded at the start of each
section to answer on behalf of the Veteran named in the invitation letter.
Web Survey. The web survey instrument was hosted on a secure web site. Sample members were
provided with a survey URL and a unique personal identification number (PIN) to log into the
survey. The Veteran’s name and appropriate pronouns were automatically merged into the survey
questions when a proxy was answering for the Veteran. Respondents were able to complete the
survey in more than one session, with their answers saved. To meet quotas in three markets, a very
small number of interviews (46 in total) were completed over the telephone. Telephone interviewers
used the web application to record Veterans’ responses.
4.4
Survey Communications Materials
Westat designed all survey communication materials (letters, postcard reminders, and envelopes) to
encourage a survey response from sampled Veterans. The black-and-white letters and reminder
postcards were clear, informative, brief, and friendly. The salutation was personalized, and the
signator was a senior VA official in the Department of Veterans Affairs. The VA logo appeared on
all communications and envelopes to reinforce the validity and importance of the survey. Examples
of all survey postal communications are included in Appendix D.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
14
Data Collection
5
The 2017 survey used the same multimode protocol as was used in 2015 and 2016. The survey was
initially launched via the web to all sampled enrollees with valid mailing addresses. All nonresponding Veterans were followed up with a paper survey. Using the 2015 and 2016 response rates,
the 2017 samples for Wave 1 and Wave 2 were selected to ensure that minimum cell targets were
met in all of the VISNs and markets. All data collection targets (i.e., number of completed interviews
in each VISN and market) were met with web and mail survey respondents, with the exception of
three markets where the number of completes came up just short of the goal. As a result,
46 interviews were completed over the telephone in order to fulfill all target numbers.
5.1
Web Survey
A secure website was developed for the survey data collection. Each sample member received a link
to the survey URL:www.surveyvha.org (i.e., the website location of the survey), and a unique 8-digit
PIN code. When Veterans logged into the web survey system, their PIN number was verified against
a database in the survey management system (SMS). Appendix B contains the annotated version of
the web survey.
Administration of the web survey began on March 29, 2017, and the web survey option was
available to respondents throughout the entire data collection period. Every Veteran selected to
participate was given the opportunity to complete the Survey of Enrollees via a web survey. Each
postal communication included instructions and a unique PIN number to securely access the web
survey.
The web survey site included a toll-free number and e-mail address if Veterans required technical
assistance. A link to the Frequently Asked Questions (FAQs) was also available on every page of the
web survey.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
15
5.2
Mail Survey
Westat conducts mail surveys using scannable forms to ensure rapid and accurate data entry. For the
Survey of Enrollees, we used TeleForm® – a software system for intelligent data capture and image
processing. TeleForm enables us to rapidly associate variable names and data capture zones with
each question included in the questionnaire. The mail survey instrument used for the Survey of
Enrollees may be found in Appendix C.
Westat sent mail surveys to all the sampled Veterans with a valid mailing address who had not
previously completed a web survey. The mail survey included a cover letter describing the
importance of the study and a business return envelope to return the completed questionnaire. Also
included were instructions for completing the web survey and a toll-free number if Veterans
required technical assistance.
5.3
Telephone Interviews
In order to meet minimum target quotas at the end of the data collection, a small number of
additional completes were needed from three markets. Westat implemented a telephone interview
protocol to meet the study’s target goals. A random selection of non-responding Veterans from the
three markets (17-e, 17-g and 23-p) were contacted during the final week of data collection. Westat
completed 46 telephone interviews over four days to satisfy sampling targets. Table 5-1 below
indicates the locations of VISNs and Markets as well as the number of completed interviews.
Table 5-1.
VISNs and markets where telephone interviews were completed
VISN
17
17
23
Total
Market
17-e
17-g
23-p
Location
Southern Texas
West Texas
West South Dakota
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
16
Completed Telephone Interviews
24
12
10
46
5.4
Field Period, Sample Size, and Sample Waves
Data collection for the 2017 Survey of Enrollees began on March 29, 2017 and ended on
July 17, 2017. In total, 124,645 Veterans with valid current mailing addresses were invited to
participate from the 131,336 records sampled. There were 6,862 records sampled that did pass
National Change of Address (NCOA) and were dropped from the sample. To allow better targeting
of stratification cells and ensure that all cell targets met their quotas, the sample was released in two
waves. Wave 1 was released on March 29, 2017, included 96,573 sampled Veterans of which 91,904
enrollees with valid addresses were mailed survey invitations. Wave 2 was released on May 8, 2017,
included an additional 34,763 sampled Veterans of which 32,741 enrollees with valid addresses were
mailed survey invitations. Completed interviews for each Wave were accepted through the end of
the data collection period (July 17, 2017).
5.5
Survey Communications
The 2017 survey communication protocol followed the strategies learned in 2015 and 2016. All
sampled enrollees were mailed an invitation letter and a list of Frequently Asked Questions (FAQ).
The same enrollees were mailed a postcard reminder one week later. Approximately two weeks later,
non-responders were sent a mail survey. The same non-responders with valid phone numbers
received Interactive Voice Response (IVR) reminder calls. Table 5-2 presents the dates and number
of communications sent in 2017.
Table 5-2.
2017 communications numbers and dates
Communication Type
Survey invitation letter
Postcard reminder
Mailing with questionnaire
Interactive voice Response (IVR)
reminder calls
Total
Number in
Wave 1
Wave 1
Date
Number in
Wave 2
Wave 2
Date
91,904
91,904
77,728
3/29
4/5
4/20
32,741
32,741
26,562
5/8
5/15
6/6
124,645
124,645
104,290
35,371
4/27
13,253
6/13
48,624
296,907
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
105,297
17
Total
402,204
5.6
U.S. Postal Service Mailings
Prior to survey administration, the U.S. Postal Service National Change of Address (NCOA)
database was used to obtain the most up-to-date mailing addresses for sample members. Of the
131,336 records sent through NCOA processing, 6,862 total records were identified as having a
change of address or an invalid address (4,969 addresses of these were in Wave 1, and 1,893
addresses were in Wave 2). These records were excluded from the sample.
All postal communications were sent via USPS first-class mail. Sampled enrollees were mailed a web
survey invitation letter. The letter informed them that they had been selected for the survey,
provided some background about the survey, and invited them to log into a secure website where
they could access the survey using their enclosed personal identification number (PIN) and complete
the survey. Each postal communications included the Information Center toll-free telephone
number for inquiries about the survey content, or computer and technical issues. For all sampled
Veterans, the web survey invitation mailing included Frequently Asked Questions (FAQs).
5.7
IVR Calls
Interactive Voice Response (IVR) reminder calls were made to survey respondents who received a
survey in the mail and had a known landline telephone number. An IVR call is an automated call
with a 30-second recorded message delivered to a list of telephone numbers. Telephone numbers
were drawn from the list file provided by the VA. Telephone numbers were available for the
majority of the sampled enrollees. The IVR calls were made to non-responders a week after the
survey mailing was sent out. Table 5-3 shows the number of sample members who received IVR
reminder calls in Wave 1 and Wave 2.
Table 5-3.
Wave 1
35,371
Number of IVR reminder calls
Wave 2
13,253
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
Total IVR Calls
48,624
18
5.7.1
Survey Communications Schedule
Table 5-4 presents the schedule for survey communications contacts. The schedule was planned to
fit within the 16-week data collection period, as dictated by the study protocol, and was adjusted to
accommodate weekends and the high volume mailings.
Table 5-4.
2017 schedule for each survey communication
Activity
Survey invitation letter
Postcard reminder
Mailing with questionnaire
IVR call
5.8
Number of Days After the
Survey Invitation Letter
0
7
Wave 1 = 21
Wave 2 = 29
Wave 1 = 28
Wave 2 = 36
Wave 1 Dates
3/29
4/5
4/20
Wave 2 Dates
5/8
5/15
6/6
4/27
6/13
Data Collection Results by Mode
At the conclusion of data collection, 43,654 Veterans had submitted or returned a completed survey.
Thirty-six percent of responders completed the web version of the questionnaire and the remaining
respondents (64%) completed the mail survey (see Table 5-5). An extremely small percentage (0.1%)
completed a telephone survey.
Table 5-5.
Number of completed surveys by mode
Web Survey
Completes
15,815
36.2%
Mail Survey
Completes
27,793
63.7%
Telephone Survey
Completes
46
0.1%
Total
Completes
43,654
100%
All targets were met for completed surveys by markets, priority groups within VISNs, and pre-/post
enrollee groups. See Appendix A for further details.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
19
Information Center
6
Westat’s Information Center for the Survey of Enrollees was staffed to answer questions from
sampled enrollees by telephone or email. Sampled enrollees could call the Information Center to
hear recorded information about the study or to speak directly with a Survey Information Center
representative. Ten persons staffed the Information Center during the fielding period and for 7 days
following the end of data collection, with additional persons assisting during periods of high volume
resulting from reminder mailings and IVR calls. Between March 29, 2017 and July 17, 2017, the
Information Center received 4,098 calls and 23 emails. Of these, 1,547 callers (38%) chose to speak
to a live representative, while the rest opted to select an option from a menu to hear a recorded
answer. The most common theme expressed in the calls and emails concerned questions about the
background and purpose of the survey. Other common reasons for contacting the Information
Center were related to requesting a paper survey, connectivity issues, and indicating they did not
have Internet or a computer.
6.1
Hours of Operation
Westat maintained a dedicated toll-free line, which was staffed from 9 a.m. to 9 p.m. Eastern
Daylight Time, 7 days a week beginning Wednesday, March 29, 2017 and ending Monday,
July 17, 2017, including holidays. During those hours when the Information Center was not staffed,
callers could leave a message and receive a return call the next business day. When calls were
returned, an outbound caller-id display of “VA HEALTH 301-255-0020” was displayed.
6.2
Answering System
When Veterans called the Information Center’s toll-free number, they were first routed to an
interactive voice response (IVR) telephone answering system menu to provide automated messages
regarding the study. The initial prompt advised that all calls would be accepted, and then the
Veterans were offered the following menu of choices:
Press 0 – To speak with an Information Center representative.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
20
Press 1 – If you do not have Internet access or a computer at home to take the web
survey.
Press 2 – To hear general information about the survey.
Press 3 – To hear how you were selected and how we obtained your name.
Press 4 – To hear who is conducting the survey.
Press 5 – To leave a voice mail message.
In order to better manage a large number of calls about no Internet or computer access, a menu
option (“Press 1”) was offered, allowing callers to hear a recorded message if they did not have
internet access or a computer. Overall, 2,100 callers selected the IVR menu option about not having
Internet access or a computer to take the web survey.
Additional information about the number of calls handled through the telephone answering system
menu can be found in Table 6-1.
During the field period, the Information Center returned 90 answering machine messages based on
the IVR choice to select option 5. The majority of these messages were either from Veterans
notifying Westat of their inability to complete the survey via web because they either did not have
access to a computer and/or had no internet access or that a Veteran was deceased.
6.3
Call Volume
The volume of calls received typically followed the timing of contacts made to sampled study
participants. For example, the highest call volumes were seen just after the invitation letters were
received by participants. A smaller increase in volume was noticed shortly after the questionnaire
mailing. An illustration of the call volume and key contact dates can be found in Figure 6-1.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
21
Table 6-1.
Week
Call volume by week according to telephone answering system menu
Date
(Mon – Sun)
1
3/29 – 4/2
2
4/3 – 4/9
3
4/10-4/16
4
4/17-4/23
5
4/24 – 4/30
6
5/1 – 5/7
7
5/8 – 5/14
8
5/15 – 5/21
9
5/22 – 5/28
10
5/29 – 6/4
11
6/5 – 6/11
12
6/12 – 6/18
13
6/19 – 6/25
14
6/26 – 7/2
15
7/3 – 7/9
16
7/10 – 7/17
Total
Percent
Figure 6-1.
Pressed 0
Speak to
Rep
Pressed 1
No Internet
Access
Pressed 2
General
Info
101
173
154
108
214
140
79
133
47
49
85
100
57
40
37
30
1,547
37.8%
144
449
367
132
203
102
103
275
63
50
57
105
20
15
7
8
2,100
51.2%
9
31
10
17
21
13
6
16
2
4
2
11
6
0
0
1
149
3.6%
Pressed 3
How we
Obtained
Your Name
8
21
10
11
53
22
2
14
5
5
3
10
2
2
3
2
173
4.2%
Total number of phone calls and key mailing dates
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
22
Pressed 4
Pressed 5
Total
Who is
Left
Calls
Conducting
Voicemail Received
Survey
1
6
269
4
9
687
4
7
552
5
5
278
9
17
517
3
6
286
1
10
201
3
12
453
1
1
119
1
5
114
1
4
152
5
6
237
1
1
87
0
1
58
0
0
47
0
0
41
39
90
4,098
1.0%
2.2%
100.0%
Table 6-1 summarizes the call volume to toll-free lines over the course of the data collection. During
the entire data collection period, 51 percent of the callers opted to listen to a pre-recorded message
regarding “no internet access” (“Press 1”). Almost 4 percent called to receive general information
(“Press 2”), and a 4 percentage of callers had questions about how we got their name (“Press 3”).
At the end of each conversation, an Information Center representative categorized the reason for
the call. Table 6-2 summarizes the reasons given for contacting the Information Center. As the table
shows, the majority of the calls were related to general information on study background/how to
proceed, requesting a mail survey, and not having access to a computer and/no internet access.
Table 6-2.
Reason for calling among those who spoke with a representative
Reason for Call
Information on study background/how to proceed
Request a survey mailing
No computer or Internet access
Refusal to participate
Sign in/connection issues with web survey
Address Update
Feedback for VA
Stated they would participate
Requests proxy/proxy needed
Request for PIN number
Question about a survey item
Name update
Telephone number update
Distressed call
Update email address
Total
Number of Calls
1,438
511
425
125
118
61
22
12
68
40
32
5
7
3
1
2,868
Percent of Calls
50.1%
17.8%
14.8%
4.4%
4.1%
2.1%
0.8%
0.4%
2.4%
1.4%
1.1%
0.2%
0.2%
0.1%
0.1%
100%
As Table 6-3 indicates, the majority of calls (95%) to the Information Center occurred during the
daytime hours between 9 a.m. and 5 p.m. EDT. Four percent (4%) of the calls occurred during the
evening hours between 5 p.m. and 9 p.m. EDT. Only 1 percent of the calls came when the
Information Center was closed between 9:01 p.m. and 8:59 a.m. EDT. The IVR telephone
answering system menu prompts satisfied a great deal of requests. The highest volume of calls
tended to occur at the beginning of the week (Monday, Tuesday, and Wednesday). In comparing the
volume between 2016 and 2017, there was little variance when the calls were received.
Table 6-3.
Information Center calls received by time of day (EDT)
Information Center Status
Percentage of Inbound Calls
9 a.m. – 5 p.m.
Open
95%
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
5 p.m. – 7 p.m.
Open
2%
23
7 p.m. – 9 p.m.
Open
2%
9 p.m. – 9 a.m.
Closed
1%
6.4
Emails Received
Study participants also had an opportunity to contact the Information Center through email. The
email address was only shown on the web survey. An extremely low number of participants opted to
reach the Information Center by email. All email inquiries received a response. Table 6-4 below
indicates the volume of emails per week and Table 6-5 contains the reason for emailing. Overall,
approximately two-thirds of the emails dealt with information on study background/how to
proceed, and PIN problems.
Table 6-4.
Total
Week
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Table 6-5.
Number of emails received per week
Date (Mon – Sun)
3/29–4/2
4/3 – 4/9
4/10 – 4/16
4/17 – 4/23
4/24 – 4/30
5/1 – 5/7
5/8 – 5/14
5/15 – 5/21
5/22 – 5/28
5/29 – 6/4
6/5 – 6/11
6/12 – 6/18
6/19 – 6/25
6/26 – 7/2
7/3 – 7/9
7/10 – 7/17
Number of emails
4
0
0
0
0
11
3
2
0
1
1
0
0
0
1
0
23
Reason for emailing the Information Center
Reason for Email
Information on study background/how to proceed
Respondent is deceased
Sign-in/connection issues with web survey
Request for PIN number
Too Sick/Incapacitated
Already completed survey
Total
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
Number of Emails
12
4
3
2
1
1
23
24
Percent of Emails
52.2%
17.4%
13.0%
8.7%
4.3%
4.3%
100.00%
6.5
Training for Information Center Representatives
Prior to data collection, Information Center representatives received training on the survey
instruments, project protocols, website operational processes, and the Survey Management System.
The purpose of the training was to prepare representatives for calls and emails from Veterans. In
order to respond to the expected volume, a total of ten Information Center representatives were
invited to train on the study. The Information Center Representatives were highly experienced as
eight of the ten Information Center representatives were previously trained and worked on the study
in 2015 or 2016. This familiarity with the study allowed Westat to customize training to
accommodate the new and seasoned representatives.
Before attending the classroom training, trainees were required to complete a self-paced online
training session. The self-paced training consisted of two hours of pre-classroom training. Modules
covered during the pre-classroom training included the following.
Study overview and background;
Frequently asked questions;
Review of help screens in the web survey;
How to properly code and summarize the contact with Veterans; and
Interviewer training/sensitivity/distressed caller protocol.
The training session was held on March 21, 2017. The first two hours of classroom training
consisted of lecture and interactive training. The final two hours of training included role play
scenarios in which Information Center training representatives were paired together to provide
hands-on practical experience in responding to anticipated questions from Veterans. Trainees were
supplied with a Survey of Enrollees Training Manual containing copies of all participant materials,
FAQs, copies of the survey, and the distress protocols. Appendix E contains copies of training
materials. Topics covered during the classroom training included the following.
An introduction of the study, the survey background and purpose;
Training on how to use the SMS;
Discussing issue and result codes to assign to records;
Protocols for trouble-shooting the Veteran’s inquiry, connectivity issues, and navigation
through the instrument; and
How to properly document each contact.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
25
Prior to interacting with Veterans, Information Center representatives were monitored and certified
by an Information Center supervisor, to ensure they were thoroughly knowledgeable and capable of
performing all Information Center functions. Once certified, the representative could look up and
verify the caller in the SMS by name, phone number, address, or PIN number. Once the call was
complete, the representative would summarize the purpose of the call and categorize the nature of
the call. This provided a historic record of all calls answered by an Information Center
representative.
6.6
Distressed Caller Protocol
Information Center staff members were trained to follow a distressed caller protocol in the event
that a Veteran became seriously distressed during an interaction with the Information Center. The
reasons for being distressed could vary greatly and dealing with these situations requires good
listening, sensitive judgment, human insight, and preparedness on the Information Center
representative’s part.
The distressed caller protocol used during the fielding of the 2015 and 2016 Survey of Enrollees was
used again in 2017. According to the protocol, the signs of distress determined the level of distress
and the urgency of the situation. Westat noted each circumstance carefully using the criteria noted in
Table 6-6 below. A level 1 situation (mild distress), was documented within the SMS and the
supervisor at the Information Center determined if it should be escalated to a higher level. Incidents
defined as level 2 or 3 situations, (moderate or severe levels of distress), required representatives to
immediately express concern and acknowledge appropriately. Then the representative was required
to contact their supervisor immediately. Supervisors at the Information Center then attempted a
“warm transfer” directly to the Veteran’s Crisis Line at 1-800-273-8255 with the Veteran’s consent.
The Veteran may refuse the transfer. If this occurred, the call was still documented in an adverse
event report and protocol was still followed. See Appendix F for the full distressed caller protocol
that was utilized with level 2 and 3 situations.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
26
Table 6-6.
Levels of distress and indicators
Level of Distress
Level 1:
Mild/Minimal
Level 2:
Moderate/In Need of Referral
Level 3:
Severe/Immediate Help
Signs or Indicators of Distress
Respondent demonstrates any of the following:
Upset about being contacted.
Change in voice tone or volume.
Changes in focus.
Hesitancy to answer questions.
Respondent expresses sadness.
Respondent demonstrates any of the following:
Respondent is too upset to continue.
Use of inappropriate language.
Provides non-relevant answers to questions.
For emotional reasons displays an unwillingness or hesitancy to
continue.
Sobbing, weeping, and/or crying on the telephone.
Displays other obvious signs of agitation.
Expresses feelings of depression about conditions such as: family,
mental and physical health status, and/or lack of services.
Respondent shows any of the following signs:
Openly states the intention to hurt themselves.
Openly states his/her intention to hurt other people.
Openly asks for help.
Suicidal or paranoid thoughts.
The Information Center’s responsibility is to immediately document the Veteran’s exact words and
level of distress in an adverse events report, so that Westat could take immediate action steps
outlined in the Distressed Caller Protocol procedures. Any adverse event reports were sent to VA
for all level 2 and 3 situations within hours of the incident. Adverse events reports included a deidentified summary of the incident, unique ID, VHA identifier, level of distress, and description of
Westat’s actions. During the fielding period, Westat received 3 distressed calls, but only one contact
from a Veteran that was determined to be an adverse event (level 2 or 3) and VA was notified of this
event following the Distressed Caller Protocol procedures.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
27
Processing Multimode Data
7
Data for the Survey of Enrollees was collected using two modes – web and mail. The 46 surveys
completed by telephone were conducted using the web survey for data entry. Westat implemented
the following data preparation steps to prepare the final data file.
Extracted web data, imported into SAS, and set out-of-range values to missing.
Extracted data from scanned questionnaires for pre-determined fields and response
categories using TeleForm software.
Imported the TeleForm data into SAS and applied cleaning logic to forward-code skip
patterns, set out-of-range values to special missing values.
Combined the web and mail data; in cases where multiple surveys from the same
respondent were receipted, Westat retained the survey that was receipted first.
Applied key item rules that determined whether each survey was complete.
Imported the survey metadata into SAS and combined it with the survey data.
Imported the sampling frame data into SAS and combined it with the survey data.
Combined the sample weights and strata with the survey data.
Dropped extraneous variables from the final delivered data set (only complete records
were retained).
Ran frequencies and cross-tabulations to check data values and skip patterns.
Created deliverable files:
1.
Final SAS data set;
2.
SAS formats catalog;
3.
Copies of annotated survey instruments; and
4.
A codebook containing the contents of the SAS data set and frequency tables of
the survey variables.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
28
7.1
Key Survey Items
Key survey items, either individual or matrix items, were identified in consultation with VA.
Table 7-1 lists the 29 survey items identified as key items, along with the associated response values.
It was determined that at least half of these key items (i.e., 15 items) needed to be answered for a
survey to be considered “complete.” The key items represent questions that all respondents receive
regardless of skip patterns or responses.
Table 7-1.
Key survey items and associated valid response values
Item Number
Q1
Q3
Q4
Q10
Q11
Q12
Q14
Q15
Q16
Q18a
Q18b
Q19
Q22
Q24a-i
Q25
Q26
Item Description
Please indicate who is completing this survey. In other words, will
you complete the survey yourself or will you ask someone to assist
you?
Are you enrolled in VA health care?
Are you covered by Medicare?
Are you currently covered by Medicaid (sometimes referred to as
“Medical Assistance”) for any of your health care?
Are you currently covered by TRICARE?
Are you currently covered by any other individual or group health
plan that you, your current or former employer, your spouse or
domestic partner’s employer, your union, or someone else, obtains
for you?
Do you have a long-term care policy that covers nursing home
care, assisted living, or long-term care services in the home?
Exclude any Medicare Supplement Policy.
Which of the following is your primary source of information about
VA health benefits and/or eligibility?
How many different prescription medications did you use in the
last 30 days? Include both VA and non-VA prescriptions. Your best
guess is fine.
From October 2016 through December 2016, how many
outpatient visits or trips did you make to any Non-VA doctor’s
office, hospital, or outpatient clinic?
From October 2016 through December 2016, how many
outpatient visits or trips did you make that were paid for fully or
partially by VA?
Have you used ANY VA health care services on or after
January 1, 2016? Services could either have been at a VA facility
or at a community provider that was paid by the VA.
Do you ever use health care services other than those provided or
paid for by VA?
Which of the following factors do you consider when selecting a
health care provider:
Please complete the following statement: I use VA services to
meet.
Below is a list of possible ways you could use VA for your health
care in the future. Please read them all, and then choose the one
that best describes the primary way you plan to use VA health care
in the future.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
29
Valid Non-Missing
Responses
1, 0
1, 0, -7
1, 0
1, 0
1, 0
1, 0
1, 0
1, 2, 3, 4, 5, 6, 7, 8
0-50
0-366
0-366
1, 0
1, 2, 0
1, 0
1, 2, 3, 4, -6
1, 2, 3, 4, 5, 6, 7
Table 7-1.
Key survey items and associated valid response values (continued)
Item Number
Q27
Item Description
Valid Non-Missing
Responses
1, 2, 3 ,4, 5
Q29
Compared with other people your age, would you say your
health is . . .
In a typical week, how much assistance from family, friends,
neighbors, or others do you need for the following daily activities or
situations? Please mark any needs you have for assistance,
whether or not you are currently receiving assistance for them.
Have you smoked at least 100 cigarettes in your entire life?
Q33
Do you use the Internet, at least occasionally?
1, 0
Q38
Do you send or receive text messages on your cell phone?
1, 0, -5
Q39
Think about any computer or mobile device available to you at
home or elsewhere that has access to the Internet. How willing
would you be to do the following on at least one of those
computers or mobile devices?
Are you aware of the My HealtheVet Web site?
2, 3, 4, -5
1, 0
Q44
Did you serve on active duty in the U.S. Armed Forces during the
following time frames?
Did you ever serve in a combat or war zone?
Q45
Which of the following best describes your current marital status?
1, 2, 3, 4, 5, 6
Q46
Not including yourself, how many dependents do you currently
have?
How would you best characterize your employment status?
0-97
Please tell us how you feel about the following statement: “I trust
VA to fulfill our country’s commitment to Veterans”
1, 2, 3, 4, 5
Q28a-q
Q40
Q43a-i
Q48
Q52
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
30
1, 2, 3, -5
1, 0
1, 0
1, 0
1, 2, 3, 4
Survey Statistics
8
The survey research industry utilizes standardized reporting of survey statistics to allow for
comparability across surveys and to assess individual survey performance on data quality. The U.S.
Federal Government, particularly the Office of Management and Budget (OMB), encourages the use
of the American Association of Public Opinion Research (AAPOR) response rate definitions. For
the 2017 Survey of Enrollees, Westat used the AAPOR 1 response rate to compute response rates.
It is also useful to examine return rates and cooperation rates to understand fully the responsiveness
of Veteran enrollees to the 2017 survey.
A summary of the 2017 calculated rates is as follows.
Response Rate = 33.4%
Return Rate = 35.2%
Cooperation Rate = 39.3%
Response Rate. This is the number of completed interviews divided by the total number of eligible
respondents. Eligible respondents are defined as Veterans in the current enrollment file who are
presumed to be alive. During the course of the data collection, it was learned that 662 sample
members were deceased prior to the 2017 survey data collection period and the number of deceased
sample members was subtracted from the number of eligible sample members. Overall, the 2017
survey response rate was 33.4 percent, calculated as follows.
Response rate = number of completes/(number of total cases released – number of deceased) * 100
= [43,654/(131,521 – 662)] * 100
= [43,654/130,859] * 100
= 33.4%
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
31
Return Rate. A study decision was made to exclude sample members who were identified as moved
or who had an invalid address, according to the National Change of Address (NCOA) database. The
return rate calculation excludes these sample members and deceased sample members from the
denominator (6,862 respondents not fielded). With these sample members excluded, the overall
return rate was 35.2 percent, calculated as follows.
Return rate = number of completes/(number of total cases released – number of deceased –
number not fielded) * 100
= [43,654/(131,521 – 662 – 6,862)] * 100
= [43,654/123,997] * 100
= 35.2%
Cooperation Rate. In addition to deceased or not fielded sample members, another group of
sample members was mailed the study materials, but the materials were returned as Postal
Non- Deliverable (PND). While these individuals were eligible for the survey, they did not receive
their invitation to participate in the survey and thus, did not have an opportunity to complete a
survey. The cooperation rate is the number of completes divided by the number of cases contacted.
There were 12,961 enrollees identified as PND. When the PND’s were excluded, the overall
cooperation rate was 39.3 percent, calculated as follows.
Cooperation rate = number of completes/(number of total cases released – number of deceasednumber not fielded – number did not receive invitation) * 100
= [43,654/(131,521 – 662 – 6,862 – 12,961)] * 100
= [43,654/111,036] * 100
= 39.3%
See Appendix G is for these calculated rates across all strata, Appendix H for rates across all
markets, and Appendix I for rates for the priority groups within each VISN.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
32
Enhancements and Implementation
Goals for Future Surveys
9
All goals were met for the 2017 Survey of Enrollees for schedule and participation. Each year of
data collection offers an opportunity to fine-tune procedures and protocols as well as “lessons
learned” that can be applied to enhance future survey rounds.
9.1
Enhancement: Reduce Sampled Veteran’s Burden
In 2015 and to a lesser extent 2016, the Information Center experienced a large number of calls
following the initial web survey invitation mailing from individuals who did not have access to a
computer or the Internet. These individuals felt technology was a barrier to participation. As a result,
Westat adapted survey communications by adding specific language in the communications that
informed the sampled Veteran they would receive a paper survey in the mail within a few weeks.
The IVR phone answering system was also modified to include new prompts explaining a survey
would be mailed within a few weeks. This change worked as expected. In 2015, 4,091 potential
respondents spoke with an Information Center representative to indicate they had “no computer or
Internet access”. The number of calls about “no computer or Internet access” dropped to 562 calls
in 2016 and dropped further to 425 calls in 2017.
9.2
Goal: Meeting Survey Quotas by Stratum While Minimizing
Overage
The survey protocol set goals for an overall number of completed surveys (42,000), as well as a
minimum number of surveys completes for each stratum. The number of completed surveys
received in 2017 was 43,654 (a decrease of 2,917 completed surveys from 2016 and 7,019 fewer than
from the 2015 survey), but still 1,654 more than required. The “overage” in 2017 can be partially
attributed to changes in the composition in VISNs and markets between 2016 and 2017 made
historical response rate comparisons more challenging.
Each year of data collection provides more information on response rates by strata and helps to
refine projections of the estimated number of completed surveys for future rounds. The more
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
33
information available, the better our estimate should become for the expected number of completes.
While the study is designed to exceed the minimum goal of 42,000 completes, the goal for next year
is to reduce further the overage by using 2015 – 2017 results to further refine distribution of the
sample.
9.3
Goal: Mail Replacement Survey Requests
A small number of sampled Veterans contacted the Information Center to request a second mail
survey be resent to them. These requests were typically the result of the mail survey being lost or
damaged, and in some cases, the enrollee was no longer at the mailing address it was originally sent.
These individuals wanted to participate in the study, but felt unable to do so. The current process to
send a second survey was not efficient enough to print a custom variable data letter and survey
booklet to allow the enrollee to receive it in a timely manner. For future rounds of the Survey of
Enrollees, it would be desirable to develop a protocol for replacement surveys to enrollees whom
request them.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
34
Quality Assurance Measures
10
Throughout the data collection process, Westat implemented a stringent set of quality control and
quality assurance (QC/QA) measures to the entire survey development and implementation process,
as well as preparation and processing of the survey data. Our QC/QA measures for the Survey of
Enrollees were applied during the following survey activities.
Development and testing of the web survey;
Development and testing of the mail survey;
Development of a Survey Management System (SMS); and
Implementation of mailing procedures.
10.1
Development and Testing of the Web Survey
Westat followed a structured, systematic, and iterative process for developing and testing the web
survey instrument. This included the following.
Testing of the survey to verify all hard and soft edits, validation rules, and skip logics.
Integration testing across all survey modules to ensure seamless integration, and that all
questions, item fills, and skip logics worked as expected.
Verifying that all question variables were appropriately named (according to client
specifications), and that all survey response values were being captured for each survey
item.
Checking that ranges and skip patterns were enforced within the survey instrument.
Checking that all required survey items were identified and worked as expected.
Checking that all validation error messages were appropriate.
Spell checking of all survey questions, survey response options, and all instructions.
Verifying that the user could navigate through the survey and break off and return to
the survey without loss of response data.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
35
Ensuring the survey worked properly on common operating systems, platforms,
resolutions, mobile devices, and smart phones.
Verifying the web survey met the 2017 refresh Section 508 compliance requirements.
Ensuring that data could be extracted from the survey database into SAS datasets.
Verifying that the SAS programs produced output datasets containing the survey
variables and response data as expected, and that the data were appropriately labeled
(both variable labels and value labels).
Rigorous systems testing were also performed to ensure that all VA enrollees who accessed the web
survey could do so without difficulty. This process included the following.
Specifications testing was performed to ensure that the instruments worked as planned
on various browsers (e.g., checking that the content, font, colors and graphics were
correct; the overall design and layout were appealing, appropriate, and consistent;
navigation, help, and print features worked well; and security measures and
confidentiality assurances were in place).
Web browser compatibility was assessed in Westat’s web lab to ensure that the web
interface worked effectively with the most common web browsers (e.g., Internet
Explorer, Firefox, Mozilla, Safari, Android, Chrome, and Opera) and browser versions
(e.g., the two most recent versions of Internet Explorer). We also tested to make sure
the survey was accessible by most device platforms (e.g., desktop PCs, mobile devices,
and tablets) and that the look and feel of the survey instrument remained stable across a
variety of monitor resolutions. Our web-based survey delivery system did not require
additional plug-ins for survey respondents or other software installation for successful
completion of the surveys. In addition, the web survey was 508-compliant according to
the 2017 refresh of compliance standards.
Beta testing from the point-of-view of users, to assess ease in accessing the web site and
logging in, understanding and following the introduction and instructions,
providing/changing responses, navigating through the survey, using features and tools,
exiting/returning to/submitting the survey, and understanding survey items.
Post-programming and evaluation testing to make sure all data were captured as
specified (e.g., yes=1, no=0) in the survey results database.
Management system testing to check that stored data were available to generate status
reports (e.g., response rate reports, VISN/market reports) and check respondent
disposition codes. Westat verified that the sample had been properly loaded. We also
conducted system tests to check that only non-respondents received follow-up
nonresponse reminders as specified. Stress testing to ensure that multiple simultaneous
users could access the survey site.
User acceptance testing after the survey was initially programmed; Westat staff tested
the web survey. Final testing was conducted after all modifications were made. Westat
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
36
staff retested web survey functionality to confirm links, logins, and navigation worked as
intended. We also reconfirmed that data capture of survey responses followed data
specifications.
10.2
Development and Testing of the Mail Survey
Westat used its in-house electronic intelligent data capture system (Teleform) to process the mail
surveys. Variable names, labels, and ranges and skip logics are specified in the scanning program.
The program is developed to follow questionnaire specifications and enforce specified skip patterns
or range checks and only record responses adhering to the intended skip patterns reflected in the
survey questions. Prior to the first survey mailing, Westat tested the instruments and validated that
responses from the survey were accurately captured.
Westat set up a central project operation to process, verify, and clean the incoming data in a secure
FISMA-High environment. We used experienced staff trained in all aspects of processing, including
receipt of the data, review, scanning and data verification, as well as quality control. When surveys
were received, processing staff reviewed each form as it was prepared for scanning. This included
the following steps:
Developed five versions of the mail survey necessary to support the 2017
methodological experiment.
Recording the receipt status of each survey, including complete, partial complete,
refusals, deceased, postal non-deliverables (PNDs), and other types of nonresponse
(when information was available on the survey).
Determining if the form was scannable. If it was not scannable for any reason (e.g., the
form was damaged during shipping), special handling of the form was arranged.
Documenting responses that were not able to be data captured.
Once scanned, 100 percent of the surveys were manually verified for any responses
where the scanner was unable to verify the survey response. In such instances, a
member of the data processing staff visually inspected an image of the questionnaire to
ascertain the intended response.
Approximately 10 percent of the surveys were reviewed during a standard quality
control process. During this step, data processing staff reviewed all items on the paper
form against the survey database.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
37
Westat performed checks to ensure that any duplicates received (i.e., multiple completed
questionnaires from the same individual, possibly from different survey modes) were identified in
the processing stream.
10.3
Survey Management System
Westat used a Survey Management System (SMS) to manage the flow of cases throughout the
sampling and data collection process.
The SMS is an important element of our quality control process. The following SMS activities
supported quality control:
Loading the sample file with contact information (i.e., name, address, phone number) of
sampled Veterans;
Assigning unique Study IDs and PIN Numbers (for the web survey) to each person
sampled;
Tracking the status and assigning the next activity for non-responders;
Sending files to the print vendor for the next mailing;
Receipting questionnaires returned through the web or mail;
Providing an interface with the Information Center to document contacts and
responses;
Recording the status of IVR reminder calls;
Allowing updates of contact information from undeliverable mail and new telephone
numbers;
Exporting VHA IDs and disposition codes requested by VA; and
Providing reports to monitor survey progress.
The SMS ensured that data collection was completed accurately and efficiently, with real-time status
information and flexible query capabilities to support both standard and ad-hoc reporting. Reports
monitored the progress of each wave, as well as the progress of the quotas required for each
sampling stratum.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
38
To ensure security, access to specific functions or data was controlled in the SMS so that each user
was only permitted to see aspects of the SMS appropriate for their role in the study.
10.4
Quality Control: Mail Procedures
When conducting surveys with personal identifying information (PII) or other confidential
information, Westat’s standard practice includes stringent quality control and internal review for all
mailings. During the 2017 survey, over 353,000 pieces of mail (invitation letters, postcard reminders,
and mail surveys) were mailed to survey respondents.
Westat utilized technologies that improved efficiency and helped minimize overall project costs,
including printing the surveys with machine-readable barcodes and unique IDs that identified the
survey piece and assisted in preparing packages and accurately receipting returned surveys.
We used both automated and manual processes to ensure that the individual components of a given
mailing were correctly assembled and labeled, that the number of survey packets for the mailing
corresponded with the number expected, and that all materials expected in the package were
included. The following steps were taken to ensure that the mailings were sent properly.
To ensure that addresses were accurate, Westat used the U.S. Postal Service National
Change of Address (USPS NCOA) database.
In Wave 1, each respondent was randomly assigned to a control or four experimental
groups that determined the reliance questions (Questions 18) they would receive.
Westat notified the printer which mail survey participants would receive and prior to
printing, confirmation was received that the actual number of items printed matched the
expected number to be printed for each group.
Using the unique identification number (ID) assigned to each sample member from our
survey management system, we had the ability to track every survey, including the date
when mailed to each respondent.
Correspondence and surveys were generated in the same production runs to ensure that
all materials had the appropriate ID codes and were packaged appropriately.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
39
Once the items were printed and assembled, staff ensured that the number of pieces
packaged were consistent with expectations, and systematically performed quality
control checks, including:
–
Verified that all materials were properly included in the mailing;
–
Verified that the barcode and the respondent’s name were properly in sync;
–
That the print quality met visual standards;
–
That the questionnaires were printed properly and were capable of being scanned
by Teleform;
–
That respondent names were properly displayed on the first question; and
–
That the number of pieces with postage matched the number anticipated.
If any errors were discovered in the process, the percentage of materials that were
reviewed were increased during that particular “run” to ensure that all materials were
correctly assembled.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
40
Weighting
11
Analysis weights were calculated and attached to the final data set. Calculation of analysis weights
consisted of the following steps: (1) calculation of base weights, (2) adjustment for nonresponse,
and (3) poststratification adjustment.
11.1
Calculation of Base Weights
The base weight for a sampled Veteran is the reciprocal of the probability the Veteran was selected
to participate in the 2017 survey. Because of the large size of the enrollee population, all the
sampling probabilities were less than 1. Hence, base weights are greater than 1. This indicates that
sampled Veterans are representing themselves, plus other Veterans who were not sampled. For
example, if a sampled Veteran’s base weight is equal to 100, this indicates that the Veteran is
representing himself or herself, plus 99 other Veterans who were not sampled.
The sampling procedures for the 2017 survey were complex. VA selected a first-phase stratified
sample, and then using the same strata, Westat selected two second-phase samples, referred to as the
Wave 1 and Wave 2 samples. For purposes of computing base weights, however, the Veterans in the
Wave 1 or Wave 2 samples can be treated as a single sample selected from the 2017 enrollee
population. In previous cycles, the base weight for each Veteran belonging to a particular sampling
stratum was equal to the stratum’s population size divided by the sum of the Wave 1 and Wave 2
sample sizes for the particular stratum.
However, beginning with the 2017 study, veterans sampled for the 2016 Survey of Enrollees were
ineligible for the 2017 Survey of Enrollees and excluded from the sampling frame. To account for
the excluded cases, Veterans who were eligible for selection in the 2016 Survey of Enrollees but not
sampled were over sampled. Aggregate counts of Veterans in three subsampling groups were
provided by VA prior to excluding any Veterans: (1) those sampled for other VA studies within the
same year (excluded from the sampling frame), (2) those eligible for other VA studies but not
sampled, and (3) those not eligible for other VA studies, i.e., Veterans new to VA. Within each
sampling stratum, the base weight for each Veteran in Group 2 was calculated to account for the
non-sampled Veterans in Group 1 within the same sampling stratum. That is, the base weight for
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
41
each Group 2 Veteran belonging to a particular sampling stratum was equal to the within stratum
population size for Groups 1 and 2 divided by the sum of the Wave 1 and Wave 2 Group 2 sample
sizes for the particular stratum. The sampled Veterans in Group 3, who were not eligible for
sampling in prior cycles, represented only Veterans in Group 3. Therefore, the base weight for each
Group 3 Veteran belonging to a particular sampling stratum was equal to the within stratum
population size for Group 3 divided by the sum of the Wave 1 and Wave 2 Group 3 sample sizes
for the particular stratum.
11.2
Adjustment for Nonresponse
The base weights for the responding Veterans were adjusted for nonresponse—that is, their base
weights were multiplied by a factor—referred to as a nonresponse adjustment factor—that was greater
than 1. This indicated that responding Veterans not only represented themselves and Veterans who
were not sampled, but they also represented sampled Veterans who did not respond. As in previous
years, the nonresponse adjustment factors were based on estimates obtained from a logistic regression
model for the propensity that a sampled Veteran was a respondent given their sampling stratum,
demographic data on the sampling frame, experimental-treatment assignment, and health-care
utilization information. Creating nonresponse adjustment cells in this manner reduces potential bias
to the extent that non-respondents and respondents with similar response probabilities are also
similar with respect to the survey statistics of interest.
The following variables were considered as predictors for developing the propensity model:
The 2017 stratification variables (VISN, priority status, market, enrollee type);
Stratification variables used in prior years (OEF/OIF/OND, gender, Hispanic);
Seven administrative health measures;
Demographic variables (age, urban/rural address); and
Experimental treatment assignment.
All sampled records were included in the model, weighted by the base weight. Stepwise logistic
regression was utilized to determine the model with the best fit. The outcome of the model is the
propensity score, the estimated probability that a sampled enrollee is in the final sample of
respondents given his or her characteristics.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
42
All but two variables, VISN and the use of VHA pharmacy services, were included in the final
model. The variables entered the model in the following order:
Age category;
Outpatient treatment: non-mental health and non-substance abuse;
Market;
Enrollee type;
Collapsed priority group;
Outpatient treatment: mental health or substance abuse;
Inpatient treatment: non-mental health and non-substance abuse;
Received long-term care benefits: institutional;
Urban vs. rural address;
Inpatient treatment: mental health and substance abuse;
Gender;
Experimental treatment assignment;
Received long-term care benefits: non-institutional;
Hispanic status; and
OEF/OIF/OND status.
All of the coefficients for the variables in the model are highly significantly different from zero
(p<0.001) except for received Hispanic status and OEF/OIF/OND which were marginally
significant (p=0.003 and p=0.0489, respectively).
After estimating each sampled enrollee’s probability of completing an interview based on the
predictor variables, respondents and non-respondents were grouped into quintiles based on their
propensity score. The sample was divided such that each cell had an equal portion of the population.
Within each quintile, the respondents’ base weights were ratio-adjusted to account for nonrespondents. The first quintile represents the enrollees with the lowest response propensity scores;
this means that these sampled enrollees are less likely to be respondents; thus, they receive the
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
43
largest adjustments. The last quintile represents the sampled enrollees with the highest response
propensity scores; this means that these sampled enrollees are more likely to be respondents; thus,
they receive the smallest adjustments. Though there were very few ineligible sampled
persons—defined as deceased, incapacitated, claimed not to be a Veteran, known ineligible persons
in the sample were adjusted with the same adjustment factor as respondents to account for
unknown ineligible persons in the sample. Because the adjustment factor for the respondents in the
quintile with the lowest response propensity was so large, it was collapsed with the neighboring
adjustment cell. Table 11-1 provides the details on the nonresponse adjustment cells.
Table 11-1.
Response
Propensity
Percentile
0-48
48-61
61-73
73-87
87-100
11.3
Nonresponse adjustment cell definitions, sizes, and adjustment factors
# of
Respondents
# of
Nonrespondents
14,922
12,867
9,616
5,382
1,568
10,519
15,237
18,918
21,410
21,082
Weighted
Count of
Respondents
998,673
790,728
570,462
340,645
117,349
Weighted
Count of
Nonrespondents
711,170
921,164
1,139,171
1,377,925
1,604,161
Adjustment
Factor Before
Collapsing
1.7
2.1
3.0
5.0
14.6
Adjustment
Factor After
Collapsing
1.7
2.1
3.0
7.5
7.5
Poststratification Adjustment
As was done in previous years, a poststratification adjustment was included as part of the weighting
to promote comparability with prior years. The poststratification of the 2017 weights was performed
along six dimensions. Five of the dimensions were the same as those used to post-stratify weights
prior to 2015, which included enrollee age. Enrollee age was categorized into seven levels: under 35,
35-44, 45-54, 55-64, 65-74, 75-84, and 85+. In 2015 and beyond, we added individual markets as a
sixth poststratification dimension because markets were used to stratify the sample.
The following were the six dimensions used to post-stratify the 2017 weights:
Age x gender (7 x 2 = 14 levels);
Hispanic status (2 levels);
Priority group x VISN (8 x 18 = 144 levels);
OEF/OIF/OND status (2 levels);
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
44
Pre/post-enrollee status (2 levels); and
Markets (96 levels).
The poststratification adjustment was implemented via a raking, or iterative proportional fitting,
algorithm. During each iteration, the non-response-adjusted weight was ratio-adjusted to match
population totals along each of the above poststratification dimensions in turn. This iterative process
continues until the weighted totals match population totals along all dimensions within a specified
tolerance (in this case, by less than 1.00). Convergence was achieved after 12 iterations, indicating a
stable adjustment. The poststratified weight was delivered with the collected data and should be used
as the analytic weight when generating population estimates.
2017 Survey of Veteran Enrollees’ Health and
Use of Health Care
45
| File Type | application/pdf |
| File Title | 2017 VA Survey of Veteran Enrollees Health and Use of Health Care - Survey Methodology Report |
| Subject | Survey of Veteran Enrollees’ Health and Use of Health Care, Survey Methodology Report, Survey of Veteran Enrollees’ Health and U |
| Author | Westat |
| File Modified | 2018-07-03 |
| File Created | 2017-10-06 |