VBA Veteran Readiness and Employment (Chapter 31) Surveys

Request for Approval under the “Generic Clearance for Improving
Customer Experience (OMB Circular A-11, Section 280
Implementation)” (OMB Control Number: 2900-0876)
TITLE OF INFORMATION COLLECTION: VBA Veteran Readiness and
Employment (Chapter 31) Surveys
PURPOSE
The Veterans Benefits Administration (VBA) is responsible for providing and/or
managing a wide array of services to Veterans and their family members. One such
responsibility is to ensure Veterans that have a service-connected disability that limits
their ability to work or prevents them from working can explore employment options and
address training or educational needs.
The Veteran Experience Office (VEO) is in partnership with VBA on many
surveys and has recently partnered with the Veteran Readiness and Employment office
(VR&E and particularly as Chapter 31 (CH 31)). VR&E assists Veterans with a serviceconnected disability explore employment options and address educational or training
needs. There are five VR&E tracks which are: employment track (help Veterans return to
a former job and support their employer in meeting their needs), rapid access to
employment track (helps a Veteran find a job or career that uses their existing skill set),
self-employment track (helps Veterans start their own business), employment through
long term services track (helps Veterans get the education or training they need to find
work in a different field), and independent living track (if a Veteran cannot return to work
right away, a Veteran may qualify for services that will allow them to live as
independently as possible). During this effort, VEO partnered with VBA to measure the
satisfaction of beneficiaries of the VR&E CH 31 program at different milestones:
•
•
•
•
1)
2)
3)

Learning about and Applying for VR&E CH 31 Survey
Participating in VR&E CH 31 Survey
Completing VR&E CH 31 Survey
Discontinuing from VR&E CH 31 Survey
The goal of service level measurements is three-fold:
To collect continuous customer experience data from VR&E CH 31 customers
To help field staff and the national office identify areas for improvement
To better understand the reasons VR&E CH 31 customers provide positive or
negative feedback

The survey questionnaire is brief and contains general Likert-scale (a scale of 1-5
from Strongly Disagree to Strongly Agree) questions to assess customer satisfaction as
well as questions assessing the knowledge, speed, and manner of the interaction. These
questions have been mapped to the OMB A-11 Customer Experience drivers. After the
survey has been distributed, recipients have two weeks to complete the survey. Invitees
will receive a reminder email after one week.
1

The purpose of this document is to define VA’s sampling methodology for
selecting potential survey respondents for this study. The population for VR&E varies
depending on the milestone the surveys are targeting. The sample size for the Learning
about and Applying for a VR&E survey and Participating in VR&E survey was
determined so that the reliability of monthly survey estimates is 4.0% and 4.5% margin of
error, respectively, at a 95% confidence level. The remaining surveys, Completing VR&E
and Discontinuing VR&E were determined so that the reliability of monthly survey
estimates is 5.0% and 5.5% margin of error at a 95% confidence level, respectively. This
report describes the methodology used to conduct the VR&E CH 31 survey. Information
about quality assurance protocols, as well as limitations of the survey methodology, is also
included in this report.
DESCRIPTION OF RESPONDENTS:
VEO has been commissioned by VBA to measure the satisfaction and experience of
customers with the Veteran Readiness and Employment (VR&E), Chapter 31 (CH 31).
VEO proposes to conduct a brief survey on customers who had experienced several
milestones in the program. Sampled customers will be contacted through an invitation
email. A link will be enclosed so the survey may be completed using an online interface,
with customized customer information. The survey itself will consist of a handful of
questions revolving around a human-centered design, focusing on such elements as trust,
emotion, effective, and ease with the care they received.
TYPE OF COLLECTION: (Check one)
[ ] Customer Comment Card/Complaint Form
[ ] Usability Testing (e.g., Website or Software
[ ] Focus Group

[X] Customer Satisfaction Survey
[ ] Small Discussion Group
[ ] Other: ______________________

CERTIFICATION:
I certify the following to be true:
1. The collection is voluntary.
2. The collection is low-burden for respondents and low-cost for the Federal
Government.
3. The collection is non-controversial and does not raise issues of concern to other
federal agencies.
4. Personally identifiable information (PII) is collected only to the extent necessary and
is not retained.
5. Information gathered is intended to be used for general service improvement and
program management purposes.
6. The collection is targeted to the solicitation of opinions from respondents who have
experience with the program or may have experience with the program in the future.
7. All or a subset of information may be released as part of A-11, Section 280
requirements on performance.gov. Additionally, summaries of the data may be
released to the public in communications to Congress, the media and other releases
disseminated by VEO, consistent with the Information Quality Act.

2

•

Name: Juan Jackson, Enterprise Measurement Project Manager, Veterans
Experience Office, VA, (202) 603-4374

To assist review, please provide answers to the following question:
Personally Identifiable Information:
1. Will this survey use individualized links, through which VA can identify particular
respondents even if they do not provide their name or other personally identifiable
information on the survey? [X] Yes [ ] No
2. Is personally identifiable information (PII) collected? [ ] Yes [X] No
3. If Yes, will any information that is collected be included in records that are subject to
the Privacy Act of 1974? [ ] Yes [ ] No [N/A]
4. If Yes, has an up-to-date System of Records Notice (SORN) been published? [ ] Yes
[ ] No [N/A]
Gifts or Payments:
Is an incentive (e.g., money or reimbursement of expenses, token of appreciation)
provided to participants? [ ] Yes [ X] No
BURDEN HOURS
Category of
Respondent
Learning about &
Applying for VR&E
Participating in VR&E
Completing VR&E
Discontinuing from
VR&E
Totals

No. of
Respondents
per month
506
203
158
130
954

No. of
Participation
Respondents Time
Per year
6,072
5 minutes

Burden

2,436
1,896
1,560

5 minutes
5 minutes
5 minutes

203 hours
158 hours
130 hours

11,964

5 minutes

997 hours

506 hours

*Q4
Please answer the following questions.
1. Are you conducting a focus group, a survey that does not employ random
sampling, user testing or any data collection method that does not employ
statistical methods?
Yes _ __
No _X _

3

If Yes, please answer questions 1a-1c, 2 and 3.
If No, please answer or attach supporting documentation that answers questions 2-8.
a. Please provide a description of how you plan to identify your potential group of
respondents and how you will select them.
•

VEO proposes to conduct a brief survey on customers who have experienced a
number of milestones once the program is completed. Sampled customers will
be contacted through an invitation email. A link will be enclosed so the survey
may be completed using an online interface, with customized customer
information. The survey itself will consist of a handful of questions revolving
around a human-centered design, focusing on such elements as trust, emotion,
effective, and ease with the care they received.

b. How will you collect the information? (Check all that apply)
[ ] Web-based or other forms of Social Media
[ ] Telephone
[ ] In-person
[ ] Mail
[X] Other- E-mail-based surveys
c. Will interviewers or facilitators be used? [ ] Yes [ X ] No
2. Please provide an estimated annual cost to the Federal government to conduct this data
collection: __$13,000______
3. Please make sure that all instruments, instructions, and scripts are submitted with the
request. This includes questionnaires, interviewer manuals (if using interviewers or
facilitators), all response options for questions that require respondents to select a
response from a group of options, invitations given to potential respondents,
instructions for completing the data collection or additional follow-up requests for the
data collection.
•

Done

4. Describe (including a numerical estimate) the potential respondent universe and any
sampling or other respondent selection methods to be used. Data on the number of
entities (e.g., establishments, State and local government units, households, or
persons) in the universe covered by the collection and in the corresponding sample are
to be provided in tabular form for the universe as a whole and for each of the strata in
the proposed sample. Indicate expected response rates for the collection as a whole. If
the collection had been conducted previously, include the actual response rate
achieved during the last collection.

4

•

Please see Statistical Sample Plan in the Appendix.

5. Describe the procedures for the collection of information, including:
a. Statistical methodology for stratification and sample selection.
b. Estimation procedure.
c. Degree of accuracy needed for the purpose described in the justification.
d. Unusual problems requiring specialized sampling procedures.
e. Any use of periodic (less frequent than annual) data collection cycles to reduce
burden.
•

Please see Statistical Sample Plan in the Appendix.

6. Describe methods to maximize response rates and to deal with issues of nonresponse.
The accuracy and reliability of information collected must be shown to be adequate for
intended uses. For collections based on sampling, a special justification must be
provided for any collection that will not yield "reliable" data that can be generalized to
the universe studied.
•

Please see Statistical Sample Plan in the Appendix.

7. Describe any tests of procedures or methods to be undertaken. Testing is encouraged
as an effective means of refining collections of information to minimize burden and
improve utility. Tests must be approved if they call for answers to identical questions
from 10 or more respondents. A proposed test or set of tests may be submitted for
approval separately or in combination with the main collection of information.
•

Please see Statistical Sample Plan in the Appendix.

8. Provide the name and telephone number of individuals consulted on statistical aspects
of the design and the name of the agency unit, contractors, grantees, or other person(s)
who will collect or analyze the information for the agency.
•

Statistical Aspects:
o Melissa Mitchell, Survey Statistician, Veterans Experience Office, VA.
(202) 437-6730

•

Evan Albert, Dir. of Measurement and Data Analytics, Veterans Experience
Office, VA, (202) 875-9478
Juan Jackson, Enterprise Measurement Project Manager, Veterans Experience
Office, VA, (202) 603-474
Robyn Trulock, Acting Supervisor for Oversight, Veterans Readiness &
Employment Service, Veterans Benefits Office, VA (202) 461-9137

•
•

5

Veteran Readiness and Employment
(Chapter 31) Survey
Sampling Methodology Report
Prepared by
Veteran Experience Office
Version 2 January 2023

6

Contents

Executive Summary ........................................................................................................................... 8
Part I – Introduction ......................................................................................................................... 10
A. Background ............................................................................................................................. 10
B. Basic Definitions ..................................................................................................................... 11
C. Application to Veterans Affairs............................................................................................... 11
Part II – Methodology ...................................................................................................................... 11
A. Target Population and Frame ................................................................................................. 11
B. Sample Size Determination .................................................................................................. 12
C. Stratification ......................................................................................................................... 14
D. Data Collection Methods ..................................................................................................... 14
E. Reporting.............................................................................................................................. 14
F. Quality Control .................................................................................................................... 15
G. Sample Weighting, Coverage Bias, and Non-Response Bias .............................................. 15
H. Quarantine Rules .................................................................................................................. 16
Part III – Assumptions and Limitations ........................................................................................... 17
A. Coverage Bias ......................................................................................................................... 17
References .................................................................................................................................... 17

7

Executive Summary
The Veterans Benefits Administration (VBA) is responsible for providing and/or
managing a wide array of services to Veterans and their family members. One such
responsibility is to ensure Veterans that have a service-connected disability that limits
their ability to work or prevents them from working can explore employment options and
address training or educational needs.
The Veteran Experience Office (VEO) is in partnership with VBA on many
surveys and has recently partnered with the Veteran Readiness and Employment office
(VR&E and particularly as Chapter 31 (CH 31)). VR&E assists Veterans with a serviceconnected disability explore employment options and address educational or training
needs. There are five VR&E tracks which are: employment track (help Veterans return to
a former job and support their employer in meeting their needs), rapid access to
employment track (helps a Veteran find a job or career that uses their existing skill set),
self-employment track (helps Veterans start their own business), employment through
long term services track (helps Veterans get the education or training they need to find
work in a different field), and independent living track (if a Veteran cannot return to work
right away, a Veteran may qualify for services that will allow them to live as
independently as possible). During this effort, VEO partnered with VBA to measure the
satisfaction of beneficiaries of the VR&E CH 31 program at different milestones:
• Learning about and Applying for VR&E CH 31 Survey
• Participating in VR&E CH 31 Survey
• Completing VR&E CH 31 Survey
• Discontinuing from VR&E CH 31 Survey
4)
5)
6)

The goal of service level measurements is three-fold:
To collect continuous customer experience data from VR&E CH 31 customers
To help field staff and the national office identify areas for improvement
To better understand the reasons VR&E CH 31 customers provide positive or
negative feedback

The survey questionnaire is brief and contains general Likert-scale (a scale of 1-5
from Strongly Disagree to Strongly Agree) questions to assess customer satisfaction as
well as questions assessing the knowledge, speed, and manner of the interaction. These
questions have been mapped to the OMB A-11 Customer Experience drivers. After the
survey has been distributed, recipients have two weeks to complete the survey. Invitees
will receive a reminder email after one week.
The purpose of this document is to define VA’s sampling methodology for
selecting potential survey respondents for this study. The population for VR&E varies
depending on the milestone the surveys are targeting. The sample size for the Learning
about and Applying for a VR&E survey and Participating in VR&E survey was
determined so that the reliability of monthly survey estimates is 4.0% and 4.5% margin of
error, respectively, at a 95% confidence level. The remaining surveys, Completing VR&E
and Discontinuing VR&E were determined so that the reliability of monthly survey
estimates is 5.0% and 5.5% margin of error at a 95% confidence level, respectively. This
report describes the methodology used to conduct the VR&E CH 31 survey. Information

8

about quality assurance protocols, as well as limitations of the survey methodology, is also
included in this report.

9

Part I – Introduction
A. Background
The Enterprise Measurement and Design team (EMD) within the Veterans
Experience Office (VEO) is tasked with conducting transactional surveys of the customer
population to measure their satisfaction with the Department of Veterans Affairs (VA)
numerous benefit services. Thus, their mission is to empower Veterans by rapidly and
discreetly collecting feedback on their interactions with such VA entities as National
Cemetery Administration (NCA), Veterans Health Administration (VHA), and Veterans
Benefits Administration (VBA). VEO surveys generally entail probability samples which
only contact minimal numbers of customers necessary to obtain reliable estimates. This
information is subsequently used by internal stakeholders to monitor, evaluate, and
improve beneficiary processes. Customers are always able to decline participation and can
opt out of future invitations. A quarantine protocol is maintained to limit the number of
times a customer may be contacted over a period of time across all VEO surveys, in order
to prevent survey fatigue.
Surveys issued by EMD are generally brief in nature and present a low amount of
burden to customers. A few targeted questions will utilize a human centered design (HCD)
methodology, revolving around concepts of Trust, Ease, Effectiveness and Emotion.
Questions will focus on a specific aspect of a service process—spanning communication,
applying for benefits, deliberation, and/or receipt of benefits. Structured questions directly
address the pertinent issues regarding each surveyed line of business. The opportunity to
volunteer open-ended text responses is provided within most surveys. This open text has
been demonstrated to yield enormous information. Machine learning tools are used for
text classification, ranking by sentiment scores, and screening for homelessness,
depression, etc. Modern survey theory is used to create sample designs which are
representative, statistically sound, and in accordance with OMB guidelines on federal
surveys.
VEO has been commissioned by VBA to measure the satisfaction and experience
of customers with the Veteran Readiness and Employment (VR&E), Chapter 31 (CH 31).
VEO proposes to conduct a brief survey on customers who had experienced several
milestones in the program. Sampled customers will be contacted through an invitation
email. A link will be enclosed so the survey may be completed using an online interface,
with customized customer information. The survey itself will consist of a handful of
questions revolving around a human-centered design, focusing on such elements as trust,
emotion, effective, and ease with the care they received.

10

B. Basic Definitions
Coverage
Measurement Error
Non-Response
Transaction
Response Rate
Sample
Sampling Error
Sampling Frame
Reliability

The percentage of the population of interest that is included in
the sampling frame.
The difference between the response coded and the true value
of the characteristic being studied for a respondent.
Failure of some respondents in the sample to provide responses
in the survey.
A transaction refers to the specific time a customer interacts
with the VA that impacts the customer’s journey and their
perception of VA’s effectiveness in servicing customers.
The ratio of participating persons to the number of contacted
persons. This is one of the basic indicators of survey quality.
In statistics, a data sample is a set of data collected and/or
selected from a statistical population by a defined procedure.
Error due to taking a particular sample instead of measuring
every unit in the population.
A list of units in the population from which a sample may be
selected.
The consistency or dependability of a measure. Also referred
to as standard error.
C. Application to Veterans Affairs

This measurement may bring insights and value to all stakeholders at VA. Frontline VA leaders can resolve individual feedback from customers and take steps to improve
the customer experience; meanwhile VA executives can receive real-time updates on
systematic trends that allow them to make changes.
1) To collect continuous customer experience data to monitor the relative success
of programs designed to improve customer experience with VR&E CH 31.
2) To help field staff and the national office identify need of the specific
population they serve
3) To better understand why VR&E CH 31 customers provide positive or
negative feedback
Part II – Methodology
A. Target Population and Frame
The target population of the VR&E surveys is all VBA customers that have
experiences one of three potential triggers.
•

11

Learning about and Applying for VR&E Survey: when veterans are in
Applicant status or exit Evaluation and Planning status and enter any other status

•

•

•

Participating in VR&E Survey: when a participant is in either Independent
Living or Rehabilitation to Employment status for 150 days or when a participant
exits Independent Living or Rehabilitation to Employment status to Interrupted or
Discontinued status (excluding those that are in discontinued status with the
following reasons: non-pursuit, death, request case closure)
Completing VR&E Survey: when a participant moves from Job Ready status to
either Rehabilitated, Interrupted, or Discontinued status (excluding those that are
in discontinued status with the following reasons: non-pursuit, death, request case
closure)
Discontinued VR&E Survey: when a participant enters Discontinued Case Status
from Interrupted Case Status and prior to entering Interrupted Case Status,
participants must have entered the following case status extended evaluation (EE)
status, independent living (IL) status, rehabilitation to the point of employment
(RTE) status, and job ready (JR) status and these participants in discontinued case
status must be discontinued for being non-pursuit. A participant may also receive
this survey if they entered discontinued case status with the reason that a
claimant/participant requested the case to be closed.

The VEO will access the population data directly from VBA’s data warehouse
(EDW). The VEO will process the data and create a corresponding invite file.

B. Sample Size Determination
For a given margin of error and confidence level, the sample size is calculated as
below (Lohr, 1999). For population that is large, the equation below is used to yield a
representative sample for proportions:

where
•
•

•

2
𝑝𝑝𝑝𝑝
𝑍𝑍𝛼𝛼/2
𝑛𝑛0 =
𝑒𝑒 2

𝒁𝒁𝜶𝜶/𝟐𝟐 = 1.96, which is the critical Z score value under the normal distribution when
using a 95% confidence level (α = 0.05).
p = the estimated proportion of an attribute that is present in the population, with
q=1-p.
o Note that pq attains its maximum when value p=0.5, and this is often used
for a conservative sample size (i.e., large enough for any proportion).
e = the desired level of precision; Also referred to as MOE.

For a population that is relatively small, the finite population correction is used to
yield a representative sample for proportions:
𝑛𝑛0
𝑛𝑛 =
𝑛𝑛
1+ 0
𝑁𝑁
Where
•
•

12

𝒏𝒏𝟎𝟎 = Representative sample for proportions when the population is large.
N = Population size.

The margin of error surrounding the baseline proportion is calculated as:
𝑁𝑁 − 𝑛𝑛 𝑝𝑝(1 − 𝑝𝑝)
�
𝑁𝑁 − 1
𝑛𝑛

𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀 𝑜𝑜𝑜𝑜 𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒 = 𝑧𝑧𝛼𝛼/2 �

Where
•
𝒁𝒁𝜶𝜶/𝟐𝟐 = 1.96, which is the critical Z score value under the normal distribution when
using a 95% confidence level (α = 0.05).
•
N = Population size.
•
n = Representative sample.
•
p = the estimated proportion of an attribute that is present in the population, with
q=1-p.
Table 2 depicts the estimated number of unique VR&E customers within a month.
Preliminary analysis of this population indicates that approximately 97% of qualifying
customers have provided an email address to VR&E CH 31. The sample sizes for each
survey vary so we to propose to randomly sample Veterans for all surveys. With current
estimates, this would result in around, 11,812 completed surveys from 78,744 invitations
per year. To account for potential estimation errors, improvement in email collection, or
changes in business volume; we are requesting approval for a maximum of 13,000
completes annually across the four surveys.
Table 2. Monthly Population and Survey Figures

Surveys
Learning
about &
Applying for
VR&E Survey
Participating
in VR&E
Survey
Completing
VR&E Survey
Discontinuing
from VR&E
Survey
1

Estimated Approximate
Total
Email
Available
Population Population Population1
12,502

12,344

Yearly
Actual Expected Error With Actual
Target Sample Response
95%
Sample
Sample Needed
Rate
Confidence Needed

8,641

3,435

3,434

2,404

1,504

1,503

1,052

1,425

1,424

997

Excluding estimated duplicates and quarantined records (30% loss)

13

506

3,376

15%

4.0%

203

1,354

15%

4.5%

158

968

15%

5.0%

130

864

15%

5.5%

40,512

16,248
11,616
10,368

C. Stratification
Stratification is used to ensure that the sample matches the population, to the extent
possible, across sub-populations. The Learning & Applying for VR&E and Participating
in VR&E surveys will be stratified by gender.
D. Data Collection Methods
The population for the survey will be drawn from the VBA’s Enterprise Data
Warehouse (EDW). VEO data analysts will access the data to download the required
fields from records that reached the one of the four milestones in the previous week. Any
record with a valid email address will be included in the survey. Email invitations are
delivered to all selected customers. Selected respondents will be contacted within 8 days
of their interaction. They will have 14 days to complete the survey. Estimates will be
accessible to data users instantly on the VSignals platform.

Table 3. Survey Mode
Mode of Data
Collection

Recruitment
Method

Time After
Transaction

Recruitment
Period

Collection
Days

Online Survey

Email
Recruitment

Within 8 days of
interaction

14 Days

Tuesday

(Reminder
after 7 Days)

E. Reporting
Researchers will be able to use the VSignals platform for interactive reporting and
data visualization. Trust, Ease, Effectiveness, and Emotion scores can be observed for
each). The scores may be viewed by various subgroups (e.g. gender) in various charts for
different perspective. They are also depicted within time series plots to investigate trends.
Finally, filter options are available to assess scores at varying time periods and within the
context of other collected variable information.
Recruitment is continuous (weekly) but the results from several weeks may be
combined into a quarterly estimate for more precise estimates, which is the recommended
reporting level.

14

F. Quality Control
To ensure the prevention of errors and inconsistencies in the data and the analysis,
quality control procedures will be instituted in several steps of the survey process. Records
will undergo a cleaning during the population file creation. The quality control steps are
as follows.
1. Records will be reviewed for missing data. When records with missing data are
discovered, they will be either excluded from the population file when required or
coded as missing.
2. Any duplicate records will be removed from the population file to both maintain
the probabilities of selection and prevent the double sampling of the same
customer.
3. Invalid emails will be removed.
The survey sample loading and administration processes will have quality control
measures built into them.
1. The extracted sample will be reviewed for representativeness. A secondary review
will be applied to the final respondent sample.
2. The survey load process will be rigorously tested prior to the induction of the
survey to ensure that sampled customers is not inadvertently dropped or sent
multiple emails.
3. The email delivery process is monitored to ensure that bounce-back records will
not hold up the email delivery process.

G. Sample Weighting, Coverage Bias, and Non-Response Bias
A final respondent sample should closely resemble the true population, in terms of
the demographic distributions (e.g. age groups). One problem that arises in the survey
collection process is nonresponse, which is defined as failure of selected persons in the
sample to provide responses. This occurs in various degrees to all surveys, but the
resulting estimates can be distorted when some groups are actually more or less prone to
complete the survey. In many applications, younger people are less likely to participate
than older persons. Another problem is under-coverage, which is the event that certain
groups of interest in the population are not even included in the sampling frame. They
cannot participate because they cannot be contacted: those without an email address will
be excluded from sample frame. These two phenomena may cause some groups to be
over- or under-represented. In such cases, when the respondent population does not match
the true population, conclusions drawn from the survey data may not be reliable and are
said to be biased.
While we are not currently planning to weight the data, survey practitioners
recommend the use of sampling weighting to improve inference on the population. This
will be introduced into the survey process as a tool that helps the respondent sample more
closely represent the overall population. Weighting adjustments are commonly applied in
surveys to correct for nonresponse bias and coverage bias. As a business rule will be

15

implemented to require callers to provide email address, the coverage bias for this survey
is expected to decrease. In many surveys, however, differential response rates may be
observed across age groups. In the event that some age groups are more represented in the
final respondent sample, the weighting application will yield somewhat smaller weights
for this age group. Conversely, age groups that are underrepresented will receive larger
weights. This phenomenon is termed non-response bias correction for a single variable.
Strictly speaking, we can never know how non-respondents would have really answered
the question, but the aforementioned adjustment calibrates the sample to resemble the full
population – from the perspective of demographics. This may result in a substantial
correction in the resulting weighting survey estimates when compared to direct estimates
in the presence of non-negligible sample error (non-response bias).
It was reported earlier that the email population comprises 97% of the full VR&E
CH 31 population. This is very respectable considering that 88% of US veterans utilize
email (National Telecommunications and Information Administration, 2020). It is
assumed that the level of customer satisfaction is not directly related to their email status
(Missing at Random).
When implemented, weighting will utilize cell weights in real time. With each
query on the VSignals platform for each respondent by dividing the target for a cell by the
number of respondents in the cell. The weighting scheme will include, where possible all
the variables used for explicit stratification, However, cells will be collapsed if the
proportion of the population is insufficient to reliably achieve a minimum of 3 completes
per month. As a result, weights may be more comprehensive for larger population
segments. For instance, in the VA, women are a smaller proportion of the populations.
Therefore, woman will have more collapsed cells than men.
As part of the weighting validation process, the weights of persons in age and
gender groups are summed and verified that they match the universe estimates (i.e.,
population totals). Additionally, we calculate the unequal weighting effect, or UWE (see
Kish, 1992; Liu et al., 2002). This statistic is an indication of the amount of variation that
may be expected due to the inclusion of weighting. The unequal weighting effect
estimates the percent increase in the variance of the final estimate due to the presence of
weights and is calculated as:

where
•
•
•

𝑠𝑠
2
= ( )2
𝑈𝑈𝑈𝑈𝑈𝑈 = 1 + 𝑐𝑐𝑐𝑐𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤ℎ𝑡𝑡𝑡𝑡
𝑤𝑤
�
cv = coefficient of variation for all weights 𝑤𝑤𝑖𝑖𝑖𝑖 .
s = sample standard deviation of weights.
1
� = sample mean of weights, 𝑤𝑤
𝒘𝒘
� = ∑𝑖𝑖𝑖𝑖 𝑤𝑤ij.
𝑛𝑛

H. Quarantine Rules

VEO seeks to limit contact with customers as much as possible, and only as
needed to achieve measurement goals. These rules are enacted to prevent excessive
recruitment attempts upon VA’s customers. VEO also monitors participation within other
16

surveys, to ensure veterans and other customers do not experience survey fatigue. All
VEO surveys offer options for respondents to opt out, and ensure they are no longer
contacted for a specific survey. VEO also monitors Veteran participation within other
surveys, to ensure customers do not experience survey fatigue.
Table 4. Quarantine Protocol
Quarantine Rule
Description
Repeated Sampling
for VR&E Survey
Other VEO Surveys
Opt Outs

Number of days between receiving/completing online
survey, prior to receiving email invitation for VR&E CH
31 experience
Number of days between receiving/completing online
survey and becoming eligible for another VEO survey
Persons indicating their wish to opt out of either phone
or online survey will no longer be contacted.

Elapsed
Time
30 Days
30 Days
N/A

Part III – Assumptions and Limitations
A. Coverage Bias
Since the VEO VR&E CH 31 Surveys are email only, there is a small population
of VBA VR&E CH 31 customers that cannot be reached by the survey. Veterans that lack
access to the internet or do not use email may have different levels of trust and satisfaction
with their service. As such, it is thought that customers in this latter category do not harbor
any tangible differences to other Veterans who do share their information. In order to
verify this, VEO plans to execute a coverage bias study to assess the amount of coverage
bias due and derive adjustment factors in the presence of non-negligible bias.
References
Choi, N.G. & Dinitto, D.M. (2013). Internet Use Among Older Adults: Association with
Health Needs, Psychological Capital, and Social Capital. Journal of Medical
Internet Research, 15(5), e97
Kalton, G., & Flores-Cervantes, I. (2003). Weighting Methods. Journal of Official
Statistics, 19(2), 81-97.
Kish, L. (1992). Weighting for unequal P. Journal of Official Statistics, 8(2), 183-200.
Kolenikov, S. (2014). Calibrating Survey Data Using Iterative Proportional Fitting
(Raking). The Stata Journal, 14(1): 22–59.
Lohr, S. (1999). Sampling: Design and Analysis (Ed.). Boston, MA: Cengage Learning.
Liu, J., Iannacchione, V., & Byron, M. (2002). Decomposing design effects for stratified
sampling. Proceedings of the American Statistical Association’s Section on Survey
Research Methods.

17

National Telecommunications and Information Administration (2020) Digital Nation Data
Explorer https://www.ntia.doc.gov/data/digital-nation-dataexplorer#sel=emailUser&demo=veteran&pc=prop&disp=chart

Wong, D.W.S. (1992) The Reliability of Using the Iterative Proportional Fitting
Procedure. The Professional Geographer, 44 (3), 1992, pp. 340-348

18