A-11 Generic Clearance Requirements_Board of Veterans Appeals

Request for Approval under the “Generic Clearance for Improving Customer Experience (OMB Circular
A-11, Section 280 Implementation)” (OMB Control Number: 2900-0876)
TITLE OF INFORMATION COLLECTION: Board of Veterans Appeals Surveys
PURPOSE
The Board of Veteran’s Appeals Survey is designed to measure Veterans’ (or their Beneficiary’s) satisfaction with the appeals process
regarding their benefit claims, as reviewed by the Board of Veterans Appeals. The Board passes final judgement on cases of denied or
diminished benefits originating within the Veterans Benefits Administration (VBA). The goal of the VEO survey operation is to
continuously provide highly reliable monthly overall estimates.
Those recently completing one of three aspects of the appeals process are eligible for the survey:
1. Initial Filing Survey
2. Hearing Proceeds Survey
3. Survey on the Reporting of the Board’s Decision
A portion of the Veterans recently engaging the Board in the three stages listed above will be contacted by email to complete a
brief online survey. This survey is conducted weekly, with reporting also available weekly, monthly, and quarterly. The questions aim
to measure Veterans’ experience with the general process, including their interactions with Board systems, staff, and judges, as well as
their clarity and understanding of the Board’s procedures and decisions. The measurement scale is a 5-point Likert, and is based on a
short list of multiple-choice questions that effectively solicit Veterans’ feedback. Once the invitation is collected, it is immediately
available in VSignals, the Medallia-based platform used by the Veterans Experience Office for CX data storage and analysis.
Participants will have two weeks to complete the survey at their convenience. Once data collection is completed, a series of internal
quality checks are implemented. The participant responses in the online survey are weighted so the samples will more closely
resemble the overall population. Sample weighting will be applied using Age Group.

DESCRIPTION OF RESPONDENTS:
The target population of the Board Survey is all Veterans who have completed a Board of Veterans’ appeals process items within the
past week. Those recently completing one of three aspects of the appeals process are eligible for the survey:
1. Initial Filing Survey

2. Hearing Proceeds Survey
3. Survey on the Reporting of the Board’s Decision
The survey frame, a listing of all appellants, is provided to VEO directly by the Board. This listing is then reconciled with an
email database from the Veteran’s Health Administration, and the survey is offered to appellants with a recent interaction for whom an
email address is listed within VHA. This will correspond to roughly 30% of the true underlying population.

TYPE OF COLLECTION: (Check one)
[ ] Customer Comment Card/Complaint Form
[ ] Usability Testing (e.g., Website or Software
[ ] Focus Group

[X] Customer Satisfaction Survey
[ ] Small Discussion Group
[ ] Other: ______________________

CERTIFICATION:
I certify the following to be true:
1. The collection is voluntary.
2. The collection is low-burden for respondents and low-cost for the Federal Government.
3. The collection is non-controversial and does not raise issues of concern to other federal agencies.
4. Personally identifiable information (PII) is collected only to the extent necessary and is not retained.
5. Information gathered is intended to be used for general service improvement and program management purposes.
6. The collection is targeted to the solicitation of opinions from respondents who have experience with the program or may have
experience with the program in the future.
7. All or a subset of information may be released as part of A-11, Section 280 requirements on performance.gov. Additionally,
summaries of the data may be released to the public in communications to Congress, the media and other releases disseminated by
VEO, consistent with the Information Quality Act.
Name: Evan Albert, Director of Measurement and Data Analytics , Veterans Experience Office [email protected] (202) 875-478

To assist review, please provide answers to the following question:
Personally Identifiable Information:

2

1. Will this survey use individualized links, through which VA can identify particular respondents even if they do not provide their
name or other personally identifiable information on the survey? [ X ] Yes [] No
2. Is personally identifiable information (PII) collected? [ ] Yes [X] No
3. If Yes, will any information that is collected be included in records that are subject to the Privacy Act of 1974? [ ] Yes [ ] No
[N/A]
4. If Yes, has an up-to-date System of Records Notice (SORN) been published? [ ] Yes [ ] No [N/A]
Gifts or Payments:
Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to participants? [ ] Yes [ X] No

BURDEN HOURS

Category of Respondent

No. of Respondents

Participation
Time
( X minutes =)

Individuals & Households
VA Form (if applicable)
Totals

Burden
(÷ 60 =)

54,360 annually

2

1,812

54,360 annually

2

1,812

Please answer the following questions.
1. Are you conducting a focus group, a survey that does not employ random sampling, user testing or any data collection
method that does not employ statistical methods?
Yes ___
No _X_
If Yes, please answer questions 1a-1c, 2 and 3.
If No, please answer or attach supporting documentation that answers questions 2-8.
a. Please provide a description of how you plan to identify your potential group of respondents and how you will select them.

3

b. How will you collect the information? (Check all that apply)
[ ] Web-based or other forms of Social Media
[ ] Telephone
[ ] In-person
[ ] Mail
[X] Other- E-mail-based surveys
c. Will interviewers or facilitators be used? [ ] Yes [ X ] No
2. Please provide an estimated annual cost to the Federal government to conduct this data collection: __$13,000______
3. Please make sure that all instruments, instructions, and scripts are submitted with the request. This includes questionnaires,
interviewer manuals (if using interviewers or facilitators), all response options for questions that require respondents to select a
response from a group of options, invitations given to potential respondents, instructions for completing the data collection or
additional follow-up requests for the data collection.
-Done
4. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection
methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons)
in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a
whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the
collection had been conducted previously, include the actual response rate achieved during the last collection.
- Please see Statistical Sample Plan in the Appendix.
5. Describe the procedures for the collection of information, including:
a. Statistical methodology for stratification and sample selection.
b. Estimation procedure.
c. Degree of accuracy needed for the purpose described in the justification.
d. Unusual problems requiring specialized sampling procedures.
e. Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

4

- Please see Statistical Sample Plan in the Appendix.
6. Describe methods to maximize response rates and to deal with issues of nonresponse. The accuracy and reliability of information
collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be
provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.
Please see Statistical Sample Plan in the Appendix.
7. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections
of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from
10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main
collection of information.
Please see Statistical Sample Plan in the Appendix.
8. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency
unit, contractors, grantees, or other person(s) who will actually collect or analyze the information for the agency.
Statistical Aspects:
Mark Andrews, Statistician, Veterans Experience Office, VA. (703) 483-5305
Collection and Analysis: Evan Albert, Dir. of Measurement and Data Analytics, Veterans Experience Office, VA (202) 875-9478
Steven Ginski, Associate Counsel of the Board of Veterans Appeals (202) 632-4894

5

Service Level Measurements – Boards of Veterans
Appeals Survey
Sampling Methodology Report
Prepared by
Veteran Experience Office
Version 1.1

6

September 2018

Contents
Executive Summary ........................................................................................................................................ 8
Part I – Introduction ....................................................................................................................................... 9
A. Background ........................................................................................................................................... 9
B. Basic Definitions ................................................................................................................................... 9
C. Application to Veterans Affairs............................................................................................................ 10
Part II – Methodology ................................................................................................................................... 10
A. Target Population and Frame .............................................................................................................. 10
B. Sample Size Determination ............................................................................................................... 10
C. Data Collection Methods .................................................................................................................. 12
D. Reporting........................................................................................................................................... 13
E. Quality Control ................................................................................................................................. 13
F. Sample Weighting, Coverage Bias, and Non-Response Bias ........................................................... 14
G. Quarantine Rules ............................................................................................................................... 15
Part III – Assumptions and Limitations ......................................................................................................... 15
A. Respondent Satisfaction Bias ............................................................................................................... 15
B. Coverage Bias....................................................................................................................................... 16
Appendix 1. List of Data Extraction Variables .................................................................................... 16
Appendix 2. Survey Questions ............................................................................................................... 17
Appendix 3. References .......................................................................................................................... 24

7

Executive Summary
The Board of Veteran’s Appeals Survey is designed to measure Veterans’ (or their Beneficiary’s) satisfaction with the appeals
process regarding their benefit claims, as reviewed by the Board of Veterans Appeals. The Board passes final judgement on cases of
denied or diminished benefits originating within the Veterans Benefits Administration (VBA). The goal of the VEO survey operation
is to continuously provide highly reliable monthly overall estimates.
Those recently completing one of three aspects of the appeals process are eligible for the survey:
4. Initial Filing Survey
5. Hearing Proceeds Survey
6. Survey on the Reporting of the Board’s Decision
A portion of the Veterans recently engaging the Board in the three stages listed above will be contacted by email to complete a
brief online survey. This survey is conducted weekly, with reporting also available weekly, monthly, and quarterly. The questions aim
to measure Veterans’ experience with the general process, including their interactions with Board systems, staff, and judges, as well as
their clarity and understanding of the Board’s procedures and decisions. The measurement scale is a 5-point Likert, and is based on a
short list of multiple-choice questions that effectively solicit Veterans’ feedback. Once the invitation is received, participants will have
two weeks to complete the survey at their convenience. Once data collection is completed, a series of internal quality checks are
implemented. The participant responses in the online survey are weighted so the samples will more closely resemble the overall
population. Sample weighting will be applied using Age Group.
This report describes the methodology used to collect and disseminate the Board of Veterans Appeals Survey. Information
about quality assurance protocols as well as limitations of the survey methodology is also included in this report.

8

Part I – Introduction
A. Background
The Enterprise Measurement and Design team (EMD) is part of the Insights and
Analytics (I&A) division within the Veterans Experience Office (VEO). The EMD team is
tasked with conducting transactional surveys of the Veteran population to measure their
satisfaction with the Department of Veterans Affairs (VA) numerous benefit services. Thus, their
mission is to empower Veterans by rapidly and discreetly collecting feedback on their
interactions with such VA entities as NCA, VHA, and VBA. VEO surveys generally entail
probability samples which only contact minimal numbers of Veterans necessary to obtain
reliable estimates. This information is subsequently used by internal stakeholders to monitor,
evaluate, and improve beneficiary processes. Veterans are always able to decline participation,
and have the ability to opt out of future invitations. A quarantine protocol is maintained to limit
the number of times a Veteran may be contacted, in order to prevent survey fatigue, across all
VEO surveys.
The Board of Veterans Appeals administered by the Department of Veterans Affairs
(VA) determines whether Veterans are entitled to claimed benefits which have initially been
denied or reduced. Its mission is to conduct hearings and pass proper judgement on appeals in a
timely manner. The vast majority of appeals involve claims for disability compensation, in
addition to other veteran benefits. the Board allows Veterans to submit medial and lay evidence
at any point during the appeals process. Veterans Law Judges will eventually review benefit
claims determinations made by local VA offices and issue decisions on appeals.
In order to continue to provide quality services to Veterans, the Veteran Experience
Office (VEO) has been solicited to measure the satisfaction of Veterans regarding the Board
appeals and hearing procedures for their benefit claims. VEO proposed to conduct a brief survey
on Veterans who recently completed various aspects of the appeals process.
B. Basic Definitions
Coverage
Measurement Error
Non-Response
Transaction

Response Rate
Sample
Sampling Error
Sampling Frame

The percentage of the population of interest that is included in
the sampling frame.
The difference between the response coded and the true value of the
characteristic being studied for a respondent.
Failure of some respondents in the sample to provide responses in
the survey.
A transaction refers to the specific time a Veteran interacts with the
VA that impacts the Veteran’s journey and their perception of VA’s
effectiveness in caring for Veterans.
The ratio of participating persons to the number of contacted
persons. This is one of the basic indicators of survey quality.
In statistics, a data sample is a set of data collected and/or selected
from a statistical population by a defined procedure.
Error due to taking a particular sample instead of measuring every
unit in the population.
A list of units in the population from which a sample may be
selected.
9

Reliability

The consistency or dependability of a measure. Also referred to as
standard error.
C. Application to Veterans Affairs

In general, customer experience and satisfaction are usually measured at three levels: the
enterprise level, the service level patterns, and point-of-service feedback. This measurement may
bring insights and value to all stakeholders at VA. Front-line VA leaders can resolve individual
feedback from Veterans and take steps to improve the customer experience; meanwhile VA
executives can receive real-time updates on systematic trends that allow them to make changes.
1) To collect continuous customer experience data that make or break the service
experience.
2) To help field staff and the national office identify areas of improvement.
3) To understand emerging drivers and detractors of customer experience.
Part II – Methodology
A. Target Population and Frame
The target population of the Board Survey is all Veterans who have completed a Board of
Veterans’ appeals process items within the past week. The survey frame, a listing of all
appellants, is provided to VEO directly by the Board. This listing is then reconciled with an
email database from the Veteran’s Health Administration, and the survey is offered to appellants
with a recent interaction for whom an email address is listed within VHA. This will correspond
to roughly 30% of the true underlying population.

Figure 1. Measurement Goals and Survey Mode

Survey
Types

• Filing
• Hearing
• Decision

Data
Collection

• Email Recruitment
• Online Survey Interface
• 2 Weeks to Complete

B. Sample Size Determination
To achieve a certain level of reliability, a sufficiently large sample is required. The
sample size for a given level of reliability is calculated below (Lohr, 1999). Note that the
10

reliability of the survey estimates is restricted by the amount of available sample. Due to
limitations on the number of Veterans that have shared an email address with VA, coupled with
the expected non-response, the the Board Survey may require to contact all eligible Veterans
who also have an email address. We proceed to depict the expected level of precision (i.e.,
reliability) against various standards.
For a population that is large, the equation below is used to yield a representative sample
for proportions:
2
𝑍𝛼/2
𝑝𝑞
𝑛0 =
𝑒2
where
•
•
•

𝒁𝜶/𝟐 = 1.96, which is the critical Z score value under the normal distribution when using a
95% confidence level (α = 0.05).
p = the estimated proportion of an attribute that is present in the population, with q=1-p.
o Note that pq attains its maximum when value p=0.5, and this is sometimes used for
a conservative sample size (i.e., large enough for any proportion).
e = the desired level of precision; in the current case, the margin of error e = 0.03, or 3%.
Also referred to as MOE.

For a population that is relatively small, the finite population correction is used to yield a
representative sample for proportions:
𝑛0
𝑛=
𝑛
1+ 0
𝑁
Where
•
•

𝒏𝟎 = Representative sample for proportions when the population is large.
N = Population size.

The margin of error surrounding the baseline proportion is calculated as:
𝑀𝑎𝑟𝑔𝑖𝑛 𝑜𝑓 𝐸𝑟𝑟𝑜𝑟 = 𝑧𝛼/2 √

𝑁 − 𝑛 𝑝(1 − 𝑝)
√
𝑁−1
𝑛

Where
•
𝒁𝜶/𝟐 = 1.96, which is the critical Z score value under the normal distribution when using
a 95% confidence level (α = 0.05).
•
N = Population size.
•
n = Representative sample.
•
p = the estimated proportion of an attribute that is present in the population, with q=1-p.
Table 2A depicts the approximate monthly population for the the Board appeals, as well
as the sample size determination with Confidence Level (CL) and Margin of Error (MOE) at
various levels. The sample size numbers listed indicate the number of responses from Veterans
required to attain the stated level of precision.
11

Based on the estimated number of appellants filing with the Board each month (6,000),
the email availability rate (30%), and the expected survey response rate (20%), the expected
number of responses from contacted Veterans would be approximately 360. This number would
represent the number of responses if every qualified Veteran/Beneficiary were sent a survey
invitation. However, this number is still much smaller (360 vs. 907) than the desired level of
precision corresponding to a 95% CL and 3% MOE (Lohr, 1999). Therefore, it is reasonable to
sample all available Veterans to produce the most reliable estimates possible – this would
correspond to reliability levels at the 90% CL and 5% MOE. A similar situation holds for the
Decision population: 7,400 monthly decisions will result in . In the Board appeals hearing
population, the expected respondents with 100% sampling rate are still below any of the sample
sizes at proposed CL and MOE, due to the small hearing population size. Therefore, all Veterans
in this population will also be sampled.
Table 2B provides the weekly and monthly sample targets for the appellant and hearing
populations, presuming a return rate of 20%. The current method obviates the need for any
stratification or formal probability sampling.
Table 2A. Target Population Figures
Population
Type

Filing
Hearing
Decision

Approximate
Monthly
Population

Approximate
Monthly
Email
Population

6,000
1,700
7,400

1,800
510
2,220

Precision
Sample:
95% CL,
3%
MOE
907
656
933

Precision
Sample:
90% CL,
5%
MOE
258
233
260

Precision
Sample:
90% CL,
8%
MOE
N/A
100
N/A

Table 2B. Proposed Sample Targets by Time Period
Population
Type
Filing
Hearing
Decision
Total

Weekly Target
75
25
100
200

Weekly
Contact
375
125
500
1,000

Monthly
Target
300
100
400
800

Monthly
Contact
1,500
500
2,000
4,000

C. Data Collection Methods
At the beginning of every weekly measurement period, the Board will review records and
send participant information to VEO for the survey distribution and data collection. Given that
Veteran’s email address is not available to the Board, these Veteran records will be mapped
using social security number to the VHA Corporate Data Warehouse (CDW) to collect email
addresses. This will permit VEO to share the online survey link to appellants who have shared
12

their email address with VHA. Note that Veterans with private or other health care providers
who do not access VHA for health benefits will not be contacted. Thus, the email population
may differ somewhat from the true underlying population, possibly introducing coverage-bias
into the survey estimates (see section below).
After the Board participants have their email address appended to their records, an
invitation file is created and stored. Emails are immediately delivered to all selected Veterans.
Respondents will have 14 days to complete the survey. A reminder email will be sent after one
week to those who have yet to complete the survey. Estimates will be accessible to data users
instantly within the Veteran Signals (VSignals).

D. Reporting
Researchers will be able to use the Veteran Signals (VSignals) system for interactive
reporting and data visualization. VA employees with a PIV card may access the system at
https://va.voice.medallia.com/sso/va/. The scores may be viewed by Age Group, Gender, and
Race/Ethnicity in various charts for different perspective. They are also depicted within time
series plots to investigate trends. Finally, filter options are available to assess scores at varying
time periods and within the context of other collected variable information.
Recruitment is continuous (weekly) but the results should be combined into a monthly
estimate for more precise estimates, which is the recommended reporting level. Weekly
estimates are less reliable for small domains, (i.e., VAMC-level) and should only be considered
for aggregated populations. Monthly estimates will have larger sample sizes, and therefore
higher reliability. Quarterly estimates are the most precise, but will take the greatest amount of
time to obtain (12 weeks of collection). However, Quarterly estimates are the most suitable for
the analysis of small populations.

E. Quality Control
To ensure the prevention of errors and inconsistencies in the data and the analysis, quality
control procedures will be instituted in several steps of the survey process. Records will undergo
a cleaning during the population file creation. The quality control steps are as follows.
1. Records will be reviewed for missing sampling and weighting variable data. When
records with missing data are discovered, they will be either excluded from the
population file or put into separate strata upon discussion with subject matter experts.
2. Any duplicate records will be removed from the population file to both maintain the
probabilities of selection and prevent the double sampling of the same Veteran.
3. Invalid emails will be removed.
The survey sample loading and administration processes will have quality control
measures built into them.
1. The survey load process will be rigorously tested prior to the induction of the the Board
Survey to ensure that sampled customers is not inadvertently dropped or sent multiple
emails.
2. The email delivery process is monitored to ensure that bounce-back records will not hold
up the email delivery process.
13

The weighting and data management quality control checks are as follows:
1. The sum of the weighted respondents will be compared to the overall population count to
confirm that the records are being properly weighted. When the sum does not match the
population count, weighting classes will be collapsed to correct this issue.
2. The unequal weighting effect will be used to identify potential issues in the weighting
process. Large unequal weighting effects indicate a problem with the weighting classes,
such as a record receiving a large weight to compensate for nonresponse or coverage
bias.

F. Sample Weighting, Coverage Bias, and Non-Response Bias
Weighting is commonly applied in surveys to adjust for nonresponse bias and coverage
bias. Nonresponse is defined as failure of selected persons in the sample to provide responses.
This is observed virtually in all surveys, in that some groups are more or less prone to complete
the survey. The nonresponse issue may cause some groups to be over or under represented.
Under-coverage is another common survey problem, in which certain group of interest in the
population is not included in the sampling frame. The reason that these people cannot participate
is because they cannot be contacted. For instance, individuals without a valid email address
cannot be reached via email recruitment. In both cases, no reliable conclusions can be drawn
from the survey data, due to nonresponse and coverage bias.
Survey practitioners recommend the use of sampling weighting to improve inference on
the population, so that the final respondent sample is closely resemble the full population.
Although all available Veterans from the appellant and hearing population are surveyed,
differential response rates may still be observed across different age groups. Weighting can help
adjust for the demographic representation by assigning larger weights to under-represented age
group and smaller weights to over represented age group. Also, because the email population
will have different demographics than the overall population, the initial sample will be selected
in a manner from the frame so that the final respondent sample resembles the overall population.
In both ways of adjustments, weighting may result in substantial correction in the final survey
estimates when compared to direct estimates in the presence of non-negligible sample error.
In summary, the Board Survey is weighted by respondent’s age, and the final respondent
sample is representative of the full population. The population values will be collected and
recorded by VEO for every data collection period. Sample weights are generated for all
estimates: weekly, monthly, and quarterly.
It was reported earlier that the email population comprises 30% of the full population.
Since 85% of older Americans utilize email (Choi & Dinitto, 2013), we can presume that most
Veterans choose not to share their email address with the VA or are simply unware of that
option. It is assumed that the level of patient satisfaction is not directly related to their email
status (Missing at Random). Since age has been observed to be a strong predictor of Veteran
satisfaction in other VA surveys, the weighting methodology outlined above will adequately
compensate for any bias introduced by the incomplete frame of population.
If we let wij denote the sample weight for the ith person in age group j (j=1, 2, and 3), then
the CW formula is:
14

𝑤𝑖𝑗 =

# 𝑉𝑒𝑡𝑒𝑟𝑎𝑛𝑠 𝑖𝑛 𝑎𝑔𝑒 𝑔𝑟𝑜𝑢𝑝 𝑗 𝑖𝑛 𝑡ℎ𝑒 𝑝𝑜𝑝𝑢𝑙𝑎𝑡𝑖𝑜𝑛
# 𝑉𝑒𝑡𝑒𝑟𝑎𝑛𝑠 𝑖𝑛 𝑎𝑔𝑒 𝑔𝑟𝑜𝑢𝑝 𝑗 𝑖𝑛 𝑡ℎ𝑒 𝑠𝑎𝑚𝑝𝑙𝑒

As part of the weighting validation process, the weights of persons in an age group is
summed and verified that it matches the universe estimates (i.e., population totals). Additionally,
we calculate the unequal weighting effect, or UWE (see Kish, 1992; Liu et al., 2002). This
statistic is an indication of the amount of variation that may be expected due to the inclusion of
weighting. The unequal weighting effect estimates the percent increase in the variance of the
final estimate due to the presence of weights and is calculated as:
𝑠
2
𝑈𝑊𝐸 = 1 + 𝑐𝑣𝑤𝑒𝑖𝑔ℎ𝑡𝑠
= ( )2
𝑤
̅
where
•
•
•

cv = coefficient of variation for all weights 𝑤𝑖𝑗 .
s = sample standard deviation of weights.
1
̅ = sample mean of weights, 𝑤
𝒘
̅ = ∑𝑖𝑗 𝑤ij.
𝑛

G. Quarantine Rules
VEO seeks to limit contact with Veterans as much as possible, and only as needed to
achieve measurement goals. These rules are enacted to prevent excessive recruitment attempts
upon Veteran inpatients. VEO also monitors Veteran participation within other surveys, to
ensure Veterans do not experience survey fatigue. All VEO surveys offer options for respondents
to opt out, and ensure they are no longer contacted for a specific survey.
Table 4. Proposed Quarantine Protocol
Quarantine Rule
Repeated Sampling
for the Board
Survey
Other Surveys
Anonymous
Opt Outs

Description
Number of days between completing online survey, and
receiving another online survey related to another
complaint.
Veterans engaged that have recently completed other VEO
surveys will not be selected for 30 days.
Callers explicitly wishing to remain anonymous will not be
contacted.
Persons indicating their wish to opt out of either phone or
online survey will no longer be contacted.

Elapsed Time
3 Months or 90
Days
1 Month or 30
Days
N/A
N/A

Part III – Assumptions and Limitations
A. Respondent Satisfaction Bias
According to the survey design, all qualified Veterans will be surveyed to understand
their satisfaction with the overall appeals/hearing process. There is possibility that Veterans’
satisfaction may be interacted with the appeals decisions they received. In other words, Veterans
who received an approval or positive review on their claim submissions may perceive higher
15

general satisfaction with the Board services than those who did not receive an approval,
regardless of the services per se. This hypothesis needs to be examined once the survey is
implemented. VEO is aware of this potential bias and will carefully review the survey results
once the data collection is completed.
B. Coverage Bias
The Veterans’ contact information provided by the Board does not contain participants’
email address, which is the primary point of contact for most VEO surveys. Social security
numbers are collected by the Board, however, and this may be used to obtain a valid email
address by accessing email records within CDW, the VHA database system. However, not all
Veterans have provided their email address to VHA. Indeed, if the there is a substantial
difference in customer satisfaction between appellants who frequent VA hospitals and those who
do not, the final estimates will suffer from coverage bias. Although we may assume that a
Veteran’s satisfaction with the appeals process is independent of whether they frequented a VA
hospital in the past, there may be differences in the socio-demographic composition between
those utilizing VA services and those accessing private (or other) healthcare sources. Coverage
bias will be mitigated through the use of sample weighting to ensure the VHA population is
modified to greater reflect the Board population, with respect to demographic variables such as
Age Group, Gender, and Race/Ethnicity.
Additionally, there is thought to be bias from the use of email addresses as the singular
method of contact with survey respondents. Choi and Dinitto (2013) showed that 86% of older
adults use the internet for email purposes. Therefore, older Veterans that lack access to the
internet cannot be reached by the survey, and it is plausible that such Veterans may be in a sociodemographic class with differing attitudes of the appeals process. Moreover, only 25% of VHA
beneficiaries with a valid email address elect to share it for future contact. In all, approximately
30% of all appellants utilizing VHA services will be matched to a valid email address. It is
possible that Veterans withholding their email address do not share similar perspectives, which
may also lead to coverage bias.
It is assumed that the coverage bias from these two sources is negligible, and will not
significantly alter customer experience estimates. In order to assess this impact VEO will plan to
conduct a study to examine customer satisfaction differences between online surveys (w/ email
recruitment) and automated telephone surveys (w/ phone recruitment).
Appendix 1. List of Data Extraction Variables
Survey Person ID
Veteran First Name
Veteran Last Name
Appellant First Name
Appellant Last Name
Date of Birth
Certification Date
Form 9 Date
Disposition
Regional Office
Home Phone
16

SSOC
Hearing Type
Work Phone
Decision Date
Trigger Date
Appeal Date
Program Type
VLJ
Appeal ID
Representative
Period of Service
Survey Type
MVI
Age
Gender
Race
Appendix 2. Survey Questions
Summary
This copy document provides email and survey copy for 2 distinct surveys measuring customer
experience during the Board of Veterans’ Appeal process. These surveys are triggered within 1-4 weeks
after the completion of the following events:
[8.1] Board of Veterans’ Appeals Hearing Experience Survey: Email survey sent after the hearing has
occurred.
[8.2] Board of Veterans’ Appeals Appellant Experience Survey: Email survey sent after appellant process
occurs.

Target Audiences
To be included in the survey sample, respondents must be:
Veterans who have gone through the Board of Veterans’ Appeals process.
Notes
Text with in the copy indicates a string from a database.
Text with [brackets] indicates notes, comments, options, dummy text or annotates browser behaviors. [or]
separates conditionals when there are choices. indicates data does not exist.
Margin comments call out details or issues that may require follow up or a discussion with a designer,
developer, or writer.
Headings label and demarcate sections of the experiences and separate surveys.

the Board APPELLANT SURVEY
the Board Experience Survey Email Copy
[From:] Veterans Experience Office
[Subject 8.1] Board of Veterans’ Appeals Hearing Experience Survey – 2 minutes
17

[Subject 8.2] Board of Veterans’ Appeals Appellant Experience Survey – 2 minutes
[Header]

Help us serve your better
[Heading]
OMB Number: XXXX-XXXX
Expiration: MM/DD/YYYY
Estimated Burden: 2 minutes
Your feedback is important to us. Please take two minutes to let us know how we are doing by answering
this short survey about
[8.1] < your recent experience with the Board of Veterans’ Appeals hearing process >.
[8.2] < your recent experience with the Board of Veterans’ Appeals appellant process>.
If you provide feedback, you may be contacted by VA. Serving you is our top priority.
[Take Survey]
Thank you,
Veterans Experience Office
Department of Veterans Affairs
[Footer]
If you wish to share your feedback, please do so by .
You received this email because you provided your email address to VA.
The Veterans Crisis Line provides free, confidential support for Veterans in crisis and their families and
friends. Dial 1 (800) 273-8255 (Press 1), or text 838255 to receive confidential support 24/7, (System of
Records Notice VA158VA10NC5) Visit https://www.veteranscrisisline.net/ for more information.
The National Call Center for Homeless Veterans (NCCHV) provides free, confidential support for
Veterans who are homeless or at risk of homelessness—and their family members, friends and supporters.
Veterans can make the call to or chat online with the National Call Center for Homeless Veterans, where
trained counselors are ready to talk confidentially 24 hours a day, 7 days a week. Dial 1 (877) 424-3838
or visit https://www.va.gov/HOMELESS/ to receive confidential support. If you would like to opt out
from receiving future surveys, please click here.
Unsubscribe from this VA Survey | VA Privacy Policy
Department of Veterans Affairs
Veterans Experience Office (30)
810 Vermont Avenue NW
Washington, DC 20420
the Board Experience Survey Reminder Email Copy
18

[SENT 1 WEEK AFTER INITIAL EMAIL IF THERE IS NO RESPONSE]
[From:] Veterans Experience Office
[Subject 8.1] Board of Veterans’ Appeals Hearing Experience Survey Reminder – 2 minutes
[Subject 8.2] Board of Veterans’ Appeals Appellant Experience Survey Reminder – 2 minutes
the Board Experience Survey Reminder Pre-Header
[8.1] < your recent experience with the Board of Veterans’ Appeals hearing process >.
[8.2] < your recent experience with the Board of Veterans’ Appeals appellant process>.
[Header]

Help us serve your better
[Heading]
OMB Number: XXXX-XXXX
Expiration: MM/DD/YYYY
Estimated Burden: 2 minutes
We are waiting for your response about your experience with the Board of Veterans’ Appeals. Your
feedback is important for identifying areas that need improvement.
Please take two minutes to let us know how we are doing by answering this short survey about
[8.1] < your recent experience with the Board of Veterans’ Appeals hearing process >.
[8.2] < your recent experience with the Board of Veterans’ appellant process >.
If you provide feedback, you may be contacted by VA. Serving you is our top priority.
[Take Survey]
Thank you,
Veterans Experience Office
Department of Veterans Affairs
[Footer]
If you wish to share your feedback, please do so by .
You received this email because you provided your email address to VA.
The Veterans Crisis Line provides free, confidential support for Veterans in crisis and their families and
friends. Dial 1 (800) 273-8255 (Press 1), or text 838255 to receive confidential support 24/7, (System of
Records Notice VA158VA10NC5) Visit https://www.veteranscrisisline.net/ for more information.
The National Call Center for Homeless Veterans (NCCHV) provides free, confidential support for
Veterans who are homeless or at risk of homelessness—and their family members, friends and supporters.
Veterans can make the call to or chat online with the National Call Center for Homeless Veterans, where
trained counselors are ready to talk confidentially 24 hours a day, 7 days a week. Dial 1 (877) 424-3838
or visit https://www.va.gov/HOMELESS/ to receive confidential support. If you would like to opt out
19

from receiving future surveys, please click here.
Unsubscribe from this VA Survey | VA Privacy Policy
Department of Veterans Affairs
Veterans Experience Office (30)
810 Vermont Avenue NW
Washington, DC 20420
the Board Services Survey Rating Scale Questions
The Veterans Crisis Line provides free, confidential support for Veterans in crisis and their families and
friends. Dial 1 (800) 273-8255 (Press 1), or text 838255 to receive confidential support 24/7. Visit
veteranscrisisline.net for more information.

8.1 Board of Veterans’ Appeals Hearing Experience Survey
Tell us about your recent experience with the Board of Veterans’ Appeals hearing process.
1.
•
•
•
•
•
•
•
•
•

Why did you request a Board of Veterans’ Appeals hearing? (Mark all that apply)
I thought it would help the outcome of my appeal
I thought it was required
I wanted to speak to someone in person
It’s my right to have a Board of Veterans’ Appeals hearing
It was recommended by a friend, family member, or other Veteran
It was recommended by a Veteran Service Organization representative
It was recommended by a personal representative
Other (specify)
Don’t know or not sure

2.
•
•
•
•

Was your hearing held as initially scheduled? (Mark only one)
Yes (if selected, move to question #5)
No (if selected, move to question #5)
No, my hearing was cancelled at least once (if selected, move to question #3)
No, my hearing was rescheduled at least once (if selected, move to question #4)

3.
•
•
•
•
•
•

Why was your hearing cancelled? (Mark all that apply) (OPTIONAL)
I canceled the hearing based on the advice of my representative.
I was advised that I could submit a statement in writing instead of having the hearing.
I was unable to travel to the hearing location.
I felt like it was delaying my appeal decision.
My request to reschedule my hearing was denied.
None of the above.

4.
•
•
•

Why was your hearing rescheduled? (Mark all that apply) (OPTIONAL)
I requested a different type of hearing.
I had a conflict with my initially scheduled date.
My representative was unable to attend my initially scheduled date.
20

•
•
•

Illness, transportation, or weather issues on the day of the hearing.
I didn't receive adequate notice of the date/time that the hearing was scheduled.
None of the above.

5.
•
•
•
•
•

Please rate your overall experience in requesting a hearing
Very Negative
Negative
Average
Positive
Very Positive

6.
•
•
•
•
•

Please rate your Overall experience with the hearing staff.
Very Negative
Negative
Average
Positive
Very Positive

7.
•
•
•
•
•

Please rate your Overall experience with the Judge.
Very Negative
Negative
Average
Positive
Very Positive

8.
•
•
•
•
•

Please rate your overall experience with the Board of Veterans’ Appeals hearing process.
Very Negative
Negative
Average
Positive
Very Positive

8.2 Board of Veterans’ Appeals Appellant Experience Survey
The Veterans Crisis Line provides free, confidential support for Veterans in crisis and their families and
friends. Dial 1 (800) 273-8255 (Press 1), or text 838255 to receive confidential support 24/7, (System of
Records Notice VA158VA10NC5) Visit https://www.veteranscrisisline.net/ for more information.
Tell us about your recent experience regarding your Board of Veterans’ Appeals appellant process.
1. Did you receive an initial letter from the Board of Veterans’ Appeals notifying you that your
appeal was received by the Board?
• Yes
21

•
•

No
Don’t know or not sure

2. Thinking about the initial Board of Veterans’ Appeals notification letter, was the purpose of the
letter clear and easy to understand?
• Not at all clear
• Somewhat clear
• Completely clear
• Don’t know or not sure
• I did not read the letter
3.
•
•
•
•
•

Which of the following online resources do you use to check the status of your appeal?
eBenefits
Vets.gov
Inquiry submitted via the Board’s Customer Service website
Other
I do not use any online resources to check the status of my appeal

4.
•
•
•
•
•
•

Please rate your experience using the Veteran websites for appeal status. (OPTIONAL)
Very Negative
Negative
Average
Positive
Very Positive
Not Applicable

5. In the past 6 months (excluding any contact with your local VA field office), did you contact
anyone at the Board of Veterans’ Appeals about the appeal process?
• Yes
• No
6.
•
•
•
•
•
•
•

Which of the following best describes the reason for your most recent contact?
Request a status update
Ask a question
Request a change to your contact information
Provide additional evidence/arguments in support of your appeal
Resolve a concern
None of the above
Not applicable

7.
•
•
•
•
•

Please rate your experience regarding the communication and information about your appeal.
Very Negative
Negative
Average
Positive
Very Positive

8. My most recent contact was resolved to my satisfaction?
22

•
•
•
•
•
•

Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
Not Applicable

9. Please Rate your experience with the customer service provided by the Board of Veterans’
Appeals.
• Very Negative
• Negative
• Average
• Positive
• Very Positive
10.
•
•
•
•
•

The Board’s decision was clear and understandable (whether I agree with it or not).
Strongly Disagree
Disagree
Neutral
Agree
Strongly Agree

11.
•
•
•
•
•

Please Rate your overall experience with your appeal process
Very Negative
Negative
Average
Positive
Very Positive

❐ Please check this box if you would like to volunteer your demographic information to help VA better
serve you, otherwise just click “Next” to submit your survey.
[Next]

[OMB Burden Response Copy]
We are asking for this information so that you can provide compliments, recommendations, or concerns
to VA. This information is collected in accordance with section 3507 of the Paperwork Reduction Act of
1995. Title 38, United States Code, allows us to ask for this information. We estimate that you will need
an average of two minutes to review the instructions and complete this survey. The results of this survey
will be used to inform opportunities for program improvement in the quality of VA services. Participation
in this survey is voluntary and your decision not to respond will have no impact on VA benefits or
services to which you may currently be receiving. By filling out this survey, you are authorizing VA
database access to retrieve veteran contact information, to follow up with you accordingly for purposes of
service recovery, potential crisis, or to learn more about feedback you have shared regarding your
experience with VA. VA cannot conduct or sponsor a collection of information unless a valid OMB
control number is displayed. You are not required to respond to a collection of information if this number
is not displayed. Valid OMB control numbers can be located on the OMB Internet Page at
23

www.reginfo.gov/public/do/PRAMain.
[Footer]
Privacy Policy
Race, Ethnicity, and Gender Questions
Help VA Improve its Services
We are working to better understand our customers. The following questions are voluntary. By
providing your data, your responses can help us improve VA care and services. Thank you for your
participation.
1. Are you Hispanic or Latino?
• Yes
• No
2.
•
•
•
•
•

What is your race? Select one or more.
American Indian or Alaska Native
Asian
Black or African American
Native Hawaiian or Other Pacific Islander
White

3.
•
•
•
•

How do you describe your gender?
Male
Female
Non-Binary / Third Gender
Prefer not to say

[Submit]

Appendix 3. References
Choi, N.G. & Dinitto, D.M. (2013). Internet Use Among Older Adults: Association with Health
Needs, Psychological Capital, and Social Capital. Journal of Medical Internet Research,
15(5), e97
Kish, L. (1992). Weighting for unequal P. Journal of Official Statistics, 8(2), 183-200.
Lohr, S. (1999). Sampling: Design and Analysis (Ed.). Boston, MA: Cengage Learning.
Liu, J., Iannacchione, V., & Byron, M. (2002). Decomposing design effects for stratified
sampling. Proceedings of the American Statistical Association’s Section on Survey
Research Methods.

24