A-11 Generic Clearance Requirements Document

Request for Approval under the “Generic Clearance for Improving Customer
Experience (OMB Circular A-11, Section 280 Implementation)” (OMB
Control Number: 2900-0876)
TITLE OF INFORMATION COLLECTION: Outpatient Services Surveys
PURPOSE
The Department of Veterans Affairs routinely surveys Veterans Health Administration (VHA)
Outpatient Services through the customer feedback survey known as Veterans Signals (VSignals).
When Veterans respond to Outpatient Services Surveys, they provide responses to Likert-scale
questions as well as an optional free-text comment. One of the Likert-scale questions asks Veterans
how they trust the VA on a scale from 1 (Strongly Disagree) to 5 (Strongly Agree). Trust is measured
at the nation-wide, hospital network, and individual VA Medical Center level.
DESCRIPTION OF RESPONDENTS:
The target population for the surveys consists of veterans and their corresponding encounters
with VHA outpatient
services. The current focus is on veterans who are seeking outpatient health services from VA
facilities across the
different encounters highlighted in Table 1. The primary sampling units are veterans and the
secondary sampling
units are the encounters with veterans. Thus, the data can be analyzed to represent either veterans
or encounters as
the unit of analysis.
It is important to note that each veteran is represented by a cluster of transactions and encounters
in the design of this
survey. Each transaction and encounter will be identified from the existing VHA databases to
represent veterans’ experience.
TYPE OF COLLECTION: (Check one)
[ ] Customer Comment Card/Complaint Form
[ ] Usability Testing (e.g., Website or Software
[ ] Focus Group

[X] Customer Satisfaction Survey
[ ] Small Discussion Group
[ ] Other: ______________________

CERTIFICATION:
I certify the following to be true:
1. The collection is voluntary.
2. The collection is low-burden for respondents and low-cost for the Federal Government.
1

3. The collection is non-controversial and does not raise issues of concern to other federal
agencies.
4. Personally identifiable information (PII) is collected only to the extent necessary and is not
retained.
5. Information gathered is intended to be used for general service improvement and program
management purposes.
6. The collection is targeted to the solicitation of opinions from respondents who have
experience with the program or may have experience with the program in the future.
7. All or a subset of information may be released as part of A-11, Section 280 requirements on
performance.gov. Additionally, summaries of the data may be released to the public in
communications to Congress, the media and other releases disseminated by VEO, consistent
with the Information Quality Act.
Name: Evan Albert, Director of Measurement and Data Analytics (Acting), Veterans Experience
Office [email protected] (202) 875-478
To assist review, please provide answers to the following question:
Personally Identifiable Information:
1. Will this survey use individualized links, through which VA can identify particular
respondents even if they do not provide their name or other personally identifiable
information on the survey? [ X ] Yes [] No
2. Is personally identifiable information (PII) collected? [ ] Yes [X] No
3. If Yes, will any information that is collected be included in records that are subject to the
Privacy Act of 1974? [ ] Yes [ ] No [N/A]
4. If Yes, has an up-to-date System of Records Notice (SORN) been published? [ ] Yes [ ] No
[N/A]
Gifts or Payments:
Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to
participants? [ ] Yes [ X] No
BURDEN HOURS
Category of Respondent

No. of Respondents

Participation
Time

Individuals & Households
VA Form (if applicable)
Totals

7,835,048 annually

3

391,752.4

7,835,048 annually

3

391,752.4

( X minutes =)

Burden
(÷ 60 =)

Please answer the following questions.

2

1. Are you conducting a focus group, a survey that does not employ random sampling,
user testing or any data collection method that does not employ statistical methods?
[ ] Yes
] No
If Yes, please answer questions 1a-1c, 2 and 3.
If No, please answer or attach supporting documentation that answers questions 2-8.
a. Please provide a description of how you plan to identify your potential group of
respondents and how you will select them.

[X

b. How will you collect the information? (Check all that apply)
[ ] Web-based or other forms of Social Media
[ ] Telephone
[ ] In-person
[ ] Mail
[X] Other- E-mail-based surveys
c. Will interviewers or facilitators be used? [ ] Yes [ X ] No
2. Please provide an estimated annual cost to the Federal government to conduct this data
collection: __$13,000______
3. Please make sure that all instruments, instructions, and scripts are submitted with the request.
This includes questionnaires, interviewer manuals (if using interviewers or facilitators), all
response options for questions that require respondents to select a response from a group of
options, invitations given to potential respondents, instructions for completing the data
collection or additional follow-up requests for the data collection.
-Done
4. Describe (including a numerical estimate) the potential respondent universe and any
sampling or other respondent selection methods to be used. Data on the number of entities
(e.g., establishments, State and local government units, households, or persons) in the
universe covered by the collection and in the corresponding sample are to be provided in
tabular form for the universe as a whole and for each of the strata in the proposed sample.
Indicate expected response rates for the collection as a whole. If the collection had been
conducted previously, include the actual response rate achieved during the last collection.
- Please see Statistical Sample Plan in the Appendix.
5. Describe the procedures for the collection of information, including:
a. Statistical methodology for stratification and sample selection.
b. Estimation procedure.
3

c. Degree of accuracy needed for the purpose described in the justification.
d. Unusual problems requiring specialized sampling procedures.
e. Any use of periodic (less frequent than annual) data collection cycles to reduce
burden.
- Please see Statistical Sample Plan in the Appendix.
6. Describe methods to maximize response rates and to deal with issues of nonresponse. The
accuracy and reliability of information collected must be shown to be adequate for intended
uses. For collections based on sampling, a special justification must be provided for any
collection that will not yield "reliable" data that can be generalized to the universe studied.
Please see Statistical Sample Plan in the Appendix.
7. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an
effective means of refining collections of information to minimize burden and improve
utility. Tests must be approved if they call for answers to identical questions from 10 or more
respondents. A proposed test or set of tests may be submitted for approval separately or in
combination with the main collection of information.
Please see Statistical Sample Plan in the Appendix.
8. Provide the name and telephone number of individuals consulted on statistical aspects of the
design and the name of the agency unit, contractors, grantees, or other person(s) who will
actually collect or analyze the information for the agency.
Statistical Aspects:
Mark Andrews, Statistician, Veterans Experience Office, VA. (703)
483-5305
Collection and Analysis: Evan Albert, Director of Measurement and Data Analytics,
Veterans Experience Office, VA (202) 875-9478
Renee Oshinski, Assistant Secretary for Health for Operations
(202) 461-7022
,

APPENDIX- STATISTICAL SAMPLE PLAN

Service Level Measurements: Outpatient
Sampling Methodology Report
June 2018

4

Contents
Part I: Introduction........................................................................................................................................ 6
A. Background ........................................................................................................................................... 6
B. Moments that Matter Survey Series..................................................................................................... 7
C. Basic Definitions .................................................................................................................................... 8
Part II: Methodology ..................................................................................................................................... 8
A. Survey Instrument............................................................................................................................. 9
Part III: Sample Design ................................................................................................................................ 13
A. Target Population and Frame ............................................................................................................ 13
B. Quality Control ................................................................................................................................ 16
C. Sampling Design ................................................................................................................................. 16
D. Sample Size ....................................................................................................................................... 19
E. Survey Invitation File .......................................................................................................................... 22
Part IV: Survey Administration .................................................................................................................... 22
Part V: Analysis and Dashboarding ............................................................................................................. 22
Part VI: Limitations and Mitigations ........................................................................................................... 22
Sequencing and Conditionality ............................................................................................................... 22
Compound Transaction ........................................................................................................................... 23
Appendix A. Services by VA Administration................................................................................................ 24
Appendix B. Eligible Stop Code List. ............................................................................................................ 25

5

Part I: Introduction

A. Background
When veterans interact with VA, they expect the highest quality of care and service,
feeling that they are honored, respected, and their needs are met. In 2016, the Enterprise
Management and Performance Improvement (EM&PI) team at the VEO created a model for
measurement to hold VA accountable to veterans to fulfill those goals.
As a program, VA’s service level measurements are based on the most impactful
interactions veterans have with VA–the moments that matter. Veterans were asked to provide
feedback using a “push and pull” method. Unsolicited feedback is pulled from veterans through
the web and social media. Email surveys are pushed to veterans after they have had an
interaction with the VA that’s been determined to be a moment that mattered. Surveys are
pushed to veterans digitally to capture continuous and real-time information about a veteran’s
experience. Moments that matter take the customer transactions and customer journey into
consideration, setting customer experience metrics based on customer experience drivers and
detractors. For the first time, VA is able to capture veteran feedback continuously, in real-time
and to amplify the voice of the veteran to the right people.
To meaningfully measure a veteran’s experience across each transaction along the
customer journey, a human-centered approach was used to design the surveys. The survey
instrument was created based on eighty-four interviews in nine locations across the country with
veterans to understand which moments mattered and what is measurable. The surveys were then
vetted again by fourteen veterans to prove comprehension, relevance, and measurable action.
As a program, we measure customer experience at three different levels: the enterprise
level, the service level patterns, and point-of-service feedback. The focus of this report is on the
second level – service level patterns.
The goal of service level measurements is three-fold:
1) To collect continuous customer experience data at moments that make or break the
service experience
2) To help field staff and the national office identify areas for improvement
3) To understand emerging drivers and detractors of customer experience.
This measurement model brings insights and value to all stakeholders at VA. After every
moment that matters transaction, veterans are asked about their experience. Front-line VA
leaders can resolve individual feedback from veterans and take steps to improve the customer
experience for others; additionally, VA executives receive real-time updates on systematic trends
that allow them to make changes. VEO intends to identify CX drivers and set measures for all
lines of business across all of VA’s agencies.
To achieve this goal, we measure transactions occurring within a service line and, where
appropriate, within a specific VA facility across veterans and across time. Service-level patterns
complement our enterprise trust trend data by providing actionable service-level understanding
of the issues that are driving the customer satisfaction and sentiment. Ultimately, they allow us
to identify performance improvements.

The purpose of this document is to define VA’s sampling methodology for veteran’s interactions
at the service-level in receiving VA care and/or services. This approach defines the overarching sampling
principals applicable for all service-level measurement surveys. The sampling design aims to allow
decisions makers at different levels to make inference per agency, across veterans, and across

6

transactions per line of business (LOB). The document provides a highlight of the overall strategy along
with the business rules to operationalize the process with focus on the outpatient LOB as a starting
point.

B. Moments that Matter Survey Series
Each line of business consists of a customer journey which can be broken out into key
customer transactions. These transactions consist of interactions that represent what customers
perceive to be part of the journey; within these journeys, a set of selected experiences are
considered for measurement—the moments that matter to the veterans.
The first line of business to be sampled is the Outpatient Journey with VHA. Within the
Outpatient Journey, our measurable moments that matter were identified. Currently, these
experiences are grouped into the following four points in time related to VHA Outpatient Service
(as shown in Table 1). This is a starting point and will expand to cover key experiences in all
services across all VA agencies (Appendix A. shows the veteran experience cycle).
Table 1. The Outpatient Moments that Matter
Moment that
Exclusions
Description
What is covered
Matters
All Stop Codes
related to Telephone,
VA-initiated appointment or Telehealth,
veteran-initiated
Compensation &
appointments
Pension, Home Based
Care are excluded for
Appointments made
the initial release of
for a veteran’s
Appointment
the Outpatient
outpatient healthcare
surveys.
visit to a VA facility
Same-day walk-in veterans
Refer to Appendix B
(the veteran sets up the
appointment at the same time for a comprehensive
list of excluded Stop
as they visited the facility)
Codes.

Visit

Lab/Imaging

The actual veteran
outpatient healthcare
visit, which could
involve multiple
encounters during a
single visit.

Refer to Appendix B
Encounters within the Visit
for a comprehensive
survey are defined by a set of list of excluded Stop
stop codes within the VHA
Codes.
databases. These codes and
their descriptions are
outlined in Appendix B.

The follow-up visits
requested during the
outpatient healthcare
visit.

Encounters within the
Lab/Imaging survey are
defined by a set of stop codes
within the VHA databases.
These codes and their

Refer to Appendix B
for a comprehensive
list of excluded Stop
Codes.

7

descriptions are outlined in
Appendix A.

Pharmacy

Prescriptions obtained
from VA pharmacies

Mail-order and in-person VA
pharmacy interactions.
Initiated by the veteran or
initiated by the VA.

C. Basic Definitions

In order to avoid confusion, we need to differentiate between a transaction and an encounter.
•

•

A transaction. A transaction refers to the specific time a veteran interacts with the VA that both
impact the veteran’s journey and the veteran’s perception of VA’s effectiveness in caring for
veterans.
An Encounter. The veteran can participate in multiple actions within a transaction. Each of these
actions is defined as an encounter and is transaction-specific.

Therefore, multiple encounters are possible within a transaction and across transactions -- e.g. an
endocrinologist visit is one of many encounters within the main healthcare visits for attending a
veteran’s needs or a cardiologist visit, lab work, and pharmacy service.
Part II: Methodology
VA is working to transform into the number one customer service agency in the federal
government through an in-depth understanding of veterans’ experiences from their point of view.
To create a successful customer experience measurement program, we begin with the customer.
For each line of business measured through this effort, we start by describing the customer
segments to be measured, identifying the differences and similarities in customer journeys, and
leveraging quantitative research from the existing data channels to understand customer pain
points and business process. We went into the field and used human-centered design research
techniques including 1-on-1 interviews, field observations, and contextual inquiry to understand
the journey from a veteran’s point of view and identify what drivers the customer experiences at
each step. We call the most impactful moments in the journey the moments that matter. Our
design research combined with quantitative analysis and stakeholder interviews helped us to
identify themes and insights into what moments matter and what to measure across the key
customer journey. This is seen on the journey map created as an artifact to visually show the
veteran’s customer experience (Figure 1. Outpatient Journey Map).
In order to create a successful transformation, impact was measured through three primary
dimensions:
 Effectiveness – “I got the service I needed.”
 Ease – “It was easy to get what I needed.”
 Emotion – “I felt like a valued customer.”
After identifying drivers of the customer experience at specific moments in the journey,
we determined how we can trigger the survey through backend data. Once information about
8

customer experiences was gathered over the course of the service, we created a survey
framework that included sampling and content guidelines. This outlined key customer
satisfaction drivers for the journey being assessed and the journey’s key impact metrics. These
measures vary in importance across the journey. Effectiveness, ease, and emotion are present at
every moment in the veterans’ journey, including the Outpatient Experience. The graphic below
shows how our survey questions and surveys mapped back to the moments that matter to
veterans.
Figure 1. Outpatient Journey Map

The Service Level Measurement Surveys are transactional surveys that identify which
measures are the most effective at certain points along the journey and are critical for us to meet
veterans’ expectations. They are designed to have minimal burden on customers and ask about
an interaction the customers had with the line of business. The invitation to the surveys is sent
out by email directing the veteran to a web survey that allows for near real-time feedback. The
structure of the surveys covers: the five customer satisfaction driver questions in chronological
order, an impact metric question, a problem resolution/follow up question, a final comment
question. Appendix D shows a copy of the outpatient experience questionnaire where the
questions cover:
a. Relevance to the customer’s experience
b. Actionable
c. Based on business objectives
d. Part of VA’s brand

A. Survey Instrument

A.1. Questionnaire Design
The overall process for instrument design was based on four steps as follows:

9

1. Discovery: Interviews with veterans, employees, and business stakeholders to identify
important aspects and feedback needed to refine the questionnaire design
2. Journey Mapping: Synthesis of veterans’ interviews and observations by identifying
themes and insights into a measurable journey map
3. Survey Framework: VEO defines which factor is most important at each moment and
what good looks like. We decide whether to focus on ease, effectiveness, or emotion. We
define the audience, sampling rules, and business rules.
4. Writing Surveys: We write easy to understand questions that measure the key moments
that matter based on the research with veterans and industry best practices.
For the Outpatient Experience, VEO and VHA started with a discovery ‘sprint’ to explore
patient experiences at the VA -- interviewing veterans across multiple geographic locations and
of various genders, races, life stages, military branches, and eras of service. A total of 84
veterans were interviewed across nine research locations for a total of 150 hours of interviews.
Figure 2 below shows the distribution of the locations across states covered.

Figure 2. Design Phase Interviews

10

The research resulted in a journey map which represented the end-to-end patient
experience as veterans seek and receive outpatient care. The map is organized into 5 phases:
before the visit, arriving at the facility, during the appointment, departing the facility, and after
the visit. Each phase included key moments that matter, bright spots, paint points, entry/exit
points, and interaction channels.
A.2. Pre-testing of the survey instrument
After the questionnaire design stage was completed, the survey validation process was
initiated. This validation step consisted of question comprehension testing with veterans,
usability testing of the survey flow, and validation with business stakeholders.
Pre-testing of the Outpatient Journey Surveys was conducted by the VEO, working with
the Midwest District and Field Office to recruit veterans from the Detroit and Ann Arbor area.
Two facilities were chosen within a close proximity that had different populations and different
facility ratings. The VEO completed two rounds of in-person small-scale iterative usability
testing with veterans who represented a diverse set of demographics. Each interview totaled 45
minutes. A total of 14 veterans were included in the testing phase at Ann Arbor VA Healthcare
System and John D. Dingall Medical Center in Detroit, Michigan. In addition to testing with
veterans, the survey was validated with 34 staff members across the two facilities. Table 3 below
shows the veteran small-scale usability testing demographic distribution for outpatient
experience surveys.

Table 3. Demographic distributions of Veterans Covered in the Pre-test
11

Gender

Demographic

Male
Female
Branch of Service/Duty
Army
Army, National Guard, Reserves
Navy
Airforce
Marines
Unrecorded
Age Group
<30
30-39
40-49
50-59
>=60
N/A
Race
Black or African American
White, Native Hawaiian or other Pacific
Islander
White
Healthcare Mix
VA
VA, Medicare
VA, Private Through Employer Plan, Medicare
VA, Private Through Employer Plan
VA, Tricare though National Guard
VA, Tricare

Number
11
3
5
1
2
4
1
1
0
4
0
1
8
5
1
8
4
3
2
3
1
1

Usability tests on four surveys and one example email invitation were conducted between
March 6-10, 2017:
•
•
•

March 6th: Veterans were interviewed
March 7th: Staff from Primary Care, Specialty Care, Labs, Patient Advocates, and
the Medical Center Director were interviewed
March 8th: Veterans were interviewed

The feedback from these sessions was used to make questions more specific and clear
and led to the decision to create a separate survey for labs. Additionally, we mapped our
questions back to CAPHS and SHEP Patient Satisfaction Survey questions. On March 8th and 9th
we spoke with more veterans and staff and used that feedback to create our final list of surveys
and questions. Ultimately, we ended up with four different surveys.
Finally, to test the user experience, a paper prototype was created and simulated the flow of the
survey. A card sort exercise was conducted to allow participants to prioritize questions on the
Healthcare Survey, and paper prototypes were used to test comprehension.

12

Part III: Sample Design
Our sampling approach is based on two key considerations:
- Limiting Bias: This is achieved by avoiding over-representing veterans who have a larger
number of VA encounters.
- Limiting the impact of quarantine rules used for reducing burden on respondents
To achieve these goals, careful design of the sampling methodology is needed to achieve
a full representation of the population of veterans and transactions while reducing the respondent
burden to the minimal. Efforts took place in parallel to cover the following:

1. Sampling: This effort covers both the frame and the sampling design.
a. Frame construction and maintenance:
i. This effort is focused on constructing a comprehensive dataset of all
transactions under considerations taking place during the targeted timeline.
It also accounts for multiple transactions per veteran.
ii. The frame maintenance consists of updating transactions on a rolling basis
and identifying/updating quarantine rules to reduce potential respondent
burden across VA-Wide and MTM rounds.
b. Sampling Design: The goal of this effort is to design a scheme for selecting veterans
and transactions from the frame towards a representative sample of customer
receiving VA services across all services.
2. Analysis plan and Dashboard: This effort ensures that the output of the survey caters to all
types of decision makers from the executive level to the operational level, also known as
role-based dashboarding. The aim is to achieve monthly analytical reports to reflect issues
or concerns on a regular basis. Furthermore, it covers measures taken to preserve veterans’
confidentiality by reducing disclosure risk.
3. Operations: This effort is focused on survey implementation logistics and requirements to
allow smooth operations of the sampling, data collections, and analysis processes within the
Medallia environment over time.
4. Risk Assessment and Quality Control: This effort covers expected challenges and discusses
potential mitigation plans. It also covers quality protocols used to reduce risk.

A. Target Population and Frame
The target population for the surveys consists of veterans and their corresponding
encounters with VHA outpatient services. The current focus is on veterans who are seeking
outpatient health services from VA facilities across the different encounters highlighted in Table
1. The primary sampling units are veterans and the secondary sampling units are the encounters
with veterans. Thus, the data can be analyzed to represent either veterans or encounters as the
unit of analysis.
It is important to note that each veteran is represented by a cluster of transactions and
encounters in the design of this survey. Each transaction and encounter will be identified from
the existing VHA databases to represent veterans’ experience. Some of the encounters within the
transactions are similar and have been aggregated into groupings of encounters (see Appendix B
covering stop codes). In other words, a given data entry will contain a unique record of each
encounter a veteran had with the VA during that week as well as a unique identifier for each
veteran. Therefore, the population file will contain multiple records for each veteran.
13

The population file is created on a weekly basis every Saturday night from queries to the
VHA databases for all outpatient encounters in the previous week. All outpatient encounters for
the week prior to sampling, with the exception of mail-order pharmacy encounters, will be
queried from the VHA databases. Mail-order pharmacy encounters in the period of eleven days
and four days prior to sampling are queried for the current sample. VEO will extract data on
veterans who received outpatient services directly from VHA’s corporate data warehouse during
the targeted week. Once the data are pulled by the VEO team, the following quality control
checks will be executed:
o
o
o

Identification of missing data in variables required for later merging and sampling
Identification of duplicate records per encounter
Deceased veterans for removal from the VHA files prior to concatenation

Constructing and cleaning of the frame, i.e. the population file, will require several steps as
shown in Figure 2.

14

Figure 2. Flowchart Frame Construction
Query VHA CDW

Deceased?

Check for
deceased
veterans

Not deceased?

Missing?
Exclude veteran
from population file

Not missing?

Check for
missing stratum
data

Previously
queried?

Assign correct
unique veteran
identifiers

Previously
sampled?

Check for
veterans
sampled for
MTM in past 13
weeks

Check for
previously queried
veterans

Not previously
queried?

Previously
sampled?

Flag for quarantine

Check for most
recent APGsampled veterans

Not previously
sampled?

Assign new unique
veteran identifier

Flag for quarantine
Not previously
sampled?

Create
encounter
identifiers and
required variables

Final population file

To reduce respondent burden, the frame will undergo a screening for eligibility process on a
weekly basis. Veterans also have the option to opt-out of completing the current and all future
surveys by requesting to be put on a “Do Not Contact” list stored in Medallia.
Thus, veterans will be quarantined from sampling based on the following rules:
•
•
•
•

Rule 1: Veterans will be excluded from sampling for any transactions if they were sampled for
another MTM survey or the APG survey in the past 30 days.
Rule 2: Veterans will be excluded in a weekly sample if they were also sampled in the previous week.
Rule 3: Veterans will be excluded from sampling within a transaction if they are sampled for another
transaction in the current round of sampling.
Rule 4: Veterans will be excluded from sampling for a transaction if they requested to be put on a
“Do Not Contact” list in the MTM Survey for that transaction.

The quarantine rules will be applied during the creation of the sampling population frame
updates on a weekly basis. The quarantined records will be examined during the weekly frame
creation to determine if the rules are correlated with demographic strata and the excluded records
could bias the final estimate. The distribution of the quarantine records will be compared to the
population distribution to identify if there are potential concerns of biasing factors that might
impact the analysis and results. Veterans are reintroduced to the sampling pool after their
corresponding quarantine period is over unless they had opted out from all future VA surveys.
15

It is important to note as well the variables and information needed in support of the sampling
process. These variables include:
•

•
•
•
•

•
•
•
•
•
•
•

A unique veteran identifier. This identifier will be maintained over all transactions a
veteran conduct.
A unique encounter identifier must be created.
A variable will be created that identifies each record based on its line of business.
A variable will be created that identifies each record based on its type of transaction
(i.e., appointment, outpatient, labs, pharmacy, etc.).
Another variable will be created that identifies the type of encounter within a
transaction (i.e., same-day appointment, cardiology visit, laboratory follow-up visit,
etc.).
A variable that identifies the sampling population in which the sampling veteran
belongs.
A variable that identifies the encounter within the sampling population that the veteran
was selected for.
A date of transaction variable needs to be created for all records.
A certainty stratum identifier.
A flag that identifies records not eligible for sampling by reason for ineligibility
(previously sampled, future transaction)
The age group variable
The gender variable

B. Quality Control

With the access to the list of necessary variables for supporting the sampling process, the
weekly sampling frames will be examined to understand if the frame distribution consistently
resembles the population distribution. The frame distribution will also be compared against the
distribution of the quarantine records to identify if any demographic and geographic strata that
may introduce biases into the weekly sampling frames. These quality control procedures will be
carried out at both veteran and transaction levels: veteran-level quality control, transaction level
quality control, and system-wide quality control

C. Sampling Design
The sampling design is a complex systematic sample design with stratification and
clusters across different stages (Figure 3). This design will allow the sample to achieve
cumulative representation of the veterans across all transactions and encounters under
investigation (currently outpatient services out of VBA). The sample will be selected across
multiple steps as follows:
Figure 3. Stratified Cluster Design

16

Transactions

Veterans

Primary
Sampling
Units

Secondary
Sampling
Units

Line of Business
Transaction
Encounter
Location - 3 Digit Station

Proportional Allocation
Using Explicit Strata

State
Age Group
Gender

Proportion Allocation
using Systematic
Sampling

1. Stage 1: Sampling Transactions
a. To reduce respondent burden, the sample is de-duplicated to keep only one
encounter per veteran. The deduplication process will be conducted in a
fashion that allows balance across encounters and locations.
b. Transactions are then grouped by line of business 1, encounters, and location
(3 digit stations). These groupings are referred to as strata.
c. Encounters will be sampled within location with a minimum of 10 encounters
per location for analytical purposes, i.e. to get enough encounters per location
for reporting and informing decision makers.

2. Stage 2: Sampling Veterans

At this point, we are only focusing on the outpatient service as the main line of business. However, in the future
additional lines of businesses will be included. This is the main reasoning behind including line of business as a
stratification/grouping variable.

1

17

a. Veterans are sorted by state, age group and gender 2. This ordering allows the
sample taken from each stratum to represent each of the three groups according to
the distribution of the email population.
b. A sample of veterans is selected where veterans are the primary sampling units.
The sample is proportionally allocated to get a suitable sample size within each
group to match the veteran email population distribution.
To avoid a more complex design, any other variables of interest that need to be accounted for
will be treated as analytical filters and used as implicit stratification variables during the analysis
in the future.
Table 2 below shows an example using a hypothetical sample of 40 encounters selected from a
email population of 243 encounters using a matrix design. Record counts and proportions (shown
in parentheses) are shown for both the email population and the sample. The distribution of each
cell within the population matrix is used to determine the sample distribution. Note that the
sample cell proportions follow the population distribution.

2

State, gender and age group are treated as implicit strata within the sampling design.

18

Encounter
Transaction 1
Transaction 2
Transaction 3
Transaction 4

Table 2. A hypothetical matrix sample design of encounters.
Counts and overall proportions (in parentheses) are shown.
Population
Sample
Small
Med.
Large
Small
Med.
Large
Facility Facility Facility
N/A
Facility Facility Facility
5
20
60
0
1
3
10
(0.021)
(0.082) (0.247) (0.000)
(0.025)
(0.075) (0.250)
0
10
30
0
0
2
5
(0.000)
(0.041) (0.123) (0.000)
(0.000)
(0.050) (0.125)
0
0
0
100
0
0
0
(0.000)
(0.000) (0.000) (0.412)
(0.000)
(0.000) (0.000)
5
6
7
0
1
1
1
(0.021)
(0.025) (0.029) (0.000)
(0.025)
(0.025) (0.025)

N/A
0
(0.000)
0
(0.000)
16
(0.400)
0
(0.000)

In the case of the MTM surveys, encounters within the sampled veterans are selected
based on the joint distribution of the population between the type of encounter and the size of the
facility. Facilities are classified into small, medium and large facilities based on the number of
encounters that facility handles over the previous year. The facility size will not change between
weekly samples. Encounters that are not tied to a physical location are classified as ‘N/A’. A
new matrix distribution based on the sample frame will be calculated for each weekly sample.
Sampled veterans will then be stratified into two groups, veterans that have exactly one
encounter in a week and veterans that have more than one encounter each week. Veterans that
have exactly one encounter are assigned to the survey to which that encounter belongs. Veterans
that have more than one encounter will be randomly assigned to an encounter based on the
population distribution of encounters to assure sample representation and balance.
D. Sample Size

The full population was analyzed to determine a workable number of records to be sampled
each week. Due to the fact some strata may not contain enough records to be sampled, the number of
records for obtaining 10 responses per 6-digit station by encounter stratum was calculated. Additionally,
the targeted sample sizes for the 3-digit station by encounter strata are included for reference. Table 3
below shows the desired number of records for each sample design.
The final transaction-level strata are 3-digit station and encounter. The 6-digit station stratum
cannot sustain successive weeks of sampling. However, each 6-digit station is mapped to the parent 3digit station by the first three digits in the station code. Furthermore, both station code variables are
included on the Medallia invitation file, thereby allowing the potential of creating estimates at the 6digit station level.

Table 3. Population and Sample Sizes

3-digit station

6-digit station
19

Total Number of
facility*encounter strata across
5,364
20,028
all transactions
Required total weekly sample size (number of veterans)
30 responses per month
452,022
1,687,753
per stratum
10 responses per month
150,674
562,584
per stratum
Percent of facility*encounter strata whose weekly population counts are
less than the target weekly stratum sample size
30 responses per month
35.6%
64.6%
10 responses per month
16.1%
46.9%
Percent of facility*encounter strata whose monthly population counts
are less than the target weekly stratum sample size
30 responses per month
12.9%
42.8%
per stratum
10 responses per month
6.7%
30.8%
per stratum
Table 4 shows the stratification level of the 3-digit station field. Table 5 shows the stratification level of
the encounter field.
Table 4. 3-digit station strata for each survey.

Table 5. Encounter strata for each survey.

Survey
Appts
Appts
Appts
Appts
Labs/Imaging
Pharmacy
Pharmacy
Pharmacy
Visit
Visit
Visit
Visit
Visit
Visit

Encounter
Regular appointments
Same-day requested and received
Same-day requested and did not receive
Walk-in
Labs/Imaging
CMOP
Mailed
Window
Allergy & Immunology
Alternative
Anesthesia
Cardiology
Dermatology
Dialysis
20

Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit
Visit

EEG/Neurology
Endocrine/Metabolic and Diabetes
Gastroenterology
General Medicine
General Surgery
Geriatric Medicine
Hematology/Oncology
Infectious Disease
Medical - All Other
Mental Health - Mental Health Clinic
Mental Health - MHSDP Homeless
Program
Mental Health - MHSDP Methadone
Treatment Program
Mental Health - Substance Abuse Clinic
MHSDP MHICM Program
Nephrology
Nuclear Medicine
Primary Care
Primary Care - Nurse/PA
Prosthetics
Psychosocial Rehabilitation and
Recovery Centers
Pulmonary/ Respiratory Disease
Radiation Therapy
Radiology
Recreational Therapy
Rehab Medicine
Rheumatology
Surgery - Cardiovascular and Thoracic
Surgery
Surgery - Colon Rectal Surgery
Surgery - ENT
Surgery - Eye Clinic
Surgery - General and All Other Surgery
Surgery - Neurological Surgery
Surgery - Obstetrics & Gynecology
Surgery - Orthopedics
Surgery - Plastic Surgery
21

Visit
Visit
Visit

Surgery - Podiatry
Surgery - Urology
Urgent Care

The sample will be analyzed to assess coverage and precision by examining the distributions and
design effects within each stratum as compared to the true and sampling population. Such reports can
be built into the sampling strategy.

E. Survey Invitation File
The sampled records will be compiled by the VA into a file called a survey invitation file
that can be uploaded by Medallia. Medallia software will use this survey invitation file to send
the veteran survey invitations, maintain the interface for the web survey, capture the data, and
calculate estimates to be presented under the embedded dashboard tool. All variables needed by
Medallia will be created and appended onto the survey invitation file after the sample is created.
Part IV: Survey Administration
On a semi-weekly basis, VEO provides the Medallia operations team with a stratified
sample identifying the veterans who will receive a survey. Using this information, Medallia will
distribute surveys to the identified sample of veterans. Service-Level Measurement (SLM)
surveys are released semi-weekly, and will allow the veteran to provide feedback anytime over
the course of two weeks. Therefore, VEO will contact Veterans 3-4 days after their interaction
with VHA. Each weekly survey will have two waves of data collections, beginning on Tuesday
with the second launch on Friday. Each wave will be independently sampled, according to the
half the sample size targets described in this document. Veterans have two weeks to provide
feedback; however, as soon as they complete their survey, their response data is immediately
available for review within Medallia.
Part V: Analysis and Dashboarding
The sampling design allows for the unit of analysis to be either the veteran or the
encounter. The analysis will be mostly model based rather than design based (no weights
calculated and only unweighted analysis will be reported).
The dashboards are intended to give VA employees insights into operation improvements
to be made at the local level and patterns occurring at the national and regional level to allow
employees to make strategic decisions. Dashboards display data from the survey scores and
allow users to view the scores by the different strata: age, gender, location. Depending on the VA
employee’s role, the user will have access to different information.
Users should be able to filter information down to the level of the service name if there are
30 or more comments for that service. Users can see score data for each individual survey and a
combined trust score from all three surveys. Dashboards will also give VE admins the
opportunity to monitor the health of their survey campaign and email campaign by displaying
analytics data on both surveys and emails.
Part VI: Limitations and Mitigations
This section highlights various challenges and how these challenges are mitigated by the
current sampling design.
Sequencing and Conditionality

Transactions are defined within each VA line of business. Therefore, transactions may be
conditional on other transactions. Given the rules for removing veterans from the sampling
pool, this conditionality will affect how records are sampled. For example, a veteran may

22

make an appointment for a set of laboratory tests the following day as an outpatient visit at
a VA hospital. At that visit, he or she is also written a prescription that he or she then fills at
a VA pharmacy. In this scenario, the veteran’s visit transaction only happens after the
appointment transaction, and the pharmacy transaction happens after both transactions.
Given that the appointment happens before both the outpatient visit and the pharmacy
visit, the sampling probabilities would otherwise be conditional for the transactions that
take place after the appointment and sequentially. This excludes a census approach
because, under a census approach, all records in the situation mentioned above are selected
for the appointment and none of them will be eligible for the outpatient visit or pharmacy
visit surveys. This also introduces a complication in the stratified sample design in that
receiving or completing the survey becomes a stratification variable.
Thus, to mitigate the issues of conditionality, the sampling methodology includes the
veteran into the sampling pools for all transactions. Then, the current matrix-based
sampling plan allows a balanced sample across all transactions and encounters without
being restricted by the chronological order of the events. For instance, a veteran in the
situation above would be included in the sampling pools for all three transactions and would
then be randomly assigned to only one of the surveys based on the types of transaction they
had.

Compound Transaction

An individual transaction may be made up of a set of encounters, each of which may
have its own survey. For example, a veteran may be directed to have a radiology test and a
bloodwork test in one outpatient visit. And with each test being handled by different
departments in a VA hospital, these tests are considered as individual transactions
separated from one and another. The above sample plan will effectively address the
compound issue by assigning a veteran to only one transaction among the three (i.e.
healthcare visit, radiology test, and bloodwork).

23

Appendix A. Services by VA Administration

24

Appendix B. Eligible Stop Code List.
Service Name
Allergy & Immunology

Sub Services

Inclusion
Status

302

Alternative

159

Anesthesia

EXCLUDE

427

Cardiology

107
231
303
311
317
333
334
369

Dental

180

Dermatology

304

Dialysis

602
603
604
606
607
608

Emergency Department
Endocrinology

Stop Codes

130
Endocrine/Metabolic
and Diabetes

Eye Clinic Ophthalmology/Optometry

305
306
438

Gastroenterology

25

EXCLUDE

EXCLUDE

Digestive/GI/Endoscopy

General Medicine

307
321
339
340
349

Geriatric Medicine

190
318
319
320
351
352
353

Hematology/Oncology

308
316
330
431

Infectious Disease

EXCLUDE

EXCLUDE

EXCLUDE

310
717

Laboratory & Pathology

Medical - All Other

Mental Health

26

108
111

EXCLUDE

329
394
420
331
328
332
336

EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE

504
507
508
511

EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE

Mental Health - Day
Treatment Center

Mental Health - Mental
Health Clinic

27

534
555
556
586
587
591
592
593
595
596
506
599
554

EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE

EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE

502
509
512
519
524
525
531
533
538
540
562
565
580
516
550
566
572
557
561
564
571

EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE

Mental Health MHSDP Community
MH Residential Care

Mental Health MHSDP Homeless
Program

Mental Health MHSDP Methadone
Treatment Program
Mental Health MHSDP Work Therapy
Program

Mental Health Psychology

Mental Health Substance Abuse Clinic

MHSDP MHICM
Program
28

121
503
505
584
583
553
582

529
522
590

EXCLUDE

EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE

EXCLUDE
EXCLUDE

523

222
535
568
573
574
575

EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE

510
558

EXCLUDE

513
514
548
547
560

EXCLUDE
EXCLUDE

552
567
Nephrology

EXCLUDE

313

Neurology

215
225
346
347
EEG/Neurology

Nuclear Medicine

EXCLUDE

106
126
128
155
315
335
345
109
146
158
145
No Codes
Identified

Nursing
Pharmacy

EXCLUDE
EXCLUDE

EXCLUDE

160
Primary Care

120
123
124
142
301
309
322
323
341
342
348
350
371
29

EXCLUDE
EXCLUDE

EXCLUDE
EXCLUDE

372
373
450
690
701
704
706
707
710
999
Primary Care Nurse/PA
Prosthetics

104
116
312

Pulmonary/ Respiratory
Disease

481

Radiation Therapy

144
149

Radiology

105
110
115
127
150
151
153
154

30

EXCLUDE
EXCLUDE

117
588
440
449
598

Psychosocial
Rehabilitation and
Recovery Centers

EXCLUDE
EXCLUDE

EXCLUDE
EXCLUDE

703
202
197
Recreational Therapy

201
205

Rehab Medicine

195
196
206
209
210
211
212
214
217
218
220
230
240
250
417
418
423
436
437
439
213
207
208
198

Rheumatology

314

Surgery

434
Surgery - Audiology

31

203

EXCLUDE

EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE

Surgery Cardiovascular and
Thoracic Surgery

Surgery - Colon Rectal
Surgery
Surgery - ENT
Surgery - Eye Clinic

Surgery - General and
All Other Surgery

Surgery - Neurological
Surgery
Surgery - Obstetrics &
Gynecology

Surgery - Orthopedics

Surgery - Plastic
Surgery

32

204

402
413
415
421
412
402
407
408
718
327
401
416
419
429
432
433
435
457
406
404
426
405
409
422

410
Surgery - Podiatry
Surgery - Urology

Urgent Care
Not Mapped

411
414
430
102

EXCLUDE

651
658
669
131
103
118

EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE

119

HomeBased
Post
Deployment
Integrated
Care
EXCLUDE
EXCLUDE
Sleep Study
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE

135
136
137
143
147
148
156
157
162
165
166
167
168
169
170
171
172
173
174
175
33

176
177
178
179
181
182
183
184
185
186
187
188
189
191
199
216
221
224
229
295
296
297
324
325
326
338
354
370
424
425
428
441
443
444
445
446
447
448
34

EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
Care/Case
Manager
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
Long Term
Care
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE

474
490
491
527
528
530
536
542
545
546
579
597
611
644
645
646
647
648
649
652
653
656
673
674
680
681
682
683
684
685
686
692
693
694
695
696
697
698
699

35

EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE

708
713
714
719
720
801
802
803
901

36

Telehealth
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE
EXCLUDE