Survey Sample Plan

Status: Original and derived artifacts are available for this document.

Download: docx | pdf | html

Service Level Measurements:  Outpatient 
Sampling Methodology Report

June 2018
Contents
Part I: Introduction	3
A. Background	3
B. Moments that Matter Survey Series	4
C. Basic Definitions	5
Part II: Methodology	5
A.	Survey Instrument	7
Part III: Sample Design	10
A.  Target Population and Frame	10
B.	Quality Control	13
C.  Sampling Design	13
D.   Sample Size	16
E.  Survey Invitation File	19
Part IV: Survey Administration	19
Part V: Analysis and Dashboarding	19
Part VI: Limitations and Mitigations	20
Sequencing and Conditionality	20
Compound Transaction	20
Appendix A. Services by VA Administration	21
Appendix B. Eligible Stop Code List.	22




Part I: Introduction
A. Background 
When veterans interact with VA, they expect the highest quality of care and service, feeling that they are honored, respected, and their needs are met. In 2016, the Enterprise Management and Performance Improvement (EM&PI) team at the VEO created a model for measurement to hold VA accountable to veterans to fulfill those goals. 
As a program, VA’s service level measurements are based on the most impactful interactions veterans have with VA–the moments that matter. Veterans were asked to provide feedback using a “push and pull” method. Unsolicited feedback is pulled from veterans through the web and social media. Email surveys are pushed to veterans after they have had an interaction with the VA that’s been determined to be a moment that mattered. Surveys are pushed to veterans digitally to capture continuous and real-time information about a veteran’s experience. Moments that matter take the customer transactions and customer journey into consideration, setting customer experience metrics based on customer experience drivers and detractors. For the first time, VA is able to capture veteran feedback continuously, in real-time and to amplify the voice of the veteran to the right people. 
To meaningfully measure a veteran’s experience across each transaction along the customer journey, a human-centered approach was used to design the surveys. The survey instrument was created based on eighty-four interviews in nine locations across the country with veterans to understand which moments mattered and what is measurable. The surveys were then vetted again by fourteen veterans to prove comprehension, relevance, and measurable action.  
As a program, we measure customer experience at three different levels: the enterprise level, the service level patterns, and point-of-service feedback. The focus of this report is on the second level – service level patterns.
The goal of service level measurements is three-fold:
    1) To collect continuous customer experience data at moments that make or break the service experience 
    2) To help field staff and the national office identify areas for improvement 
    3) To understand emerging drivers and detractors of customer experience. 
This measurement model brings insights and value to all stakeholders at VA. After every moment that matters transaction, veterans are asked about their experience. Front-line VA leaders can resolve individual feedback from veterans and take steps to improve the customer experience for others; additionally, VA executives receive real-time updates on systematic trends that allow them to make changes. VEO intends to identify CX drivers and set measures for all lines of business across all of VA’s agencies.
To achieve this goal, we measure transactions occurring within a service line and, where appropriate, within a specific VA facility across veterans and across time. Service-level patterns complement our enterprise trust trend data by providing actionable service-level understanding of the issues that are driving the customer satisfaction and sentiment.  Ultimately, they allow us to identify performance improvements.
The purpose of this document is to define VA’s sampling methodology for veteran’s interactions at the service-level in receiving VA care and/or services.  This approach defines the overarching sampling principals applicable for all service-level measurement surveys. The sampling design aims to allow decisions makers at different levels to make inference per agency, across veterans, and across transactions per line of business (LOB). The document provides a highlight of the overall strategy along with the business rules to operationalize the process with focus on the outpatient LOB as a starting point.

B. Moments that Matter Survey Series
Each line of business consists of a customer journey which can be broken out into key customer transactions. These transactions consist of interactions that represent what customers perceive to be part of the journey; within these journeys, a set of selected experiences are considered for measurement—the moments that matter to the veterans. 
The first line of business to be sampled is the Outpatient Journey with VHA. Within the Outpatient Journey, our measurable moments that matter were identified. Currently, these experiences are grouped into the following four points in time related to VHA Outpatient Service (as shown in Table 1). This is a starting point and will expand to cover key experiences in all services across all VA agencies (Appendix A. shows the veteran experience cycle).
Table 1. The Outpatient Moments that Matter
Moment that Matters
Description
What is covered
Exclusions
Appointment
Appointments made for a veteran’s outpatient healthcare visit to a VA facility
VA-initiated appointment or veteran-initiated appointments
All Stop Codes related to Telephone, Telehealth, Compensation & Pension, Home Based Care are excluded for the initial release of the Outpatient surveys.

Refer to Appendix B for a comprehensive list of excluded Stop Codes.




Same-day walk-in veterans (the veteran sets up the appointment at the same time as they visited the facility)


Visit
The actual veteran outpatient healthcare visit, which could involve multiple encounters during a single visit.  


Encounters within the Visit survey are defined by a set of stop codes within the VHA databases.  These codes and their descriptions are outlined in Appendix B.
Refer to Appendix B for a comprehensive list of excluded Stop Codes.
Lab/Imaging
The follow-up visits requested during the outpatient healthcare visit.
Encounters within the Lab/Imaging survey are defined by a set of stop codes within the VHA databases.  These codes and their descriptions are outlined in Appendix A.
Refer to Appendix B for a comprehensive list of excluded Stop Codes.

Pharmacy
Prescriptions obtained from VA pharmacies
Mail-order and in-person VA pharmacy interactions.



Initiated by the veteran or initiated by the VA.


C. Basic Definitions 
In order to avoid confusion, we need to differentiate between a transaction and an encounter.

    • A transaction. A transaction refers to the specific time a veteran interacts with the VA that both impact the veteran’s journey and the veteran’s perception of VA’s effectiveness in caring for veterans.  
    • An Encounter.  The veteran can participate in multiple actions within a transaction.  Each of these actions is defined as an encounter and is transaction-specific.  
Therefore, multiple encounters are possible within a transaction and across transactions -- e.g. an endocrinologist visit is one of many encounters within the main healthcare visits for attending a veteran’s needs or a cardiologist visit, lab work, and pharmacy service.
Part II: Methodology
VA is working to transform into the number one customer service agency in the federal government through an in-depth understanding of veterans’ experiences from their point of view. To create a successful customer experience measurement program, we begin with the customer. For each line of business measured through this effort, we start by describing the customer segments to be measured, identifying the differences and similarities in customer journeys, and leveraging quantitative research from the existing data channels to understand customer pain points and business process. We went into the field and used human-centered design research techniques including 1-on-1 interviews, field observations, and contextual inquiry to understand the journey from a veteran’s point of view and identify what drivers the customer experiences at each step. We call the most impactful moments in the journey the moments that matter. Our design research combined with quantitative analysis and stakeholder interviews helped us to identify themes and insights into what moments matter and what to measure across the key customer journey. This is seen on the journey map created as an artifact to visually show the veteran’s customer experience (Figure 1. Outpatient Journey Map).
In order to create a successful transformation, impact was measured through three primary dimensions:
            ▪ Effectiveness – “I got the service I needed.”
            ▪ Ease – “It was easy to get what I needed.”
            ▪ Emotion – “I felt like a valued customer.”

After identifying drivers of the customer experience at specific moments in the journey, we determined how we can trigger the survey through backend data. Once information about customer experiences was gathered over the course of the service, we created a survey framework that included sampling and content guidelines. This outlined key customer satisfaction drivers for the journey being assessed and the journey’s key impact metrics. These measures vary in importance across the journey. Effectiveness, ease, and emotion are present at every moment in the veterans’ journey, including the Outpatient Experience. The graphic below shows how our survey questions and surveys mapped back to the moments that matter to veterans.
Figure 1. Outpatient Journey Map


The Service Level Measurement Surveys are transactional surveys that identify which measures are the most effective at certain points along the journey and are critical for us to meet veterans’ expectations. They are designed to have minimal burden on customers and ask about an interaction the customers had with the line of business. The invitation to the surveys is sent out by email directing the veteran to a web survey that allows for near real-time feedback. The structure of the surveys covers: the five customer satisfaction driver questions in chronological order, an impact metric question, a problem resolution/follow up question, a final comment question. Appendix D shows a copy of the outpatient experience questionnaire where the questions cover: 
a. Relevance to the customer’s experience 
b. Actionable 
c. Based on business objectives
d. Part of VA’s brand

    A.  Survey Instrument
A.1.  Questionnaire Design
The overall process for instrument design was based on four steps as follows:
    1. Discovery: Interviews with veterans, employees, and business stakeholders to identify important aspects and feedback needed to refine the questionnaire design
    2. Journey Mapping: Synthesis of veterans’ interviews and observations by identifying themes and insights into a measurable journey map
    3. Survey Framework: VEO defines which factor is most important at each moment and what good looks like. We decide whether to focus on ease, effectiveness, or emotion. We define the audience, sampling rules, and business rules.
    4. Writing Surveys: We write easy to understand questions that measure the key moments that matter based on the research with veterans and industry best practices. 
For the Outpatient Experience, VEO and VHA started with a discovery ‘sprint’ to explore patient experiences at the VA -- interviewing veterans across multiple geographic locations and of various genders, races, life stages, military branches, and eras of service. A total of 84 veterans were interviewed across nine research locations for a total of 150 hours of interviews. Figure 2 below shows the distribution of the locations across states covered. 














Figure 2. Design Phase Interviews


The research resulted in a journey map which represented the end-to-end patient experience as veterans seek and receive outpatient care. The map is organized into 5 phases: before the visit, arriving at the facility, during the appointment, departing the facility, and after the visit. Each phase included key moments that matter, bright spots, paint points, entry/exit points, and interaction channels. 
A.2.  Pre-testing of the survey instrument
After the questionnaire design stage was completed, the survey validation process was initiated. This validation step consisted of question comprehension testing with veterans, usability testing of the survey flow, and validation with business stakeholders. 
Pre-testing of the Outpatient Journey Surveys was conducted by the VEO, working with the Midwest District and Field Office to recruit veterans from the Detroit and Ann Arbor area. Two facilities were chosen within a close proximity that had different populations and different facility ratings. The VEO completed two rounds of in-person small-scale iterative usability testing with veterans who represented a diverse set of demographics. Each interview totaled 45 minutes. A total of 14 veterans were included in the testing phase at Ann Arbor VA Healthcare System and John D. Dingall Medical Center in Detroit, Michigan. In addition to testing with veterans, the survey was validated with 34 staff members across the two facilities. Table 3 below shows the veteran small-scale usability testing demographic distribution for outpatient experience surveys.

Table 3. Demographic distributions of Veterans Covered in the Pre-test
Demographic
Number
Gender

Male
11
Female
3
Branch of Service/Duty

Army
5
Army, National Guard, Reserves
1
Navy
2
Airforce
4
Marines
1
Unrecorded
1
Age Group

<30
0
30-39
4
40-49
0
50-59
1
>=60
8
N/A

Race

Black or African American
5
White, Native Hawaiian or other Pacific Islander
1
White
8
Healthcare Mix

VA
4
VA, Medicare
3
VA, Private Through Employer Plan, Medicare
2
VA, Private Through Employer Plan
3
VA, Tricare though National Guard
1
VA, Tricare
1
Usability tests on four surveys and one example email invitation were conducted between March 6-10, 2017: 
    • March 6th: Veterans were interviewed
    • March 7th: Staff from Primary Care, Specialty Care, Labs, Patient Advocates, and the Medical Center Director were interviewed
    • March 8th: Veterans were interviewed
The feedback from these sessions was used to make questions more specific and clear and led to the decision to create a separate survey for labs. Additionally, we mapped our questions back to CAPHS and SHEP Patient Satisfaction Survey questions. On March 8th and 9th we spoke with more veterans and staff and used that feedback to create our final list of surveys and questions. Ultimately, we ended up with four different surveys.
Finally, to test the user experience, a paper prototype was created and simulated the flow of the survey. A card sort exercise was conducted to allow participants to prioritize questions on the Healthcare Survey, and paper prototypes were used to test comprehension. 
Part III: Sample Design
Our sampling approach is based on two key considerations:
    • Limiting Bias: This is achieved by avoiding over-representing veterans who have a larger number of VA encounters.
    • Limiting the impact of quarantine rules used for reducing burden on respondents

To achieve these goals, careful design of the sampling methodology is needed to achieve a full representation of the population of veterans and transactions while reducing the respondent burden to the minimal.  Efforts took place in parallel to cover the following: 
    1. Sampling: This effort covers both the frame and the sampling design.
        a. Frame construction and maintenance: 
            i. This effort is focused on constructing a comprehensive dataset of all transactions under considerations taking place during the targeted timeline. It also accounts for multiple transactions per veteran. 
            ii. The frame maintenance consists of updating transactions on a rolling basis and identifying/updating quarantine rules to reduce potential respondent burden across VA-Wide and MTM rounds.
        b. Sampling Design: The goal of this effort is to design a scheme for selecting veterans and transactions from the frame towards a representative sample of customer receiving VA services across all services.
    2. Analysis plan and Dashboard: This effort ensures that the output of the survey caters to all types of decision makers from the executive level to the operational level, also known as role-based dashboarding. The aim is to achieve monthly analytical reports to reflect issues or concerns on a regular basis. Furthermore, it covers measures taken to preserve veterans’ confidentiality by reducing disclosure risk.
    3. Operations: This effort is focused on survey implementation logistics and requirements to allow smooth operations of the sampling, data collections, and analysis processes within the Medallia environment over time. 
    4. Risk Assessment and Quality Control: This effort covers expected challenges and discusses potential mitigation plans. It also covers quality protocols used to reduce risk.
A.  Target Population and Frame
The target population for the surveys consists of veterans and their corresponding encounters with VHA outpatient services. The current focus is on veterans who are seeking outpatient health services from VA facilities across the different encounters highlighted in Table 1.  The primary sampling units are veterans and the secondary sampling units are the encounters with veterans. Thus, the data can be analyzed to represent either veterans or encounters as the unit of analysis. 
It is important to note that each veteran is represented by a cluster of transactions and encounters in the design of this survey. Each transaction and encounter will be identified from the existing VHA databases to represent veterans’ experience. Some of the encounters within the transactions are similar and have been aggregated into groupings of encounters (see Appendix B covering stop codes). In other words, a given data entry will contain a unique record of each encounter a veteran had with the VA during that week as well as a unique identifier for each veteran. Therefore, the population file will contain multiple records for each veteran. 
The population file is created on a weekly basis every Saturday night from queries to the VHA databases for all outpatient encounters in the previous week.  All outpatient encounters for the week prior to sampling, with the exception of mail-order pharmacy encounters, will be queried from the VHA databases.  Mail-order pharmacy encounters in the period of eleven days and four days prior to sampling are queried for the current sample.  VEO will extract data on veterans who received outpatient services directly from VHA’s corporate data warehouse during the targeted week. Once the data are pulled by the VEO team, the following quality control checks will be executed:
        ◦ Identification of missing data in variables required for later merging and sampling
        ◦ Identification of duplicate records per encounter
        ◦ Deceased veterans for removal from the VHA files prior to concatenation
Constructing and cleaning of the frame, i.e. the population file, will require several steps as shown in Figure 2.
Figure 2. Flowchart Frame Construction

To reduce respondent burden, the frame will undergo a screening for eligibility process on a weekly basis. Veterans also have the option to opt-out of completing the current and all future surveys by requesting to be put on a “Do Not Contact” list stored in Medallia. 
Thus, veterans will be quarantined from sampling based on the following rules:
    • Rule 1: Veterans will be excluded from sampling for any transactions if they were sampled for another MTM survey or the APG survey in the past 30 days.
    • Rule 2: Veterans will be excluded in a weekly sample if they were also sampled in the previous week.
    • Rule 3: Veterans will be excluded from sampling within a transaction if they are sampled for another transaction in the current round of sampling.  
    • Rule 4: Veterans will be excluded from sampling for a transaction if they requested to be put on a “Do Not Contact” list in the MTM Survey for that transaction. 
The quarantine rules will be applied during the creation of the sampling population frame updates on a weekly basis. The quarantined records will be examined during the weekly frame creation to determine if the rules are correlated with demographic strata and the excluded records could bias the final estimate. The distribution of the quarantine records will be compared to the population distribution to identify if there are potential concerns of biasing factors that might impact the analysis and results.  Veterans are reintroduced to the sampling pool after their corresponding quarantine period is over unless they had opted out from all future VA surveys.  
It is important to note as well the variables and information needed in support of the sampling process. These variables include: 
    •  A unique veteran identifier.  This identifier will be maintained over all transactions a veteran conduct.  
    • A unique encounter identifier must be created.  
    • A variable will be created that identifies each record based on its line of business.
    • A variable will be created that identifies each record based on its type of transaction (i.e., appointment, outpatient, labs, pharmacy, etc.).  
    • Another variable will be created that identifies the type of encounter within a transaction (i.e., same-day appointment, cardiology visit, laboratory follow-up visit, etc.). 
    • A variable that identifies the sampling population in which the sampling veteran belongs.
    • A variable that identifies the encounter within the sampling population that the veteran was selected for. 
    • A date of transaction variable needs to be created for all records. 
    • A certainty stratum identifier.
    • A flag that identifies records not eligible for sampling by reason for ineligibility (previously sampled, future transaction)
    • The age group variable
    • The gender variable
    B.  Quality Control
With the access to the list of necessary variables for supporting the sampling process, the weekly sampling frames will be examined to understand if the frame distribution consistently resembles the population distribution. The frame distribution will also be compared against the distribution of the quarantine records to identify if any demographic and geographic strata that may introduce biases into the weekly sampling frames. These quality control procedures will be carried out at both veteran and transaction levels: veteran-level quality control, transaction level quality control, and system-wide quality control

C.  Sampling Design
The sampling design is a complex systematic sample design with stratification and clusters across different stages (Figure 3).  This design will allow the sample to achieve cumulative representation of the veterans across all transactions and encounters under investigation (currently outpatient services out of VBA).  The sample will be selected across multiple steps as follows:

Figure 3. Stratified Cluster Design 

    1. Stage 1: Sampling Transactions
    a. To reduce respondent burden, the sample is de-duplicated to keep only one encounter per veteran. The deduplication process will be conducted in a fashion that allows balance across encounters and locations.  
    b. Transactions are then grouped by line of business1, encounters, and location (3 digit stations). These groupings are referred to as strata.
    c. Encounters will be sampled within location with a minimum of 10 encounters per location for analytical purposes, i.e. to get enough encounters per location for reporting and informing decision makers.  


    2. Stage 2: Sampling Veterans
        a. Veterans are sorted by state, age group and gender2. This ordering allows the sample taken from each stratum to represent each of the three groups according to the distribution of the email population. 
        b. A sample of veterans is selected where veterans are the primary sampling units. The sample is proportionally allocated to get a suitable sample size within each group to match the veteran email population distribution.
To avoid a more complex design, any other variables of interest that need to be accounted for will be treated as analytical filters and used as implicit stratification variables during the analysis in the future.
Table 2 below shows an example using a hypothetical sample of 40 encounters selected from a email population of 243 encounters using a matrix design. Record counts and proportions (shown in parentheses) are shown for both the email population and the sample. The distribution of each cell within the population matrix is used to determine the sample distribution. Note that the sample cell proportions follow the population distribution.  

Table 2.  A hypothetical matrix sample design of encounters.
Counts and overall proportions (in parentheses) are shown.
Encounter
Population

Sample

Small Facility
Med. Facility
Large Facility
N/A

Small Facility
Med. Facility
Large Facility
N/A
Transaction 1
5 
(0.021)
20 
(0.082)
60 
(0.247)
0 
(0.000)

1 
(0.025)
3 
(0.075)
10 
(0.250)
0 
(0.000)
Transaction 2
0 
(0.000)
10 
(0.041)
30 
(0.123)
0 
(0.000)

0 
(0.000)
2 
(0.050)
5 
(0.125)
0 
(0.000)
Transaction 3
0 
(0.000)
0 
(0.000)
0 
(0.000)
100 
(0.412)

0 
(0.000)
0 
(0.000)
0 
(0.000)
16 
(0.400)
Transaction 4
5 
(0.021)
6 
(0.025)
7 
(0.029)
0 
(0.000)

1 
(0.025)
1 
(0.025)
1 
(0.025)
0 
(0.000)

In the case of the MTM surveys, encounters within the sampled veterans are selected based on the joint distribution of the population between the type of encounter and the size of the facility. Facilities are classified into small, medium and large facilities based on the number of encounters that facility handles over the previous year. The facility size will not change between weekly samples.  Encounters that are not tied to a physical location are classified as ‘N/A’. A new matrix distribution based on the sample frame will be calculated for each weekly sample.  
Sampled veterans will then be stratified into two groups, veterans that have exactly one encounter in a week and veterans that have more than one encounter each week. Veterans that have exactly one encounter are assigned to the survey to which that encounter belongs. Veterans that have more than one encounter will be randomly assigned to an encounter based on the population distribution of encounters to assure sample representation and balance. 
D.   Sample Size
The full population was analyzed to determine a workable number of records to be sampled each week. Due to the fact some strata may not contain enough records to be sampled, the number of records for obtaining 10 responses per 6-digit station by encounter stratum was calculated. Additionally, the targeted sample sizes for the 3-digit station by encounter strata are included for reference.  Table 3 below shows the desired number of records for each sample design.

The final transaction-level strata are 3-digit station and encounter.  The 6-digit station stratum cannot sustain successive weeks of sampling.  However, each 6-digit station is mapped to the parent 3-digit station by the first three digits in the station code.  Furthermore, both station code variables are included on the Medallia invitation file, thereby allowing the potential of creating estimates at the 6-digit station level.




Table 3. Population and Sample Sizes

3-digit station
6-digit station
Total Number of facility*encounter strata across all transactions
                   5,364
                   20,028
Required total weekly sample size (number of veterans)
30 responses per month per stratum
              452,022
             1,687,753
10 responses per month per stratum
              150,674
                 562,584
Percent of facility*encounter strata whose weekly population counts are less than the target weekly stratum sample size
30 responses per month
35.6%
64.6%
10 responses per month
16.1%
46.9%
Percent of facility*encounter strata whose monthly population counts are less than the target weekly stratum sample size
30 responses per month per stratum
12.9%
42.8%
10 responses per month per stratum
6.7%
30.8%

Table 4 shows the stratification level of the 3-digit station field.  Table 5 shows the stratification level of the encounter field.

Table 4.  3-digit station strata for each survey.
  

Table 5.  Encounter strata for each survey.
Survey
Encounter
Appts
Regular appointments
Appts
Same-day requested and received
Appts
Same-day requested and did not receive
Appts
Walk-in
Labs/Imaging
Labs/Imaging
Pharmacy
CMOP
Pharmacy
Mailed
Pharmacy
Window
Visit
Allergy & Immunology
Visit
Alternative
Visit
Anesthesia
Visit
Cardiology
Visit
Dermatology
Visit
Dialysis
Visit
EEG/Neurology
Visit
Endocrine/Metabolic and Diabetes
Visit
Gastroenterology
Visit
General Medicine
Visit
General Surgery
Visit
Geriatric Medicine
Visit
Hematology/Oncology
Visit
Infectious Disease
Visit
Medical - All Other
Visit
Mental Health - Mental Health Clinic
Visit
Mental Health - MHSDP Homeless Program
Visit
Mental Health - MHSDP Methadone Treatment Program
Visit
Mental Health - Substance Abuse Clinic
Visit
MHSDP MHICM Program
Visit
Nephrology
Visit
Nuclear Medicine
Visit
Primary Care
Visit
Primary Care - Nurse/PA
Visit
Prosthetics
Visit
Psychosocial Rehabilitation and Recovery Centers
Visit
Pulmonary/ Respiratory Disease
Visit
Radiation Therapy
Visit
Radiology
Visit
Recreational Therapy
Visit
Rehab Medicine
Visit
Rheumatology
Visit
Surgery - Cardiovascular and Thoracic Surgery
Visit
Surgery - Colon Rectal Surgery
Visit
Surgery - ENT
Visit
Surgery - Eye Clinic
Visit
Surgery - General and All Other Surgery
Visit
Surgery - Neurological Surgery
Visit
Surgery - Obstetrics & Gynecology
Visit
Surgery - Orthopedics
Visit
Surgery - Plastic Surgery
Visit
Surgery - Podiatry
Visit
Surgery - Urology
Visit
Urgent Care

The sample will be analyzed to assess coverage and precision by examining the distributions and design effects within each stratum as compared to the true and sampling population. Such reports can be built into the sampling strategy. 
E.  Survey Invitation File 
The sampled records will be compiled by the VA into a file called a survey invitation file that can be uploaded by Medallia. Medallia software will use this survey invitation file to send the veteran survey invitations, maintain the interface for the web survey, capture the data, and calculate estimates to be presented under the embedded dashboard tool. All variables needed by Medallia will be created and appended onto the survey invitation file after the sample is created.
Part IV: Survey Administration
On a semi-weekly basis, VEO provides the Medallia operations team with a stratified sample identifying the veterans who will receive a survey. Using this information, Medallia will distribute surveys to the identified sample of veterans. Service-Level Measurement (SLM) surveys are released semi-weekly, and will allow the veteran to provide feedback anytime over the course of two weeks. Therefore, VEO will contact Veterans 3-4 days after their interaction with VHA. Each weekly survey will have two waves of data collections, beginning on Tuesday with the second launch on Friday. Each wave will be independently sampled, according to the half the sample size targets described in this document. Veterans have two weeks to provide feedback; however, as soon as they complete their survey, their response data is immediately available for review within Medallia.
Part V: Analysis and Dashboarding
	The sampling design allows for the unit of analysis to be either the veteran or the encounter.  The analysis will be mostly model based rather than design based (no weights calculated and only unweighted analysis will be reported). 
The dashboards are intended to give VA employees insights into operation improvements to be made at the local level and patterns occurring at the national and regional level to allow employees to make strategic decisions. Dashboards display data from the survey scores and allow users to view the scores by the different strata: age, gender, location. Depending on the VA employee’s role, the user will have access to different information.
Users should be able to filter information down to the level of the service name if there are 30 or more comments for that service. Users can see score data for each individual survey and a combined trust score from all three surveys. Dashboards will also give VE admins the opportunity to monitor the health of their survey campaign and email campaign by displaying analytics data on both surveys and emails.
Part VI: Limitations and Mitigations
This section highlights various challenges and how these challenges are mitigated by the current sampling design. 
Sequencing and Conditionality
Transactions are defined within each VA line of business. Therefore, transactions may be conditional on other transactions.  Given the rules for removing veterans from the sampling pool, this conditionality will affect how records are sampled.  For example, a veteran may make an appointment for a set of laboratory tests the following day as an outpatient visit at a VA hospital. At that visit, he or she is also written a prescription that he or she then fills at a VA pharmacy. In this scenario, the veteran’s visit transaction only happens after the appointment transaction, and the pharmacy transaction happens after both transactions.  
Given that the appointment happens before both the outpatient visit and the pharmacy visit, the sampling probabilities would otherwise be conditional for the transactions that take place after the appointment and sequentially. This excludes a census approach because, under a census approach, all records in the situation mentioned above are selected for the appointment and none of them will be eligible for the outpatient visit or pharmacy visit surveys. This also introduces a complication in the stratified sample design in that receiving or completing the survey becomes a stratification variable.
Thus, to mitigate the issues of conditionality, the sampling methodology includes the veteran into the sampling pools for all transactions. Then, the current matrix-based sampling plan allows a balanced sample across all transactions and encounters without being restricted by the chronological order of the events. For instance, a veteran in the situation above would be included in the sampling pools for all three transactions and would then be randomly assigned to only one of the surveys based on the types of transaction they had.

Compound Transaction
An individual transaction may be made up of a set of encounters, each of which may have its own survey. For example, a veteran may be directed to have a radiology test and a bloodwork test in one outpatient visit. And with each test being handled by different departments in a VA hospital, these tests are considered as individual transactions separated from one and another. The above sample plan will effectively address the compound issue by assigning a veteran to only one transaction among the three (i.e. healthcare visit, radiology test, and bloodwork).  
	
Appendix A. Services by VA Administration


Appendix B. Eligible Stop Code List.
Service Name
Sub Services
Stop Codes
Inclusion Status
Allergy & Immunology





302

Alternative





159
EXCLUDE
Anesthesia





427

Cardiology





107



231



303



311



317



333



334



369

Dental





180

Dermatology





304

Dialysis





602



603



604
EXCLUDE


606



607



608

Emergency Department





130
EXCLUDE
Endocrinology




Endocrine/Metabolic and Diabetes




305



306

Eye Clinic - Ophthalmology/Optometry





438

Gastroenterology




Digestive/GI/Endoscopy




307



321

General Medicine





339



340



349

Geriatric Medicine





190
EXCLUDE


318



319



320



351
EXCLUDE


352



353

Hematology/Oncology





308



316



330



431
EXCLUDE
Infectious Disease





310



717

Laboratory & Pathology





108



111
EXCLUDE
Medical - All Other





329



394



420



331
EXCLUDE


328
EXCLUDE


332
EXCLUDE


336
EXCLUDE
Mental Health





504
EXCLUDE


507
EXCLUDE


508
EXCLUDE


511
EXCLUDE


534



555
EXCLUDE


556
EXCLUDE


586
EXCLUDE


587
EXCLUDE


591



592



593
EXCLUDE


595
EXCLUDE


596
EXCLUDE


506
EXCLUDE


599
EXCLUDE

Mental Health - Day Treatment Center




554
EXCLUDE





Mental Health - Mental Health Clinic
502



509



512



519



524



525



531



533



538



540



562



565



580



516
EXCLUDE


550
EXCLUDE


566
EXCLUDE


572
EXCLUDE


557
EXCLUDE


561
EXCLUDE


564
EXCLUDE


571
EXCLUDE





Mental Health - MHSDP Community MH Residential Care




121
EXCLUDE


503



505



584
EXCLUDE


583
EXCLUDE


553
EXCLUDE


582
EXCLUDE

Mental Health - MHSDP Homeless Program




529



522
EXCLUDE


590
EXCLUDE

Mental Health - MHSDP Methadone Treatment Program




523


Mental Health - MHSDP Work Therapy Program




222
EXCLUDE


535
EXCLUDE


568
EXCLUDE


573
EXCLUDE


574
EXCLUDE


575
EXCLUDE

Mental Health - Psychology




510



558
EXCLUDE

Mental Health - Substance Abuse Clinic




513



514



548



547
EXCLUDE


560
EXCLUDE

MHSDP MHICM Program




552



567
EXCLUDE
Nephrology





313

Neurology





215
EXCLUDE


225
EXCLUDE


346



347
EXCLUDE

EEG/Neurology




106



126



128



155



315



335



345

Nuclear Medicine





109



146



158



145
EXCLUDE
Nursing

No Codes Identified

Pharmacy





160

Primary Care





120
EXCLUDE


123



124
EXCLUDE


142



301



309



322



323



341



342



348
EXCLUDE


350



371
EXCLUDE


372



373



450
EXCLUDE


690
EXCLUDE


701



704



706



707



710
EXCLUDE


999
EXCLUDE





Primary Care - Nurse/PA




117

Prosthetics





588



440
EXCLUDE


449



598
EXCLUDE
Psychosocial Rehabilitation and Recovery Centers





104



116



312

Pulmonary/ Respiratory Disease





481

Radiation Therapy





144



149

Radiology





105



110



115



127



150



151



153



154



703



202



197





Recreational Therapy





201



205

Rehab Medicine





195



196
EXCLUDE


206



209



210



211



212



214



217



218



220



230



240



250



417



418



423



436



437



439



213
EXCLUDE


207
EXCLUDE


208
EXCLUDE


198
EXCLUDE




Rheumatology





314

Surgery





434


Surgery - Audiology




203



204


Surgery - Cardiovascular and Thoracic Surgery




402



413



415



421


Surgery - Colon Rectal Surgery




412


Surgery - ENT




402


Surgery - Eye Clinic




407



408



718


Surgery - General and All Other Surgery




327



401



416



419



429



432



433



435



457


Surgery - Neurological Surgery




406


Surgery - Obstetrics & Gynecology




404



426


Surgery - Orthopedics




405



409



422


Surgery - Plastic Surgery




410


Surgery - Podiatry




411


Surgery - Urology




414



430

Urgent Care





102
EXCLUDE
Not Mapped





651
EXCLUDE


658
EXCLUDE


669
EXCLUDE


131
EXCLUDE


103
EXCLUDE


118
EXCLUDE


119
Home-Based


135
Post Deployment Integrated Care


136
EXCLUDE


137
EXCLUDE


143
Sleep Study


147
EXCLUDE


148
EXCLUDE


156
EXCLUDE


157
EXCLUDE


162
EXCLUDE


165
EXCLUDE


166
EXCLUDE


167
EXCLUDE


168
EXCLUDE


169
EXCLUDE


170
EXCLUDE


171
EXCLUDE


172
EXCLUDE


173
EXCLUDE


174
EXCLUDE


175
EXCLUDE


176
EXCLUDE


177
EXCLUDE


178
EXCLUDE


179
EXCLUDE


181
EXCLUDE


182
EXCLUDE


183
EXCLUDE


184
Care/Case Manager


185
EXCLUDE


186
EXCLUDE


187
EXCLUDE


188
EXCLUDE


189
EXCLUDE


191
EXCLUDE


199
EXCLUDE


216
EXCLUDE


221
EXCLUDE


224
EXCLUDE


229
EXCLUDE


295
EXCLUDE


296
EXCLUDE


297
EXCLUDE


324
EXCLUDE


325
EXCLUDE


326
EXCLUDE


338
EXCLUDE


354
EXCLUDE


370
Long Term Care


424
EXCLUDE


425
EXCLUDE


428
EXCLUDE


441
EXCLUDE


443
EXCLUDE


444
EXCLUDE


445
EXCLUDE


446
EXCLUDE


447
EXCLUDE


448
EXCLUDE


474
EXCLUDE


490
EXCLUDE


491
EXCLUDE


527
EXCLUDE


528
EXCLUDE


530
EXCLUDE


536
EXCLUDE


542
EXCLUDE


545
EXCLUDE


546
EXCLUDE


579
EXCLUDE


597
EXCLUDE


611
EXCLUDE


644
EXCLUDE


645
EXCLUDE


646
EXCLUDE


647
EXCLUDE


648
EXCLUDE


649
EXCLUDE


652
EXCLUDE


653
EXCLUDE


656
EXCLUDE


673
EXCLUDE


674
EXCLUDE


680
EXCLUDE


681
EXCLUDE


682
EXCLUDE


683
EXCLUDE


684
EXCLUDE


685
EXCLUDE


686
EXCLUDE


692
EXCLUDE


693
EXCLUDE


694
EXCLUDE


695
EXCLUDE


696
EXCLUDE


697
EXCLUDE


698
EXCLUDE


699
EXCLUDE


708
Telehealth


713
EXCLUDE


714
EXCLUDE


719
EXCLUDE


720
EXCLUDE


801
EXCLUDE


802
EXCLUDE


803
EXCLUDE


901
EXCLUDE