Telemedicine

Request for Approval under the “Generic Clearance for Improving
Customer Experience: OMB Circular A-11, Section 280
Implementation”
(OMB Control Number: )
TITLE OF INFORMATION COLLECTION: VAMC or CBOC Appointment Survey
PURPOSE OF COLLECTION:
Telehealth is an effective and convenient way for patients to
receive, and for clinicians to provide quality care management. The
initiative is a way to incorporate telecommunications technology to
improve and modernize health care offered by the Veteran Health
Administration (VHA). In order to assess patient satisfaction with
the program and identify areas for intervention or further
evaluation, the Telehealth Services Office within VHA enlisted the
services of the VEO. The Veteran Telehealth Survey is designed to
measure Customer Experience associated with utilizing VA electronic
health services within the three major aspects, or modalities, of
Telehealth: Clinical Video Telehealth (CVT), Home Telehealth (HT),
and Store and Forward (SFT). The purpose of this report is to
document the survey methodology and sampling plan of the survey.
Information about quality assurance protocols, as well as
limitations of the survey methodology, is included in this report.
Once data collection is completed, the participant responses in the
online survey will be weighted so that the samples will be more
representative of the overall population. Iterative proportional
fitting to create sample weights will be applied using variables:
Modality/Stage, Gender, Age Group (18-39, 40-59, 60+), and
District.
Once the data is collected, it is immediately available in
Vsignals, the Medallia-based platform used by the Veterans
Experience Office for CX data storage and analysis. Survey weights
are incorporated into the system at the close of every weekly
survey. The interface allows data users to analyze the survey
results using interactive charts and sub-populations. Survey data
may also be reviewed over differing time periods, ranging from
weekly, to monthly, to quarterly estimates.
One of the surveys, Telehealth Clinic Appointment: VAMC or CBOC is
updating the language of five questions to reflect feedback from
the field and to lower confusion from survey respondents; these are
phrasing updates and the A-11 Section 280 CX domains remain the
same.

TYPE OF ACTIVITY: (Check one)
[ ] Customer Research (Interview, Focus Groups, Surveys)
[ x] Customer Feedback Survey

[

] Usability Testing of Products or Services

ACTIVITY DETAILS
1. If this is a survey, will the results of this survey be
reported to Touchpoints as part of quarterly reporting
obligations specified in OMB Circular A-11 Section 280?
[ ] Yes
[ x ] No
[ ] Not a survey
2. How will you collect the information? (Check all that apply)
[x ] Web-based or other forms of Social Media
[ ] Telephone
[ ] In-person
[ ] Mail
[ ] Other, Explain
3. Who will you collect the information from?
The target population of the TH survey is all Veterans having an
Outpatient CVT, HT, or SFT event in the past 7 days. The
identification of Telehealth patients utilizes weekly data
extracts from the Corporate Data Warehouse (CDW), which houses
the operational records of VHA. Each Telehealth event eligible
for a VEO survey will be associated with one of these three
modalities. The classification of TH events into a modality is
based on a combination of primary and/or secondary stop codes.
As indicated by VSSC documentation. Under each modality, three
types of surveys are designed to inquire about veterans’
experience in terms of different VA service domains.
Patients scheduling CVT appointments are derived from the
general Outpatient scheduling database table from CDW. When
either an actual CVT or SFT appointment occurs, the distinction
between Offsite appointments (home, mobile, or non-VA
facilities) vs. appointments taking place within a VAMC or CBOC
is based on secondary stop codes. A subset of veterans in each
modality and subtype will be randomly selected to participate in
the survey. However, the subtypes of are sparsely populated so
these will be selected into each weekly sample with certainty.
Telehealth subtypes that are less than 10% of the modality
population will be selected with certainty. In total, there will
be 9 total sets of survey questions.

4. How will you ask a respondent to provide this information?

Patients scheduling CVT appointments are derived from the
general Outpatient scheduling database table from CDW. When
either an actual CVT or SFT appointment occurs, the distinction
between Offsite appointments (home, mobile, or non-VA
facilities) vs. appointments taking place within a VAMC or CBOC
is based on secondary stop codes. A subset of veterans in each
modality and subtype will be randomly selected to participate in
the survey. However, the subtypes of are sparsely populated so
these will be selected into each weekly sample with certainty.
Telehealth subtypes that are less than 10% of the modality
population will be selected with certainty. In total, there will
be 9 total sets of survey questions. Patients will complete
these email invitation-based surveys on a voluntary basis. The
burden times range from 3-5 minutes for completion.
5. What will the activity look like?
Patients will complete these email invitation-based surveys on a
voluntary basis. The burden times average 3 minutes for
completion.
6. Please provide your question list.
See Attached.
7. When will the activity happen?
These are all ongoing surveys in which invitation surveys are
sent out on a weekly basis to telehealth customers.
XXX
8. Is an incentive (e.g., money or reimbursement of expenses,
token of appreciation) provided to participants?
[ ] Yes [x ] No
If Yes, describe:
XXX
BURDEN HOURS
Category of Respondent

No. of
Respondents

Participation
Time

Burden
Hours

Individuals

100,000

2 minutes

3,333 hours
3,333 hours

Totals

100,000

2 minutes

CERTIFICATION:
I certify the following to be true:
1. The collections are voluntary;
2. The collections are low-burden for respondents (based on
considerations of total burden hours or burden-hours per
respondent) and are low-cost for both the respondents and the
Federal Government;
3. The collections are non-controversial;
4. Any collection is targeted to the solicitation of opinions
from respondents who have experience with the program or may
have experience with the program in the near future;
5. Personally identifiable information (PII) is collected only to
the extent necessary and is not retained;
6. Information gathered is intended to be used for general
service improvement and program management purposes
7. Upon agreement between OMB and the agency aggregated data may
be released as part of A-11, Section 280 requirements only on
performance.gov. Summaries of customer research and user
testing activities may be included in public-facing customer
journey maps.
8. Additional release of data will be coordinated with OMB.
Name and email address of person who developed this survey/focus
group/interview:
Name: ___Brian Brown_________
Email address: [email protected]__________
All instruments used to collect information must include:
OMB Control No. XXXX-XXXX
Expiration Date: XX/XX/XXXX

HELP SHEET
(OMB Control Number: XXXX-XXXX)
TITLE OF INFORMATION COLLECTION: Provide the name of the collection that is
the subject of the request. (e.g. Comment card for soliciting feedback on
xxxx)
PURPOSE: Provide a brief description of the purpose of this collection and
how it will be used. If this is part of a larger study or effort, please
include this in your explanation.
TYPE OF COLLECTION: Check one box. If you are requesting approval of other
instruments under the generic, you must complete a form for each instrument.
CERTIFICATION: Please read the certification carefully. If you incorrectly
certify, the collection will be returned as improperly submitted or it will
be disapproved.
Personally Identifiable Information: Agencies should only collect PII to the
extent necessary, and they should only retain PII for the period of time that
is necessary to achieve a specific objective.
BURDEN HOURS:
Category of Respondents: Identify who you expect the respondents to be in
terms of the following categories: (1) Individuals or Households;(2) Private
Sector; (3) State, local, or tribal governments; or (4) Federal Government.
Only one type of respondent can be selected per row.
No. of Respondents: Provide an estimate of the Number of respondents.
Participation Time: Provide an estimate of the amount of time required for a
respondent to participate (e.g. fill out a survey or participate in a focus
group)
Burden: Provide the Annual burden hours: Multiply the Number of responses
and the participation time and divide by 60.

Service Level Measurements: Telehealth
Survey
Sampling Methodology Report
Prepared by
Veteran Experience Office
Version 2 May 2023

Contents
Executive Summary ...................................................................................................................................... 8
Part I – Introduction ...................................................................................................................................... 9
A. Background .......................................................................................................................................... 9
B. Basic Definitions ................................................................................................................................ 10
C. Application to Veterans Affairs ......................................................................................................... 10
Part II – Methodology ................................................................................................................................. 11
A. Target Population and Frame ............................................................................................................ 11
B.

Sample Size Determination............................................................................................................. 11

C.

Stratification.................................................................................................................................... 14

D.

Data Collection Methods ................................................................................................................ 14

E.

Reporting......................................................................................................................................... 15

F.

Quality Control ............................................................................................................................... 15

G.

Sample Weighting, Coverage Bias, and Non-Response Bias ......................................................... 16

H.

Quarantine Rules ............................................................................................................................. 17

Part III – Assumptions and Limitations ...................................................................................................... 19
A. Coverage Bias .................................................................................................................................... 19
C. Other Issues ........................................................................................................................................ 19
Part IV - Appendices ................................................................................................................................... 20
Appendix 1. References ........................................................................................................................ 20

Executive Summary
Telehealth is an effective and convenient way for patients to receive, and for clinicians to
provide quality care management. The initiative is a way to incorporate telecommunications
technology to improve and modernize health care offered by the Veteran Health Administration
(VHA). In order to assess patient satisfaction with the program and identify areas for
intervention or further evaluation, the Telehealth Services Office within VHA enlisted the
services of the VEO. The Veteran Telehealth Survey is designed to measure Customer
Experience associated with utilizing VA electronic health services within the three major
aspects, or modalities, of Telehealth: Clinical Video Telehealth (CVT), Home Telehealth (HT),
and Store and Forward (SFT). The purpose of this report is to document the survey methodology
and sampling plan of the survey. Information about quality assurance protocols, as well as
limitations of the survey methodology, is included in this report.

Part I – Introduction
A. Background
The Enterprise Measurement and Design team (EMD) within the Veterans Experience
Office (VEO) is tasked with conducting transactional surveys of the customer population to
measure their satisfaction with the Department of Veterans Affairs (VA) numerous benefit
services. Thus, their mission is to empower Veterans by rapidly and discreetly collecting
feedback on their interactions with such VA entities as National Cemetery Administration
(NCA), Veterans Health Administration (VHA), and Veterans Benefits Administration (VBA).
VEO surveys generally entail probability samples which only contact minimal numbers of
participants necessary to obtain reliable estimates. This information is subsequently used by
internal stakeholders to monitor, evaluate, and improve processes. Participants are always able to
decline participation and can opt out of future invitations. A quarantine protocol is maintained to
limit the number of times a customer may be contacted over a period of time across all VEO
surveys, in order to prevent survey fatigue.
Surveys issued by EMD are generally brief in nature and present a low amount of burden
on participants. Structured questions directly address the pertinent issues regarding the surveyed
population. The opportunity to volunteer open-ended text responses is provided within most
surveys. This open text has been demonstrated to yield enormous information. Machine learning
tools are used for text classification, ranking by sentiment scores, and screening for
homelessness, depression, etc. Modern survey theory is used to create sample designs which are
representative, statistically sound, and in accordance with OMB guidelines on federal surveys.
VA uses a wide variety of technologies to facilitate quality healthcare to its beneficiaries.
Telehealth services are a critical aspect to modernizing the VA health care system. Telehealth
(TH) increases access to high quality services by using information technology and
telecommunication for Veterans, especially those that live in remote areas or are incapacitated.
In FY 2017, over 700,000 patients received care via the three central telehealth modalities1.
Clinical Video Telehealth (CVT) is the use of real-time interactive video conferencing to
assess, treat and provide patient care remotely. Veterans may be linked to physicians from a local
clinic or even from home, for over 50 clinical applications, ranging from primary care to
numerous specialties (e.g. tele-dermatology). Home Telehealth (HT) is applied to high-risk
Veterans with chronic disease requiring long-term care. Care management is augmented through
such technologies as in-home and mobile monitoring, messaging, and/or video conferencing. The
goal of HT is to reduce complications, hospitalizations, and clinical/ER visitations, so at-risk
patients may remain in their own homes. Finally, Store and Forward Telehealth (SFT)
concerns the acquisition and storage of electronic patient information (e.g., images, sounds, and
video) collected at a VA clinic or medical center. The information is forwarded and retrieved by
healthcare professionals at another VA medical facility where an assessment is performed.
The Veteran Experience Office (VEO) has been commissioned by the Veteran Health
Administration (VHA) to measure the satisfaction of Telehealth recipients regarding their
electronic interaction with physicians, nursing professionals, and other medical staff. It also
seeks Veteran input on the quality of the treatment they received via the three modalities listed
above. VEO proposes to conduct a brief transactional survey on Veterans who utilized the
1

VA Telehealth Services Fact Sheet FY17, Office of Connected Care, VHA, VA

service within the past week. A subset of veterans will be randomly selected to participate.
Sampled patients will be contacted through an invitation email. A link will be enclosed so the
survey may be completed using an online interface, with customized patient information. The
survey itself will consist of a handful of questions revolving around a human-centered design,
focusing on such elements as trust, emotion, effective, and ease with the care they received.
B. Basic Definitions
Coverage
Measurement Error
Non-Response
Transaction

Response Rate
Sample
Sampling Error
Sampling Frame
Reliability

The percentage of the population of interest that is included in the
sampling frame.
The difference between the response coded and the true value of the
characteristic being studied for a respondent.
Failure of some respondents in the sample to provide responses in
the survey.
A transaction refers to the specific time a Veteran interacts with the
VA that impacts the Veteran’s journey and their perception of VA’s
effectiveness in caring for Veterans.
The ratio of participating persons to the number of contacted
persons. This is one of the basic indicators of survey quality.
In statistics, a data sample is a set of data collected and/or selected
from a statistical population by a defined procedure.
Error in estimation due to taking a particular sample instead of
measuring every unit in the population.
A list, map, or other specification of units in the population from
which a sample may be selected.
The consistency or dependability of a measure. Also referred to as
standard error.
C. Application to Veterans Affairs

This measurement may bring insights and value to all stakeholders at VA. Front-line VA
staff can resolve individual feedback from participant and take steps to improve their experience;
meanwhile VA executives can receive real-time updates on systematic trends that allow them to
make changes.
1) To collect continuous participant experience data to monitor the relative success of
programs designed to improve Telehealth service delivery
2) To help field staff and the national office identify need of the specific population they
serve
3) To better understand why veterans provide positive or negative feedback about
telehealth services

Part II – Methodology
A. Target Population and Frame
The target population of the TH survey is all Veterans having an Outpatient CVT, HT, or
SFT event in the past 7 days. The identification of Telehealth patients utilizes weekly data
extracts from the Corporate Data Warehouse (CDW), which houses the operational records of
VHA. Each Telehealth event eligible for a VEO survey will be associated with one of these three
modalities. Under CVT and SFT modalities, three types of surveys are designed to inquire about
veterans’ experience in terms of different VA service domains—HT only has 1 type of survey
(see Table 1).
The sample frame is limited to those that have a valid email address and excludes anyone
that has been invited to take any vSignals survey in the prior 30 days or has opted out from
receiving surveys.
Table 1. Survey Types under Telehealth Modalities
Telehealth Modality
Clinical Video
Telehealth
Store and Forward

Subtype 1
Appointment Scheduling

Subtype 2
Clinic Appointment

Clinic Appointment

Home or Mobile
Appointment

Home Telehealth

Continuing Patient
Survey

Subtype 3
Home or Mobile
Appointment
Result

B. Sample Size Determination
For a given margin of error and confidence level, the sample size is calculated as below
(Lohr, 1999):
For population that is large, the equation below is used to yield a representative sample
for proportions:
2
𝑍𝛼/2
𝑝𝑞
𝑛0 =
𝑒2
where
•
•
•

𝒁𝜶/𝟐 = 1.96, which is the critical Z score value under the normal distribution when using a
95% confidence level (α = 0.05).
p = the estimated proportion of an attribute that is present in the population, with q=1-p.
o Note that pq attains its maximum when value p=0.5, and this is sometimes used for
a conservative sample size (i.e., large enough for any proportion).
e = the desired level of precision; in the current case, the margin of error e = 0.03, or 3%.
Also referred to as MOE.

For a population that is relatively small, the finite population correction is used to yield a
representative sample for proportions:

𝑛=

𝑛0
𝑛
1 + 𝑁0

Where
•
•

𝒏𝟎 = Representative sample for proportions when the population is large.
N = Population size.

The margin of error surrounding the baseline proportion is calculated as:
𝑀𝑎𝑟𝑔𝑖𝑛 𝑜𝑓 𝑒𝑟𝑟𝑜𝑟 = 𝑧𝛼/2 √

𝑁 − 𝑛 𝑝(1 − 𝑝)
√
𝑁−1
𝑛

Where
•
𝒁𝜶/𝟐 = 1.96, which is the critical Z score value under the normal distribution when using
a 95% confidence level (α = 0.05).
•
N = Population size.
•
n = Representative sample.
•
p = the estimated proportion of an attribute that is present in the population, with q=1-p.
Sample sizes was originally calibrated to ensure monthly a 3% MOE at a 95%
Confidence Level at the modality level. This represents an industry standard for reliability
widely used by survey administrators (Lohr, 1999). With the expansion of Telehealth services
more localized measurement was desired to evaluate at the Medical Center level (STA3N) and/or
the VISN level. Each survey’s target was set as either a fixed number of invites or as a set
sample rate. For the 2 surveys with the largest population the target was fixed at 64,092 invites
per month. For the Telehealth Store & Forward Result survey a fixed sample of 13,036 was
chosen to maximize the returns for key measures. While we try to minimize the sampling rate of
surveys to below 50%, the Home Telehealth Continuing Patient survey is a census to maximize
returns for this modality. The Telehealth Store & Forward at the Clinic Appointment &
Telehealth Store & Forward at Home or Mobile Appointment surveys targets were fixed at a
40% sample rate to assure that, after quarantine, sufficient sample remained for the Telehealth
Store & Forward Result survey.
Table 2 shows the estimated monthly population and survey targets. Annually we expect
252,000 survey responses from roughly 2.1 million invites.
Table 2. Monthly Population and Survey Targets
Survey
CVT

Telehealth Appointment Scheduling

CVT

Telehealth at the Clinic Appointment

CVT
HT

Telehealth at Home or Mobile Appointment
Home Telehealth Continuing Patient

SFT

Telehealth Store & Forward at the Clinic Appointment

SFT

Telehealth Store & Forward at Home or Mobile Appointment

SFT

Telehealth Store & Forward Result

Total

Available
Email
Population
306,520

Sample
Rate

Return
Rate

Responses

64,092

21%

10.8%

6,930

Invites

9,850
250,179
23,709

4,925

50%

18.5%

64,092
23,709

26%
100%

11.2%
18.2%

11,378

4,551

40%

12.2%

40%

7.6%

60%
28%

8.1%
12.0%

1,680
21,639
624,955

672
13,036
175,077

914
7,159
4,324
554
51
1,051
20,983

Source: Telehealth Survey 4/1/2022 through 3/31/2023

C. Stratification
As noted in the section above, stratification is employed to ensure that sufficient number of
Veterans will be sampled for each of the seven surveys. These strata, whether define by fixed
targets or by a fixed sample rate, are considered explicit strata.
To ensure samples are balanced with respect to the following demographic variables: Age
Group, Gender, District and VAMC/CBOC, the random selection of patients within each stratum
will follow a systematic sampling design. The Veterans are sorted according to the demographic
variables, and every nth patient will be selected for survey invitation at a randomly selected
starting point—the value of n will change with each explicit strata so that all cases have an equal
probability of selection. This mechanism ensures that resulting respondent sample resembles the
email population with respect to the demographic variables. Since these stratification variables
do not have explicit targets for each permutation, they are deemed to be implicit stratification
variables.
Although we do not expect differences between the email population and the general
population with regard to geography, email populations tend to skew somewhat younger and
more female. Since these groups are less represented in the Veteran population, it is not
problematic for these demographics to be marginally oversampled – sample weighting calibrated
to the general population will ensure valid representation and correct for any imbalances.
Stratification Type
Variables
Explicit
Survey Type
Implicit
Age Group, Gender, District, VAMC, CBOC

D. Data Collection Methods
At the beginning of every measurement period, VEO data analysts will access the Corporate
Data Warehouse (CDW), which contains the governmental database for nearly all VHA
interactions. The remaining population will be extracted from the Cerner database that is being
piloted at a number of sites. This will become a larger proportion of the population as Cerner
implementation progesses. The telehealth target population will be extracted and recorded with
each new iteration. Those veterans with a valid email address will be included in the survey
frame. A new random sample, according to the stratification and quarantine protocol defined
below will be used to create an invitation file. Emails are immediately delivered to all selected
patients. Selected respondents will be contacted within 8 days of their Telehealth interaction.
They will have 14 days to complete the survey. Estimates will be accessible to data users
instantly, with the final results available 14 days after the beginning of the survey.

Table 3. Survey Mode
Mode of Data
Collection
Online Survey

Recruitment
Method
Email
Recruitment

Time After
Transaction
Within 8 days after
Telehealth Interaction

Recruitment
Period
14 Days

Invitation Days
Friday

(Reminder after 7 Days)

E. Reporting
Researchers will be able to use the vSignal (powered by Medallia) for interactive
reporting and data visualization. Trust, Ease, Effectiveness, and Emotion scores can be observed
for each Modality and Subtype (or Survey Type). The scores may be viewed by Age Group,
Gender, and Race/Ethnicity in various charts for different perspective. They are also depicted
within time series plots to investigate trends. Finally, filter options are available to assess scores
at varying time periods and within the context of other collected variable information.
The survey results become available in vSignals in real-time. Cell based weights are
applied at the time each query is run based on targets set at the beginning of each month. Targets
are calculated by dividing the target proportion of the cell by the total number respondents
within the cell. Weight cells are defined by survey, age group, gender, and district.
Recruitment is continuous (weekly) but the results from several weeks may be combined
into a monthly estimate for more precise estimates, which is the recommended reporting level.
Weekly estimates are unweighted, but allow analysts to review scores more quickly and within
smaller time intervals. Weekly estimates are less reliable for small domains, and should only be
considered for aggregated populations. Monthly estimates will have larger sample sizes, and
therefore higher reliability set to a 3% MOE at the 95% Confidence level (at the Modality Level
for Veterans 18+). Monthly estimates are also weighted for improved representation and less
bias (non-response and coverage, see section G on Sample Weighting). Quarterly estimates are
the most precise, but will take the greatest amount of time to obtain (12 weeks of collection).
However, Quarterly estimates are the most suitable for the analysis of small populations (e.g.
VAMC, Female Veterans 18-29, etc.).

F. Quality Control
To ensure the prevention of errors and inconsistencies in the data and the analysis, quality
control procedures will be instituted in several steps of the survey process. Records will undergo
a cleaning during the population file creation. The quality control steps are as follows.
1. Records will be reviewed for missing data. When records with missing data are
discovered, they will be either excluded from the population file when required or coded
as missing.
2. Any duplicate records will be removed from the population file to both maintain the
probabilities of selection and prevent the double sampling of the same customer.
3. Invalid emails will be removed.

The survey sample loading and administration processes will have quality control
measures built into them.
1. The extracted sample will be reviewed for representativeness. A secondary review will be
applied to the final respondent sample.
2. The survey load process will be rigorously tested prior to the induction of the survey to
ensure that sampled participants is not inadvertently dropped or sent multiple emails.
3. The email delivery process is monitored to ensure that bounce-back records will not hold
up the email delivery process.
The weighting and data management quality control checks are as follows:
1. The sum of the weighted respondents will be compared to the overall population
count to confirm that the records are being properly weighted. When the sum
does not match the population count, weighting classes will be collapsed to
correct this issue.
2. The unequal weighting effect will be used to identify potential issues in the
weighting process. Large unequal weighting effects indicate a problem with the
weighting classes, such as a record receiving a large weight to compensate for
nonresponse or coverage bias.

G. Sample Weighting, Coverage Bias, and Non-Response Bias
Weighting is commonly applied in surveys, to adjust for nonresponse bias and/or
coverage bias. Nonresponse is defined as failure of selected persons in the sample to provide
responses. This is observed virtually in all surveys, in that some groups are more or less prone to
complete the survey. The nonresponse issue may cause some groups to be over- or underrepresented. Coverage bias is another common survey problem in which certain groups of
interest in the population are not included in the sampling frame. The reason that these
beneficiaries cannot participate is because they cannot be contacted (no email address available).
In both cases, the exclusion of these portions of beneficiaries from the survey contributes to the
measurement error. The extent that the final survey estimates are skewed depends on the nature
of the data collection processes within an individual line of business and the potential alignment
between beneficiary sentiment and their likelihood to respond.
Survey practitioners recommend the use of sample weighting to improve inference on the
population so that the final respondent sample more closely resembles the true population. It is
likely that differential response rates may be observed across different age and gender groups.
Weighting can help adjust for the demographic representation by assigning larger weights to
underrepresented group and smaller weights to over-represented group. Stratification can also be
used to adjust for nonresponse by oversampling the subgroups with lower response rates. In both
ways of adjustments, weighting may result in substantial correction in the final survey estimates
when compared to direct estimates in the presence of non-negligible sample error.
The Telehealth Survey will also rely on what are often referred to as design weights—
weights that correct for disproportional sampling where respondents have different probabilities
of selection. Therefore, the weights are applied to make the explicit strata (the Survey Type)
proportional to the number of beneficiaries.

Weights are updated live within the VSignals reporting platform2. Proportions are set
based on the monthly distribution of the previous month.3
If we let wij denote the sample weight for the ith person in group j (j=1, 2, and 3), then the
CW formula is:

𝑤𝑖𝑗 =

% 𝑉𝑒𝑡𝑒𝑟𝑎𝑛𝑠 𝑖𝑛 𝑝𝑜𝑝𝑢𝑙𝑎𝑡𝑖𝑜𝑛 𝑖𝑛 𝑔𝑟𝑜𝑢𝑝 𝑗
# 𝑉𝑒𝑡𝑒𝑟𝑎𝑛𝑠 𝑖𝑛 𝑔𝑟𝑜𝑢𝑝 𝑗 𝑖𝑛 𝑡ℎ𝑒 𝑠𝑎𝑚𝑝𝑙𝑒

As part of the weighting validation process, the weights of persons in an age and gender
group are summed and verified that they match the universe estimates (i.e., population
proportion). Additionally, we calculate the unequal weighting effect, or UWE (see Kish, 1992;
Liu et al., 2002). This statistic is an indication of the amount of variation that may be expected
due to the inclusion of weighting. The unequal weighting effect estimates the percent increase in
the variance of the final estimate due to the presence of weights and is calculated as:
𝑠
2
𝑈𝑊𝐸 = 1 + 𝑐𝑣𝑤𝑒𝑖𝑔ℎ𝑡𝑠
= ( )2
𝑤
̅
where
•
•
•

cv = coefficient of variation for all weights 𝑤𝑖𝑗 .
s = sample standard deviation of weights.
1
𝒘
̅ = sample mean of weights, 𝑤
̅ = 𝑛 ∑𝑖𝑗 𝑤 ij.

H. Quarantine Rules
VEO seeks to limit contact with participants as much as possible, and only as needed to
achieve measurement goals. These rules are enacted to prevent excessive recruitment attempts
upon VA’s participants. All VEO surveys offer options for respondents to opt out, and ensure
they are no longer contacted for a specific survey. VEO also monitors participation within other
VEO surveys, to ensure participants do not experience survey fatigue.
Table 4. Quarantine Protocol
Quarantine Rule

Description

Repeated Sampling
Number of days between receiving/completing online
for Telehealth Survey survey, prior to receiving email invitation for the
Telehealth Survey
Other VEO Surveys
Number of days between receiving/completing online
survey and becoming eligible for another VEO survey
2

Elapsed
Time
30 Days

30 Days

Realtime weighting may cause some distortions at the beginning of each cycle due to empty cells or random
variance in small sample distributions.
3
Using previous months data is a design option for handling the problem of setting targets prior to fielding each
month. An alternative design is to set targets off annualized estimates to create more stability month to month. If the
population is known to fluctuate from month to month, past month population estimates may not be the optimal
solution.

Opt Outs

Persons indicating their wish to opt out of either phone
or online survey will no longer be contacted.

N/A

Part III – Assumptions and Limitations
A. Coverage Bias
Since the Telehealth Survey is email only, there is a substantial population of qualifying veterans
that cannot be reached by the survey. Veterans that lack access to the internet or do not use email may
have different levels of Trust and satisfaction with their service. As such, it is thought that Veterans in this
latter category do not harbor any tangible differences to other program participants who do share their
information.

C. Other Issues
The telehealth service may have limited use to the diagnosis and treatment of common illnesses
and conditions. Veterans who have complex disease types, such as cancer or tumor, may not choose to
use telehealth to pursuit the medical care even if they are located in the remote area. The telehealth
service users do not cover Veterans with a wide spectrum of diseases. Therefore, the Veteran respondent
types should be incorporated into consideration when interpreting the survey results and applications.

The telehealth service rating may require Veterans to be familiar with and have access to
modern technologies (e.g., Apps, Mobil Appt, Online Video Chat). Therefore, Veterans who use
the telehealth services and respond to the survey may be younger in age. The demographic
distribution of the survey respondents will be reviewed by the VEO when receiving the survey
results.
Home Telehealth is designed to provide medical care and services to high-risk Veterans
with chronic disease. When such patients receive the survey, their family members, caregivers,
or nurses are likely to respond to the survey on behalf of them. Therefore, the feedback and
information from the primary source may be missing. VEO will continue to identify these
responses in the VA databases and assess the effect of them on the Telehealth Survey estimates.

Part IV - Appendices
Appendix 1. References
Choi, N.G. & Dinitto, D.M. (2013). Internet Use Among Older Adults: Association with Health
Needs, Psychological Capital, and Social Capital. Journal of Medical Internet Research,
15(5), e97
Kalton, G., & Flores-Cervantes, I. (2003). Weighting Methods. Journal of Official Statistics,
19(2), 81-97.
Kish, L. (1992). Weighting for unequal P. Journal of Official Statistics, 8(2), 183-200.
Kolenikov, S. (2014). Calibrating Survey Data Using Iterative Proportional Fitting (Raking). The
Stata Journal, 14(1): 22–59.
Lohr, S. (1999). Sampling: Design and Analysis (Ed.). Boston, MA: Cengage Learning.
Liu, J., Iannacchione, V., & Byron, M. (2002). Decomposing design effects for stratified
sampling. Proceedings of the American Statistical Association’s Section on Survey
Research Methods.
Wong, D.W.S. (1992) The Reliability of Using the Iterative Proportional Fitting Procedure. The
Professional Geographer, 44 (3), 1992, pp. 340-348

20