Telehealth Sample Design

VA Telehealth Service transactional survey: Sample design brief

Introduction

The Department of Veterans Affairs (VA) operates the nation’s largest health care system. It uses a wide variety of technologies to facilitate quality healthcare to its beneficiaries. Telehealth services are a critical aspect to modernizing the VA health care system. Telehealth (TH) increases access to high quality services by using information technology and telecommunication for Veterans, especially those that live in remote areas or are incapacitated. In FY 2017, over 700,000 patients received care via the three central telehealth modalities¹. Telehealth is an effective and convenient way for patients to receive, and for clinicians to provide quality care management.

Clinical Video Telehealth (CVT) is the use of real-time interactive video conferencing to assess, treat and provide patient care remotely. Veterans may be linked to physicians from a local clinic or even from home, for over 50 clinical applications, ranging from primary care to numerous specialties (e.g. dermatology). Home Telehealth (HT) is applied to high-risk Veterans with chronic disease requiring long-term care. Care management is augmented through such technologies as in-home and mobile monitoring, messaging, and/or video conferencing. The goal of HT is to reduce complications, hospitalizations, and clinical/ER visitations, so at-risk patients may remain in their own homes. Finally, Store and Forward Telehealth (SFT) concerns the acquisition and storage of electronic patient information (e.g., images, sounds, and video) collected at a VA clinic or medical center. The information is forwarded and retrieved by healthcare professionals at another VA medical facility where an assessment is performed.

VEO Transactional Surveys

The Veteran Experience Office (VEO) has been commissioned by the Veteran Health Administration (VHA) to measure the satisfaction of Telehealth recipients regarding their electronic interaction with physicians, nursing professionals, and other medical staff. It also seeks Veteran input on the quality of the treatment they received via the three modalities listed above.

VEO proposes to conduct a brief transactional survey on persons who utilized the service within the past week. A subset of veterans will be randomly selected to participate. Sampled persons will be contacted through an invitation email. A link will be enclosed so the survey may be completed using an online interface, with customized patient information. The survey itself will consist of a handful of questions revolving around a human-centered design, focusing on such elements as trust, emotion, effective, and ease with the care they received.

The purpose of this document is to outline the planned sample design and provide a description of the data collection and sample sizes necessary for proper reporting. A revised version of this document will be issued after survey implementation, containing the finalized sampling methodology.

Measurement Goals

The goal of the VEO survey operation is to continuously provide highly reliable monthly overall estimates of veteran utilization within the three major aspects of telehealth: CVT, HT, and SFT (Weekly and Quarterly survey estimates will also be available to data users). The number of persons contacted each month will be restricted to limit the burden on the patient population. In order to prevent survey fatigue, limitations will also be imposed on the number of times veterans may be contacted for this survey (or other active VEO surveys). Therefore, the sample sizes will represent a suitable compromise between reporting precision and intrusiveness on beneficiaries. Appropriate sample stratification and weighting methodology will be levied to ensure that the final estimates are truly representative of the at-large Telehealth population.

Sample Design Elements

• Data Collection Process

• Target Population and Sample Size Determination

• Stratification and Weighting

• Quarantine Protocol

• Quality Control Processes

Data Collection Process

At the beginning of every measurement period, VEO data analysts will access the Corporate Data Warehouse (CDW), which contains the governmental database for all VHA interactions. The telehealth target population will be extracted and recorded with each new iteration. Those veterans with a valid email address will be included in the survey frame. A new random sample, according to the stratification and quarantine protocol defined below will be used to create an invitation file. Emails are immediately delivered to all selected patients.

Selected respondents will be contacted within 3-4 days of their Telehealth interaction. They will have 14 days to complete the survey. Estimates will be accessible to data users instantly, with the final weighted results available 14 days after the beginning of the survey.

Table 1. Survey Mode

Mode of Data Collection	Recruitment Method	Time After Transaction	Recruitment Period	Collection Days
Online Survey	Email Recruitment	Within 3-4 days after Telehealth Appointment	14 Days (Reminder after 7 Days)	Tuesday and Friday (Tentative)

Target Population and Sample Size Determination

The target population of the TH survey is all Veterans having a CVT, HT, or SFT event in the past 7 days. Sample sizes are calibrated to ensure monthly reports have at least 3% Margin of Error (MOE) at a 95% Confidence Level. This represents a standard for reliability widely used in the survey³. In order to improve measurements at the facility-level, the sample sizes are increased beyond the 3% MOE threshold, but are still kept low enough to prevent excessive repeated contacts.

Table 2 depicts the number of unique Telehealth patients that received care in fiscal year 2017, along with the approximate monthly populations. Preliminary analysis of the Telehealth patient population indicates that approximately 30% of such patients have provided an email address to the VHA. This represents the frame population for the survey (see section below for information on possible bias due to frame under-coverage).

Assuming a 20% response rate, the reliability target for HT is unattainable at the FY2017 levels of usage. HT recipients are fewer and will have less turnover month-to-month (HT patients receive care on a more permanent basis) and may be prone to also have CVT or SFT episodes. For these reasons, the HT target is set lower and HT patients are prioritized in the sample selection. Table 3A shows the sample targets from each TH modality for the three reporting periods, while Table 3B provides the expected number of Veterans that need to be invited to achieve the sample targets, presuming a return rate of 20%.

Table 2. Target Population Figures

Survey Stratum	Unique Population in FY 2017	Approximate Monthly Population	Approximate Monthly Email Population	Precision at 3% MOE	Precision at 2% MOE
Clinical Video	336,000	60,000	18,000	1,049	2,309
Home Telehealth	145,000	81,000	24,300	1,054	2,332
Store and Forward	306,000	27,000	8,100	1,027	2,205

Table 3A. Proposed Sample Targets by Time Period

Survey Stratum	Weekly Target	Monthly Target	Quarterly Target
Clinical Video	500	2,000	6,000
Home Telehealth	250	1,000	3,000
Store and Forward	500	2,000	6,000
Total	1,250	5,000	15,000

Table 3B. Proposed Number of Invited Telehealth Patients, by Time Period

Survey Stratum	Weekly Contacts	Monthly Contacts	Quarterly Contacts
Clinical Video	2,500	10,000	30,000
Home Telehealth	1,250	5,000	15,000
Store and Forward	2,500	10,000	30,000
Total	7,250	25,000	75,000

Stratification and Weighting

Because the email population will have different demographics than the overall population, the initial sample will be selected in a manner from the frame so that the final respondent sample resembles the overall population. Stratification may also adjust for non-response (occurring when certain subpopulations are less prone to participate). Targets will be established for every permutation of the following stratification variables. As such, population values will be collected and recorded by VEO for every data collection period.

Stratification Variables

• Telehealth Type (CVT, HT, SFT)

• Gender

• Age Group (18-34, 35-54, 55+)

• Geographic Region (Tentative)

The stratification scheme above will result in a representative sample (w.r.t to the full population). Weighting will then be applied so that the sample is more fully matched to the population. Sample weights will be generated for Weekly, Monthly, and Quarterly estimates.

It was reported earlier that the email population comprises 30% of the full Telehealth population. Since 85% of older Americans utilize email², we can presume that most veterans choose not to share their email address with VHA or are simply unware of that option. It is assumed that the level of patient satisfaction is not directly related to their email status (Missing at Random). Since age and gender have been observed to be strong predictors of patient satisfaction in other VA health surveys, the stratification and weighting methodology outlined above will adequately compensate for any bias introduced by the incomplete frame of population.

Quarantine Rules

VEO seeks to limit contact with Veterans as much as possible, and only as necessary to achieve measurement goals. These rules are enacted to prevent excessive recruitment attempts upon Telehealth patients. VEO also monitors veteran participation within other surveys, to ensure veterans do not experience survey fatigue. All VEO surveys offer options for respondents to opt out, and ensure they are no longer contacted for a specific survey.

Table 4. Proposed Quarantine Protocol

Quarantine Rule	Description	Elapsed Time
Repeated Sampling for Telehealth Survey	Number of days between receiving/completing online survey, prior to receiving email invitation for a separate Telehealth experience	60 Days
Other VEO Surveys	Number of days between receiving/completing online survey and becoming eligible for another VEO survey	90 Days
HT Prioritization	Veterans enrolled in Home Telehealth will be excluded from CVT and SFT selection.	N/A
Lifetime Omission	No Veteran will be selected for the TH survey more than 10 times.	N/A
Opt Outs	Persons indicating their wish to opt out of either phone or online survey will no longer be contacted.	N/A

Quality Control

To ensure the prevention of errors and inconsistencies in the data and the analysis, quality control procedures will be instituted in several steps of the survey process. Records will undergo a cleaning during the population file creation. The quality control steps are as follows.

1. Records will be reviewed for missing sampling and weighting variable data. When records with missing data are discovered, they will be either excluded from the population file or put into separate strata upon discussion with subject matter experts.

2. Any duplicate records will be removed from the population file to both maintain the probabilities of selection and prevent the double sampling of the same veteran.

3. Invalid emails will be removed.

The survey sample loading and administration processes will have quality control measures built into them.

1. The survey load process will be rigorously tested prior to the induction of the TH Survey to ensure that sampled customers is not inadvertently dropped or sent multiple emails.

2. The email delivery process is monitored to ensure that bounce-back records will not hold up the email delivery process.

The weighting and data management quality control checks are as follows:

1. The sum of the weighted respondents will be compared to the overall population count to confirm that the records are being properly weighted. When the sum does not match the population count, weighting classes will be collapsed to correct this issue.

2. The unequal weighting effect will be used to identify potential issues in the weighting process. Large unequal weighting effects indicate a problem with the weighting classes, such as a record receiving a large weight to compensate for nonresponse or coverage bias.

4

1. VA Telehealth Services Fact Sheet FY17, Office of Connected Care, VHA, VA

2. Internet Use Among Older Adults: Association With Health Needs, Psychological Capital, and Social Capital, J Medical Internet Research, 2013 May; 15(5): e97

3. Lohr, S. (1999). Sampling: Design and Analysis (Ed.). Boston, MA: Cengage Learning.