2022 HIO Supporting Statement B_Final

2. Statistical Methods

Respondent Universe and Sampling Method

Given the relatively small number of HIOs in the U.S. (<200), the proposed approach is to achieve nationally representative survey results by conducting a census. Under this approach, UCSF would use their current comprehensive list of all HIOs in the nation that was constructed for the 2019 ONC HIO Survey and collect survey responses from all HIOs on the list.  

 

UCSF will also ensure the list is up to date by working with CIVITAS to review their membership and add any new HIOs that are not on the master list or remove HIOs that are no longer active. In addition, UCSF plans to reach out to a contact in each state, using a directory compiled by the Office of the National Coordinator for Health IT, to verify the list of organizations for their states. To be consistent with previous years, UCSF does not plan to include HIE networks led by single vendors or a consortium of vendors, such as Epic’s CareEverywhere Network or the CommonWell Health Alliance. 

 

For any new HIOs identified, UCSF will conduct a web search to identify the lead person (e.g., CEO, executive director) and their contact information. If needed, state HIT contacts can be leveraged to fill in any missing contact information. Past survey efforts have shown that there is a moderate amount of turnover in HIO leadership. Therefore, once the sampling frame and contact list is populated, UCSF will send an email to the HIO contact to confirm that the information is current. Any bounce-backs will trigger online searching and reaching out to UCSF’s network to identify the most up-to-date contact. Cleaning the contact list early in the project will save substantial time during the survey data collection phase. 

 

Procedures for the Collection of Information 

The final survey will be programmed into the online survey tool Qualtrics©. This tool has been used for past surveys and it has strong capabilities to support complex survey design (e.g., branching logic) as well as respondent communication and tracking. UCSF will extensively test the tool to ensure accuracy of branching and skip logic, accuracy of piped text, clarity of question display, and adherence to other survey usability guidelines. The contact list will also be loaded into Qualtrics to support communication and response tracking (i.e., by generating a unique survey link for each target respondent). 

 

UCSF plans to use their standard survey data collection methodology, modified based on any changes requested by ONC. This methodology involves an email to respondents 10-14 days before the survey is sent to them letting them know to expect it. UCSF then sends the survey via email (through Qualtrics) and follows up with all non-respondents one week later to ensure receipt. Follow-up emails will then be sent through Qualtrics every 1-2 weeks to all non-respondents. After at least three email reminders, UCSF begins weekly phone call reminders. In every communication, respondents are offered the choice of response modality and mention the response incentive. It typically takes 8-10 weeks in the field to achieve the target response rate. 

 

Using Qualtrics features, UCSF will track responses daily and update response status in a separate MS Excel tracking spreadsheet based on the HIO sampling frame. The MS Excel spreadsheet will allow UCSF to update the response rate regularly as well as document contacts that take place outside of Qualtrics. We will coordinate with SHIEC as needed. 

 

As responses are received, they will be reviewed for completeness, internal consistency, and accuracy. Beyond general quality assurance (QA) techniques (e.g., examining systematic patterns in responses), based on prior experience UCSF has developed a more specific set of QA checks. For example, a typical review checks for high outlier values in the number of hospitals and number of ambulatory practices participating in their HIO as well as for broad geographic coverage in non-contiguous states. UCSF will follow up with individual respondents to correct any errors. At the end of this process, UCSF will have a QA-ed data set for analysis. 

Methods to Maximize Response Rates and Deal with Nonresponse  

UCSF has typically tracked two types of response rates. First, of all HIOs in the sampling frame, for what proportion can their status be determined as: operational, planning, defunct. This response rate is critical to tracking the number of efforts over time, and UCSF will aim to achieve a 100% response rate as with prior surveys. The second response rate that is tracked is, of all operational HIOs, what proportion respond to the survey. For this response rate, UCSF has had substantial success achieving a more than 80% response rate and that will be the goal for the current survey. Given that the scope of the survey is somewhat broader and will include sensitive questions about information blocking, it is likely that at least an 80% response rate to the survey module that asks about key HIO demographics (e.g., governance, key activities, size, geographic coverage, support for reform efforts) is achievable. For modules related to standards and information blocking, UCSF will work to achieve an equally high response rate but could end up with a response rate closer to 65% (what was achieved in the first information blocking survey). However, by collecting the demographic measures for the broader sample, UCSF will be able to assess whether HIOs that respond to the standards and information blocking modules look systematically different from the overall group of HIOs. UCSF will also calculate individual item responses and conduct targeted follow up with respondents if any items have substantially lower response rates (i.e., 10 percentage points or more) than the overall response rate.  

 

UCSF will implement a robust approach to achieving high response rates based on prior experience surveying HIOs. First, they will email respondents to let them know to expect the survey. Then, after respondents receive the survey, UCSF will conduct weekly/bi-weekly email follow-up for approximately 4 weeks. Next, UCSF will begin calling non-responders to ensure they received the survey and answer any questions. They will continue calls and emails until achieving the target response rate. UCSF has found that this contact strategy is most effective when: (1) offered with the choice for the respondent to complete the survey via MS Word or over the phone if they prefer that to the online platform and (2) offered with a financial incentive for survey completion. As with prior years, UCSF will offer respondents a small incentive ($10) for completing the screening questions that enable us to determine whether they are operational, planning, or defunct and a larger incentive ($50) for completing the entire survey if they are eligible (i.e., not defunct). This approach will be further supplemented with communications from SHIEC to their members to raise awareness of the survey and encourage completion.  

 

Tests of Procedures or Methods to be Undertaken

UCSF and ONC pilot tested the survey in order to calculate an estimate of time required for completion of the survey (for OMB burden estimate), identify any potential concerns that could impede survey completion (e.g., selection of mandatory questions), and obtain input on the clarity of the instrument. We worked with CIVITAS to identify 7 of their members to serve as pilot respondents. UCSF then followed their standard approach to survey pilot testing. Specifically, they sent the pilot respondents the survey and instructions that ask them to complete the survey, track the time it takes, and note any questions that are confusing or otherwise problematic. Then, they conducted a phone interview with each respondent in which the respondent was asked each survey question and follow-up questions designed to elicit their understanding of the underlying concepts. The goals of the interview were to ensure: (1) respondents understand the questions as intended, and (2) the questions are written in a manner that respondents can answer. Through interviews, UCSF and ONC identified potential response errors as well as errors in question interpretation. Survey questions were updated after an initial set of feedback from 4 pilot sites interviews so that, for the second set of 3 interviews, the questions were increasingly clear and comprehensible and captured the correct concepts. After each wave of pilot testing, UCSF discussed with ONC and with CIVITAS the results of the pilot testing and recommendations for survey modifications.

 

Survey Content: 

The survey draws from the 2019 ONC HIO Survey and the CIVITAS survey. Questions from these instruments were reviewed and adapted based on current priorities, as described below. New questions were added to ensure all key content is covered. As with prior surveys, UCSF first asks the respondent screening questions to determine whether, as of a given date, the organization was facilitating clinical data exchange among independent entities. Respondents whose organization meets these criteria are then prompted to complete the remainder of the survey, with the following specific content in the five focal domains: Standards; Information Blocking; Sustainability, Health Reform Engagement, and Related HIO Demographics; Early Experiences with TEFCA; and Services Provided to Support COVID-19 pandemic.

Once the 2019 ONC HIO Survey and the CIVITAS survey were integrated, UCSF solicited overall feedback from ONC on priorities for any additional topics or questions and used that feedback to guide the development of a complete survey draft – informed by the latest literature and in collaboration with CIVITAS. UCSF conducted detailed rounds of engagement with ONC and targeted individuals with expertise in specific domains (e.g., standards, information blocking, governance, and trusted exchange). 

 

UCSF and ONC refined the survey iteratively, such that each individual reviews the most up-to-date version. It was also decided which questions should be mandatory (i.e., respondent cannot proceed until the question is answered), which is typically used for only a small number of critical questions to facilitate survey completion. This process yielded a complete first draft of the survey.   

Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The information for this study is being collected by the UCSF Center for Clinical Informatics and Improvement Research, on behalf of ONC. With ONC oversight, UCSF is responsible for the study design, instrument development, data collection, analysis, and report preparation.

The instrument for this study and the plans for statistical analyses were developed by Dr. Julia Adler-Milstein with input from ONC. The staff team is composed of Dr. Julia Adler-Milstein, Principal Investigator and Natalya Maisel, Analytic Lead. Contact information for these individuals is provided below.

Name	Email	Number
Julia Adler-Milstein	[email protected]	415-476-9562
Natalya Maisel	[email protected]	415-502-3729