Supporting Statement

The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see http://www.ahrq.gov/hrqa99.pdf), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:

1. research that develops and presents scientific evidence regarding all aspects of health care; and

2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and

3. initiatives to advance private and public efforts to improve health care quality.

Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.

In support of the mission, AHRQ has a strategic focus around advancing the use of patient-reported outcomes (PROs) in clinical practice, and has funded projects to understand how health information technology (HIT) might advance the collection and use of PROs in ambulatory care settings. PROs are assessments that directly capture the medical symptoms of the patient based on their own perception including measures such as physical function, pain, and health-related quality of life, without interpretation of the patient’s responses by a clinician.^¹² PROs are critical to the effective care of patients with PRO data informing provider practice as well as supporting important research on patient outcomes.^³ Traditionally, PROs are collected via pen and paper which inconveniences patients and providers, and makes the data difficult to access and analyze. Technical tools, such as software applications (apps), have the potential to improve the collection and analysis of PRO data. However, this technology must be usable by patients, providers, and researchers.³ Further, there must be clear standards driving the development of PRO apps and the integration of data with EHRs, so that PRO apps can be consistently implemented in a seamless fashion and the resulting data can be easily combined for large-scale analyses. Without such standards, collecting PRO data via apps could require extensive IT resources at each clinical site, making app implementation and data collection burdensome with technology of suboptimal usability.

The Office of the National Coordinator for Health Information Technology (ONC) developed standards to advance the collection and integration of PRO data. AHRQ is hosting a Challenge Competition to develop apps using standards developed by ONC. AHRQ also awarded a contract to MedStar to modify an existing app (OBERD app) by incorporating the standards developed by ONC. The OBERD app is used in some of MedStar’s specialty care practices to collect PRO data.

The purpose of this request is to conduct pilot tests to assess whether the Challenge Competition winning app and the OBERD app can be successfully used in practices to collect and integrate PRO data with different EHRs. AHRQ will assess the usability of the apps including whether patients can successfully navigate the app and answer the PRO questions. AHRQ will also assess whether providers use PRO data collected via the app and whether the data are informative for clinical care. The proposed effort is a critical step toward testing the use of standards for PRO app development, implementation, and effective use of data, by providers and patients. A report will be produced to identify success factors, barriers, and facilitators to implementing the PRO apps, usability of the apps, and recommendations for future activities relevant to the key stakeholders. This report will be instrumental to inform AHRQ’s objective of advancing the collection and use of standardized PRO data.

This research has the following goals:

1) Understand the usability and functional requirements of stakeholders involved in the development, implementation, and use of two PRO apps

2) Evaluate the use of two PRO apps.

The following steps and data collections are involved in the pilot tests:

Recruitment: A total of 18 practices will be recruited for the pilot tests (nine practices per app). These pilot sites will be selected from MedStar Health primary care and specialty care practices as well as the Capital Area Primary Care Research Network Practice Based Research Network (CAPRICORN PBRN) primary care practices in Washington DC, Maryland, and Virginia. A total of 10 patients will be recruited at each pilot site. Based on initial site visits, feedback from providers suggest this is an appropriate number of patients to identify and enroll while being minimally disruptive to the clinics. The site coordinators at both MedStar and CAPRICORN sites will work with clinic staff to identify eligible patients using existing patients’ records prior to their appointments with the following inclusion criteria:
1. 65+ (practices can target patients based on identifying Medicare enrollment)
2. Post-procedure, rehabilitating patients of any age
3. English proficient ( the pilot app will only be available in English)

2) PROMIS® items: Both the Challenge Competition winning app and the OBERD app will be used to collect PRO physical function data using the Patient-Reported Outcomes Measurement Information System (PROMIS®) items (Attachment A). The PROMIS Physical Function v2.0 is the core measure that will be administered. Each practice site has the option to administer additional PROMIS items including the Mobility 2.0, Upper Extremity 2.0, Physical Function for Samples with Mobility Aid Users 1.0, and Physical Function short form 10a. Target patient populations include patients who are 65+, post-procedure, rehabbing, and English proficient. Ten patients will be selected from each practice site. Training materials (Attachment B) have also been created to aid participants in ease of completion of this app.

3) Process Usage Worksheet (Patients): Prior to the app implementation, each practice site will complete a standard usage and process worksheet (Attachment C) electronically every week for the duration of the 6-month period. Worksheets will track: 1) # of patients eligible to receive the PRO app; 2) # of patients receiving the PRO app; 3) # of patients submitting PRO data; and 4) # of patients who decline the PRO app. The worksheet will take fewer than five minutes to complete. In addition, the worksheet will be used to examine passive usage data collected from the app such as the amount of time a patient uses the app or if they abandon it.

4) Process Usage Worksheet (Providers): To understand the provider organizations view of how the PRO measures are used by the clinical team, each site lead will complete a brief usage process worksheet collecting two data points each week (Attachment D). The worksheet will track: 1) # of patients whose PRO data was accessed by the provider; and 2) A rating of how informative the PRO data were to the goals of the provider accessing the information. Providers will be asked to rate this on a 1 to 5 scale with 5 being highly informative to their goals.

This study is being conducted by AHRQ through its contractor, MedStar Health, pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services and with respect to quality measurement and improvement (42 U.S.C. 299a(a)(1) and (2)).

2. Purpose and Use of Information

To understand how the two PRO apps work in clinical settings, this project will assess functionality and usability of the apps as well as app and PRO data usage. Pilot testing the two apps at different sites will provide diverse testing conditions including variability in clinical workflow, patient demographics, and EHRs. The findings from this study will inform future modification of the two PRO apps that are being tested.

The proposed effort is a critical step toward testing PRO data collection through apps designed with standards to promote data integration with EHRs. Lessons learned from the pilot tests including success factors, barriers, and facilitators to implementing the PRO apps as well as usability of the apps will be instrumental to inform AHRQ’s objective of developing user-friendly apps to collect standardized PRO data.

3. Use of Improved Information Technology

MedStar’s technical team will work with each site to ensure that PRO data are seamlessly integrated with each provider’s EHR to make data available during clinic visits. The patient-facing apps will be easily accessible by patients on a mobile device. The team will ensure the apps are correctly configured with the desired PROMIS measure. To minimize data collection burden on the patient, the Computer Adaptive Test (CAT) of the selected PROMIS physical function measure will be used. An optimally designed app interface will allow patients to submit PRO data with minimum effort.

Rapid on-site integration testing will take place to ensure appropriate functioning in the specific clinical environment. The usability core team will work with patients and providers at each site to ensure their needs and workflows are being met prior to apps go-live.

After completion of the six-month pilot, it will be necessary to remove any aspects of the technology that altered patient and provider workflows which the provider no longer wishes to retain.

4. Efforts to Identify Duplication

No similar data have been gathered by the research team or are available from other sources known to the research team.

5. Involvement of Small Entities

It is unlikely that any sites participating in this pilot test will be classified as small businesses.

6. Consequences if Information Collected Less Frequently

Data collected through the PRO apps will be collected only once from each patient.

Data collected through the patient and provider process and usage worksheets will be collected once a week until full patient enrollment. Practice sites have indicated that in many cases, the target enrollment could be obtained in just a few days or weeks, and as such the usage worksheets will be collected once a week for the period that patients are actively being enrolled. The worksheets will take fewer than five minutes to complete which does not constitute a burden to the sites. If the patient and provide process and usage worksheets are not collected weekly, this may be a larger time burden for the site coordinators.

7. Special Circumstances

This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.

8. Federal Register Notice and Outside Consultations

8.a. Federal Register Notice

A Federal Register notice is not required for this generic clearance.

8.b. Outside Consultations

A technical expert panel (TEP) composed of patients, providers, researchers, and developers from industry and academia will inform all aspects of the project to ensure robust stakeholder input. The TEP represents expertise in HIT implementation and usability, application development, patient safety, and patient needs. The TEP will convene for three two hour-long virtual meetings, led by Dr. Rollin (Terry) Fairbanks, who is AVP of Ambulatory Quality and Safety at MedStar Health and is an expert in HIT, human factors engineering, and patient safety. The panel will provide input on the implementation plan and pilot testing findings

9. Payments/Gifts to Respondents

Patients completing the PROMIS items on the PRO apps:

A small incentive for the patients in the form of a $5 gift certificate to Starbucks (or equivalent) will be offered as a token of appreciation for their time. In prior experience recruiting participants for stakeholder feedback, providing incentives of even relatively modest amounts significantly improves recruitment and retention. This is especially the case for patients, who otherwise have competing interests of time, and providers who tend to be extremely challenging to recruit given their schedules and availability. For example, on a previous ACTION III TO3, (patient and family engagement) incentives provided to primary care providers and patients were critical to recruiting sufficient numbers for semi-structured interviews during both the environmental scan, and the pre- and post-intervention stakeholder feedback sessions. In fact, oftentimes even with incentives available it proved challenging to recruit providers given the constraints of their schedules in between seeing patients and the potential time lost if they are not available to patients at any time point during their day. Additionally, we have extensive experience recruiting patients for interviews for similar interviews and surveys. In each instance recruitment and retention would not have proven successful without the use of financial incentives. In another study conducted in partnership with the American Medical Association, emergency medicine physicians were compensated for their time spent completing a usability evaluation and interview session. Further, there is a body of literature that supports the use of incentives to effectively recruit and retain research participants in studies such as this^⁴⁵⁶.

Data provided by Providers: Providers participating in this study will not receive any payment or remuneration. Participating sites will have valuable PRO data for key patient populations integrated to their EHR systems for providers to use during patients’ clinic visits. This was expressed as valuable in earlier provider interviews. Additionally, upon completion of data collection, each site may retain the elements of the technical setup that they wish to continue with.

10. Assurance of Confidentiality

Individuals and organizations will be assured of the confidentiality of their replies under Section 944(c) of the Public Health Service Act. 42 U.S.C. 299c-3(c). That law requires that information collected for research conducted or supported by AHRQ that identifies individuals or establishments be used only for the purpose for which it was supplied.

Information that can directly identify the respondent, such as name and/or social security number will not be collected.

11. Questions of a Sensitive Nature

PROMIS items can be completed without the need to ask questions of a sensitive nature. If any question is expressed to be of a sensitive nature during data collection, these questions will be modified to address the concern.

12. Estimates of Annualized Burden Hours and Costs

Exhibit 1. Estimated annualized burden hours

Form Name	Number of respondents	Number of responses per respondent	Hours per response	Total burden hours
PROMIS items	180 (100%)	1	0.25	45
Process Usage Worksheet (Patients)	18 (100%)	26	0.08	37.4
Process Usage Worksheet (Providers)	18 (100%)	26	0.08	37.4
Total	216			119.8

Exhibit 2. Estimated annualized cost burden

Form Name	Number of respondents	Total burden hours	Average hourly wage rate*	Total cost burden
PROMIS items	180	45	$24.34^a	$1,095.30
Process Usage Worksheet (Patients)	18	37.4	$38.83^b	$1,452.24
Process Usage Worksheet (Providers)	18	37.4	$38.83^b	$1,452.24
Total	216			$3,999.78

* National Compensation Survey: Occupational wages in the United States May 2017, “U.S. Department of Labor, Bureau of Labor Statistics.”

^a Based on the mean wages for all occupations (00-0000)

^b Based on the mean wages for all Healthcare Practitioners and Technical Occupations (29-0000)

13. Estimates of Annualized Respondent Capital and Maintenance Costs

There are no direct costs to respondents other than their time to participate in the study.

14. Estimates of Total and Annualized Cost to the Government

Exhibit 3a. Estimated Total and Annualized Cost

Cost Component	Total Cost	Annualized Cost
Project Development	$15,111.08	$15,111.08
Data Collection Activities	$13,5109.8	$13,5109.8
Data Processing and Analysis	$9,735.281	$9,735.281
Publication of Results	$4,867.639	$4,867.639
Project Management	$10,491.75	$10,491.75
Overhead	$10,1683.1	$10,1683.1
Total	$27,6998.8	$27,6998.8

Exhibit 3b. Federal Government Personnel Cost

Activity	Federal Personnel	Hourly Rate	Estimated Hours	Cost
Data Collection Oversight	Health Scientist Administrator GS 14	$60.40	25	$1,510
Review of Results	Health Scientist Administrator GS 14	$60.40	75	$4,530

Annual salaries based on 2018 OPM Pay Schedule for Washington/DC area: https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2018/DCB_h.pdf

15. Changes in Hour Burden

This is a new collection of information.

16. Time Schedule, Publication and Analysis Plans

Once OMB approval is received, pilot testing is scheduled to begin in December 2018. The pilot tests for the Challenge Competition winning app and the OBERD app involve very similar tasks including app implementation and general evaluation of app usage. Each pilot test will last about six months. However, the pilot test for the OBERD app will start and end earlier. The timeline of scheduled tasks is provided below:

Conduct Pilot Test for the OBERD app
1. App Integration, Implementation (December 2018-January 2019)
2. App Usage (January-June 2019)
3. Evaluation (June-July 2019)
Report Pilot Test Findings for the OBERD app
1. Draft Pilot Test Findings Report (August 30, 2019)
2. Expert Panel Review and Input on Findings (August 2019)
3. Final Pilot Test Findings Report (September 30, 2019)
Conduct Pilot Test for the Challenge Competition winning app
1. Orientation of the app/Adapt pilot plan (February-March 2019)
2. App integration, implementation (March-April 2019)
3. App Usage (March-August 2019)
4. Evaluation (August-September 2019)
Report Pilot Test Findings for the Challenge Competition winning app
1. Draft Pilot Test Findings Report (September-October 2019)
2. Final Pilot Test Findings Report (September-October 2019)

The quantitative app usage data will be analyzed using appropriate descriptive statistical methods by a biostatistics researcher.

17. Exemption for Display of Expiration Date

AHRQ does not seek this exemption.

List of Attachments:

Attachment A. PROMIS Item Bank V2.0

Attachment B: Training Materials

Attachment C: Process Usage Worksheet (Patients)

Attachment D: Process Usage Worksheet (Providers)

1 Acquadro C, Berzon R, Dubois D. Incorporating the Patient’s Perspective into Drug Development and Communication: An Ad Hoc Task Force Report of the Patient-Reported Outcomes (PRO) Harmonization Group Meeting at the Food and Drug Administration. Value Heal. 2003;6(5):522-531.

2 Weldring T, Smith S. Patient-Reported Outcomes (PROs) and Patient-Reported Outcome Measures (PROMs). Heal Serv Insights. 2013;6:61-68.

3 Snyder C, Jensen R, Segal J, Wu A. PATIENT-REPORTED OUTCOMES (PROs): PUTTING THE PATIENT PERSPECTIVE IN PATIENT-CENTERED OUTCOMES RESEARCH. Med Care. 2013;51:S73-S79.

4Turnbull AE, O'Connor CL, Lau B, Halpern SD, Needham DM Allowing Physicians to Choose the Value of Compensation for Participation in a Web-Based Survey: Randomized Controlled Trial J Med Internet Res 2015;17(7):e189 URL: https://www.jmir.org/2015/7/e189 DOI: 10.2196/jmir.3898 PMID: 26223821 PMCID: 4705363

5Cook DA, Wittich CM, Daniels WL, West CP, Harris AM, Beebe TJ Incentive and Reminder Strategies to Improve Response Rate for Internet-Based Physician Surveys: A Randomized Experiment J Med Internet Res 2016;18(9):e244 URL: https://www.jmir.org/2016/9/e244 DOI: 10.2196/jmir.6318 PMID: 27637296 PMCID: 5045523

6Pit SW, Vo T, Pyakurel S. The effectiveness of recruitment strategies on general practitioner’s survey response rates – a systematic review. BMC Medical Research Methodology. 2014;14:76. doi:10.1186/1471-2288-14-76.

File Type	application/msword
File Title	OMB Clearance Application
Author	hamlin-ben
Last Modified By	SYSTEM
File Modified	2018-11-29
File Created	2018-11-29

Supporting Statement