SUPPORTING STATEMENT
Part A
Embedded Research in Care Delivery Systems
Version: July 12, 2019
Agency for Healthcare Research and Quality (AHRQ)
Draft OMB supporting statement A
Table of contents
A. Justification
1. Circumstances that make the collection of information necessary
2. Purpose and use of information
3. Use of Improved Information Technology
4. Efforts to Identify Duplication
5. Involvement of Small Entities
6. Consequences if Information Collected Less Frequently
7. Special Circumstances
8. Consultation outside the Agency
9. Payments/Gifts to Respondents
10. Assurance of Confidentiality
11. Questions of a Sensitive Nature
12. Estimates of Annualized Burden Hours and Costs
13. Estimates of Annualized Respondent Capital and Maintenance Costs
14. Estimates of Annualized Cost to the Government
15. Changes in Hour Burden
16. Time Schedule, Publication and Analysis Plans
17. Exemption for Display of Expiration Date
List of Attachments
1. Circumstances that make the collection of information necessary
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999, is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:
1. Research that develops and presents scientific evidence regarding all aspects of health care; and
2. The synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and
3. Initiatives to advance private and public efforts to improve health care quality.
Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.
In support of its mission and the goal of the Department of Health and Human Services of promoting affordable health care, AHRQ’s strategic agenda includes “catalyzing the evolution of learning health systems” through research, training, tools, and data. The Learning Health System is a model in which internal data and experience are systematically integrated with external evidence, and that knowledge is put into practice. As a result, patients get higher quality, safer, more efficient care, and health care delivery organizations become better places to work.
One way to foster this type of system learning is to embed researchers within care delivery systems. Embedded researchers are either employed by a delivery system or invited to work in them on a specified project. Embedded researchers contribute to health system learning by collaborating with delivery system stakeholders to identify and evaluate innovations and practice changes that can improve the outcomes of individual and populations and health system performance. AHRQ is developing tools and findings to support embedded research, and is funding training of researchers to conduct this embedded research.
Evidence about strategies for deploying embedded researchers can contribute directly to AHRQ’s strategic initiatives in support of embedded research and the learning health system and to related programs conducted by the federally-funded Patient Centered Outcomes Research Institute (PCORI); the National Academy of Medicine; AcademyHealth and other non-governmental entities.
Most available information on embedded research and learning systems comes from descriptions offered by the participants. There is little independent research on ways that delivery systems currently organize and use embedded research to promote improved care delivery.
To provide research background on the current state of embedded research, the investigators for this project conducted an environmental scan. It included preliminary, key informant interviews with six nationally recognized experts on embedded research and consultations with several staff members at AHRQ and AcademyHealth, who contribute to programs supporting embedded research and learning health systems. Additionally, the researchers scanned published and grey literature on embedded research and reviewed summaries of discussions at the inaugural meeting of the Embedded Researcher’s Interest Group within AcademyHealth (June, 2019) and a national conference on Accelerating the Development of Learning Healthcare Systems through Embedded Research (February, 2019). This conference was jointly funded by AHRQ, PCORI and the Department of Veterans Affairs Health Services Research & Development Service (VA HSR&D).
None of the available sources on embedded research systematically characterizes organizational strategies for deploying this type of research and documents and illustrates such strategies in practice; nor do available sources examine potential contributions and challenges associated with distinctive strategies for managing embedded research.
In conducting this study, AHRQ aims to conduct preliminary, exploratory case studies that illustrate alternative strategies for funding, managing, and using embedded research in care delivery systems and explores challenges associated with different ways of managing embedded research.
This investigation will be restricted to research where (1) the primary investigator is closely affiliated with the care organization or care system that is a major site of the research project, and (2) the research at least partially addresses operational concerns of the system (e.g. ways to improve care quality, value, or other aspects of system performance (such as patient and staff satisfaction).
This data collection aims at developing illustrative case studies. It does not intend to create a comprehensive inventory of current practice in embedded research or to provide a representative sample of these practices.
In light of the exploratory nature of the collection, the appropriate data gathering techniques for this collection are qualitative, rather than quantitative (Sofaer, 1998.) (See Attachment F for Reference). Through qualitative case studies of sites identified during the scan (e.g., recommended by key informants) the proposed study will develop carefully documented examples that illustrate promising ways that delivery systems can conduct embedded research.
These illustrative case studies will likely stimulate further research, along with discussion at AHRQ and elsewhere about how to prepare researchers to conduct embedded research. Additionally, the case studies may provide insights for health research funding agencies about ways that funding criteria can influence the conduct of embedded research. The case studies may also provide health care leaders with illustrations of some of the potential benefits of supporting embedded research and some of the challenges of incorporating such research into care delivery systems.
Based on the environmental scan, the researchers will select six to eight systems for case studies. Each system will have the following characteristics:
The health system employs people engaged in embedded research;
The system has been engaged in embedded research for at least two fiscal years (i.e. began embedded research no later than fiscal year prior to the fiscal year of interviews);
The system appears to have a distinctive approach to embedded research or is recognized as a leader in this field.
Among these systems, at least one system will have a mission and a commitment to serve AHRQ’s priority populations. Attachment A provides further details on how the delivery systems will be selected for study.
This project has the following goals:
Select health systems that currently apply diverse strategies to funding, initiating, managing, and using embedded research.
Document and describe how embedded research is prioritized, funded, managed and conducted in each system, and how research findings are used by the system.
Specify several promising strategies for organizing and conducting embedded research.
Provide summaries of study findings that will stimulate consideration of current and future strategies for embedded research among funders, trainers, and delivery system leaders.
To achieve the goals of this project, the following data collection activities are planned:
1. Main interviews
2. Supplementary Interviews
3. Document Review
1. Main Interviews
After obtaining human subjects approval, for each of the targeted systems, the investigators will interview up to eight people in each of six-to-eight health selected systems. The interview subjects in each delivery system will include at least one occupant of each of the following roles:
Executive-level manager;
Person exercising the most oversight over embedded research activities;
Person from a service line or care sector in which several embedded research projects have been carried out;
Lead investigator on one or more embedded research projects.
The interviews will be conducted by one or both investigators and will last about one hour. Nearly all interviews will be conducted by telephone. In some instances, more than one member of a site may take part in a single interview. Notes will be taken during interviews and supplemented by transcripts based on recordings of the interview (unless interview subjects do not agree to recording). See Attachment B for details on selection of subjects for interviews. The procedures to be used in contacting the subjects before the interview are described in Attachment C (Advance Letters). Interview subject consent forms appear in Attachment D.
2. Supplementary Interviews. If, after conducting these interviews, the investigators determine that additional information is needed on specific projects mentioned during the interviews, they may conduct up to three supplementary group interviews with staff that were heavily involved in three selected projects. These supplementary group interviews will use similar contact and consent procedures to those described above and detailed in Attachments B, C, and D. Up to four people will take part in each supplementary interview. Some will be individuals who were interviewed previously, and others will be people who were recommended by the original interview subjects within a system. The burden and cost estimates that appear later in this statement take into account the possibility of these three supplementary interviews.
All interviews will be based on an interview guide (see Attachment E) containing open- ended questions about the main topics of interest. This guide was pretested during the preliminary interviews with key informants. The interviews will be conducted following standard methodological protocols for qualitative, organizational interviews (McCracken, 1998). If supplementary interviews are needed to obtain details on specific research projects, these interviews will use the same dimensions for project description that were used in the main interviews (see Attachment E, Question # 23.)
3. Document review. Note: There is no public burden for this collection activity. The investigators will collect and review any documents (e.g., summary of findings from an embedded research study) that add details on embedded research and related activities within each system. These documents will typically be provided or recommended by the interview subjects.
The following procedures will be used in analysis of the interview data:
With the aid of qualitative analysis software, code interviews using deductive, qualitative coding based on the interview guide and then modify or supplement codes based on emergent themes (Saldaña, 2013).
Review these codes, interview protocols, and documents related to each system and construct case studies portraying the ways that embedded research has been conducted in each system. These case studies will describe the origins, conduct, findings, and uses of embedded research.
By comparing cases and project descriptions, develop a typology or a set of thematic summaries of alternative strategies for funding, managing, and using embedded research, along with the system contributions of these strategies and challenges associated with implementing them.
The intent of this work is to provide in-depth case examples of embedded research procedures. This data collection does not aim to produce generalizable information about embedded research in particular types of care delivery settings or across the United States.
The information obtained from the above analysis will be provided to three different audiences, and the form and content of these communications will be tailored to the audience’s interests and needs.
First, AHRQ senior leadership will receive an internal report. That report will provide the preliminary findings and case study summaries. These will complement the sources of input that AHRQ leadership currently consults in developing policies for further initiatives related to learning health systems and embedded research. For example, the findings from this research study may stimulate further consideration within AHRQ of procedures and priorities for supporting training of embedded researchers; areas for soliciting investigator-initiated applications; and topics for targeted extramural funding of more comprehensive and systematic research.
Second, through submissions to peer-reviewed journals, the researchers will target health services researchers and entities like AcademyHealth that support health services research. These manuscripts will use case study findings to illustrate some promising strategies for conducting embedded research in delivery systems and some challenges to this type of research. The discussion section in these empirical papers will explicitly note the methodological limitations of the highly selective case studies and will emphasize that further, systematic research will be needed to test the generalizability of this study’s findings. The discussion will also consider implications for how embedded and non-embedded researchers might make their research more useful to clinicians, managers, and senior leaders in delivery systems. These scientific publications will also seek to encourage additional, broader studies of embedded research and learning health systems.
Third, AHRQ will use blogs, such as AHRQ Views or NEJM Catalyst, to reach policy makers and research funders of embedded research projects. These blogs will use the study findings to illustrate how embedded research can contribute to system learning and care improvement and will describe continuing or new AHRQ’s initiatives in these areas.
Because of the exploratory nature of this study and its qualitative methodology, the questions asked of subjects will be open-ended and will sometimes be rephrased to fit the subject’s organizational position or experience with embedded research. Hence, data collection will not use electronic or internet-based technologies.
There are no comparable investigations under way or completed in the United States. This conclusion is based on the investigators conversations with key informants and staff at ARHQ and AcademyHealth; summaries of recent meetings on embedded research; and our literature scan. There have been valuable studies of embedded research in the United Kingdom, and some additional studies are underway there. Unlike this proposed data gathering, which concentrates on how delivery systems deploy researchers on a full-time basis, the focus in the UK studies is on academic researchers who collaborate on a temporary, part-time basis with members of a care delivery organization. This arrangement will not be explored in this proposed study.
Most or all of the information to be collected will come from large healthcare delivery systems. AHRQ will strive to keep data collection and impact to a minimum for all entities with which the investigators interact, including any small, care delivery organizations.
A one-time collection is planned.
This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.
8.a. Federal Register Notice
As required by 5 CFR 1320.8(d), notice was published in the Federal Register on July 29, 2019 on page 36606 for 60 days (see Attachment G). AHRQ received one comment that was outside of the scope of 5 CFR 1320.8(d)(1) and made no changes in response to this comment.
This project does not relate to any other federal entity. It reflects accepted procedures for exploratory qualitative research. Neither the researchers, nor senior AHRQ leaders whom the researchers consulted, identified any unresolved issues that need to be addressed through consultation.
No payments or gifts will be provided to respondents.
Individuals and organizations will be assured of the confidentiality of their replies under Section 944(c) of the Public Health Service Act. 42 U.S.C. 299c-3(c). The researchers will take notes during interviews and record and transcribe the interviews. The interview transcriptions and case study summaries will identify the organization but will not include the names of interview subjects. Transcriptions will be numbered and stored in password-protected files that are only accessible to the researchers. The names, email addresses, and contact phone numbers of potential and actual interview subjects will be stored in a separate, password-protected file. This file will not contain the number of the subject’s interview. Subjects will be provided with assurance of confidentially (see Attachment C and D).
The study will not include questions of a sensitive nature. The only personally identifiable information retained will be the subject’s name, email, and contact phone number, which is needed to make contact with the individual and to schedule the telephone interview. Following the guidelines of AHRQ’s IRB, the researchers will provide subjects with assurances of confidentiality, will explain the scientific nature of the study and its potential value for knowledge and as a stimulus to decision making by research funders and users. The confidentiality statement will note that there are no foreseeable risks to participating in the study and no cost for the subject other than time. (See Attachment D for Subject Consent form).
Exhibit 1 is based on the following assumptions: No more than 8 subjects will participate in the main round of interviews in each system (site). There will be a maximum of 8 sites. If supplementary information is needed on selected projects, no more than 3 supplementary interviews will be conducted. Each supplementary interview will include 3-4 participants, with a total of no more than 10 participants in the whole set of supplementary interviews.
Exhibit 1. Estimated annualized burden hours
Collection Activity- Interviews |
Number of respondents |
Number of responses per respondent |
Hours per response |
Total burden hours |
Interviews with executive-level subjects |
10 |
1 |
1 |
10 |
Interviews with physicians |
22 |
1 |
1 |
22 |
Interviews with researchers and other operations staff |
42 |
1 |
1 |
42 |
Total |
|
|
|
Exhibit 2. Estimated annualized cost burden
Interview Participants |
Number of respondents |
Total burden hours |
Average hourly wage rate* |
Total cost burden |
Executive level (code 11-1011) |
10 |
10 |
$96.22 |
$962.20 |
Physicians (code 29-1060 |
22 |
22 |
$101.43 |
$2,231.46 |
Researchers and other operations staff (based on Operations Research Analysts code 15-2031) |
42 |
42 |
$42.48 |
$1,784.16 |
|
|
|
|
|
Total |
|
|
|
$4,977.82 |
* National Compensation Survey: Occupational wages in the United States May 2018 “U.S. Department of Labor, Bureau of Labor Statistics.”
There are no direct costs to respondents other than their time to participate in the study.
14. Estimates of Total and Annualized Cost to the Government
The total estimated costs to the government are $16,379.43.
The costs include payments to a government contractor for transcription of the interviews and federal personnel costs for all other data collection activities.
The contractor costs are shown in Exhibit 3a. These are estimated at $8,400, based on the following assumptions. 48, one-hour interviews (including 3 supplementary group interviews); 5 hours transcription time per hour of interview content; $35 per hour transcription labor charge. Note that only 48 hours of interviews are anticipated (an average of 6 interviews per case for 8 cases), since some of the interviews will have multiple participants, and a few selected subjects may decline to be interviewed or forced to cancel the planned interview. The estimated number of interviews includes the possibility of three supplementary interviews, as described in Section 1 above.
Exhibit 3a. Estimated Total Contractor Cost
Cost Component |
Total Cost |
Interview transcription 48hr@ (@$35hr; 5 hr. per interview) |
$8,400 |
Total |
$8,400 |
Government Personnel Cost:
The data gathering, analysis, and write up will be conducted by two AHRQ staff members. One staff member is a Sociologist at the GS-15 grade level, Step 10; hourly rate of $79.78. The other staff member is a Health Scientist Administrator, at the GS-13 Step 9, Hourly rate of $60.19.
The Federal hourly salary information is available on the OPM website at https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2019/DCB.pdf.
The following assumptions were used in estimating the Government Personnel Costs:
a. Interviewing: Total hours for conducting the interviews is assumed to be 48 (see explanation of assumptions for Exhibit 3a.). Some interviews will be conducted by both the Sociologist (GS 15) and the Health Scientist (GS 13). Others will be conducted solely by just one of the two investigators.
b. Analysis: the cost to analyze the data is assumed to be .5 hr. (½ hour) per interview, including analysis of any associated documentation provided by interview subjects. The senior researcher (the Sociologist) will analyze some interviews to allow for a check on coding reliability.
Exhibit 3b. Federal Government Personnel Cost
Activity |
Federal Personnel |
Hourly Rate |
Estimated Hours |
Cost |
Interviewing |
Sociologist |
79.78 |
25 |
1,194.50 |
Interviewing |
Health Scientist |
60.19 |
33 |
1,986.27 |
Analysis |
Sociologist |
79.78 |
6 |
478.68 |
Analysis |
Health Scientist |
60.19 |
24 |
1,444.56 |
Report preparation |
Sociologist |
79.78 |
30 |
2,393.40 |
Report preparation |
Health Scientist |
60.19 |
20 |
1,203.80 |
Total |
$8,701.21 |
|||
The estimated total annualized cost for this activity is $17,101. 21. This cost includes contractor costs ($8,400) and Federal personnel costs ($8,701.21).
Collection of data February 2020-March 2020
Coding and analysis April –May 2020
Report preparation and submission
to AHRQ June 2020
Writing of scientific manuscripts
and blogs July- September, 2020
Posting of summary on AHRQ website September, 2020
AHRQ does not seek this exemption.
List of Attachments:
Selection of Sites
Selection of Interview Subjects
Advance Letters
Interview Subject Consent Form
Interview Protocol
References
Federal Register Notice
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Harrison, Michael (AHRQ/CDOM) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-15 |