SUPPORTING STATEMENT
Part A
Evaluation of the GuideLines Into Decision Support (GLIDES)
Version: February 3rd, 2010
Agency of Healthcare Research and Quality (AHRQ)
Table of contents
A. Justification 3
1. Circumstances that make the collection of information necessary 3
2. Purpose and use of information 4
3. Use of Improved Information Technology 5
4. Efforts to Identify Duplication 5
5. Involvement of Small Entities 5
6. Consequences if Information Collected Less Frequently 5
7. Special Circumstances 6
8. Consultation outside the Agency 6
9. Payments/Gifts to Respondents 6
10. Assurance of Confidentiality 6
11. Questions of a Sensitive Nature 6
12. Estimates of Annualized Burden Hours and Costs 6
13. Estimates of Annualized Respondent Capital and Maintenance Costs 8
14. Estimates of Annualized Cost to the Government 8
15. Changes in Hour Burden 8
16. Time Schedule, Publication and Analysis Plans 8
17. Exemption for Display of Expiration Date 9
List of Attachments 9
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see Attachment A), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:
1. research that develops and presents scientific evidence regarding all aspects of health care; and
2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and
3. initiatives to advance private and public efforts to improve health care quality.
Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.
With this project AHRQ proposes to evaluate how the translation of clinical knowledge into clinical decision support can be routinized in practice and taken to scale in ways that improve the quality of healthcare delivery for children in the U.S. Previously in the GLIDES project AHRQ designed and implemented decision support tools based on guidelines for the prevention of pediatric overweight and obesity and the management of chronic asthma in the pediatric population (publication forthcoming). In this phase of the project physicians will be surveyed about their experiences with the decision support tools developed in the previous phase. The participating study institutions (Yale University and Nemours) are geographically and organizationally diverse, and include a wide range of patients from a variety of social, economic and ethnic backgrounds. This project directly addresses AHRQ’s mission of “improving health systems practices,” in particular for priority populations including low-income groups, minority groups, children, and individuals with chronic diseases.
The evaluation plan includes a physician survey component (see Attachments B to E), in-person interviews (see Attachments F and G), and an extraction of electronic medical record data (see Attachments H and I). Participating physicians will be surveyed about their experiences with the decision support tools developed for this project. This will allow AHRQ to evaluate the fulfillment of knowledge transformation goals and the effectiveness of the decision support tools in improving the quality of health care at the chosen sites. Without such an evaluation, it would be difficult to determine whether this project has met AHRQ’s goals of enhancing the “quality, appropriateness and effectiveness of health services.” Consequently, it is necessary to collect this information to fulfill AHRQ’s mission.
The information collected during this project will be collated and analyzed by AHRQ in order to determine the effectiveness of the decision support tools in improving the quality of health care at the chosen sites. Aggregate results will be disseminated more broadly in the form of public reports, research papers and national meetings. AHRQ understands that the sample is not nationally representative and the results will not be presented as parameter estimates. However, the lessons learned from this research will be helpful in both refining these tools and implementing them more broadly.
Electronic medical record data will be extracted into an electronic spreadsheet for analysis. This extraction will occur at regular intervals to ensure continued maintenance and uptake of the tool. Utilization of the decision support tools at the provider and site level will be assessed based on the rate of electronic chart documentation. This is important to determine the rate of uptake of the intervention, as well as to determine whether there are any flaws in the design of the tool. Congruence of actual practice with guideline recommendations will be assessed based on automatically generated disagreement flags in the electronic medical record as well as by manual chart review. This data collection, including the manual chart review, will be performed by project staff and will not impose a burden on the participating sites.
Self-administered questionnaires will be used to elicit physicians’ general opinions of guideline-based care and clinical decision support tools on a five point Likert-type scale. Results from low-utilizing physicians will be compared to high-utilizing physicians to determine whether general opinions of guidelines and technology correlate with actual practice. Results will also be analyzed by demographic characteristics included in the survey questionnaire to determine whether opinions vary by age, degree of computer experience and skill, level of training and professional degree. These analyses will be important to future studies and decision support designers because they will help us understand whether interventions need to be targeted differently to different audiences. For example, senior level specialists may have less desire or need for clinical decision support tools than novice generalists have.
The questionnaire will also ask physicians about their general workflow during clinic sessions. These results will be reported in aggregate and by demographic characteristics to determine whether the decision support tools need to be targeted differently to populations with different workflows. For example, senior level physicians may be better able to incorporate technology into their workflow because they are faster and more experienced at clinical care; on the other hand, they may be less inclined to use technology because of ingrained practice.
Finally, physicians will be asked questions about the utility and efficacy of the specific decision support tools designed for this study. Results from low-utilizing physicians will be compared to high-utilizing physicians to determine whether usage of the system and perceptions of its utility are correlated. Similarly, the results will be analyzed by demographic characteristics to determine whether the decision support tools were perceived differently by different groups of physicians. Again, these results will help to both modify the existing tools and design future tools to be more effective.
The surveys will be available on a commercial web-based survey collection site such as SurveyMonkey.com. However, in order to maximize response rates, we will also have paper copies of the survey available for those who prefer paper-based forms or who can be reached directly in clinic. The survey has been designed to require minimal work and is almost entirely check-boxes rather than lengthy text responses. The online questionnaires for Yale can be seen at:
http://www.surveymonkey.com/s.aspx?sm=9Ry3EZzJr_2fNFu8UteHMRxQ_3d_3dttp
http://www.surveymonkey.com/s.aspx?sm=npDYMdGNQVOt_2fOZXesub9A_3d_3dttp
Nemours will do their online questionnaires using an internal tool (see Attachments J and K).
There has not been an evaluation of these particular clinical decision support tools. Therefore, this research is not duplicative of any previous research.
Some of the clinical practices involved in this research may be considered small businesses or other small entities. The information being requested has been held to the absolute minimum required.
This data collection is a one-time collection.
This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.
8.a. Federal Register Notice
As required by 5 CFR 1320.8(d), notice was published in the Federal Register on November 27th, 2009 for 60 days (see Attachment L). No comments were received.
The surveys have been pilot-tested with 9 or fewer physicians involved in clinical activity at the study institutions. The surveys have been altered, shortened, reframed and revised according to their views.
There will be no payment or gift to respondents associated with this data collection.
Individuals and organizations will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). They will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose.
Information that can directly identify the respondent, such as name and/or social security number will not be collected.
No questions in this data collection are of a sensitive nature. We are not collecting Social Security numbers, Medicare numbers or other personal identifiers for this study.
Exhibit 1 shows the estimated annualized burden hours for the respondents' time to participate in this research. The Asthma Management and Clinical Decision Support System Usability and User Satisfaction Survey (asthma questionnaire) will be completed by 172 health care professionals across 3 sites and is expected to require about 6 minutes to complete. The Obesity Prevention and Clinical Decision Support System Usability and User Satisfaction Survey (obesity questionnaire) will be completed by 82 health care professionals across 2 sites and is expected to require about 6 minutes to complete. The in-person interviews will be conducted with a total of 50 clinicians at 3 sites and are expected to last 30 minutes each. The total burden is estimated to be 51 hours.
Exhibit 2 shows the estimated annualized cost burden associated with the respondents’ time to participate in this research. The total cost burden is estimated to be $2,781.
Exhibit 1. Estimated annualized burden hours
Form Name |
Number of Sites |
Number of Responses per Site |
Hours per Response |
Total Burden Hours |
Asthma questionnaire -- Yale |
2 |
31 |
6/60 |
6 |
Asthma questionnaire -- Nemours |
1 |
110 |
6/60 |
11 |
Obesity questionnaire -- Yale |
1 |
57 |
6/60 |
6 |
Obesity questionnaire -- Nemours |
1 |
25 |
6/60 |
3 |
In-person interviews – Yale |
2 |
15 |
30/60 |
15 |
In-person interviews – Nemours |
1 |
20 |
30/60 |
10 |
Total |
8 |
na |
na |
51 |
Exhibit 2. Estimated annualized cost burden
Form Name |
Number of Sites |
Total Burden hours |
Average Hourly Wage Rate* |
Total Cost Burden |
Asthma questionnaire -- Yale |
2 |
6 |
$59.83 |
$359 |
Asthma questionnaire -- Nemours |
1 |
11 |
$59.83 |
$658 |
Obesity questionnaire -- Yale |
1 |
6 |
$47.25 |
$284 |
Obesity questionnaire -- Nemours |
1 |
3 |
$47.25 |
$142 |
In-person Interviews – Yale |
2 |
15 |
$53.54 |
$803 |
In-person Interviews – Nemours |
1 |
10 |
$53.54 |
$535 |
Total |
8 |
51 |
na |
$2,781 |
*Based upon the mean of the average wages for other physicians and surgeons, general pediatricians, and pediatric trainees (asthma questionnaire), and general pediatricians and pediatric trainees (obesity questionnaire), National Compensation Survey: Occupational wages in the United States 2008, “U.S. Department of Labor, Bureau of Labor Statistics,” and Yale Pediatric Residency Program, 2008.
Capital and maintenance costs include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. There are no direct costs to respondents other than their time to participate in the study.
Exhibit 3 shows the total and annualized cost for this research. Since this project will not exceed one year the total and annualized costs are identical. The total cost is estimated to be $5,703.
Exhibit 3. Estimated Total and Annualized Cost
Cost Component |
Total Cost |
Annualized Cost |
Project Development |
$1,406 |
$1,406 |
Data Collection Activities |
$416 |
$416 |
Data Processing and Analysis |
&780 |
$780 |
Publication of Results |
$1,601 |
$1,601 |
Project Management |
$200 |
$200 |
Overhead |
$1,299 |
$1,299 |
Total |
$5,703 |
$5,703 |
This is a new collection of information.
Overall utilization will be assessed by review of electronic medical record data and will be described using descriptive statistics. Disagreement with guideline recommendations will be identified electronically and by manual chart review, and will be analyzed qualitatively to determine the main reasons for disagreement. We will use descriptive statistics to describe survey results, and will use chi-square analyses to examine differences between sub-groups (such as differences between sites or level of training). We plan to analyze and report these results in a peer-reviewed journal. We expect to begin data collection upon receiving OMB clearance, and will complete the data collection within 2 months. We expect that the analysis will be complete 3 months after the data collection, and that reports will be ready for submission for publication 6 months after the data collection. Actual publication dates are contingent upon external reviewers’, journal timelines and other factors outside our control.
AHRQ does not seek this exemption.
List of Attachments:
Attachment A -- Healthcare Research and Quality Act of 1999
Attachment B – Asthma questionnaire – Yale
Attachment C – Asthma questionnaire – Nemours
Attachment D – Obesity questionnaire – Yale
Attachment E – Obesity questionnaire – Nemours
Attachment H -- Electronic Data Extraction – Obesity
Attachment I -- Electronic Data Extraction – Asthma
Attachment J -- Online Asthma Questionnaire – Nemours
Attachment K -- Online Obesity Questionnaire - Nemours
Attachment L -- Federal Register Notice
SUPPORTING STATEMENT
Part B
Evaluation of the GuideLines Into Decision Support (GLIDES)
Version: February 3rd, 2010
Agency of Healthcare Research and Quality (AHRQ)
Table of contents
B. Collections of Information Employing Statistical Methods 3
1. Respondent universe and sampling methods 3
2. Information Collection Procedures 4
3. Methods to Maximize Response Rates 5
4. Tests of Procedures 5
5. Statistical Consultants 5
B. Collections of Information Employing Statistical Methods
1. Respondent universe and sampling methods
This study is a component of the evaluation plan for the AHRQ-funded contract, GLIDES, or GuideLines Into DEcision Support. In the GLIDES project, AHRQ has designed and implemented decision support tools based on guidelines for the prevention of pediatric overweight and obesity and the management of chronic asthma in the pediatric population. The evaluation plan includes a physician survey, in which participating healthcare providers (primary care pediatricians and pediatric pulmonologists, as well as advanced practice nurses) will be surveyed about their experiences with the decision support tools developed for this project. In this way AHRQ can evaluate the fulfillment of knowledge transformation goals and the effectiveness of the decision support tools in improving the quality of health care at the chosen sites. This information will be used to determine the effectiveness of the tools. In addition, subgroup analyses, such as those by site, level of experience, and rate of intervention uptake, will assist in determining whether these decision support tools should be designed, targeted and implemented differently for different audiences. This evaluation is not designed to be nationally representative of all pediatricians or pediatric pulmonologists, but is rather designed to assess the effects of a site-specific intervention. The lessons learned from this research will be helpful in both refining these tools and implementing them more broadly.
Questionnaire Sample:
There are a total of 262 clinicians who are eligible for the questionnaire by virtue of taking care of patients for whom the CDS tool is designed, 49 of which are also eligible for in-person interviews. . Since this number is not very large, particularly considering prespecified subgroup analyses by site and demographics, all will be contacted for participation in the survey. Based on prior survey work in this population at these sites, a 75% response rate is anticipated, resulting in about 254 completed questionnaires from 197 clinicians.
Interview Sample:
For the interview, we will purposefully select high and low utilizers to ensure a diversity of opinion in the interviews, based on a query looking at use of the decision support tools. There were 3 sites in Florida where the asthma tool was rolled out, making at least 6 potential interviews; there were 7 sites in which the obesity tool was rolled out at Nemours, making at least 14 potential interviews: approximately 20 interviews altogether at Nemours. If we have not reached thematic saturation at this point (i.e. we are still generating new ideas and themes), we will continue interviews, again purposefully selecting high and low users. At Yale there are only 9 physicians in the pulmonology practice so we will interview all of them to obtain a reasonable sample. In the primary care setting at Yale, we will select 10 primary care attending physicians and 10 primary care house staff to conduct interviews about obesity and asthma tool utilization, again using the purposeful sampling method of high and low users for the interviews. Respondents will be selected using data from the clinical information system to identify high and low users; this is to ensure a representative sampling for the interviews. Among these groups selection will be convenience-based - who is available and willing to talk. Qualitative interviews are not random samplings. Every effort will be made as per standard scientific protocol to ensure a wide diversity of opinion in the interviews.
Questionnaire and Interview Potential Respondents:
The distribution of potential respondents among type and site is as follows (note that since some clinicians are eligible for both questionnaires and the in-person interview, the total number of responses (387) exceeds the total number of unique respondents (262):
Pulmonology attending or fellow Allergist Primary care pediatrician Pediatric house staff Advanced Practice Registered Nurse Total
Yale
Asthma questionnaire 6 1 12 50 14 83
Obesity questionnaire 0 0 12 50 14 76
In-person interview 8 0 10 10 1 29
Nemours
Asthma questionnaire 14 5 50 68 9 146
Obesity questionnaire 0 0 27 0 6 33
In-person interview 6 0 14 0 0 20
Total 387
2. Information Collection Procedures
Since the universe of potential respondents is small there will be no stratifying or sampling of respondents. The analysis will have power to detect a 0.5 point difference in Likert-type scale responses between major subtypes of respondents (i.e. Yale sites vs. Nemours sites) at a 0.05% confidence level. Previous work has found that differences between categories of respondents (for example, between primary care pediatricians and pulmonologists) on these topics are more on the order of 1.0 points on a Likert-type scale. The analysis will have sufficient power to detect differences between both the physical sites and clinician types.
The questionnaires will be self-administered without aid from investigators. There will be no in-person interviews, advance appointments/letters or visits from investigators. The majority of data will be self-entered by respondents on a web-based survey tool. Some paper-based questionnaires may be collected, which will be entered into the study database by a trained research assistant with random checks initiated by the physician evaluation director.
The analysis of the qualitative data will follow standard qualitative analytic techniques using the inductive, grounded theory approach as described in Glaser's seminal work: Glaser BG, Strauss AL. The Discovery of Grounded Theory: Strategies for Qualitative Research. Chicago, IL: Aldine; 1967.
Participants will not be re-interviewed or re-contacted. We will not impute missing data. Analyses will be conducted using SAS 9.1.2 (SAS Institute, Cary, NC).
3. Methods to Maximize Response Rates
We will maximize response rates by sending an e-mail-based survey request to each group of participants from a local champion who is known and respected by that group of respondents (i.e. a pediatric pulmonologist for the pediatric pulmonology group; a chief resident for pediatric house staff). A maximum of three reminders may be sent to non-respondents. If necessary, questionnaires will be distributed in paper-based format at clinic sessions in which physicians are present in person. This study includes all practicing clinicians in each office; thus paper-based copies of the questionnaire will not be inadvertently distributed to non-study clinicians.
The universe of respondents is divided into 4 groups: pediatric pulmonologists at Yale University School of Medicine, primary care pediatric faculty and house staff at Yale-New Haven Hospital, pediatric pulmonologists at Nemours in Florida and primary care pediatricians at Nemours in Delaware. All are physicians or nurse practitioners; all will have had direct clinical experience with decision support tools intended to assist them with prevention of childhood overweight/obesity and management of chronic asthma. The questionnaire for each condition is 5 pages long, comprised primarily of Likert-type scale questions, and addresses physicians’ perceptions of guidelines, of decision support systems in general, of their workflow, and of the decision support tools in this study in particular. There will be one questionnaire for overweight/obesity and one questionnaire for asthma. Pulmonologists will receive the asthma questionnaire. Primary care physicians will receive the overweight/obesity questionnaire. Primary care physicians at Yale will receive both questionnaires (each at a different time, or combined as one 8 page questionnaire) because they will be exposed to both decision support tools. Data collection will be web-based as much as possible with paper as a back-up. There will be no remuneration for participation.
4. Tests of Procedures
The questionnaires have been pilot tested with < 10 respondents to refine and revise the instrument. We consider this adequate.
5. Statistical Consultants
Leora Horwitz, MD, MHS and Gabriela Ramírez-Garnica, PhD, MPH, are the evaluation leads at Yale and Nemours, respectively. Each has advanced training in research design and statistics and will be collecting and analyzing the information for AHRQ. Dr. Horwitz can be reached at (203) 688-5678. Dr. Ramirez can be reached at (407) 650-7062.
| File Type | application/msword |
| File Title | SUPPORTING STATEMENT |
| File Modified | 2010-05-25 |
| File Created | 2010-05-25 |