SUPPORTING STATEMENT
Part A
Evaluation of the GuideLines Into Decision Support (GLIDES)
Version: February 3rd, 2010
Agency of Healthcare Research and Quality (AHRQ)
Table of contents
A. Justification 3
1. Circumstances that make the collection of information necessary 3
2. Purpose and use of information 4
3. Use of Improved Information Technology 5
4. Efforts to Identify Duplication 5
5. Involvement of Small Entities 5
6. Consequences if Information Collected Less Frequently 5
7. Special Circumstances 6
8. Consultation outside the Agency 6
9. Payments/Gifts to Respondents 6
10. Assurance of Confidentiality 6
11. Questions of a Sensitive Nature 6
12. Estimates of Annualized Burden Hours and Costs 6
13. Estimates of Annualized Respondent Capital and Maintenance Costs 8
14. Estimates of Annualized Cost to the Government 8
15. Changes in Hour Burden 8
16. Time Schedule, Publication and Analysis Plans 8
17. Exemption for Display of Expiration Date 9
List of Attachments 9
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see Attachment A), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:
1. research that develops and presents scientific evidence regarding all aspects of health care; and
2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and
3. initiatives to advance private and public efforts to improve health care quality.
Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.
With this project AHRQ proposes to evaluate how the translation of clinical knowledge into clinical decision support can be routinized in practice and taken to scale in ways that improve the quality of healthcare delivery for children in the U.S. Previously in the GLIDES project AHRQ designed and implemented decision support tools based on guidelines for the prevention of pediatric overweight and obesity and the management of chronic asthma in the pediatric population (publication forthcoming). In this phase of the project physicians will be surveyed about their experiences with the decision support tools developed in the previous phase. The participating study institutions (Yale University and Nemours) are geographically and organizationally diverse, and include a wide range of patients from a variety of social, economic and ethnic backgrounds. This project directly addresses AHRQ’s mission of “improving health systems practices,” in particular for priority populations including low-income groups, minority groups, children, and individuals with chronic diseases.
The evaluation plan includes a physician survey component (see Attachments B to E), in-person interviews (see Attachments F and G), and an extraction of electronic medical record data (see Attachments H and I). Participating physicians will be surveyed about their experiences with the decision support tools developed for this project. This will allow AHRQ to evaluate the fulfillment of knowledge transformation goals and the effectiveness of the decision support tools in improving the quality of health care at the chosen sites. Without such an evaluation, it would be difficult to determine whether this project has met AHRQ’s goals of enhancing the “quality, appropriateness and effectiveness of health services.” Consequently, it is necessary to collect this information to fulfill AHRQ’s mission.
The information collected during this project will be collated and analyzed by AHRQ in order to determine the effectiveness of the decision support tools in improving the quality of health care at the chosen sites. Aggregate results will be disseminated more broadly in the form of public reports, research papers and national meetings. AHRQ understands that the sample is not nationally representative and the results will not be presented as parameter estimates. However, the lessons learned from this research will be helpful in both refining these tools and implementing them more broadly.
Electronic medical record data will be extracted into an electronic spreadsheet for analysis. This extraction will occur at regular intervals to ensure continued maintenance and uptake of the tool. Utilization of the decision support tools at the provider and site level will be assessed based on the rate of electronic chart documentation. This is important to determine the rate of uptake of the intervention, as well as to determine whether there are any flaws in the design of the tool. Congruence of actual practice with guideline recommendations will be assessed based on automatically generated disagreement flags in the electronic medical record as well as by manual chart review. This data collection, including the manual chart review, will be performed by project staff and will not impose a burden on the participating sites.
Self-administered questionnaires will be used to elicit physicians’ general opinions of guideline-based care and clinical decision support tools on a five point Likert-type scale. Results from low-utilizing physicians will be compared to high-utilizing physicians to determine whether general opinions of guidelines and technology correlate with actual practice. Results will also be analyzed by demographic characteristics included in the survey questionnaire to determine whether opinions vary by age, degree of computer experience and skill, level of training and professional degree. These analyses will be important to future studies and decision support designers because they will help us understand whether interventions need to be targeted differently to different audiences. For example, senior level specialists may have less desire or need for clinical decision support tools than novice generalists have.
The questionnaire will also ask physicians about their general workflow during clinic sessions. These results will be reported in aggregate and by demographic characteristics to determine whether the decision support tools need to be targeted differently to populations with different workflows. For example, senior level physicians may be better able to incorporate technology into their workflow because they are faster and more experienced at clinical care; on the other hand, they may be less inclined to use technology because of ingrained practice.
Finally, physicians will be asked questions about the utility and efficacy of the specific decision support tools designed for this study. Results from low-utilizing physicians will be compared to high-utilizing physicians to determine whether usage of the system and perceptions of its utility are correlated. Similarly, the results will be analyzed by demographic characteristics to determine whether the decision support tools were perceived differently by different groups of physicians. Again, these results will help to both modify the existing tools and design future tools to be more effective.
The surveys will be available on a commercial web-based survey collection site such as SurveyMonkey.com. However, in order to maximize response rates, we will also have paper copies of the survey available for those who prefer paper-based forms or who can be reached directly in clinic. The survey has been designed to require minimal work and is almost entirely check-boxes rather than lengthy text responses. The online questionnaires for Yale can be seen at:
http://www.surveymonkey.com/s.aspx?sm=9Ry3EZzJr_2fNFu8UteHMRxQ_3d_3dttp
http://www.surveymonkey.com/s.aspx?sm=npDYMdGNQVOt_2fOZXesub9A_3d_3dttp
Nemours will do their online questionnaires using an internal tool (see Attachments J and K).
There has not been an evaluation of these particular clinical decision support tools. Therefore, this research is not duplicative of any previous research.
Some of the clinical practices involved in this research may be considered small businesses or other small entities. The information being requested has been held to the absolute minimum required.
This data collection is a one-time collection.
This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.
8.a. Federal Register Notice
As required by 5 CFR 1320.8(d), notice was published in the Federal Register on November 27th, 2009 for 60 days (see Attachment L). No comments were received.
The surveys have been pilot-tested with 9 or fewer physicians involved in clinical activity at the study institutions. The surveys have been altered, shortened, reframed and revised according to their views.
There will be no payment or gift to respondents associated with this data collection.
Individuals and organizations will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). They will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose.
Information that can directly identify the respondent, such as name and/or social security number will not be collected.
No questions in this data collection are of a sensitive nature. We are not collecting Social Security numbers, Medicare numbers or other personal identifiers for this study.
Exhibit 1 shows the estimated annualized burden hours for the respondents' time to participate in this research. The Asthma Management and Clinical Decision Support System Usability and User Satisfaction Survey (asthma questionnaire) will be completed by 172 health care professionals across 3 sites and is expected to require about 6 minutes to complete. The Obesity Prevention and Clinical Decision Support System Usability and User Satisfaction Survey (obesity questionnaire) will be completed by 82 health care professionals across 2 sites and is expected to require about 6 minutes to complete. The in-person interviews will be conducted with a total of 50 clinicians at 3 sites and are expected to last 30 minutes each. The total burden is estimated to be 51 hours.
Exhibit 2 shows the estimated annualized cost burden associated with the respondents’ time to participate in this research. The total cost burden is estimated to be $2,781.
Exhibit 1. Estimated annualized burden hours
Form Name |
Number of Sites |
Number of Responses per Site |
Hours per Response |
Total Burden Hours |
Asthma questionnaire -- Yale |
2 |
31 |
6/60 |
6 |
Asthma questionnaire -- Nemours |
1 |
110 |
6/60 |
11 |
Obesity questionnaire -- Yale |
1 |
57 |
6/60 |
6 |
Obesity questionnaire -- Nemours |
1 |
25 |
6/60 |
3 |
In-person interviews – Yale |
2 |
15 |
30/60 |
15 |
In-person interviews – Nemours |
1 |
20 |
30/60 |
10 |
Total |
8 |
na |
na |
51 |
Exhibit 2. Estimated annualized cost burden
Form Name |
Number of Sites |
Total Burden hours |
Average Hourly Wage Rate* |
Total Cost Burden |
Asthma questionnaire -- Yale |
2 |
6 |
$59.83 |
$359 |
Asthma questionnaire -- Nemours |
1 |
11 |
$59.83 |
$658 |
Obesity questionnaire -- Yale |
1 |
6 |
$47.25 |
$284 |
Obesity questionnaire -- Nemours |
1 |
3 |
$47.25 |
$142 |
In-person Interviews – Yale |
2 |
15 |
$53.54 |
$803 |
In-person Interviews – Nemours |
1 |
10 |
$53.54 |
$535 |
Total |
8 |
51 |
na |
$2,781 |
*Based upon the mean of the average wages for other physicians and surgeons, general pediatricians, and pediatric trainees (asthma questionnaire), and general pediatricians and pediatric trainees (obesity questionnaire), National Compensation Survey: Occupational wages in the United States 2008, “U.S. Department of Labor, Bureau of Labor Statistics,” and Yale Pediatric Residency Program, 2008.
Capital and maintenance costs include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. There are no direct costs to respondents other than their time to participate in the study.
Exhibit 3 shows the total and annualized cost for this research. Since this project will not exceed one year the total and annualized costs are identical. The total cost is estimated to be $5,703.
Exhibit 3. Estimated Total and Annualized Cost
Cost Component |
Total Cost |
Annualized Cost |
Project Development |
$1,406 |
$1,406 |
Data Collection Activities |
$416 |
$416 |
Data Processing and Analysis |
&780 |
$780 |
Publication of Results |
$1,601 |
$1,601 |
Project Management |
$200 |
$200 |
Overhead |
$1,299 |
$1,299 |
Total |
$5,703 |
$5,703 |
This is a new collection of information.
Overall utilization will be assessed by review of electronic medical record data and will be described using descriptive statistics. Disagreement with guideline recommendations will be identified electronically and by manual chart review, and will be analyzed qualitatively to determine the main reasons for disagreement. We will use descriptive statistics to describe survey results, and will use chi-square analyses to examine differences between sub-groups (such as differences between sites or level of training). We plan to analyze and report these results in a peer-reviewed journal. We expect to begin data collection upon receiving OMB clearance, and will complete the data collection within 2 months. We expect that the analysis will be complete 3 months after the data collection, and that reports will be ready for submission for publication 6 months after the data collection. Actual publication dates are contingent upon external reviewers’, journal timelines and other factors outside our control.
AHRQ does not seek this exemption.
List of Attachments:
Attachment A -- Healthcare Research and Quality Act of 1999
Attachment B – Asthma questionnaire – Yale
Attachment C – Asthma questionnaire – Nemours
Attachment D – Obesity questionnaire – Yale
Attachment E – Obesity questionnaire – Nemours
Attachment H -- Electronic Data Extraction – Obesity
Attachment I -- Electronic Data Extraction – Asthma
Attachment J -- Online Asthma Questionnaire – Nemours
Attachment K -- Online Obesity Questionnaire - Nemours
Attachment L -- Federal Register Notice
| File Type | application/msword |
| File Title | SUPPORTING STATEMENT |
| File Modified | 2010-06-04 |
| File Created | 2010-06-04 |