SUPPORTING STATEMENT
Part A
Evaluation of AHRQ's Effective Health Care Program
February 19, 2010
(Revised)
Agency for Health Care Research and Quality (AHRQ)
U.S. Department of Health and Human Services
Table of contents
A. Justification 1
1. Circumstances that make the collection of information necessary 1
2. Purpose and use of information 4
3. Use of Improved Information Technology 4
4. Efforts to Identify Duplication 5
5. Involvement of Small Entities 5
6. Consequences if Information Collected Less Frequently 5
7. Special Circumstances 5
8. Consultation outside the Agency 5
9. Payments/Gifts to Respondents 6
10. Assurance of Confidentiality 6
11. Questions of a Sensitive Nature 7
12. Estimates of Annualized Burden Hours and Costs 7
13. Estimates of Annualized Respondent Capital and Maintenance Costs 8
14. Estimates of Annualized Cost to the Government 8
15. Changes in Hour Burden 9
16. Time Schedule, Publication and Analysis Plans 9
17. Exemption for Display of Expiration Date 10
Attachments 11
The Agency for Health Care Research and Quality (AHRQ) requests that the Office of Management and Budget (OMB) approve, under the Paperwork Reduction Act of 1995, AHRQ’s intention to collect information from stakeholders involved with the Effective Health Care (EHC) program. This data collection is part of the evaluation of the EHC program’s governance structure, methods for engaging stakeholders and approaches to setting national research priorities. The evaluation will be conducted by IMPAQ International and its subcontractor, Abt Associates, henceforth referred to as the “IMPAQ team”, under contract to AHRQ.
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see Attachment A), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:
1. Research that develops and presents scientific evidence regarding all aspects of health care; and
2. The synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and
3. Initiatives to advance private and public efforts to improve health care quality.
Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.
AHRQ’s EHC program originates from Section 1013 of the Medicare Prescription Drug, Improvement, and Modernization Act (MMA) of 2003. This statute authorizes AHRQ to “conduct and support research to meet the priorities and requests for scientific evidence and information identified by such programs with respect to – (i) the outcomes, comparative clinical effectiveness, and appropriateness of health care items and services (including prescription drugs); and (ii) strategies for improving the efficiency and effectiveness of such programs, including the ways in which such items and services are organized, managed, and delivered under such programs” (MMA 2003). The EHC program provides a critical step in AHRQ’s mission to improve health care outcomes by encouraging the use of evidence-based information in the making of informed treatment decisions and choices.
The EHC program offers a platform for reviewing and synthesizing published and unpublished scientific evidences on the clinical effectiveness of pharmaceuticals and other health care interventions, promoting and generating new scientific evidence and analytic tools in an accelerated and practical format, as well as compiling, translating and disseminating scientific findings into meaningful messages for a wide variety of audiences including consumers, health care providers, payers, and policymakers. In additional to its program staff at AHRQ, the EHC program relies on a group of Research Centers to generate, synthesize and disseminate evidence: the Evidence-based Practice Centers (EPCs), the Developing Evidence to Inform Decisions about Effectiveness (DEcIDE) Research Network, the John M. Eisenberg Clinical Decisions and Communications Science Center, and the Centers for Education & Research on Therapeutics (CERTs).
Transparency and stakeholder input are also hallmarks of the EHC program. The program is transparent, i.e., open about its processes and allowing for communication and accountability. It is designed to welcome input from stakeholders by engaging the public and providing extensive opportunities for public comment and involvement. Public involvement helps ensure that the program responds to the most pressing issues and that its products are useful for health care decision-makers.
The purpose of this project is to evaluate the governance structure of the EHC program at AHRQ. To achieve that purpose, information will be collected to identify strengths and weaknesses in the current EHC program’s governance structure, methods for engaging stakeholders, and approaches to setting priorities for the research conducted by the EHC program. The second phase of the evaluation will be to contrast the EHC program with international programs of similar purpose.
To implement this evaluation, the IMPAQ team will conduct the following information collections:
1) Key Informant Interviews -- semi-structured, key informant interviews (in-person and by phone) with key informants will be used to understand the EHC program’s governance components and structure, from the vantage point of individuals governing the program, governed by the program, contributing to the program in various capacities, or impacted by the program’s activities. Thirteen EHC Research Centers Staff, two EHC Stakeholder Group Members, and twenty-seven EHC Program Users and Stakeholders will be interviewed about the governance structure of the EHC program.
Additional in-depth key informant interviews with twenty five EHC Program Users and Stakeholders will be used to collect more detailed information on the success or impact of the EHC program product that results from its governance element or approach, or about a specific, important governance element.
All key informant interviews will be tape recorded to improve data capture, with prior permission from the participants.
Interview guides for the key informant interviews, a process manual for the interviews, and letters, emails and scripts for phone calls associated with the interviews are provided in Attachments C1-C4, D, and H-L. Additionally, an advance letter about the evaluation that will be sent to all participants in the interviews and survey is provided in Attachment G.
2) Online Survey -- A structured, web-based online survey of EHC program Research Centers Staff and EHC program Users and Stakeholders will be used to gather information about the EHC program. The survey will provide a robust view of the EHC governance system by providing feedback from a broad group of individuals whose work is related to the program. Specifically, the survey will collect data about these individuals’ engagement and involvement with the EHC program; perceptions of the program’s governance; experiences with the development, production, dissemination, and use of EHC products; and their beliefs regarding the quality and nature of the collaborative work, including public-private partnerships, being done within centers, across centers, and between centers and stakeholders.
The data collection instrument for the online survey and letters and emails associated with the survey are provided in Attachments E and M-P. Additionally, an advance letter about the evaluation that will be sent to all participants in the interviews and survey is provided in Attachment G.
3) Appreciative Inquiry Workshop -- Small- and large-group discussions as part of an Appreciative Inquiry workshop will be designed to encourage EHC decision-makers (AHRQ staff, EHC program staff, AHRQ project officers for each of the Research Center networks, principal investigators or other representatives from each of the Research Center networks) and key program stakeholders or users to consider and decide which are the preferred alternative governance models or elements for which roadmaps should be developed. Appreciative Inquiry (AI) approach is an organizational development process that engages individuals within an organization in renewal, change, and focused performance. The AI approach focuses on successes and opportunities to improve things by looking forward, rather than looking back on the problems or issues. The AI workshop is expected to facilitate consensus among decision-makers to contribute to the endorsement of the roadmap(s), and to encourage utilization of the evaluation findings. The workshop will involve a creative thinking process that will build on existing successes, identify and rank preferred alternatives, and ultimately develop a plan to strengthen the EHC program’s governance system.
An agenda for the Appreciative Inquiry Workshop is provided in Attachment F.
4) Document Review (will not impose a burden on research participants) and
5) Interviews with staff at international organizations of similar purpose (9 persons or less; will not impose a burden on U.S. citizens).
The latter two activities do not require OMB approval and are not discussed further. The information collected will ultimately be used to develop a roadmap, including at least three alternative models of governance and operation, to be submitted to AHRQ that could be used to help guide future programmatic development.
These data collection activities are necessary because the program’s stakeholders possess knowledge and information that cannot be obtained through other existing means.
AHRQ understands that information collection may not proceed until approved by OMB. An annual summary of activities conducted under this clearance will be provided to OMB on the anniversary of the approval date.
The purpose of the information collection is to study the governance structure of the EHC program at AHRQ. The findings are intended primarily for internal use by AHRQ staff in the EHC program, but findings may be shared with key government policy and management officials, other AHRQ staff, and other members of the EHC program.
The information will be used to identify strengths and weaknesses in the current EHC program’s governance structure, methods for engaging stakeholders and ensuring the transparency of the program, and approaches to national priority-setting. The information will ultimately be used to develop a roadmap to strengthen the EHC program’s governance system.
Information collections conducted under this clearance are not required by regulation. Surveys will be entirely voluntary, and information provided by respondents will be presented in aggregate reports so that no individually identifiable information will be released.
Improved electronic technology (e.g. web-based materials) will be used whenever possible to reduce the burden on the public. The voluntary survey, for example, will be administered via a web-based/online instrument called Survey Monkey, an intelligent survey software designed to create professional online surveys quickly and easily – and at low cost. The software provides encryption capabilities for ensuring confidential responses and for the exporting of responses for future analyses. The software also allows for the identification of responders and non-responders without compromising the confidentiality of responses. Web-based surveys have several advantages over other types of surveys, including the ability to monitor the response rate in real-time and send customized reminder emails to participants, with easier-to-follow skip patterns, and ensure confidential and secure data.
AHRQ’s EHC program staff, through extensive contacts with and knowledge of organizations and individuals in both the private and public sectors, indicated to the IMPAQ team that there has never been a study of the EHC program governance; therefore, the proposed information collection is not a duplication.
The survey instruments and procedures for completing the instruments have been designed to minimize the burden on all respondents and will not have a significant impact on small businesses or other small entities. Participation is entirely voluntary. The information being requested has been held to the absolute minimum required for the intended use.
This evaluation only uses a one-time data collection process. The consequence of foregoing data collection will be reduced efficiency or a less effective governance structure of the EHC program. This could lead to a program that is less responsive to the most pressing issues for stakeholders and produce products that are less useful for health care decision-makers as well as other users.
This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.
As required by 5 CFR 1320.8(d), notice was published in the Federal Register on April 24th, 2009 for 60 days (see Attachment B).
Persons outside of AHRQ were not consulted since this project is an internal study of a specific program’s governance within AHRQ, it does not relate to, or impact in any way, other Federal programs within or outside the U.S. Department of Health and Human Services (DHHS).
Each participant in the one-day Appreciative Inquiry workshop who is not a Federal employee will be provided a seventy-five dollar (US $75) honorarium to compensate participants for their time participating in the workshop. In addition, participants’ travel expenses and per diem will be reimbursed in accordance with General Services Administration (GSA) policies. Honoraria of $75 per day are warranted to enhance the participation of senior level managers and executives in the workshop. The provision of honoraria is a standard industry practice, intended to confer distinction on, or to symbolize respect, esteem, or admiration for, the participants in an activity that benefits the project. (NIH Manual 1130, Delegations of Authority, Acquisition #5, “Rates of Compensation (Honoraria) Under Professional Services Orders.”) However, this honorarium size will be re-evaluated and increased if the response rate becomes concerning.
No payments or gifts will be provided to participants of the other data collection activities (key informant interviews and online survey) because the time burden is minimal. However, if the survey response rate becomes concerning during the data collection stage, AHRQ will assess whether providing some incentives will boost the response rate.
Individuals and organizations will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). Participants will be advised that participation is voluntary. They will be informed about the purposes for which the information is collected, and that, in accordance with this statute, individually identifiable information about them will not be used or disclosed for any other purpose.
Individuals and organizations contacted will be further assured of the confidentiality of their replies under 42 U.S.C. 1306, and 20 CFR 401 and 4225 U.S.C.552a (Privacy Act of 1974). In instances where respondent identity is needed, the information collection will fully comply with all aspects of the Privacy Act.
Participants will not be identified in any publications of the study’s findings, including interim reports generated by the contractor, IMPAQ International, LLC, for AHRQ. All findings reported to AHRQ will be aggregate-level data to ensure the confidentiality of participants’ responses.
Identifying information, such as name, title, address, phone number, and email address, will be used only to send the participant a copy of the study’s findings, if AHRQ decides to make public the findings of this evaluation. Identifying information will be stored in locked file cabinets and secure electronic storage by the contractor, IMPAQ International, LLC. All identifying information will be destroyed upon completion of the study.
Key informant interviews will be audio recorded to improve data capture, only with the participants’ prior consent. Audio recordings will be used to create accurate transcripts of the interviews. Digital audio files will be saved in secure electronic storage and accessed only by password-protected computers. All audio files will be destroyed upon completion of the study.
The study protocol has been reviewed by the Institutional Review Board (IRB) at Abt Associates (see Attachment Q).
Participants will not be asked questions of a sensitive nature.
Exhibit 1 shows the estimated annualized burden hours for the respondents to participate in this evaluation. Key informant interviews will be conducted about the governance structure of the EHC program and will last about one hour. The on-line survey will be completed by 115 EHC program Research Centers Staff and 205 EHC Program Users and Stakeholders and will require about 20 minutes to complete. The Appreciative Inquiry workshop will be conducted with 20 participants and will last about 6 hours. The total burden hours are estimated to be 293 hours.
Exhibit 2 shows the estimated annualized cost burden based on the respondents' time to participate in the evaluation. The total cost burden is estimated to be $14,600.
Exhibit 1. Estimated Annualized Burden Hours
Activity Name |
Number of Respondents |
Number of Responses per Respondent |
Hours per Response |
Total Burden hours |
Key Informant Interviews with EHC Research Centers Staff |
13 |
1 |
1 |
13 |
Online Survey with EHC Research Centers Staff* |
115 |
1 |
20/60 |
38 |
Key Informant Interviews with EHC Stakeholder Group Members |
2 |
1 |
1 |
2 |
Key Informant Interviews with EHC Program Users and Stakeholders |
27 |
1 |
1 |
27 |
Online Survey with EHC Program Users and Stakeholders* |
205 |
1 |
20/60 |
68 |
In-Depth Key Informant Interviews with EHC Program Users and Stakeholders |
25 |
1 |
1 |
25 |
Appreciative Inquiry Workshop |
20 |
1 |
6 |
120 |
Total |
407 |
na |
na |
293 |
Note: The survey will be conducted on all individuals in the most updated AHRQ EHC program contacts database (the universe) at the time of the survey, which is estimated to be about 400. Since the database is not finalized yet, the exact number of individuals for these two types of survey respondents (Center staff or Program users/Stakeholders) may change (not increase). However, we do not expect the total number of surveys and resulting public burdens will exceed the current estimate.
Exhibit 2. Estimated Annualized Cost Burden
Activity Name |
Number of Respondents |
Total Burden hours |
Average Hourly Wage Rate* |
Total Cost Burden |
Key Informant Interviews with EHC Research Centers Staff |
13 |
13 |
$54.27 |
$706 |
Online Survey with EHC Research Centers Staff |
115 |
38 |
$54.27 |
$2,062 |
Key Informant Interviews with EHC Stakeholder Group Members |
2 |
2 |
$43.52 |
$87 |
Key Informant Interviews with EHC Program Users and Stakeholders |
27 |
27 |
$46.73 |
$1,262 |
Online Survey with EHC Program Users and Stakeholders |
205 |
68 |
$46.73 |
$3,178 |
In-Depth Key Informant Interviews with EHC Program Users and Stakeholders |
25 |
25 |
$46.73 |
$1,168 |
Appreciative Inquiry Workshop |
20 |
120 |
$51.14 |
$6,137 |
Total |
407 |
293 |
na |
$14,600 |
* Wage rates were calculated using the following data: (1) For the Governance Interviews and the Online Survey with EHC Research Centers Staff the hourly rate is a weighted average for physicians ($58.76 per hour) and medical and health services managers ($37.82); (2) for the Governance Interviews with EHC Stakeholder Group Members the hourly rate is the rate for average for medical and health services managers ($37.82); (3) for the Governance Interviews and the Online Survey with EHC Program Users and Stakeholders the hourly rate is a weighted average for physicians ($58.76 per hour), general and operations managers ($43.52 per hour), medical and health services managers ($37.82 per hour), and social and community service managers ($24.73 per hour); (4) for the Workshop the hourly rate is a weighted average for physicians ($58.76 per hour) and general and operations managers ($43.52 per hour) from the mean of the average wages, National Compensation Survey: Occupational Wages in the United States 2006, U.S. Department of Labor, Bureau of Labor Statistics.
There are no direct costs to respondents other than their time to participate in the study.
Exhibit 3 shows the estimated cost of the project, including the cost of developing the methodology and data collection instruments, collecting, and analyzing the data, publishing the results, etc. The work will be carried out by IMPAQ International and Abt Associates under contract to the Agency for Healthcare Research and Quality.
Exhibit 3. Estimated Annualized Cost to the Government
Cost Component |
Total Cost* |
Annualized Cost |
Project Development |
$137,901 |
$55,196 |
Data Collection Activities |
$181,672 |
$77,930 |
Data Processing and Analysis |
$173,077 |
$69,506 |
Publication of Results |
$63,686 |
$27,392 |
Project Management |
$97,236 |
$40,229 |
Total |
$653,572 |
$270,253 |
*Please note the costs include fully loaded costs (overhead, G&A).
This is a new collection of information, so this item is not applicable.
The time schedule for the project is described in Exhibit 4 below:
Exhibit 4: Tentative Schedule for Major Evaluation Activities
Activity |
Estimated Schedule |
Draft Evaluation Plan Submitted to AHRQ |
November 12, 2008 |
Comments on Evaluation Plan from AHRQ |
November 25, 2008 |
Final Evaluation Plan Submitted to AHRQ |
December 23, 2008 |
60-Day Federal Register Notice Published |
April 2009 to June 2009 |
IRB Clearance Waiver Received (Estimated) |
Fall 2009 |
OMB Clearance Received (Estimated) |
Approx February 2010 |
Start of Data Collection |
When OMB Clearance is received (~ February 2010) |
Key Informant Interviews Conducted |
Beginning when OMB Clearance is received, ending approx 3 months later (~ March 2010 to May 2010) |
Survey Administered |
Beginning approx 2 months after when OMB Clearance is received, ending approx 1 month later (~ May 2010 to June 2010) |
In-Depth Key Informant Interviews Conducted |
Beginning at conclusion of Key Informant Interviews (approx 3 months after when OMB Clearance is received), ending approx 2 months later (~ June 2010 to August 2010) |
Appreciative Inquiry Workshop Held |
Beginning approx 1 month after conclusion of In-Depth Key Informant Interviews (approx 6 months after when OMB Clearance is received), ending 1 day later. (~ August 2010) |
End of Data Collection |
Approx 6 months after OMB Clearance is received (~ August 2010) |
The analyses and findings produced in this project are intended primarily for internal use but may be shared with key government policy and management officials, AHRQ staff, and other members of the EHC program.
Key Informant Interviews. Content analysis, which aggregates chunks of texts into content categories based on a coding scheme, is the most appropriate analytical technique for this type of data and will be used to analyze the key informant interviews.
Additionally, the information collected from the 25 detailed interviews on the success or impact of an EHC program product that results from its governance element or approach, or about a specific, important governance element will be integrated into a detailed timeline of events. Then, each step along the chain of events will be analyzed for its dependency on the step preceding it, for any competing dependencies, or other competing results. Process tracing methods will be used to search not only for data patterns that fit the stated hypothesis in terms of both the effect being the result of the causal process, but also that each link in the process played its expected role.
Survey. For the responses to categorical survey questions, statistical tools will be used to summarize and analyze the data for notable patterns. For multiple choice questions and true/false questions, the analysis will include pie graphs or bar charts whose divisions represent the proportion of respondents that selected each alternative. For questions associated with a rating scale, descriptive statistics will be presented. Further statistical analysis will be conducted to determine the significance of the results. The responses to open-ended questions will be evaluated using content analysis. Survey responses will be aggregated and analyzed separately for the two target audiences: EHC program Research Centers Staff and EHC program stakeholders. In addition, staff responses will be aggregated at the EHC program level (e.g., 13 DEcIDE centers combined), and the stakeholder responses at the group level (e.g., Federal and state policymakers, provider groups). Cross-tabulations will display any differences among the groups.
Appreciative Inquiry Workshop. The results of the workshop will be summarized in a report that will be used to complete the implementation phase of the project, the development of a roadmap for future programmatic development alternatives.
AHRQ does not seek this exemption.
Attachment E: Online Survey
Attachment F: Appreciative Inquiry Workshop
Attachment G: Advance Letter about the Evaluation
Attachment H: Cover Email for Interview
Attachment I: Scheduling Email for Interview
Attachment J: Reminder Scheduling Email for Interview
Attachment K: Thank You Email for Interview
Attachment L: Script for Follow Up Scheduling Phone for Interview
Attachment M: Cover Email for Survey
Attachment N: Email Invitation for Survey
Attachment O: Follow-up Reminder Email for Survey
Attachment P: Thank You at End of Survey
Attachment Q: IRB Waiver Determination Letter
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | OMB Clearance Application |
| Author | hamlin-ben |
| File Modified | 0000-00-00 |
| File Created | 2021-02-03 |