Supporting Statement
Section A
Evaluation of the Technical Assistance to
ARRA Complex Patient Grantees Project
July 11th, 2011
Agency for Healthcare Research and Quality
Table of Contents
A.1. Circumstances Making the Collection of Information Necessary 3
A.2. Purpose and Use of the Information Collected 5
A.3. Use of Improved Information Technology and Burden Reduction 6
A.4. Efforts to Identify Duplication and Use of Similar Information 6
A.5. Involvement of Small Businesses or other Small Entities 6
This project does not involve or impact any small entities. 6
A.6. Consequences if Information Collected Less Frequently 6
A.8. Federal Register Notice and Outside Consultations 7
A.9. Payments/Gifts to Respondents 7
A.10. Assurance of Confidentiality 7
A.11. Questions of a Sensitive Nature 8
A.12. Estimates of Annualized Burden Hours and Costs 8
A.14. Estimates of Annualized Cost to the Government 9
A.15. Changes in Hour Burden 9
A.16. Time Schedule, Publication and Analysis Plan 10
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see http://www.ahrq.gov/hrqa99.pdf), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions.
According to its authorizing legislation, AHRQ shall promote health care quality improvement by conducting and supporting:
1. research that develops and presents scientific evidence regarding all aspects of health care;
2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and
3. initiatives to advance private and public efforts to improve health care quality.
Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.
Under the American Recovery and Reinvestment Act (ARRA) of 2009, the Agency for Health Research and Quality (AHRQ) awarded $473 million in grants and contracts to support patient-centered outcomes research. As part of this investment, AHRQ funded fourteen R21 (exploratory) grants and thirteen R24 (infrastructure development) grants to generate new knowledge on individuals with multiple chronic conditions. This work is critical to improve the understanding of how to prioritize evidence-based services for patients with multiple co-morbidities and to suggest appropriate adaptations to guidelines for their care.
In order to support the R21 and R24 complex patient grantees, AHRQ funded a Learning Network and Technical Assistance Center (LN&TAC) to encourage collaboration among the researchers and help them share research methods, definitions and products through in-person meetings, small workgroups and network facilitation. The LN&TAC will provide the grantees with technical assistance regarding research design, data collection, data analysis, public use dataset development, and dissemination.
Through the LN&TAC AHRQ will support work to:
Create and support a Learning Network of the complex patient grantees to facilitate advancement of infrastructure development, as well as to leverage developments and learning across the program. The Learning Network will give these grantees the opportunity to share information with and learn from other research teams, provide resources for data management and other research-related issues, and synthesize and disseminate findings that transcend individual projects.
Provide both group and individual technical assistance to grantees as they address issues of ARRA reporting, infrastructure development, data sharing, and creation of public use data sets.
Disseminate results, including developing materials targeted to researchers and policy-makers to describe study results and facilitate future use of newly created datasets. This will include a marketing plan to advertise availability of datasets and promote their use.
Develop and implement an evaluation of the above activities throughout the project.
The purpose of this Information Collection Request is to evaluate the effectiveness of the LN&TAC. The goals of the evaluation are to:
1) Ascertain whether expected outcomes of the LN&TAC were achieved;
2) Assess whether the LN&TAC met the needs and expectations of the grantees;
3) Identify challenges and lessons learned, and determine the feasibility and advisability of developing similar project models in the future.
To meet the goals of this evaluation the following data collections will be implemented:
1) LN Meeting Evaluation -- Grantees who attend the three annual in-person Learning Network meetings will be asked to complete the LN Meeting Evaluation (Attachment A) to provide immediate feedback about their level of satisfaction with the meeting (including session topics and speakers) and make suggestions about how the meeting could be improved.
2) Group TA Evaluation -- Grantees who participate in group technical assistance activities, such as webinars and the TA given at annual meetings, will be asked to complete the Group TA Evaluation (Attachment B) to provide feedback about their level of satisfaction with the group TA (including session leader), how effective the TA was, and make suggestions about how the TA session could have been better.
3) Individual TA Evaluation -- Grantees who request individual technical assistance will be asked to complete the Individual TA Evaluation (Attachment C) to provide feedback about their level of satisfaction with the TA (including session leader), how effective the TA was, and make suggestions about how the TA session could have been better.
4) Annual Survey -- All 27 Complex Patient grantees will be asked to complete the Annual Survey (Attachment D) once a year. This survey is designed to measure whether, due to their participation in the project, grantees have experienced changes in knowledge, confidence or attitudes related to research activities and grant requirements, changes in their research itself (design, methods, and/or analyses), and/or if participation has increased collaboration (e.g., sharing methods, developing new coding, merging data sets) among the Complex Patient researchers, as well as satisfaction with the LN&TAC in general.
5) Annual Interview -- The Annual Interview (Attachment E) will be administered with a small subset of 5 grantees per year, and will be used to augment the Annual Survey with more in-depth qualitative data. Therefore, similar questions will be asked in the Annual Interview as are asked in the Annual Survey, but the interview will allow for probing and clarification of answers. Different grantees will be asked to participate in the interview each year, such that no grantee participates in the Annual Interview more than once during the three year contract.
This study is being conducted by AHRQ through its contractor, Abt Associates, pursuant to AHRQ’s statutory authority to “conduct and support research, evaluations, and training, support demonstration projects, research networks and multidisciplinary centers, provide technical assistance, and disseminate information on healthcare and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services. 42 U.S.C. 299a(a)(1).
The purpose of the evaluation of the Technical Assistance to ARRA Complex Patient Grantees project will be to review and document the activities of the LN&TAC, ascertain whether expected outcomes were achieved, and inform AHRQ of the feasibility and advisability of developing similar project models in the future. The evaluation will assess whether the LN&TAC was relevant to, and met the needs of, the grantees; and will identify challenges and lessons learned.
This evaluation will use information collected as part of regular project implementation, as well as data collected specifically for purposes of the evaluation.
These evaluation instruments are designed to capture a combination of quantitative and qualitative data
No claim is made that the results from this study will be generalizable in the statistical sense. Rather, this evaluation is aimed at determining the effectiveness of this particular program.
Whenever feasible, we will use information technology to minimize respondent burden. To this end, the Group TA Evaluation, Individual TA Evaluation, and the Annual Survey will all be administered as web-based surveys, using Checkbox software from Prezza Technologies. The LN Meeting Evaluation will be administered at each in-person meeting, and therefore will be administered as a pencil and paper survey, and the Annual Interview will be administered via telephone interviews.
This is an evaluation of the current Complex Patient LN&TAC, and therefore the instruments are specific and unique to this project. Also, a literature review was conducted and no previous studies evaluating a learning network (or learning collaborative) of researchers who study patients with multiple chronic conditions could be found, so we are assured that the evaluation of this project is not a duplicative effort.
Grantees will be asked to complete the LN Meeting Evaluation at the end of each of the three in-person Learning Network Meetings. It is important to get feedback from the participants after each meeting so that we can understand what aspects of the meetings worked well from their perspective, and which did not, so that future meetings can be adapted to best meet the grantees’ needs. Group TA Evaluations will be distributed to all participants when group technical assistance activities (such as webinars) take place, and Individual TA Evaluations will be distributed whenever a grantee’s request for technical assistance has been completed. Because these TA activities will vary widely (the topics, leaders and format will all differ), it will be important to get feedback from the participants after each activity so that the team can understand what the participants thought was successful and what was not, again so that future activities can be adapted accordingly. The Annual Survey and Annual Interview will take place during each of the three years of the contract. It is important that we conduct these each year so that all project related activities which took place during that calendar year are still fresh in minds of the grantees, allowing them to report fully on the range of their project related activities, including any collaborations. It is also important that we measure the grantees’ attitudes, behaviors and knowledge at multiple points throughout the project so that we can determine if their participation in this project has had an effect over time. There are no legal obstacles to reduce the burden.
This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.
As required by 5 CFR 1320.8(d), notice was published in the Federal Register on August 3rd, 2011 for 60 days and again on October 5th, 2011 for 30 days (see Attachment F). No comments were received.
Abt Associates is the contractor who will lead the data collection and analysis for the evaluation of the Complex Patient LN&TAC for AHRQ, along with their subcontractors MacColl and ICPSR. Since its founding in 1965, Abt Associates has helped shape U.S. healthcare policy through research, studies and analyses, comprehensive evaluations, and technical assistance. Abt partners with federal, state and local governments, and foundations to evaluate and implement key health initiatives.
No honoraria or incentives will be offered to the grantees for their participation in evaluation activities.
Individuals and organizations will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). They will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose.
No confidential or protected health information will be collected for this project. Additionally, no demographic information will be collected. Abt IRB has reviewed this project and determined that these activities are exempt from IRB review. Participation in all evaluation activities is voluntary. Some of the data for this evaluation will be collected with personal identifiers for the purpose of tracking changes over time. Participants will be assured that their identified data will not be shared outside of the research group, that data will only be reported in the aggregate, and where quotes are used for illustrative purposes the source will not be identified.
The evaluation instruments do not contain any questions concerning sexual behavior and attitudes, religious beliefs, income or proprietary business information, nor will we collect demographic information from grantees.
Exhibit 1 shows the estimated annualized burden hours for the grantees' time to participate in the surveys and interviews. The LN Meeting Evaluation will be completed be about 22 grantees and takes about 20 minutes to complete. The Group TA Evaluation will be completed by 8 grantees 4 times a year, although not necessarily the same 8 persons each time and will take 5 minutes to complete. The Individual TA Evaluation will be completed by about 15 grantees annually and takes 5 minutes to complete. The Annual Survey will be completed by 22 grantees and will take about 10 minutes to complete. Annual Interviews will be conducted with 5 persons annually and will last 45 minutes. The total annualized burden hours are estimated to be 19 hours.
Exhibit 2 shows the estimated annualized cost burden for the grantees' time to provide the requested data. The estimated total cost burden is about $774.
Exhibit 1. Estimated
annualized burden hours
Form Name |
Number of respondents |
Number of responses per respondent |
Hours per response |
Total burden hours |
LN Meeting Evaluation |
22 |
1 |
20/60 |
7 |
Group TA Evaluation |
8 |
4 |
5/60 |
3 |
Individual TA Evaluation |
15 |
1 |
5/60 |
1 |
Annual Survey |
22 |
1 |
10/60 |
4 |
Annual Interview |
5 |
1 |
45/60 |
4 |
TOTAL |
72 |
na |
na |
19 |
Exhibit 2. Estimated annualized cost burden
Form Name |
Number of respondents |
Total burden hours |
Average hourly wage rate* |
Total cost burden |
LN Meeting Evaluation |
22 |
7 |
$40.75 |
$285 |
Group TA Evaluation |
8 |
3 |
$40.75 |
$122 |
Individual TA Evaluation |
15 |
1 |
$40.75 |
$41 |
Annual Survey |
22 |
4 |
$40.75 |
$163 |
Annual Interview |
5 |
4 |
$40.75 |
$163 |
TOTAL |
72 |
19 |
$40.75 |
$774 |
* Based upon the mean hourly wage rate for Medical Scientists, Except Epidemiologists, from the National Compensation Survey: Occupational wages in the United States May 2009, “U.S. Department of Labor, Bureau of Labor Statistics,” accessed on April 26, 2011.
A.13. Estimates of Annualized Respondent Capital and Maintenance Costs
Capital and maintenance costs include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. There are no direct costs to respondents other than their time to participate in the study.
The total cost of this contract to the government is $178,137 over the three years of the project (September 27, 2010 to September 26, 2013). Therefore, the annualized cost to the government of the evaluation of the Complex Patient LN&TAC is $59,379.
Exhibit 3. Estimated Total and Annualized Cost
Cost Component |
Total Cost |
Annualized Cost |
Project Development |
$70,247 |
$23,416 |
Data Collection Activities |
$54,636 |
$18,212 |
Data Processing and Analysis |
$31,220 |
$10,406 |
Overhead |
$22,034 |
$7,345 |
Total |
$178,137 |
$59,379 |
This is a new information collection effort.
Time Schedule
The project timeline is shown in Exhibit 4 below.
Exhibit 4. Project Timeline
Activity |
Time Schedule |
LN Meeting Evaluation |
Meeting 1: March 2011 (completed)* Meeting 2: September 2011 Meeting 3: September 2012 |
Group TA Evaluation |
Ongoing: As soon as OMB Clearance is obtained |
Individual TA Evaluation |
Ongoing: As soon as OMB Clearance is obtained |
Annual Survey |
Year 1: August 2011 Year 2: August 2012 Year 3: May 2013 |
Annual Interview |
Year 1: August 2011 Year 2: August 2012 Year 3: May 2013 |
Final Report |
September 2013 |
*OMB Clearance is not required for meeting evaluations, and therefore the LN Meeting Evaluation instrument was used to evaluate the first Complex Patient Learning Network meeting in March of 2011, prior to OMB Clearance. The LN Meeting Evaluation instrument is included in this package for the sake of thoroughness.
Publication Plan
Study results will be disseminated through peer-reviewed publications; presentations at professional conferences, including AHRQ’s Annual Conference; and AHRQ’s website. Manuscripts and presentations will clearly state the limitations of the study findings including the lack of generalizability of the specific results associated with the research methods.
Analysis Plan
Descriptive statistics will be calculated for all survey items. Survey responses will be aggregated and frequency distributions will be compiled. Changes over time will be measured using t-tests to compare the differences in means for continuous variables and chi-square or Fisher’s exact tests will be performed to compare differences for categorical data. Responses to open-ended questions will be coded using content analytic techniques.
AHRQ does not seek this exemption.
Attachment A: LN Meeting Evaluation
Attachment B: Group TA Evaluation
Attachment C: Individual TA Evaluation
Attachment D: Annual Survey
Attachment E: Annual Interview
Attachment F: Federal Register Notice
| File Type | application/msword |
| Author | SubramanianR |
| Last Modified By | DHHS |
| File Modified | 2011-10-07 |
| File Created | 2011-10-05 |