SUPPORTING STATEMENT
Voluntary Questionnaire and Data Collection Testing
to Pilot Test Proposed Medical Office Surveys on Patient Safety
A. JUSTIFICATION
1. Need for Information
The Agency for Healthcare Research and Quality (AHRQ) requests that the Office of Management and Budget (OMB) approve, under the Paperwork Reduction Act of 1995, AHRQ’s intention to pilot test survey questions for two new survey instruments, the Medical Office Surveys on Patient Safety, in preparation of work to be performed by Westat in collaboration with AHRQ.
As mandated in the Healthcare Research and Quality Act of 1999 (P.L. 106-129), the mission of AHRQ is to enhance the quality, appropriateness, and effectiveness of health services and access to such services, through the establishment of a broad base of scientific research [Section 901 (b)]. With respect to its role as the principal agency for health care research and quality and providing scientific and technical support for private and public efforts to improve health care quality, AHRQ shall promote health care quality improvement by conducting and supporting research that develops and presents scientific evidence regarding all aspects of health care--including the outcomes, effectiveness, and cost-effectiveness of health care practices, including preventive measures; and methods for measuring quality and strategies for improving quality; as well as initiatives to advance private and public efforts to improve health care quality. [Section 901, (b) (1) (B) and (F) and (3)].
In carrying out its mission and addressing the full continuum of care, AHRQ shall conduct and support research activities with respect to clinical practice, including primary care and practice-oriented research [Section 902 (a) (4)], and support strategies and mechanisms to link research to practice improvement [Section 911 (b) (1)]. The importance of the ambulatory care setting as a focus of safety and quality improvement is demonstrated by AHRQ’s recent grant announcements of new funding for four areas of research aimed at improving the safety and quality of ambulatory health care.
Furthermore, AHRQ shall conduct and support research to identify the causes of preventable health care errors and patient injury in health care delivery; develop, demonstrate, and evaluate strategies for reducing errors and improving patient safety; and disseminate such effective strategies throughout the health care industry. [Section 912, (c) (1) (2) and (3)].
To help AHRQ meet these goals as they pertain to the ambulatory healthcare setting, pilot data
collection under this request will be done for the following purposes:
a) Developing the Medical Office Surveys on Patient Safety—Two surveys—one for providers and one for all other office staff—will be pilot tested in 97 ambulatory outpatient medical offices. A provider is a physician, physician assistant, or nurse practitioner who diagnoses, treats patients, and prescribes medications. Staff refers to all other people working in the office. The office manager of every medical office will be asked to complete several additional questions about the background characteristics of the office (number of staff, whether it is a single or multi-specialty office, etc.)
b) Psychometric Analysis—Provide information for the revision and shortening of the final surveys based on an assessment of the reliability and construct validity of each survey’s items and composites.
The goal of the final surveys is to make them available for use in the public domain by researchers, healthcare systems, and medical offices to assess patient safety culture.
These preliminary research activities are not required by regulation, and will not be used by AHRQ to regulate or sanction its customers. They will be entirely voluntary and the confidentiality of individual respondents and their responses will be preserved.
2. How, by Whom, and for What Purpose Information Will Be Used
The information collected will be used to test draft survey items from the Medical Office Surveys on Patient Safety described in item #1. The end result will be improvement in the final survey instruments which will be made available to the public for use in ambulatory outpatient medical offices to assess their safety culture from the perspectives of their providers and staff. The surveys can be used by outpatient medical offices to identify areas for patient safety culture improvement. Researchers are also likely to use the ambulatory surveys to assess the impact of ambulatory medical office patient safety culture improvement initiatives. These ambulatory medical office surveys are an expansion of AHRQ’s Hospital Survey on Patient Safety Culture (HSOPSC) which was pilot tested (OMB # 0935-0115, exp 1/31/2004) and made available to the public in November 2004 (www.ahrq.gov/qual/hospculture). It has since been used by hundreds of hospitals across the U.S. to assess patient safety culture. The medical office surveys contain new and revised questions and composites that more accurately apply to the outpatient ambulatory medical office setting.
Psychometric analysis will be conducted on the pilot data to examine item non-response, item response variability, reliability, and construct validity of the items included in the surveys. Because the survey items are being developed to measure specific aspects of patient safety culture in the ambulatory setting, the factor structure of the survey items will be evaluated through multi-level confirmatory factor analysis. After dropping items or factors based on the data analysis, final versions of the surveys will be released for use in the public domain.
3. Use of Improved Information Technology
Data collection will not involve the use of any information technology. This data collection effort is a pilot study designed to gather information to examine the psychometric properties (internal consistency reliability, response variability, etc.) of paper-and-pencil ambulatory medical office safety culture survey instruments. The use of information technology is not needed for this study and is not likely to result in a reduction of burden at this time.
4. Efforts to Avoid Duplication
There are a number of different types of surveys for the ambulatory outpatient medical office setting that currently exist. One type focuses on the use of information technology in medical offices, given that many outpatient settings are moving toward adoption of electronic medical records and updating their practices using other technological innovations. There are also surveys designed specifically for physicians, that assess physician satisfaction with their work or other issues. There are surveys for patients that ask about patient satisfaction with their provider and their healthcare. Of the surveys that focus on safety issues, many focus on a single specific safety issue, such as medications. Another type of safety survey is more like a group self-assessment checklist of the policies and practices that should ideally be implemented in the outpatient setting. Still other surveys measure general organizational culture in the outpatient setting.
In our review of all of these surveys, there is one non-proprietary survey that obtains physician and staff opinions about patient safety culture in ambulatory settings; however the survey items were originally developed for the hospital setting and were only modified slightly for use in the outpatient setting. Therefore, we did not find a survey that assesses patient safety culture (which is a more specific area of an organization’s overall culture) that was developed specifically for the ambulatory setting (rather than the hospital setting) that seeks to obtain input from the perspective of providers as well as non-provider staff. The proposed pilot study will allow for the development and pilot testing of patient safety culture surveys specifically designed to measure issues relevant to patient safety in the ambulatory outpatient medical office setting. The pilot data collection will provide information about the reliability and construct validity of the instruments, with the final goal of making the surveys available in the public domain to be used by medical offices and healthcare organizations as an assessment tool that can provide input toward their patient safety culture improvement efforts.
5. Small Businesses
The survey instruments and procedures for completing the instruments are designed to minimize burden on individual provider and office staff respondents and will not have a significant impact on small businesses or other small entities.
6. Consequences of Less Frequent Collection
This effort is a one-time pilot test.
7. Consistency with the Guidelines in 5 CFR 1320.5(d)(2)
The data collection efforts will be consistent with the guidelines at 5 CFR 1320.5(d)(2).
8. Applicability to 5 CFR 1320.8(d)
a. Notice was published in the Federal Register on Wednesday, Dec. 6, 2006—Attachment C. Thus far, one individual comment has been received but was received too late to be addressed in the final 30 day Federal Register notice:
“dhs ahrq pilot study of proposed medical office survey on patient safety
“i think this is a waste of tax dollars. i do not think the survey will turn up anything of value to American citizens. your proposal certainly does not say what you will do with the information you want to gather. why gather if there is no purpose to it. just to
tell us in a press release what you found. how does that help?
“i think this project does not help. efforts instead must be focused on making them safer.
the offices surveyed will be expert in returning the survey showing how ‘medically right’ they are.”
In response to this comment, the purpose of the survey data collection from medical office physicians and staff will be to pilot test newly developed surveys on patient safety and healthcare quality in medical office settings. The pilot test will be done on only 97 medical offices for the purposes of pretesting the instruments, identifying which items work the best to assess patient safety culture, and developing final surveys that will be made available in the public domain. The goal of the pilot test is to not to have the government conduct a national survey of medical offices and report on the results, but to collect data for the purposes of testing and finalizing surveys that can be used by medical offices and health systems for their own self-assessment, as well as providing a tool for researchers interested in examining patient safety culture in ambulatory settings.
The surveys are intended to educate staff in medical offices about important patient safety issues they need to be aware of, and to provide a tool for self-assessment so medical offices and health systems can determine how physicians and staff view patient safety and quality in their medical offices and identify areas for improvement. When the final surveys are used by medical offices and health systems, they will provide valuable information to identify areas for patient safety and quality improvement.
A similar patient safety culture survey has already been developed for hospitals and has been widely used by hundreds of hospitals across the U.S. and has even been translated into many different languages for use in hospitals internationally. Hospitals have found the hospital patient safety culture survey a very useful tool in their patient safety improvement efforts. AHRQ believes the ambulatory patient safety culture surveys are just as important and needed as the hospital survey because many more patients are seen in ambulatory settings.
b. The contractor for this project has subcontracted with a prominent ambulatory patient safety researcher, John Hickner, MD, to serve as a consultant throughout the survey development and pilot testing of this project. A number of other physicians, patient safety researchers, and stakeholders in healthcare organizations were also consulted for background interviews on the need for a patient safety culture survey for the ambulatory setting as well as for review and comment on the draft survey domains and items—see Attachment A for a list of those consulted outside the Agency thus far.
9. Remuneration of Respondents
In consultation with the study’s patient safety research consultant, Dr. John Hickner, and representatives from Practice-Based Research Networks (PBRNs) with whom we are working to develop the pilot test data collection plan, we have determined that some type of incentive will be necessary to recruit medical offices to participate in the study which asks all providers and staff to complete a 15-minute paper-based survey. These sources have indicated, through their prior experience with research in medical practices, that to ensure high response rates for our paper survey, it is typical and expected for medical offices to receive some type of remuneration for participation in research studies.
Because it is too expensive for us to consider an individual-level incentive, and given that physicians would expect more remuneration for completing a survey than staff (which would be problematic), we have determined that a site-level incentive would be the best way to ensure site participation as well as providing some benefit to all the individuals within an office. Our plan is to recruit a site-level point of contact (POC) in each outpatient medical office who will manage data collection at that office (compile sample information, distribute surveys, promote survey response, etc.). We plan to provide the site as a whole with a one-time office-level incentive of $300, which we would encourage them to use toward the provision of lunch or something else that would benefit each potential respondent within the office.
The American Academy of Family Physicians (AAFP), one of our pilot study partners, typically provides remuneration for participation in their research studies and comments on the proposed $300 incentive for the present study:
“The American Academy of Family Physicians National Research Network (AAFP NRN) generally pays participating practices approximately $100/hour for time required to complete a research study. We estimate that each practice will have to devote approximately 4.5 - 5.5 hours in order to complete this study, as follows: 1 hour - training of practice Study Coordinator [our POC]; 1 hour - put together sample information (names of staff and physicians and their positions); 30 minutes - complete survey of basic background characteristics about the practice; 2-3 hours - conduct data collection and follow-up for nonresponse. Consequently, the $300 proposed practice stipend is less than the amount generally paid by the AAFP NRN to participating practices for a study such as the Medical Office Surveys on Patient Safety.” [Email communication from Debbie G. Graham, MSPH, Associate Research Director, AAFP National Research Network, on January 5, 2007.]
We asked another partner, the PROS (Pediatric Research in Office Settings) Network, which is a practice-based research network that was established by the American Academy of Pediatrics (AAP), to describe their usual practice and their rationale for why a site-level incentive is necessary for this study:
“We are unaware of actual experimental evidence to back up the use of office-based rather than individual incentives, but we do have some relevant network-based experience. It is the customary practice of the PROS network to compensate practices (in part) for the time and effort spent on study completion by offering partial remuneration for time spent by the practice. We have anecdotal but consistent experience within the network over time suggesting its effectiveness in boosting our participation rate in studies, particularly for those studies requiring the completion of lengthy study instruments by numerous practice respondents.
“We generally send a check for a set amount (ideally made out to the practice or clinic site, to avoid tax consequences) to the contact practitioner at each site, suggesting that s/he oversee the use of the funds in a way appropriate to that site. Sites have done different things with the money. We have reports of celebratory feedings (e.g., pizza for everyone today), boxes of chocolates, flowers, and donations to favored charities. We have received feedback from practices indicating that allowing each site latitude in how the funds are spent, creates the best impact.” [Email communication from Eric J. Slora, PhD, Director, AAP Division of Primary Care Research, American Academy of Pediatrics, on February 1, 2007.]
Our proposed site-level remuneration of $300 therefore, appears to be on the low end as far as usual practice for research participation by medical offices, but some site-level remuneration appears to be a recognized practice when administering surveys in ambulatory outpatient medical office settings, without which, it would be very difficult to achieve appropriate and adequate participation at the site level and individual level.
10. Assurance of Confidentiality
Individuals and organizations contacted will be assured of the confidentiality of their replies under Section 924(c) of the Healthcare Research and Quality Act of 1999.
11. Questions of a Sensitive Nature
We do not believe there are questions of a particularly sensitive nature included in the surveys, but if during cognitive testing sensitivities are discovered, such questions will be modified to ensure they are not of a sensitive nature.
12. Estimates of Annualized Hour Burden
The surveys will be distributed to a total of approximately 2,340 individuals from 97 medical offices (about 592 providers and 1,748 medical office staff), with a target response rate of 70%, or 1,638 completed surveys (414 completed provider surveys and 1,224 medical office staff surveys). Standard techniques like using a cover letter of support from the medical office, a reminder postcard, and distribution of a second survey will be used to achieve the target response rate. Respondents should take approximately 15 minutes to complete the survey. Therefore, in Table 1 we estimate that the total respondent burden for completing the survey will be 410 hours (1,638 completes multiplied by 0.25 hours per completed survey).
TABLE 1 ANNUAL BURDEN ESTIMATE FOR PILOT OF MEDICAL OFFICE SURVEYS
Type of Respondent |
Estimated Number of Respondents |
Number of Responses per Respondent |
Estimated Time per Respondent (hours) |
Estimated Total Respondent Burden (Total Hours) |
Avg. Medical Office Wage per Hour |
Annualized Cost Burden to Respondents |
Provider |
414 |
1 |
0.25 |
104 |
$61.09 per hour |
$6,353.36 |
Medical office staff |
1,224 |
1 |
0.25 |
306 |
$15.65 per hour |
$4,788.90 |
Total |
1,638 |
----------- |
----------- |
410 |
------------ |
$11,142.26 |
We used the U.S. Department of Labor, Bureau of Labor Statistics 2005 National Occupational Employment and Wage Estimates (http://www.bls.gov/oes/current/oes_nat.htm#b29-0000) to estimate a mean hourly wage for providers and medical office staff.
The mean hourly wage for providers (Family/General Practice; General Internist; OB/GYN; General Pediatrics; Physicians and Surgeons, All Other; Physician Assistant; and Nurse Practitioner) is estimated to be $61.09 per hour.1 The mean hourly wage for all other medical office staff (Registered Nurse; Medical and Clinical Laboratory Technician; LPN/LVN; Medical Records and Health Information Technicians; Nursing Aide; Medical Assistant; Medical Transcriptionist; and Healthcare Support Workers, All Other) is estimated to be $15.65 per hour.
13. Estimates of Total Annual Cost Burden to Respondents
The only cost to the respondent will be that associated with their time to respond to the information collection, as shown in Table 1. There will not be any costs for capital equipment or operational expenses.
14. Estimates of Annualized Cost to the Government
The total one-time cost for this contracted survey development and analysis effort for the provider survey is approximately $257,000, and for the medical office staff survey is approximately $268,000. These estimates include the costs of background literature reviews, survey development, cognitive testing, preparation of survey administration procedures, pilot data collection, incentives, data analysis, and preparation of a final report. The estimated cost of only the data collection components is approximately $250,000, which includes incentive costs, labor costs, fringe expenses, administrative expenses, and costs associated with copying, postage, and telephone expenses for both surveys.
15. Change in Burden
This is a new activity.
16. Plans for Analyses
The purpose of this survey effort is to develop and pilot test two new surveys—Medical Office Surveys on Patient Safety—and to perform a psychometric analysis of the pilot test data. Psychometric analyses of the pilot data will involve an examination of item-level non-response and variability. In addition, since the survey items were developed to measure apriori domains of patient safety culture, multi-level confirmatory factor analyses will be conducted to determine whether the survey items appear to assess these domains at both the individual and medical office levels. ICC(1)s and goodness-of-fit indices will be examined. Any problematic items or factors will be dropped. The final surveys will be made available in the public domain for use by researchers, healthcare systems, and medical offices to assess patient safety culture.
17. Exemption for Display of Expiration Date
No exemption is being requested.
18. Certifications
These activities will comply with the requirements of 5 CFR 1320.9.
Attachment A-List of Patient Safety Expert Reviewers and Consultants
Attachment B-Survey Materials
Cover letter
Reminder postcard
Survey
Attachment C- Federal Register Notice (separate file attachment)
Andy Brown, MD, MPH University of Mississippi Medical Center |
Gene Burke, MD Vice President of Medical Affairs Sentara Norfolk General Hospital |
Nancy C. Elder, MD, MSPH Associate Professor Department of Family Medicine University of Cincinnati |
David N.
Gans, FACMPE |
Debbie G. Graham, MSPH Associate Research Director, National Research Network American Academy of Family Physicians |
Sheldon Greenfield, MD Professor of Medicine University of California, Irvine |
Terry
Hammons MD |
John Hickner, MD, MSc Professor of Family Medicine The University of Chicago Pritzker School of Medicine |
Randall Huss, MD President, St. John's Clinic |
Julie K.
Johnson, MSPH, PhD |
Charles Kilo, MD, MPH President of GreenField Health System |
John E. Kralewski, PhD Wallace Professor Division of Health Services Research and Policy University of Minnesota |
Anton Kuzel, MD Virginia Commonwealth University Chair, Department of Family Practice |
Mark Linzer, MD Chief, Section of General Internal Medicine University of Wisconsin School of Medicine |
John
T. Lynch ProHealth Physicians Research Network |
Linda Baier Manwell, MS University of Wisconsin-Madison Medical School Section of General Internal Medicine |
Wilson D. Pace, MD Department of Family Medicine University of Colorado Health Sciences Center |
Alison Peterson, MD Allina Hospitals & Clinics |
Douglas Roblin, PhD Research Department Kaiser Permanente |
Jonathan Rosen, MD Chair, Connecticut Center for Primary Care |
Gurdev Singh, PhD SUNY Buffalo Director, Patient Safety Research Center |
Ranjit Singh, MD SUNY Buffalo Patient Safety Research Center |
Eric J.
Slora, PhD |
Margaret Toth, MD Chief Quality Officer Delmarva Foundation for Medical Care Inc. |
Richard C. "Mort" Wasserman, MD, MPH Professor of Pediatrics, University of Vermont Director, Pediatric Research in Office Settings American Academy of Pediatrics |
Donna Woods, EdM, PhD Research Assistant Professor Northwestern University |
Attachment B
Cover Letter of Support from Medical Office
[Insert medical office logo here]
Hello,
[Insert name of medical office] is participating in an important survey sponsored by the Agency for Healthcare Research and Quality. The survey asks for your opinions about patient safety and healthcare quality in our office. Westat, a private research organization located in Rockville, Maryland, is conducting the survey.
The survey should take about 15 minutes to complete. Your feedback will help us find ways to improve patient safety and healthcare quality in our office. If you do not wish to answer a question, you may leave it blank. Westat will keep your individual responses to this survey strictly confidential. Only group results will be reported.
Please complete the survey as soon as possible and return it in the enclosed postage-paid envelope. If the return envelope is missing, please contact [insert medical office point of contact (POC) name here] for another envelope.
If you have any questions or concerns, please contact me at [insert medical office POC phone here]. We value and appreciate your participation in this important research!
Sincerely,
[Insert medical office manager/managing physician name here]
[Office Manager/Managing Physician]
Attachment B
Reminder Postcard Text
We want to hear from you!
Medical Office Survey on Patient Safety and Healthcare Quality
Recently, a survey was distributed to you. The survey is part of a research project sponsored by the Agency for Healthcare Research and Quality (AHRQ) to assess provider and staff views on patient safety and healthcare quality issues in medical offices.
If you have already completed your survey and mailed it back to [pilot study data collection coordinator], THANK YOU VERY MUCH!
If you have not yet had a chance to complete your survey, please take a few minutes to fill it out and mail it back to [pilot study data collection coordinator] in the postage-paid envelope that was provided. Your opinions are important to us. Thank You!
If you have any questions, please call
«NAME» at «PHONE»
1 Because the Bureau of Labor Statistics source above did not include nurse practitioners, we obtained an average hourly wage for them from the 2006 Nurse Practitioner Salary & Practice Survey, in The Nurse Practitioner: The American Journal of Primary Health Care, May 2006, Volume 31 Number 5, pages 39-43.
| File Type | application/msword |
| File Title | SUPPORTING STATEMENT |
| Author | AHCPR |
| Last Modified By | wcarroll |
| File Modified | 2007-02-21 |
| File Created | 2007-02-21 |