Page 4 of the supporting statement (part A) says that hypotheses will be tested. What are the hypotheses and how will they be tested? Will causal inferences be made? What are the study limitations (both internal validity and external validity) and how will they be acknowledged? What kind of conclusions will be drawn from this research?
It is important to clarify that this evaluation is not an empirical study of the PSIC program’s impact with experimental conditions or control and comparison groups. Rather, it is a descriptive analysis of participant feedback, and feedback from organizations they may have influenced, several months to several years after participation. The goal of this evaluation is not to identify causal relationships (i.e., formal hypothesis testing) or draw definitive conclusions of impact but rather to examine processes and summarize patterns and themes emerging from participants’ experiences post-training.
Specifically, AHRQ is interested in the extent to which PSIC participants were able to (1) use the tools and concepts they learned from the training program and (2) spread these tools and concepts to others in their organizations and/or beyond their organizations. Participants applied as teams with each member representing a different type of organization (i.e., State Health Department, hospital or provider organization, and Quality Improvement Organization, QIO). Due to the differences among the types of organizations participating in the program, each participant has a different potential to apply tools and concepts within and/or beyond their home organizations. For example:
In their regulatory role, State Health Department participants will be more likely than some other participants to use PSIC materials as a means of influencing practices across a range of organizations (e.g., setting new State policies on standards for conducting RCAs, determining the type of acceptable or required follow-up). As a result, they are more likely to have a broad impact overall. State Health Departments may also have a more direct impact to the extent that they work closely with hospitals to disseminate PSIC material and information and/or to investigate patient safety events.
Hospital participants are more likely than other participants to implement the PSIC material on a daily basis and will be more likely to affect specific work processes being conducted within an organization. As a result, hospital participants are likely to have a focused and specific impact within that organization only.
Similar to State Health Department participants, QIO participants will be more likely to have both an in-depth and broad impact assuming that they use the PSIC materials to assist a particular organization in their patient safety activities, as well as to provide general patient safety guidance to a large number of organizations.
To clarify the differences among the participants, we developed a logic model that highlights the roles of the different types of participants, the types of activities in which they are likely to engage post-training, and the potential outcomes that may stem from these activities. The logic model served as a guide for developing questions for the web-based questionnaires and qualitative interviews to ensure that we could capture participant and leadership feedback as thoroughly and accurately as possible.
Some of the limitations of this study, which will also be noted in the final report, include the following:
Retrospective analysis of post-training experiences. This is not an empirical study; therefore, neither causality nor generalizability can be established;
Survey non-response, which can limit our understanding of the concerns and issues participants encountered when trying to apply PSIC tools and concepts in their home organizations or organizations that they support;
Lack of access to contact information for non-participants, except on the rare occasion when our site visit contacts identify non-participants to include in the interviews during the site visits;
Inclusion of only six out of 52 states and territories for the site visits and qualitative interviews due to budgetary constraints. As a result, we will not be able to generalize findings beyond these individual cases;
Limited number of interviews that can be conducted at each site visit due to time and budgetary constraints; and
Inability to interview participants who have changed jobs since PSIC. Because up to five years have passed for some of the PSIC participants, they may no longer work for the organization they were employed by at the time of participation or may no longer work in the field of patient safety; therefore, we may have fewer than anticipated participants.
Despite the limitations in this study, the qualitative data we collect will provide AHRQ with valuable information that will help them as they develop tools and training that effectively support organizations’ efforts to improve patient safety. By understanding the factors that facilitate or inhibit the use of tools or the spread of knowledge, AHRQ will better understand the needs of these organizations and will be better prepared to address their future concerns, issues, and needs for improving patient safety.
Page 4: One of the selection criteria for the semi-structured interviews is “degree of training spread.” Isn’t this an outcome that you’re trying to test (i.e. the extent to which PSIC participants spread their training)?
This evaluation is retrospective and is not designed as an experiment in which we can “test” degree of spread empirically. Because many years have elapsed since the advent of the PSIC program, both AHRQ and the Veteran’s Administration’s National Center for Patient Safety (NCPS) have had continued contact and communications with many participants. This anecdotal information, in concert with relevant participant characteristics (e.g., types of organizations represented by the participating teams), serves as the basis for identifying potential sites as having either a perceived high or low degree of spread post-training. The interviews themselves will more accurately reveal the degree of training spread for the teams included.
Page 5: What will the “final product” of this study be? Will it be for internal agency purposes only?
The final product for this evaluation will be a report to AHRQ that documents the background, methodology, results including any patterns or themes emerging from the data, limitations of the study, and recommendations for future training programs and tool development. The results of this evaluation will help AHRQ understand the extent to which participants and participating organizations have been able to employ various PSIC tools and concepts and the barriers and facilitators they encountered. This information will help guide AHRQ in developing and refining other patient safety tools and future training programs for patient safety and other areas.
Page 5: The main goal of this study seems to be to learn the extent to which AHRQ has “spread the PSIC material.” As such, training non-participants seem to be critical to this study. Please clarify which questions they will be asked and how many there will be in this study and how they will be selected. Was there thought given to increasing the number of non-participants in this study?
Because the main goal of the PSIC program is to ultimately build a national infrastructure to support patient safety efforts, the focus of the current evaluation is to examine whether PSIC tools and concepts have been easily spread and implemented since attending the training program. We can best gauge the ease of spread and ability to implement the tools and concepts by asking training participants questions like: Did they share any information or tools with others in their home organizations or with others that they support? If they did, which tools/concepts were most useful? What challenges did they encounter? As a result, we designed a study that would (1) systematically collect information from all participants and participating organizations about what they have been doing since their participation in the program through web-based questionnaires and (2) collect more detailed information through site visits and qualitative interviews.
We will recruit 15 non-training participants for the qualitative interviews. These individuals were selected by reviewing documents from participating teams regarding their ongoing patient safety activities and by talking with a designated contact1 from the team about organizations with whom they have been partnering for post-PSIC activities.
To include non-participants in the web-based questionnaires would have required access to the e-mail addresses of people who serve in similar roles within organizations not participating in the PSIC program. This information would have been fairly difficult and costly to obtain and was beyond the scope of AIR’s contract with AHRQ.
Page 5: What are the OMB control numbers for the participant feedback questionnaires referenced on this page?
The participant feedback questionnaires noted on page 5 of Part A of our OMB submission refer to Level 1 training feedback questionnaires administered during the PSIC training program as a means of obtaining participant feedback on the quality of instruction and training materials. This information was collected in an effort to learn issues and elements of the program that AHRQ can improve for future administrations of the program. This data collection effort did not require OMB clearance.
Page 6: Will AHRQ’s pledge of confidentiality withstand a FOIA request?
AHRQ’s authority will withstand a FOIA request of information that would identify a person or establishment.
Page 3 (part B): What will be done if an 85% response rate is not attained?
Because response rates for any questionnaire can vary widely, our goal is to attain an acceptable response rate for conducting descriptive analyses to identify basic trends and patterns in the data. To that end, we hope to achieve a response rate acceptable for census questionnaires such as these. Best practice and extant literature (Barclay, Todd, Finlay, Grande, & Wyatt, 2002)2 point to acceptable response rates for post-hoc training evaluation questionnaires to be approximately 40 percent of the target population (e.g., training participants). However, we estimated a 75 percent or higher response rate, as noted in Part B of the OMB submission package and not 85 percent as indicated in the question, based on the overall interest and enthusiasm expressed by many PSIC training participants to engage in patient safety activities, as informally reported by AHRQ and NCPS to AIR.
Despite this perceived level of interest, AHRQ and its contractor, AIR still anticipates the need to bolster participation in the questionnaires. To do so, we will begin with an initial e-mail to invite participants to complete the questionnaire and follow-up with reminder notices3 at periodic intervals during the questionnaire response period. Our goal remains to obtain the highest possible response rate that can be achieved by reminding potential respondents of the opportunity.
Recruitment materials and questionnaire introduction: Since it is important for respondents to give candid feedback, OMB would suggest making it clear that they are not themselves being evaluated as far as their progress in implementing PSIC or spreading it.
We greatly welcome this suggestion. To address this more clearly, we have added the following paragraph after the introductory purpose paragraph in all recruitment materials and questionnaire introductory sections4:
“It is important to note that AHRQ is interested in learning about the ease of using and implementing the various PSIC program tools, resources, information, concepts, and techniques in your real world settings. This is not an evaluation of you or your progress in implementing the PSIC tools and concepts or your ability to share them with others. Rather, the information you provide regarding these tools, resources, information, concepts, and techniques will be used to enhance AHRQ’s patient safety tools and resources as well as future training initiatives. Your candid feedback is the only way that these enhancements can be achieved.”
Participant questionnaire: Are all participants given all the PSIC concepts/tools/ info/technique/resource listed on the instrument starting on page 7?
Yes, all training participants are shown all of the PSIC concepts, tools, information, techniques, and resources. The list starting on page 7 of the training participant questionnaires is the complete list of PSIC course concepts, tools, information, techniques, and resources. Every respondent will have the opportunity to indicate whether they have used the tool or applied the concepts listed.
Participant questionnaire: It seems important to be able to distinguish the “spread” of PSIC within an organization and outside the organization (page 13).
This is absolutely correct. We will also address this distinction by modifying one item on the participant questionnaire: question 4 under the post-PSIC activities starting on page 7 of Attachment C. As currently drafted, this item asks participants the following question: “To what extent has it [a given PSIC concept, tool, information, technique, or resource] been incorporated?” This item is currently designed to vaguely assess how well PSIC course content has been incorporated within one’s organization or another organization without distinguishing between the two.
To address the need to distinguish between spread of PSIC within and outside the organization, we will alter this item by making it two sub-items that will ask participants to identify: (1) the extent to which a given PSIC concept, tool, information, technique, or resource has been incorporated throughout their own organization and (2) the extent to which a given PSIC concept, tool, information, technique, or resource has been incorporated by organizations they support. The two sub-items will read as follows:
“To what extent has it (a given PSIC concept, tool, information, technique, or resource) been incorporated in your organization?”
“To what extent has it (a given PSIC concept, tool, information, technique, or resource) been incorporated in other organizations you support?”
This minor modification will allow us to distinguish between spread within the respondent’s organization, as well as across other organizations that respondents may support, and is not expected to significantly increase response time.
In addition, this issue also impacts the semi-structured interview guide that we will use during our site visits. As a result, we will revise the question 5 and associated probes on page 7 of Attachment G. These questions refer to the extent or level of spread of PSIC tools and concepts within the organization and outside the organization. The revised question 5 and its associated probes will read as follows:
“Have you been able to train others within your organization, community, and/or state about the tools and skills you learned the PSIC program?
What did the training(s) consist of?
What types of people attended the training?
Which organization hosted the training?”
“Have you shared the resources or tools with others within your organization, community, and/or state that you accessed through the PSIC program?”
1 Site visit contacts were identified by AHRQ and NCPS.
2 Barclay, S., Todd, C., Finlay, I., Grande, G. & Wyatt, P. (2002). Not another questionnaire! Maximizing the response rate, predicting non-response and assessing non-response bias in postal questionnaire studies of GPs. Family Practice, 19, 105-111.
3 Invitations, reminders, and thank you e-mails are provided in Attachments E and F of our original OMB application.
4The paragraph will be inserted, for example, on page 1 of the participant and leadership questionnaires after paragraph 1. The questionnaires may be found in Attachments C and D of our original OMB application.
| File Type | application/msword |
| File Title | OMB Clearance Application |
| Author | hamlin-ben |
| Last Modified By | Doris |
| File Modified | 2008-10-22 |
| File Created | 2008-10-22 |