SUPPORTING STATEMENT
Part B
Evaluation of the Implementation of the Guide to Patient and Family Engagement in Health Care Quality and Safety in the Hospital Setting
April 12, 2011
Agency for Healthcare Research and Quality (AHRQ)
Table of contents
B. Collections of Information Employing Statistical Methods 3
1. Respondent universe and sampling methods 3
2. Information Collection Procedures 7
3. Methods to Maximize Response Rates 11
4. Tests of Procedures..................................................................................................12
5. Statistical Consultants..............................................................................................13
For the Guide implementation, we will select three hospitals to serve as demonstration sites. Selected hospitals will vary in terms of size, location, teaching status, and ownership. Hospitals will be selected to participate in this project with a focus on variability, not representativeness. Selection criteria for the hospitals include the following:
Hospital size: 50-500 beds. We will exclude hospitals with fewer than 50 beds and more than 500 beds since hospitals of this size account for a small percentage of the total hospitals in the United States.1
Geographic location: While we will not place restrictions on location, the goal will be to select three hospitals in different geographic locations.
Teaching status: We will recruit both academic medical centers and non-academic medical centers. We will recruit no more than one academic medical center to ensure variability across hospitals.
Ownership: We will recruit both for-profit and not-for-profit hospitals. The goal will be to have at least one for-profit and at least one not-for-profit hospital.
Military and specialty hospitals: We will exclude military and specialty hospitals (e.g., pediatrics, cardiac, or cancer hospitals) due to the highly specialized nature of these institutions.
Agreement to participate: Hospitals must agree to implement Component 2 of the Guide and at least one of the strategies from Component 1 over a 9-month implementation period. Hospitals also must agree to participate in all of the data collection activities as detailed in this clearance package.
In selecting three hospitals to participate in this study, we are not intending to make population estimates from a representative sample. Rather, this study is intended to examine the utility and efficacy of the Guide in order to make improvements to it, which includes incorporating lessons learned. This project is not designed to evaluate outcomes or demonstrate effectiveness, nor will the results be used to make inferences that are generalizable to other hospitals. To do so would require extensive data collection in comparison hospitals, activities that are beyond the scope of this project. Thus, the results of this study will be illustrative rather than definitive and generalizable.
To fulfill the goals of this study and to examine the utility and efficacy of the Guide, a rich description of the implementation of the Guide and associated outcomes is needed. As such, we will collect qualitative data to describe the implementation, to gather lessons learned, and to provide context on the impacts on hospital staff and patients. The qualitative data collection activities we will conduct for this project include semi-structured interviews with hospital staff and leaders; collection of documentation; semi-structured interviews with implementation coordinators; and focus groups with patients and families.
We also will collect quantitative data in the form of staff surveys to assess changes in organizational culture and staff knowledge, attitudes, and behaviors, and patient surveys to assess changes in experiences of care.
Exhibit 1 describes the data collection activities that will occur at the three hospitals. For each planned data collection activity, Exhibit 1 details the following: the timing of the data collection, the description of and number in the potential respondent universe, the sample description and size, and method of sampling or selecting respondents for each data collection activity.
Exhibit 1. Description of data collection activities, sampling universe and strategy
Data Collection Activity |
Description of and Number in Potential Respondent Universe |
Description and Size of Sample |
Sampling Method and Anticipated Response |
Semi-structured
leader interviews |
2 Hospital Leaders (HL) will be identified by each participating hospital to oversee the implementation of the Guide and serve as points of contact for the project. A Hospital Leader may be the Chief Nursing officer, Chief Medical Officer, nurse manager or supervisor, patient safety officer, or quality improvement officer, or other high level administrator designated by the respective hospital. |
Maximum of 12 hospital leaders: 2 at each of 3 hospitals, pre-and post implementation. The leaders may be the same at pre- and post-implementation, in which case there may be as few as 6 HLs, however, because they could change due to turnover or change in job responsibilities, we estimated the maximum number who might be interviewed. |
Both designated HLs will be interviewed. No sampling will be used. |
Semi-structured
staff interviews |
The staff interviews will include doctors and nurses from the medical surgical units implementing the intervention, hospital pharmacists, and social workers. Because the hospitals have not yet been identified, we do not know the exact number in the respondent universe. |
Maximum of 24 health professionals (physicians, nurses, social workers, or pharmacists): 4 at each of 3 hospitals, pre- and post-implementation. The staff may be the same at pre- and post-implementation, in which case some staff may be interviewed twice (once pre- and once post-implementation). In theory, there could be as few as 12 total staff interviewed or as many as 24. |
This will be a purposive, rather than random, sample of staff. Staff will be selected by the hospitals based on their interest, knowledge or experience with using the intervention materials. |
Collection of documentation |
Not applicable. |
Not applicable. |
Not applicable. |
Bi-weekly semi-structured interviews |
3 Implementation Coordinators (1 in each of 3 hospitals) |
3 Implementation Coordinators (1 in each of 3 hospitals) |
No sampling involved, all 3 coordinators are included. |
Observation of Guide implementation |
The observations will include doctors and nurses from the medical surgical units implementing the intervention, hospital pharmacists, and social workers. Because the hospitals have not yet been identified, we do not know the exact number in the respondent universe. |
Maximum of 30 health professionals (up to 10 per hospital). |
This will be a purposive, rather than a random sample. Staff will be selected to observe based on their interest and availability during the site visit time period. |
Focus
groups with Patients and Family Members |
An estimated 5,184 patients will be treated in medical surgical units over the course of the intervention: 48 patients per week for 36 weeks at 3 hospitals. |
Maximum 72 participants: 3 groups of 6-8 participants at each of 3 hospitals. |
A list of potential patients to contact will be obtained from the hospital. If the hospital does not provide such information, a recruitment firm in the area will be used and will screen patients to see if they stayed at the hospital in the past 6 months. |
Staff
Survey |
The staff surveys will include doctors and nurses from the medical surgical units implementing the intervention, hospital pharmacists, and social workers. Because the hospitals have not yet been identified the exact number in the respondent universe is unknown. |
Maximum of 300 health professionals (physicians, nurses, social workers, and pharmacists) involved in implementing the Guide: 50 staff at each of 3 hospitals, pre- and post-implementation. The staff may be the same at pre- and post-implementation, in which case some staff may be interviewed twice (once pre- and once post-implementation). In theory, there could be as few as 150 total staff surveyed or as many as 300. |
No sampling method will be employed. We will attempt to survey all staff in the respondent universe. We anticipate at least a 50% response rate for the staff survey based on our experience administering similar surveys. |
Patient
survey |
Over two four week periods (one pre and one post-implementation), an estimated 1,768 patients will be seen and discharged from medical surgical units in the 3 participating hospitals. |
Surveys will be distributed to 1,770 patients at discharge: 885 pre- and 885 post-implementation. |
The survey will include all patients on Medical Surgical units over a four week period prior to implementation and post implementation. We anticipate at least a 50% response rate. |
This
section summarizes the methodology to be employed for each of the
data collection activities. Detailed protocols and data collection
instruments are included as Attachments.
For all data collection activities, AIR’s IRB will approve all protocols, recruitment, and interview procedures before any contacts are made or any data are collected. The notice of IRB approval is included as Attachment T.
Semi-structured interviews with hospital leaders and hospital staff (Attachments E, F, G, and H).
Objectives. Pre-implementation, the purpose of these interviews is to collect data to describe staff knowledge, attitudes, and beliefs around patient and family engagement and on the current organizational culture and climate surrounding patient and family engagement.
Post-implementation, the purpose of these interviews is to understand the hospital’s experiences implementing the Guide interventions, including how easy or difficult the Guide was to implement; the perceived effects of the Guide interventions; and the sustainability of the Guide interventions.
This data will be used to describe changes at each hospital during the implementation period, to inform revisions to the Guide interventions, and to incorporate lessons learned from the implementation into the final Guide.
Participants. Interviews will be conducted with hospital leaders (2 at each hospital during each site visit) and key staff (4 at each hospital at each site visit).
Procedures. In-person interviews with hospital leaders and key staff will be conducted during the pre- and post-implementation hospital site visits at each hospital. Each in-person interview will last up to 60 minutes and will be guided by a semi-structured interview topic guide. The topic guides (Attachments E, F, G, and H) were developed based on the literature scan conducted as part of this project and on discussions among the project team, including the AHRQ Project Officer. The topic guides include questions that need to be covered, as well as probes designed to ensure that all critical topics are addressed.
Data analysis. All qualitative data, including the transcripts from the semi-structured interviews, will be managed and analyzed using NVIVO 8.0 or Atlas.ti, qualitative data analysis software programs. We will define and apply a coding system for the notes from the interviews and focus groups. This coding system will be applied using the qualitative software to facilitate the retrieval, summary, and synthesis of data. The data are coded by assigning labels to the units, clustering the codes into categories and hypothesizing about relationships among the categories.
Collection of documentation (Attachment I).
Objectives. To gather documentation of the implementation of the Guide and to document hospital policies and procedures related to patient and family engagement. The data will be used to help describe changes at each hospital over the 9 month implementation period.
Participants. Project team staff will work in conjunction with the implementation coordinators at each hospital to collect documentation. Documents may include formal policies, reports, memos, training materials for hospital staff, or materials for patients.
Procedures. Data collection will occur both pre- and post-implementation at each site. The implementation coordinator at each hospital will gather data on existing hospital policies, procedures, documents, and strategies related to patient and family engagement or patient- and family-centered care. To the extent that it is available, the following types of documentation will be collected:
Background on organizational structure and vision
Policies and procedures related to Component 1 and Component 2 strategies
Tools used to foster communication between patients, family members and health care team
Policies and procedures related to patient and family engagement, patient- and family-centered care, quality and safety
Data analysis. Documents collected from hospital sites will be managed and analyzed using NVIVO 8.0 or Atlas.ti. As described above, we apply the defined coding system to the documents.
Bi-weekly semi-structured interviews with implementation coordinators (Attachment J).
Objectives. The purpose of this interview is to check in on progress during the implementation period of the Guide, track Guide implementation in real time, and gather information on successes and challenges thus far. The data will be used to inform revisions to the implementation handbooks included with the Guide.
Participants. Bi-weekly interviews will be conducted with the implementation coordinator (i.e., the person identified by the hospital to be responsible for overseeing implementation activities and serving as a primary point-of-contact).
Procedures. These interviews will occur soon after Guide implementation and will continue throughout the implementation period. These telephone interviews will last up to 30 minutes. The topic guide (Attachment J) includes questions that need to be covered, as well as probes designed to ensure that all critical topics are addressed.
Data analysis. Transcripts from the bi-weekly interviews will be managed and analyzed using NVIVO 8.0 or Atlas.ti. As described above, we will apply the defined coding system to the documents and analyze relationships between the codes.
Observation of Guide Implementation. (Attachment K)
Objectives. To observe practices and behaviors on the units where the Guide has been implemented. We will use the information gathered during the observations to supplement and inform other data collected about the hospital’s experiences implementing the Guide strategies. We also will use the observations to inform follow-up questions about how easy or difficult the Guide interventions have been to implement and sustain.
Participants. Observations will occur at each site and will be of physicians and nurses who work on the hospital units where the Guide strategies have been implemented. We do not anticipate that more than 10 staff members per hospital will be observed interacting with patients and/or family members and using the Guide.
Procedures. We will conduct two observations at each of the three hospitals during the both the pre- and post-implementation periods, for a total of 12 observations. Each observation will last for 4 hours, during which time we will observe select hospital staff. Observers will complete an observation form for each patient-provider encounter observed.
Data analysis. We will enter all data from the site visit observation protocol into an Excel spreadsheet. We then will analyze the data to describe how each strategy was executed at a given hospital and also to compare how strategies were implemented across hospitals (similarities, differences, etc.).
Focus groups with patients and family members at each of the participating sites (see Attachments L, M, and N).
Objectives. To elicit information about patients’ and families’ experiences of care at the hospital along with their reactions to tools in the Guide and their implementation. This data will be used to inform any revisions to the Guide tools and implementation handbooks.
Participants. Three focus groups will be conducted with up to 8 individuals at each hospital who were discharged from hospital units implementing the Guide. One focus group at each hospital will be conducted with patients only, one with family members only and one with patients and family members together.
Procedures. Focus groups will be conducted post-implementation. Each focus group will last up to 120 minutes and will be guided by a semi-structured interview topic guide. The topic guides (Attachments L, M, and N) include questions that need to be covered, as well as probes designed to ensure that all critical topics are addressed.
Data analysis. Transcripts from the focus groups will be managed and analyzed using NVIVO 8.0 or Atlas.ti. As described above, we will apply the defined coding system to the documents and analyze relationships between the codes.
Staff survey (see Attachments O and P).
Objectives. The purpose of the staff survey is to assess changes in organizational culture related to patient safety and engagement, and to assess significant changes in staff knowledge, attitudes, and behaviors. The survey will be built around items from the Medical College of Georgia Patient- and Family-Centered Care Culture Survey (MCG) and the Army Medical Department Climate Survey (AMDCS). We will supplement these items with questions from AHRQ’s Hospital Survey on Patient Safety (HSOP) survey. The data from the survey will be used to describe how organizational culture and staff knowledge, attitudes, and beliefs changed over the implementation period at each hospital.
Participants. Participants will include all staff members who work on the units where the Guide is implemented during the pre- and post-intervention periods. Because inpatient units typically use temporary nursing staff, a minimum number of required shifts will be established for inclusion in the survey (to avoid surveying individuals who were only minimally exposed to the Guide). Because the hospital sites have not yet been selected, the exact number of staff on each unit where the Guide will be implemented is unknown. However, based on staffing estimates from the Health Research and Educational Trust (HRET), it is estimated that 50 staff members will receive the survey at each hospital (assuming implementation on one unit) for a total of 150 participants.
Procedures. The staff survey will be conducted pre- and post-implementation. The survey will be an online survey, programmed via Survey Monkey. Hospitals will supply staff email addresses. Invitations to participate, containing a hypertext survey link, will be sent to each staff member who works on the hospital units where the Guide is being / was implemented. Staff members can choose to complete the survey or not. The Staff Survey Instrument and the Introductory Email Text are included as Attachments L and M. Attachment L documents the source of each survey item.
Data analysis. The staff survey items will be used to create continuous scales defining the constructs of interest. We will report the survey results descriptively, and we will compare mean scores on the constructs between the pre-test (baseline) and post-test (follow-up).The regression analyses will test whether, holding personal and provider characteristics constant, there is a statistically significant difference in our indicators of staff behavior (described above under “Objectives”) after implementation of the Guide.
Patient survey (see Attachments Q and R).
Objectives. The purpose of the patient survey is to identify changes in patient experience since implementation of the Guide. The patient questionnaire will be built around the CAHPS® Hospital Survey (HCAHPS) domains that assess aspects of patient-physician interaction around the hospital stay (Communication with Nurses, Communication with Doctors, Communication about Medicines, Responsiveness of Hospital Staff, and Discharge Information) and the Patient Activation Measure (PAM). These scales directly assess the aspects of the relationship that the Guide is intended to affect. A small set of additional questions addressing aspects the Guide that are not adequately addressed by the HCAHPS composites were developed.
Participants. Participants will include all patients seen and discharged from the units at each hospital implementing the Guide over two four-week periods (one pre- and one post-implementation). It is estimated that 1,770 patients will be seen and discharged during this time period: 885 pre- and post-implementation.
Procedures. The patient survey will be conducted both pre- and post-implementation. A self-administered paper and pencil questionnaire will be distributed to patients at discharge. The questionnaire is intended to be completed by patients at home. As such, the surveys will include a postage-paid, addressed envelope that participants can use to mail the completed surveys to researchers at AIR. A cover letter inviting patients to participate and providing answers to “frequently asked questions” will be included with the survey. The Patient Survey Instrument and Cover Letter are included as Attachments Q and R. The source of each survey item is documented in Attachment Q.
Data analysis. The patient survey items will be used to create continuous scales defining the constructs of interest. We will report the survey results descriptively, and we will compare mean scores on the constructs between the pre-test (baseline) and post-test (follow-up). The regression analyses will test whether, holding personal and provider characteristics constant, there is a statistically significant difference in our indicators of patient experience of care (described above under “Objectives”) after implementation of the Guide.
As described below, methods to maximize response rates vary based on each data collection method:
Semi-structured interviews with hospital leaders. To maximize participation from hospital leaders, we will schedule interviews at times convenient to hospital leaders and will provide at least 6 weeks of lead time in scheduling these interviews.
Semi-structured interviews with hospital staff. To maximize participation from hospital staff, we will schedule interviews at times convenient to hospital staff and will provide at least 6 weeks of lead time in scheduling these interviews.
Collection of documentation. To ensure collection of documents from hospitals, we will provide several mechanisms for the submission of these documents, including email, regular mail, or collection in-person during site visits.
Bi-weekly semi-structured interviews with hospital coordinators. To maximize participation from hospital implementation coordinators, we will schedule interviews at times convenient to the coordinators and at recurring times (e.g., every other Wednesday). We will conduct these calls at times when implementation coordinators can ask questions regarding Guide implementation and receive informal technical assistance; as such, we expect implementation coordinators to be motivated to participate in these calls.
Observation of Guide implementation. To ensure that observation can occur for up to four hours, we will schedule site visits at times convenient to hospitals and with at least 6 weeks of lead time.
Focus groups with patients and family members. To maximize participation in the focus groups, participants will be given an incentive of up to $75 for participation. We will also hold the focus groups at times convenient for patients and families (e.g., during lunch hours or after work).
Staff survey. To maximize the response rate on the staff survey, we will distribute the survey online to make it convenient for staff to access and complete. We will also provide information in an email invitation that describes the overall project and the importance of participating in this project and completing this survey.
Patient survey. To maximize the response rate on the patient survey, we will include a postage-paid, addressed envelope to return the survey. We also will provide a $2 incentive in the form of a $2 bill that will be included in the survey packet. The American Association of Public Opinion Research lists monetary incentives as a procedure to consider in ensuring adequate survey response rates. Research has shown that an incentive as small as one to two dollars mailed with the first questionnaire can increase response rates 5 to 8 percent2.
The qualitative research methods proposed have been used in prior research by this project team. The team has extensive experience with the qualitative methods proposed. Nevertheless, prior to data collection, the topic guides will be submitted to 2-3 selected individuals from the participating organizations for review. These individuals will be familiar with patient and family engagement and/or patient-and family-centered care, but will not be the key informants interviewed during data collection. The individuals will be asked if there are any confusing questions or if different language should be employed. They also will be asked if any key areas are missing from the topic guides. Our topic guides are currently based on guides used in previous engagement and evaluation projects.
During data collection, questions which are difficult to answer, unclear, present substantial response burden, or are otherwise problematic will be noted by the interviewer with a description of the concerns. The questions will be modified to address these concerns and subsequently re-reviewed.
The items and measures selected for the patient and staff surveys were taken from existing measures of the constructs of interest and have been used in prior research; therefore the measures are virtually identical to existing measures. A small set of items specific to the intervention were developed for both the Patient and Staff Survey. These newly developed items will be reviewed by 2-3 selected individuals from the participating hospitals. These individuals will be employees and will be familiar with the Guide implementation, but will not be the key informants interviewed during data collection. The individuals will be asked if there are any confusing questions or if different language should be employed.
The following persons were consulted regarding the quantitative and qualitative methods used in this project. The AIR personnel listed below will collect and analyze the data.
Name (Affiliation) |
Telephone Number |
|
Marjorie Shofer (AHRQ) |
301-427-1259 |
|
Kristin Carman (AIR) |
202-403-5090 |
|
Amy Windham (AIR) |
202-403-5518 |
|
Steven Garfinkel (AIR) |
919-918-2306 |
|
Lauren Smeeding (AIR) |
202-403-5067 |
|
Maureen Maurer (AIR) |
919-918-2308 |
|
Pam Dardess (AIR) |
919-918-2311 |
|
Ushma Patel (AIR) |
919-918-2321 |
|
Maulik Joshi (HRET) |
312-422-2622 |
11 AHA Annual Survey 2002. http://www.hhs.gov/nvpo/meetings/PowerPoints/SimoneNVACApril2005.ppt
2 James, J.M. and Bollstein, R. “The effect of Monetary Incentives and Follow-Up Mailings on the Response Rate and Response Quality in Mail Surveys.” Public Opinion Quarterly. 54:346-361 (1990).
| File Type | application/msword |
| File Title | SUPPORTING STATEMENT |
| Author | wcarroll |
| Last Modified By | william.carroll |
| File Modified | 2011-04-18 |
| File Created | 2011-04-18 |