Supporting Statement Part B_ESI Final

Supporting Statement Part B_ESI Final.doc

Assessment of the Emergency Severity Index (ESI)

OMB: 0935-0139

Document [doc]

Download: doc | pdf

SUPPORTING STATEMENT

Part B

Evaluation of the Emergency Severity Index

Version 1Revised - March 27July 13, 2008

Agency of Healthcare Research and Quality (AHRQ)

Table of contents

B. Collections of Information Employing Statistical Methods 3

1. Respondent universe and sampling methods 3

2. Information Collection Procedures 4

3. Methods to Maximize Response Rates 6

4. Tests of Procedures 6

5. Statistical Consultants 7

B. Collections of Information Employing Statistical Methods

1. Respondent universe and sampling methods

AHRQ received approximately 10,000 requests for the ESI tool. Requests for the ESI tool were received by AHRQ via telephone, USPS mail, email, and in-person at various conferences. Of the 10,000 individuals who requested the ESI tool, 1,014 individuals agreed to participate in a survey about the ESI. Requesters either verbally agreed to participate in a future survey about the ESI tool, or mailed a postcard to AHRQ (included in the ESI tool materials) that indicated they would be interested in participating in a future survey.

The majority of people requested the ESI tool via USPS mail and e-mail. Of these requesters, only those who returned the postcard are included in the respondent universe.

A smaller number of people placed telephone requests for the ESI tool, AHRQ recorded requesters’ contact information and verbally asked if requesters would like to participate in a future ESI survey. Of the telephone requesters, only those individuals who verbally agreed to participate in a survey are included in the respondent universe.

Finally, AHRQ received in-person requests for the ESI tool at various conferences. AHRQ did not obtain contact information for these individuals. Only those who returned the postcard to AHRQ (included in the ESI materials) were included in the respondent universe.

Of the 10,000 requesters, the respondent universe includes only those 1,014 requesters who actively volunteered to participate in a future survey. While approximately 10,000 people requested the ESI tool from AHRQ, Wwe plan towill draw a sample from the 1,014 ESI tool requesters who agreed to be contacted to participate in a survey about the ESI tool. This sample selection procedure was chosen because AHRQ’s ESI tool requester database only contains contact information for people who agreed to be contacted to participate in a survey about the ESI tool. We do not have contact information for the tool requesters who did not agree to be contacted to participate in a survey. As a result, we cannot include these people in the respondent universe. We recognize that the sample selection procedure – sampling from the population of tool requesters that agreed to be contacted to participate in the survey – has the potential to introduce a selection bias into the study. The sample of 1,014 requesters may not be representative of the entire population of 10,000 people who requested the ESI toolrequesters. Requesters who volunteered to participate in a future survey may be different than requesters who did not volunteer.. However, as the main purpose of this study is gather information on people’s perceptions of the materials and the ESI methodology, it is more likely that those who agreed to participate in the study are those who are familiar with ESI and will have the knowledge and experience that enables them to complete the survey.

In order to explore that possibilitypotential differences between those who volunteered to participate in a survey and those who did not, , we plan towill analyze any data available from the AHRQ Data Clearinghouse on the requesters who did not volunteer to participate in a future survey. To the extent to which it possible, we will use available data (e.g., requester’s name, organizational affiliation, credentials, and address) to distinguishassess whether there are differences between requesters who volunteered to participate in a future survey and those who did not. Results of this analysis will be used to interpret the roverall findings.esults from the ESI survey.

However, wWe still believe that a study of the 1,014 requesters that can be contactedhave volunteered to participate will yield a sample that satisfies the study’s goals. This population is likely to be responsive to the survey and have opinions about the ESI materials that will be valuable to AHRQ. Any findings from this project will, of course, apply primarily to the requesters who agreed to be contacted.

The proposed sample design using this frame includes two key requirements. First, we will use a stratified sample based on the type of survey that the respondent requested. Of the 1,014 requesters, 96 people requested a telephone-based survey, 351 requested a mail-based survey, and 567 requested an email-based survey. The goal is to survey 50% of the total number of people who requested the ESI tool from AHRQ and provided contacted information, or 507 people. The sample of 507 people is weighted by the proportion of people requesting each type of survey.

For the survey, we will aim for an 80% response rate (or 405 completed surveys). Of the 405 completed surveys, we estimate that 227 will be completed via email, 140 will be completed via mail, and 38 will be completed via telephone. While we expect a few hard refusals to the survey, we have found that there are many individuals who put the web survey aside to complete at a later date, often forgetting to complete the survey. Providing busy professionals with multiple options for completing a survey is the most effective means of increasing response rates. Exhibit 5 provides an overview of the sample, including the number of respondents we anticipate surveying based on the type of survey requested.

Exhibit 5. Overview of Sample

Type of Survey Requested	Number of People Requesting Each Type of Survey	Percentage of People Requesting Survey/ Total Number of Respondents (n=1,014)	Number Selected in Sample (n=507)	Number of Respondents Surveyed (80% Response Rate)
Mail	567	55.9%	283.5	227*
Email	351	34.6%	175.5	140*
Telephone	96	9.5%	48	38*
Total	1,014	100.0%	507	405

*Numbers have been rounded.

We will conduct four focus groups with individuals who work in an emergency department (ED) and are familiar with/are users of the ESI, including ED nurses, physicians, and medical and health services managers. Each focus group will have eight participants, for a total of 32 participants. To recruit focus group participants, we will utilize the Emergency Nurses Association (ENA) member mailing lists of emergency department managers and emergency nurses, and the American College of Emergency Physicians (ACEP) membership list. We will purchase the smallest denomination of names available (1,000 names), requesting names from two geographic areas—the Washington, D.C. metropolitan area and another U.S. metropolitan area (to be determined).

We will use a convenience sample of 32 people for the focus groups. A total of four focus group meetings will be held, comprised of 8 people each. To recruit focus group participants, we will make a general announcement at national ED conferences, targeting individuals who are ED physicians, medical directors, triage nurses, or nurse administrators. Holding focus groups at national ED conferences is an effective mechanism to solicit feedback from ED professionals for two reasons. First, recruiting participants that are attending national ED conferences enables us to maximize the diversity of the focus groups; participants will likely be coming to the conference from a variety of EDs across the country that are different in size, composition, population served, and geographic location. Second, by recruiting participants that are already attending a national ED conference, we will minimize the burden to participants, as they have already taken time away from their jobs to attend the conference. Thus, holding focus groups at national ED conferences enables us to maximize the diversity of the focus groups while also minimizing resource use and participant burden.

Focus groups will likely include individuals who have implemented the ESI and those who are aware of the ESI but selected to not implement the ESI, providing a range of perspectives. We will also ask those who are unaware of the ESI tool to volunteer to participate in the focus groups. Of the 32 participants, we estimate that 45% of focus group respondents will be nurses, 45% will be ED managers/ administrators, and 10% will be physicians.

2. Information Collection Procedures

The primary mode of information collection of information will use be a survey self administered questionnaire (SAQ). The survey will be completed by 405 people who have requested a copy of the ESI training tools from AHRQ, and indicated that they would be willing to be contacted to participate in a survey about the ESI tool. Individuals who requested the ESI tool (handbook and DVDs) provided their contact information and indicated their preferred survey mode. (Of the 1,014 requesters, 96 people requested a telephone-based survey, 351 requested a mail-based survey, and 567 requested an email-based surveyhard-copy via the mail, email, and telephone). Given that each person listed in the AHRQ ESI tool requester’s database listed a preferred survey mode, we will begin by sending a self-administered questionnaire (SAQ) tocontacting each person according to their type of survey requested mode of communication (i.e., email, mail, or by calling the respondent to conduct the survey via telephone).

Accompanying the questionnaire for participants completing the survey via the hard-copy or web formats will be a cover letter, explaining the importance of the survey (Attachment F). Participants that selected to complete a telephone survey will be mailed a modified version of the cover letter that explains the importance of the survey, and also indicates that they will be contacted within two weeks to complete the interview via telephone with a trained telephone interviewer (Attachment G).

All participants will have an opportunity to complete a hard-copy, email, or telephone-based survey, regardless of their initial stated preference. We will also accommodate participants’ preferences should they change, in order to ensure that they complete the mode of the SAQ survey that is most convenient for them. It is anticipated that respondents will choose the option of least personal burden, thereby reducing the overall burden of the study.

The field period for the survey will be approximately three months long. Follow-up pPrompts to non-responders will be based on the preferred mode of communication provided by each individual. Prompts will begin sent after one week to those who have not completed the questionnaire, and then occur again three-weeks later. One and three-week fFollow-up prompts to non-responders will be based on the preferred mode of communication provided by each individual: mail (Attachments I and L); email (Attachments J and M); and telephone (Attachments K and N). Non-responders that requested a mail-based survey will receive a postcard reminder (Attachment I). Non-responders that requested a web-based survey will receive an email reminder (see Attachment J). Non-responders that requested a telephone-based survey will receive a phone call, asking them to complete the survey via telephone (see Attachment K).

Following the third week of the field period, we will send the second reminder to non-responders, which reminds them to complete the survey and also provides them with an opportunity to complete the survey via another mode. Non-responders that requested a mail-based survey will receive a postcard reminder (see Attachment L). Non-responders that requested a web-based survey will receive an email reminder (see Attachment M). Non-responders that requested a telephone-based survey will receive a phone call (see Attachment N). Subsequent reminders for remaining non-responders will be sent at three-week intervals until a total of three reminders have been sent (all subsequent reminders will use the same scripts as provided in Attachments L, M, and N).

If the respondent has lost or misplaced the hardcopy SAQquestionnaire, and indicates a preference for hardcopy completion, NORC will mail or fax the respondent a new hardcopy SAQquestionnaire. If the respondent opts to complete the survey by telephone, the interviewer will access the respondent’s case online and enter responses directly into the online survey.

Project investigators will use an electronic receipt control system using case ID numbers to track the initial electronic survey mailing, mailing of follow-up paper surveys, address updates, remailing of questionnaires, mailing of postcard reminders, and complete and incomplete questionnaire returns. Reports from the system will identify the sample members which require prompting for completion of the survey.

All data from the completed surveys will be keyed (data entered) to create the analytic data file. Ten percent of the questionnaires will be randomly selected for keying a second time (double entry). The accuracy of the data entry process will be verified by comparing the data from the first entry with the data from the second entry. The double keying verification process will allow researchers to report the rate of accuracy to the Project Officer. The questionnaires will be processed in two batches. Data entry of the second and final batch will be completed within two weeks of the close of data collection. We will not impute missing data.

To minimize burden on participants, focus groups participants will be recruited while attending national or regional conference. ED professionals attending meetings have taken time away from their clinical schedules to network and engage in professional development activities. We will use a flyer to recruit participants (see Attachment O).

We will utilize the Emergency Nurses Association (ENA) member mailing lists of emergency department managers and emergency nurses, and the American College of Emergency Physicians (ACEP) membership list to recruit focus group participants. We will request contact information for members from two geographic areas—the Washington, D.C. metropolitan area and another U.S. metropolitan area (to be determined). Members in these selected geographic areas will be mailed a flyer (Attachment O) and letter (Attachment P) inviting them to participate in a focus group about their experiences with the ESI. The letter will also specify that we are interested in speaking with individuals who work in an ED and are familiar/or have experience working with the ESI. The letter will instruct recipients who fulfill these criteria to call a toll-free number or send an email to [email protected] if they are willing to participate in the focus group. The letter will also inform recipients about the honorarium for participation. We will sort the first 32 people who contact us and fulfill the two criteria (e.g., work in an ED, and have experience with the ESI) into four focus groups, and follow up with each participant to provide them with information about the location and date of the focus group.

3. Methods to Maximize Response Rates

The investigators will use a number of proven methods to maximize participation in the study. First, it is important to note that the ESI requesters database (the sample for the survey) only contains individuals who were willing to provide contact information to AHRQ and agreed to be contacted to participate in a survey about the ESI. Second, the survey instrument itself is designed to maximize response rates. The style of the survey is inviting and user friendly, with a maximum of 39 45 questions. The instructions for the survey are straightforward. Third, the questionnaire will be pilot tested with six respondents from the sampling frame, and questions will were be amended to reflect suggested improvements from these respondents. Fourth, each survey respondent will receive a cover letter encouraging participation in the survey (Attachments F and G). The cover letter will convey the importance of the study to AHRQ. In addition, the cover letter will be signed by Bruce Siegel, MD, MPH of the George Washington University, a highly recognized and well-respected researcher in the ED community, and Daniel S. Gaylin, MPA, Executive Vice President for Health Research at NORC.

Finally, telephone and email prompts will be sent after one week to those who have not completed the questionnaire; subsequent reminders for remaining non-responders will be sent at the three-week interval until a total of three reminders have been sent. Non-respondents will be contacted to confirm that the hardcopy SAQ questionnaire has been received, and to inquire whether the respondent would like to complete the survey online or by telephone. If the respondent has lost or misplaced the hardcopy completion, NORC will fax and or/mail the respondent the new copy of the questionnaireSAQ. If the respondent opts to complete the survey by telephone, the interviewer will access the respondent’s case online and enter responses directly into the online survey.

4. Tests of Procedures

A We conducted a pretest of the survey will be conducted with six (6) ED professionals during the initial OMB review period. Based on the results of this pretest, we made several minor revisions and amendments to the survey instrument. The revised survey instrument is attached. Revisions are as follows:

Section I (Background):
- Question 7 now asks for zip code of the respondent’s hospital rather than asking the respondent to classify the market of their hospital as rural, suburban, or urban.
- Question 10 on boarding was deleted.
- Questions 10, 11, 12, 13 are new in the revised survey instrument.
- Questions 15 added additional skip pattern, guiding nonusers who are considering implementation of the ESI to skip to Section III, and nonusers who are not considering implementing the ESI to Section IV.
Section II:
- Responses were added to Question 5.
- Question 8 was changed to ask “Has your ED monitored the impact of the ESI on any of the following?”
- Question 10 in the original survey was deleted.
- Questions 18, 19, 20, 21, 22, 26, and 27 are new in the revised survey instrument.
Section III:
- Section III was added for those respondents who are nonusers of the ESI considering ESI implementation.
Section IV (formerly Section III):
- Questions 3 and 4 combined into 3a and 3b.
- Question 5 combines Questions 7 and 8 from the original survey.
Section V (formerly Section IV):
- Question 1 revised to closed-ended question.
- Question 5 deleted.
- Questions 6 and 7 combined into new Question 5.