EVALUATION OF THE EFFECTIVENESS OF THE SMOKE ALARM INSTALLATION AND FIRE SAFETY EDUCATION (SAIFE) PROGRAM
Supporting Statement for OMB Review 0920-0730
PART B
August 24th, 2009
We will utilize households identified by NCDHHS and individual fire departments that would normally receive the SAIFE program as potential participants. These households contain populations that are at higher risk for fire injuries, such as those with children under 5 years of age, with persons aged 65 or older, and those with median per capita incomes below the arithmetic average. We will select the first 425 households from the pool of all eligible households (as provided by the fire departments) to receive the evaluation, provided they consent to do so.
Consistent with the current protocol for the SAIFE/North Carolina Get Alarmed program, NCDHHS will prepare and disseminate a Request for Proposal (RFP) to all eligible fire departments. Fire departments can join together in application for funds as long as they do not serve more than a combined population of 50,000. In the event that no department responds, sites will actively be recruiting for participation, with assistance from NCDHHS staff.
Because respondents will be receiving up to $75 in free safety equipment (i.e., one to three smoke alarms) from participation in the SAIFE program, and because the self-administered survey will be delivered to respondents’ by their local fire department, we expect a very high cooperation rate for the pre-intervention survey. Additionally, based on our experience in the pilot/cognitive testing and with similar evaluations, we expect the response rate to the SAIFE Evaluation surveys to be approximately 85 percent of the 425 households that will be asked to participate.
Upon entry into a potential respondent’s home, firefighters will ask to speak to an adult (18 years or older) in the home who can provide approval for the installation of smoke alarms and participation in a survey. Once a suitable respondent is identified, the firefighter will explain that in addition to their participation in the SAIFE program, they are being asked to assist with an evaluation of the program. Firefighters will provide potential respondents with information on the nature and purpose of the program and evaluation. They will read an informed consent statement to respondents, explaining the smoke alarm installation and fire safety education procedures, the purpose of the evaluation survey, and the voluntary nature of participating in either part of the program (Appendix E). Then respondents will be able to participate in the intervention (i.e., smoke alarm installation and fire safety education) or the survey, or both. Signing the evaluation consent form (Appendix E) indicates participation in the intervention and evaluation, while not signing the consent form indicates only the intervention. This screening process will take approximately 5 minutes to complete per household.
For those who want only the intervention component of the program, the firefighter will proceed to install the smoke alarms and provide the fire safety education to the respondent. Should respondents agree to participate in the evaluation as well, the self-administered pre-intervention survey will be left with respondents while the firefighter proceeds to install the appropriate number of smoke alarms in the home. The time to complete the pre-intervention survey is estimated to average 15 minutes. Upon completion of the survey, respondents will be asked to immediately place the survey in a pre-addressed business reply envelope and hand it to the firefighter. Firefighters will be asked to return all surveys to RTI on a weekly basis for processing and analysis.
To ensure that firefighters participating in the study follow the above procedures, fire departments involved in the evaluation will undergo a modified version of the North Carolina “Get Alarmed” training program focused on proper implementation of the SAIFE program intervention and evaluation tasks. In addition to the normal training topics of installation and fire safety, firefighters (and other project staff) will be provided specific instructions on how to implement the SAIFE intervention (i.e., smoke alarm installation and fire safety education) and the pre-intervention survey. It will be critical to emphasize the importance of not discussing fire safety issues with respondents until the survey has been completed and sealed in its return envelope. In addition, firefighters will be trained in how to respond to possible questions or requests for help with the survey in a nondirective manner that will not influence survey answers. To ensure that the intervention and evaluation components of the program are properly administered, RTI staff will accompany firefighters from each fire district on approximately 5 percent of the home visits to observe procedures, checking for fidelity to the data collection protocol.
Approximately 6 months after completion of the pre-intervention survey, respondents will be notified by mail that the evaluation team will be contacting them soon to complete a short telephone interview (Appendix F1). An interviewer from RTI will contact respondents during the afternoon and evening to facilitate reaching them at a convenient time. Respondents who agree to participate will be administered the 6-month post-intervention survey over the phone using a pre-programmed CATI instrument. When possible, the survey will be administered to the same person who gave consent and participated in the pre-intervention survey. The estimated time to complete the 6-month post-intervention telephone survey is 15 minutes.
During each year of data collection, a sub-sample of respondent households—approximately 10 percent or 36 households each year—will be selected for completion of the 6-month post-intervention survey in-person, in addition of the CATI telephone survey. This household visit will take one hour to complete. To facilitate these interviews, a member of the project team will visit a randomly selected sub-sample of 36 households that completed in the pre- and post-intervention surveys. Respondent selected for in-person 6-month post-intervention surveys will be notified by mail (Appendix F2) that the evaluation team will be contacting them soon to schedule an in-person visit.
At the point of contact with the respondent, the interviewer will schedule an in-person visit. When possible, the survey will be administered to the same person who gave consent and participated in the pre-intervention survey. For those who do not agree to participate, the interview will be terminated and the respondent will be thanked for his or her time. For respondents who agree, the interview will proceed as planned. The interviewer will administer the 6-month post-intervention telephone survey using a paper and pencil version of the survey (Appendix D2). The data from the in-person survey will be used to validate the 6-month post-intervention telephone survey. In addition, all smoke detectors installed at the point of the intervention will be inspected and tested to ensure that they are still properly installed and functioning.
Because obtaining acceptable response rates to surveys and evaluations has become more difficult over the years, RTI will take several steps during data collection to maximize participation rates. This section describes the procedures RTI will follow to meet the highest standards of data collection quality in conducting the SAIFE Evaluation. It also provides an overview of the response rate expectations and the steps that will be taken to deal with non-response.
Based on our experience in the pilot/cognitive testing and with similar evaluations, we expect the response rate to the SAIFE Evaluation surveys to be approximately 85 percent.
A frequent concern expressed by participants in surveys is whether the survey is research or a disguised marketing or sales call. With the increase in sales calls and the use of more call screening, participants are often wary of receiving calls from organizations that they have not initiated. Often, this wariness can be allayed by giving respondents information about the nature of the study before the first call has been made and in all subsequent interactions.
To counteract any respondent concerns about the legitimacy of the survey, the pre-intervention survey will be administered by a firefighter during the home visit. The survey will be accompanied by a copy of the letter (Appendix G) from the project director and an informed consent form (Appendix E) that explains the nature and purpose of the survey. In our experience, this type of approach results in a high degree of compliance with requests to participate in in-person surveys. At the stage of the 6-month post-intervention telephone survey, however, respondents may not remember participating in the pre-intervention survey. To help alleviate this problem, pre-notification letters (Appendix F1) will be sent to respondents 14 days prior to calling them for the 6-month post-intervention survey. The letter includes a reminder of the purpose of the project, provides information on the project sponsor and RTI, and informs the respondent that they will be called soon for an interview. The letter also provides a toll-free number for respondents to call if they have any questions about the study and informs them of a $10 gift they will receive on completion of the survey.
Avoiding refusals to participate, and converting a significant portion of those who initially decline, will play an important part in RTI’s efforts to meet the response rate goal. The first step to avoid refusals begins with interviewer training. Both firefighter and telephone interviewer training in this area will focus on two key elements: maintaining interaction with respondents and tailoring the information provided to respondents (Groves and Couper, 1998).
To maintain positive interaction with respondents, the firefighters and telephone interviewers must quickly and professionally establish rapport with participants. Although this technique is critical to avoiding refusals in both, in-person and telephone modes, maintaining interaction is difficult in telephone interviewing. Often the first step of telephone interaction is simply assuring respondents that the purpose of the call is legitimate. RTI’s training and supervision of interviewers emphasizes key techniques for maintaining interaction. Because the only tools telephone interviewers have to convey meaning are the words they speak and the tone they use, RTI trains interviewers to develop an effective tone of voice. Generally, an effective tone for telephone interviewers conveys confidence, preparedness, friendliness, professionalism, and sincerity.
Tailoring communication to respondents is the second important dimension of avoiding refusals. The requirement for firefighters and telephone interviewers to obtain permission from respondents to conduct the interview is probably the most important issue in avoiding refusals. RTI will train firefighters and telephone interviewers to tailor their communication to respondents to address their questions or concerns about the study. A list of frequently asked questions will prepare firefighters and telephone interviewers to respond to questions directly and effectively.
The basic techniques for avoiding refusals also will be applied when firefighters and telephone interviewers try to encourage potential refusers to reconsider participating in the study. RTI’s basic refusal conversion procedures follow the steps outlined below.
1. Be prepared to provide information needed to address the concern(s).
2. Anticipate future concerns of respondents by taking note of reasons given by initial refusals.
3. Focus information on the importance of the respondent’s participation.
4. Maintain effective tone throughout interaction.
5. Always act as a professional working in association with RTI, NCDHHS, and CDC.
The key element of effective refusal conversion is using active listening techniques, in which firefighters and telephone interviewers listen to the respondent’s questions or concerns, paraphrase and summarize the message, and provide a direct response to the message. In addition, we will also train firefighters and telephone interviewers to distinguish respondents who are reluctant to participate in the study from those who are likely to refuse outright. In general, reluctance is indicated by vague questions or concerns, whereas refusal entails more direct objections. Given the number of different research, solicitation, and other calls that respondents may be receiving at home, RTI trains firefighters and telephone interviewers that it is reasonable to expect some reluctance and to be prepared to address it effectively. At the same time, firefighters and telephone interviewers are trained to recognize clear refusal behavior so that they do not let those kinds of situations escalate to a negative interaction.
Following standardized techniques is the key to conducting reliable and valid survey data collection. Standardized telephone interviewing techniques include reading survey questions exactly as written; using neutral probes to clarify responses; maintaining a professional, nonbiased rapport with respondents; and recording responses exactly as provided. To ensure that telephone interviewers maintain standardized interviewing procedures, RTI will monitor at least 10 percent on average of the hours spent conducting CATI interviews. Interviewer monitoring reduces interviewer errors, improves interviewer performance by reinforcing positive interviewer behaviors, and collects information on the quality of the data being collected.
Monitoring will detect deviations from standardized interviewing and allow monitors and supervisors to work with interviewers to correct these deviations. Furthermore, monitors provide positive feedback to interviewers to reinforce appropriate standardized interviewing behaviors.
The preceding sections described RTI’s approach to collecting the data needed for the SAIFE evaluation program. In combination, the procedures we will establish for data collection anticipate and address the major challenges of such an undertaking.
Two surveys were developed for the SAIFE evaluation: a pre-intervention paper and pencil survey (Appendix D1) and a 6-month post-intervention telephone survey (Appendix D2). The two survey forms contain similar questions and are designed to assess change over time (6 months) as a result of the SAIFE program intervention. The follow-up telephone survey also contains additional questions inquiring about any fires that may have occurred in the household since the intervention and a few questions that ask respondents what they remember from the fire safety component of the intervention. The survey forms are based on the surveys used by the “Get Alarmed” fire safety education and smoke alarm installation program administered by the North Carolina Department of Health and Human Services (NCDHHS). Additional items were added to these survey forms to collect information necessary to the evaluation.
The project’s evaluation task team prepared the survey forms. The forms were subjected to review by the team members and expert review by survey methodologists. A test form of the surveys, containing all core questions and new items added to the survey forms, was developed for cognitive testing. The survey, along with a cognitive testing guide and testing protocol were submitted and approved by the RTI Institutional Review Board (IRB) for pilot/cognitive testing.
Pilot/cognitive testing of the survey was conducted with six adults who were currently receiving the North Carolina “Get Alarmed” program in two different municipalities in North Carolina. This program utilizes a fire safety education and smoke alarm installation program similar to the one to be implemented as part of the evaluation. Respondents were asked to participate in a 30-minute in-person interview to test the proposed surveys. The testing focused on (1) clarity of instructions, (2) question comprehension, (3) appropriateness of response categories, (4) information recall, and (5) skip patterns. The testing entailed reading the survey questions and response options (where indicated) to respondents, obtaining their response, and posing follow-up questions. A survey methodologist met with each participant to provide informed consent and to observe the interview. The survey methodologist listened in and stopped the interview at appropriate times to ask follow-up questions or probe on specific issues. Respondents were allowed to ask for clarification whenever they had difficulty comprehending the questions or the response options.
Respondents to the pilot/cognitive testing provided feedback identifying minor issues in the wording of instructions, questions, and response categories. In addition to the feedback obtained from these respondents, the surveys and planned methods were reviewed by two safety officers at fire departments participating in the current North Carolina “Get Alarmed” program. The feedback from the safety officers was also helpful in identifying minor issues in procedures and question wording within the survey. As a result of the feedback from the pilot/cognitive testing and review by the safety officers, the survey received a number of minor wording changes, which improved the clarity of the survey.
RTI, an independent survey research organization, provided CDC with consultation on statistics and data collection. The RTI lead investigator, responsible for the study design, is Dr. David Driscoll. Dr. Lei Li and Mr. Walter Boyle assisted with power calculations and statistical issues. Mr. Jon Poehlman assisted with potential issues related to community program evaluation. Mr. Murrey Olmsted designed the evaluation instruments for the study and advised Dr. Driscoll on study design. Ms. Kristine Rae assisted Dr. Driscoll in study design and statistical analyses.
Dr. Michael Bowling, from the University of North Carolina Injury Prevention Research Center, assisted with study design and statistical analyses.
Ms. Jeanne Givens and Ms. Sherri Troop, from the NCDHHS, were consulted on issues regarding study participation by local fire departments.
Dr. Driscoll will oversee all data analysis efforts, with assistance on sampling, analysis, and other tasks as noted above. Dr. Michael Ballesteros of CDC will be the lead agency representative responsible for receiving and approving contract deliverables.
Dr. David Driscoll: (919) 541-6565
Dr. Lei Li: (919) 541-8821
Mr. Walter Boyle: (919) 316-3928
Dr. Jon Poehlman: (919) 541-7068
Mr. Murrey Olmsted: (919) 485-5506
Ms. Kristine L. Rae: (919) 541-8035
Dr. Michael Bowling: (919) 966-7021
Ms. Jeanne Givens: (919) 715-6448
Ms. Sherri Troop: (919) 715-6450
Dr. Michael Ballesteros: (770) 488-1308
B-
| File Type | application/msword |
| Author | hak5 |
| Last Modified By | fmc7 |
| File Modified | 2009-08-24 |
| File Created | 2009-08-24 |