PRA Application Supporting Statement
OMB Control #0693-0078
Expiration Date: 07/31/2022
NIST Generic Clearance for Community Resilience Data Collections
HURRICANE MARIA EMERGENCY COMMUNICATIONS INVESTIGATION:
FINAL HOUSEHOLD SURVEY
FOUR STANDARD SURVEY QUESTIONS
1. Explain who will be surveyed and why the group is appropriate to survey.
The National Institute of Standards and Technology (NIST) has the responsibility to investigate cases of serious failures of the built environment (buildings and infrastructure) under the National Construction Safety Team Act, signed into law in 20021. Members of the National Construction Safety Team (NCST) based at NIST were tasked in early 2018 with the investigation of impacts from Hurricane Maria, which hit Puerto Rico on September 20, 2017. NCST duties include:
“...establish the likely technical cause or causes of the building failure;
evaluate the technical aspects of evacuation and emergency response procedures;
recommend, as necessary, improvements to building standards, codes, and practices based on the findings;
recommend any research and other appropriate actions needed to improve the structural safety of buildings, and improve evacuation and emergency response procedures, based on the findings of the investigation.”
NIST is conducting multiple projects that represent all of these areas for investigation. However, the survey for which this application is written pertains to the second and fourth duties of the NCST listed above. The goal of this specific portion of the investigation is to evaluate the effectiveness of emergency communication as well as behavioral aspects of evacuation response across the population of Puerto Rico. This investigation is not considered research but is instead a fact-finding mission to establish the role that emergency communication played in effective or ineffective evacuation behavior. The purpose of NCST investigative activities is to make recommendations, based directly on findings, that can help prevent future deaths and injuries across the United States.
To complete this investigation, NIST has partnered with contractors who can utilize trained personnel local to Puerto Rico to complete data collection activities. The contractors, in consultation with NIST scientists and engineers, have used U.S. Census data to develop a sampling strategy for the survey (1,500 respondents) whereby data collection will occur in four specific regions of Puerto Rico that facilitate representation of key hurricane-impacted areas, but is still representative of the island’s overall population. Efforts to understand our target population is informed by pilot testing the survey (20 respondents) which concluded in December 2020 and yielded helpful feedback to make items more appropriate and easier to understand for our target audience. With feedback from respondents and analysis of pilot data from the contractors now completed, NIST is well positioned to meet its NCST duties with an excellent final survey instrument.
In line with our investigation goals, this household survey instrument will elicit data on emergency communication systems, emergency messages, and factors influencing evacuation behavior, as well as actual evacuation behavior on the island of Puerto Rico for Hurricane Maria from a sample of the general public. This survey is also a key component among two related data collection activities including interviews with providers of emergency communications messages (PRA approved; ongoing), as well as follow-up interviews with a sample of the public to learn more about their emergency preparedness and response (in development).
2. Explain how the survey was developed including consultation with interested parties, pre-testing, and responses to suggestions for improvement.
The survey instrument was developed by NIST scientists with backgrounds in sociology, anthropology, communication, and psychology through a series of development and quality control activities. First, a review of previous hurricane literature and survey instruments was conducted to better understand the types of factors that influence public response during these types of disaster events and develop the basis of our survey. When possible, established scales and items were used from previous hurricane and/or evacuation and protective action research.
Next, after a draft questionnaire was developed, NIST investigators elicited informal feedback from numerous content experts in hurricane forecasting, emergency communication, and evacuation behavior from experts at The National Oceanic and Atmospheric Administration (NOAA), as well as professors from the University of Puerto Rico and other academic colleagues across the U.S.. Additional revisions were made that focused on the content of the survey questions (i.e. which concepts were measured and included), and the instrument was condensed to reduce its length.
NIST investigators then disseminated the revised questions to a few survey experts for final refinement. These included experts here at NIST, as well as project contractors. A final round of revisions was made focusing on details of the content, including issues with item clarity, question ordering, and consistency in wording. These efforts resulted in the pilot survey which received previous PRA approval, which formed the last phase of instrument testing and refinement. The pilot survey was conducted verbally, over the phone, from November 6th to December 14th with 20 respondents. Results from the 20 responses, both quantitative data in terms of response trends, and qualitative data in terms of item feedback (e.g. items participants found confusing) were summarized by the contractors and presented to the NIST investigation team. Following a thorough review of this feedback, the investigation team made edits to survey items to improve clarity and consistency of wording and help ensure proper understanding of question meaning. For example, text for several questions was simplified, and changed to dichotomous response options to simplify complexity, and also to enhance respondent accuracy. This revised version was reviewed by contractors and finalized into the version being submitted with this application (please see two attached surveys for the phone and online versions).
3. Explain how the survey will be conducted, how customers will be sampled if fewer than all customers will be surveyed, expected response rate, and actions your agency plans to take to improve the response rate.
The contractors, in consultation with NIST scientists and engineers, used U.S. Census data to develop a sampling strategy for the final survey of 1,500 respondents. Data collection will occur in four regions of Puerto Rico that represent key hurricane-impacted areas. More specifically, the contractors used FEMA’s Risk Mapping, Analysis, and Planning (Risk MAP) website and USGS landslide data for Puerto Rico to identify census tracts within those four regions that are in flood and landslide-prone areas. Sampling for these hazard risks will ensure we receive data from those who were most likely to have needed emergency communications during Hurricane Maria. A random stratified sample of neighborhoods was then selected from a set of possible addresses from the Topologically Integrated Geographic Encoding and Referencing system (TIGER) data produced by the U.S. Census Bureau. Although we are including hazard exposure in sample considerations, contractors are also ensuring that samples will be representative of the island’s overall population in terms of demographic stratification.
Originally, surveying was supposed to be done in person. However, due to COVID an alternate strategy has been developed to invite respondents to participate either (1) online or (2) via phone. To initiate survey data collection, team members in Puerto Rico will follow the sequence of randomly selected neighborhoods and, on foot, deliver postcard-sized invitations (using social distancing protocols) to households. (See attachment for text of invitation postcard). Instructions will be written for the teams including walking rules, residence selection, rules for MDU’s (multiple dwelling units), and rules for revisiting non-responding households for additional contact attempts. Team members will also be given forms and instructions for documenting households where postcards were left and any other interactions and will receive appropriate safety training. We anticipate that 50% of respondents will complete the online survey and 50% will complete the survey via telephone.
As distribution of survey invitations and data collection proceeds, the contractor team will monitor response rates and may selectively re-invite in certain areas to improve the final sample size and composition. To reach our target sample of 1,500 respondents, 375 completed surveys are required in each of the four sampled regions within Puerto Rico. If necessary, contractors are willing to distribute up to 10 times the number of postcards as responses that we are seeking, but the initial plan is to distribute three times the number of postcards in each region as the targeted responses. Our hope is that since the contractor is associated with a University local to Puerto Rico, that residents will be more likely to respond than they would if they were being contacted directly by a federal agency.
Each postcard will have a unique participant code, which will be used to track responses and follow-ups, and will enable us to have no PII, such as address, in the final data set. The participant code will be used to ensure that individuals from the same address cannot take the survey multiple times; improving data quality while protecting their privacy. If a participant chooses to take the survey online, a link will enable them to access the Qualtrics survey with their unique passcode. If a participant chooses to take the survey over the phone, they will be directed to a professional call center proficient in English and Spanish, coordinated by the contractor where the interviewer will input responses into a Qualtrics version of the survey. Appropriate consent and screening questions will be asked before respondents are able to complete with the survey.
Time burden is calculated to be 1500 respondents * 25 (minutes) / 60 (minutes in an hour) = 625 burden hours.
PII is collected in this instrument, but information is not retrieved by personal identifiers in the system. Therefore, this is not a Privacy Act System and SORN and Privacy Act Notice are not applicable.
4. Describe how the results of the survey will be analyzed and used to generalize the results to the entire customer population.
After survey data collection has closed and 1,500 complete responses have been obtained, the contractors will share the final dataset with the investigation team as a .csv file via secure NIST file transfer system. Data will contain participant responses to the survey but will remain anonymous. Address data and contact information will not be shared, and home location information will be recorded only by Census Block.
Following data checks and cleaning, descriptive statistics and regression analyses will be used to answer a series of specific investigation questions, including the types of emergency messages received by individuals, what individual characteristics are most closely associated with the decision to evacuate, and what material or cognitive factors were influential on their decision to evacuate. Our sample demographics (e.g. average age, income) will also be compared to population-level Census data to confirm the representativeness of the sample and the generalizability of our results. These results will be summarized in a report informing the effectiveness of emergency communication as well as behavioral aspects of evacuation response across the population of Puerto Rico.
The purpose of an NCST investigation is to help prevent future deaths and injuries across the U.S. by recommending actions that can influence codes, standards, and practices. Lessons learned from Puerto Rico regarding the use and effectiveness of emergency communications, and their influence on evacuation behaviors, can be useful not only to better understand the impacts from this particular storm, but also can also be relevant for other hurricane prone regions and in other hazard conditions. For example, lessons learned by NIST’s NCST investigation of the Joplin, M.O. tornado in 2006 have been applied to help standardize siren emergency communications across the U.S.. The entire NIST Hurricane Maria investigation team is hopeful that this kind of impact will also be possible based on the results of our household survey regarding emergency communications in Puerto Rico.
1 https://www.nist.gov/system/files/documents/public_affairs/releases/hr46871.pdf
| File Type | application/zip |
| Author | Walpole, Emily H. (Fed) |
| File Modified | 0000-00-00 |
| File Created | 2022-09-28 |