PAR Survey Part B Statistical Methods

FINAL SUPPORTING STATEMENT

FOR

NRC SURVEY OF PUBLIC RESPONSE TO EMERGENCIES

(3150-XXXX)

NEW COLLECTION

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Respondent Description

The primary objective of this project is to gain a broader and deeper understanding of people’s views and reactions to protective action strategies within the 10 mile EPZ of nuclear power plants, and to assess the degree of public acceptance of those strategies. Among the most important factors affecting the viability of sheltering and evacuation strategies are the views and reactions of the public. Do people fully understand the various strategies or are the strategies seen as confusing? Does the public view protective action instructions as credible and practicable and do people have confidence that undertaking the prescribed actions will make them safer? Does the public see the adoption of multiple strategies as ineffective? Do people intend to undertake protective actions, or will they ignore them? All of these questions are crucial to determining whether evacuation and sheltering strategies will be effective. Thus, any consideration of sheltering and evacuation strategies needs to be grounded in a detailed understanding of public views and reactions.

Once the data is obtained, it will be used to inform a decision process which will determine whether enhancements to protective actions should be implemented. The extent of the use of the data is fully dependent upon the results. The decision on enhancements in the protective action regime should be substantiated through public input and stakeholder involvement. Information from the survey may also support decisions on an approach to management of shadow evacuations which includes evacuees that leave, but are not within the evacuation area. The information is also expected to provide insights on where evacuees intend to go if ordered to evacuate. Current plans include establishing congregate care centers to accommodate evacuees. The survey may provide a basis that supports estimating the population that these centers would be designed to accommodate.

The potential respondent universe is approximately 5 million members of the public who reside within the boundaries of 62 EPZs around nuclear power plants. The number of respondents to be contacted includes approximately 2,500. Approximately 1,580 will answer the phone and not agree to take the survey. Approximately 930 are expected to agree to complete the survey and approximately 120 of these are expected to drop out of the survey during the course of questioning. The remaining 800 are expected to be completed surveys. These values are based on the experience of professionals who conduct telephone surveys of this length. No similar collection has been previously conducted.

Describe Procedures for Collecting the Information

The telephone survey contractor conducts surveys using a computer-assisted telephone interviewing (CATI) system and multi-station survey laboratory. Each working phone number is called 5 times. These calls will be at different times of day and days of the week. This process is controlled by the CATI phone room software developed by the contractor. Trained interviewers will conduct the survey under supervision using an industry standard survey protocol. The intent of this protocol is to maximize both the survey response and cooperation rates, and the consistency of implementation to assure maximum data validity and reliability. The telephone survey samples the non-institutionalized adult (over age 18) residential population. To identify the sample area, radii are calculated by determining the zip+4's that lie within the desired radius distance. Any zip+4 having its centroid located within the radius is included. The centroid is the discrete weighted population point within the zip+4. For RDD to be produced, all area code / exchanges (e.g. 480-812-XXXX) within a qualifying zip+4 are used to generate RDD numbers according to the contractor’s standard RDD methodology. Because area code / exchange geography in some areas may be larger than zip+4 geography, RDD numbers will spill in/out of the defined radius with that amount varying depending on the phone company wiring of local numbers. As part of the initial screening process in the interview, the contractor verifies that each respondent lives within the EPZ.

The contractor uses a computerized sampling program to draw a sample of randomly generated phone numbers from a frame that includes the numbers of all households with working telephones. To accomplish this, a computerized sampling program randomly selects numbers from the working blocks of numbers within each active telephone prefix in a given region. Because the selection covers the full range of each working block, unlisted phone numbers are included in the sample. The ranges of working numbers are updated regularly by telephone sample providers to assure continued accuracy of the sample phone lists.

The standard sample frames are developed by sample providers using a random phone number generation program that relies on a specially designed telephone number database. This database allows for the random production of the numbers attached to valid prefixes in any given region designated for potential inclusion in a given study. Databases generally contain all working prefixes in a given region and, within prefixes, all "blocks" of 100 telephone "suffixes" that contain working residential numbers. As specified in the sampling frame, each prefix has 100 blocks, beginning with the block containing numbers 0000-0099 and ending with the block containing numbers 9900-9999. The random phone number generator draws non-repeating random telephone numbers from within the blocks and prefixes with working residential numbers in designated regions to produce a working frame for implementation based on a predetermined sample size.

Because the list is in random order, any contiguous segment of the list itself constitutes a random sample of phone numbers. Lists contain percentages of numbers within each of the working prefixes that are proportionate to the number of working blocks within that prefix. Thus, a full prefix (in which all 100 blocks have working numbers) will be represented in the list twice as frequently as would a prefix in which only half of the blocks have working residential numbers.

Blocks with no working residential numbers are screened out to increase the efficiency of the lists. Nevertheless, the effective "density" - or percentage of working residential numbers - will vary across blocks. Typically, new prefixes or those that serve rural areas have blocks with lower densities than do those that have been in service longer or that serve urban areas. To compensate for the variance in density, the frequency of numbers drawn by block is kept constant. Thus, due to the random generation of numbers within blocks, low-density blocks will produce a higher frequency of invalid numbers. The invalid numbers are screened out in the interviewing process. In this way, the proportion of valid numbers within each prefix is retained, thereby preserving the equality of probability of inclusion within the list for any household with a (single) residential phone line.

For statistical purposes, it is necessary that (a) each household with a telephone and (b) each individual interviewed have an identifiable probability of inclusion in the sample. The frame assures that each working residential telephone number has an equal probability of being contacted. The number of working residential phone lines at each residence contacted is recorded in the interview in order to obtain weights to correct for multiple-phone households. In addition, the respondent is selected at random from among those eligible within the household. Thus, the sampling procedure assures that the household and the respondent within the household are selected at random. The interview protocol assures that the data needed to develop weights to correct for differences in the probability that a household would be contacted (the number of working residential phone lines) and that a respondent would be selected for the interview (the number of eligible respondents within the household) are collected. The weights can be readily applied to obtain household population frequencies from the results of the samples.

The selection of the approximate 800 completed surveys will be collected, providing a margin of error of +/- 4% at the 97% confidence level. There are no potential problems in conducting this survey.

3. Describe Methods to Maximize Response Rates and to Deal with Statistical Issues of Non-response

To assure that the samples used during the data collection process permit reliable statistical inference, the contractor implement’s quality control procedures. This involves extensive review of the survey instrument where the survey is checked for biased or misleading questions, or questions that may be culturally insensitive or threatening to different socio-demographic groups. This process assures that the survey itself does not inadvertently induce respondents from different groups or classes to dropout before completing the survey. The survey process is designed to maximize response rate by using a contact design that minimizes refusals, employs up to 10 call backs. This includes the concise and informative introduction, employing a short 15 minute survey and respondent tracking. Respondent tracking is employed and a conversion protocol is in place for individuals who are initially categorized as soft refusals. This approach has been demonstrated in market analysis to maximize response rates and yield reliable data that can be generalized to the universe studied.

The response rate will be in the range of 10-15% according to the AAPOR formula. The actual rate can be calculated once the total number of participants selected for the survey is identified. It is anticipated that a typical 15:1 RDD phone sample will be purchased to begin the research process. To assess non-response bias, the contractor will compare our respondents’ demographics to comparable census data. To support the non-response bias assessment, one question has been added to obtain the age range of the respondents. Discussion and assessment of non-response will be included in the final reporting.

4. Describe Test or Procedures

Early testing of the survey instrument was conducted in-house. Results of the in-house testing concluded that the open ended questions caused the survey to require more time than desired and required additional burden in the analysis of the data. Through review and editing of the survey instrument, the open ended questions were removed. Where practical, some of these questions were rewritten with items from which the respondent can select one or more. This approach reduced the time to conduct the survey to 15 minutes and significantly reduced the burden in coding and analyzing the data from the open ended questions.

The procedure requires that the survey instrument be first programmed into the computer assisted telephone interviewing (CATI) system. This includes all tracking, skip and randomization protocols. Then a verbal protocol of the survey instrument is conducted to test the efficacy of the questions and skip patterns. Once the survey is programmed and the verbal protocol complete, the next step involves training the interviewers to properly execute the survey. This process entails oral reading of the survey in several group training sessions to make sure that proper and consistent emphasis is given to the various words and phrases specified in the survey, to assure that respondents are interviewed using consistent phrasing, emphasis and protocols during the data collection process. Data collection does not begin until the interviewers have demonstrated thorough competence with the survey instructions and the reading of the survey. Internal testing of the telephone survey was conducted to determine the length of the survey and assure the automated prompts were correct. The survey length is approximately 15 minutes.

The data will be captured in Excel and SPSS format. All personal information identifying respondents is removed by the market research firm prior to submittal of data to Sandia. There are no open ended questions in the survey which simplifies the statistical analysis. Once the data is received, Sandia will perform statistical analysis to determine the appropriate use of the information to the decision making process. This will include frequency distributions and measures of central tendency for questions asked in the survey.

5. Name and Telephone Number of Individual Consulted on Statistics

Statistician consulted for the statistical aspects of the survey design:

Carl Axness
505-844-0084
Sandia National Laboratories