1660-0146 Supporting Statement B 2021 12 15 clean

SUPPORTING STATEMENT B FOR

Post Disaster Survivor Preparedness Research

OMB Control No.: 1660-0146

COLLECTION INSTRUMENT(S): FEMA Form FF-008-FY-21-112

(formerly FEMA Form 519-0-54)

B. Collection of Information Employment Statistical Methods

The agency should be prepared to justify its decision not to use statistical methods in any case where such methods might reduce burden or improve accuracy of results. When Item 17 on the Form OMB 83-I is checked, "Yes," the following documentation should be included in the Supporting Statement to the extent that it applies to the methods proposed:

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The universe for this study will consist of all adults (18 years of age or older) residing in counties eligible for applying for individual and/or public assistance from FEMA as a result of a disaster, such as flood or hurricane (e.g. 2018 hurricanes Harvey, Irma, and Maria). Respondents who report that they resided in impacted counties at the time of the event will be considered eligible for this study. As needed, other selection criteria will be utilized, such as census track overlays and oversamples or quotas for specific demographics. As desired, depending on the populations affected by disasters, screener questions can be employed to oversample underserved populations or other population factors.

For surveys in this collection, a household-based telephone survey will be conducted to complete about 500 interviews for the telephone survey. The expected sample ratio of landlines to cell phones is 30 to 70, depending on the specific target population. This ratio is shaped based on the ever-increasing proportion of households that are wireless only or wireless primary, but for example, a greater proportion of landlines would be used to target the underserved population of adults 75 or older.

For the focus groups and cognitive interviews, depending on the desired audience, for example the underserved audience that will be the focus of the initial collection, the sampling methods will consist of recruitment through relevant FEMA programs, recruitment through local partners, a purchased listed sample, or other network contacts, as appropriate. Recruitment will be done through a screener call, email, or direct recruitment from contacts, in order to obtain an appropriate sample of these hard to reach audiences. Focus groups will consist of 10 participants per group, while cognitive interviews will have 20 participants.

Surveys for this collection had a response rate of 20-40%. For both focus groups and cognitive interviews, response rate was higher than was sufficient to fill the maximum participants. FEMA has been able to get high response rates on national preparedness surveys conducted previously with trained interviewers. The goal will be to maximize the response rate by taking necessary steps as outlined in section B3 on “Methods to maximize response rates.”

As stated, the general population parameters are those individuals living in the counties eligible to apply for individual and/or public assistance from FEMA. Depending on the target population within this eligible population, these parameters (proportions or means) will be estimated at the overall population level for sampled counties or based on the population of the targeted subgroup such as the underserved socio-demographic groups. As appropriate, the corresponding estimates at subgroup level will be computed and the precision associated with those estimates will depend on the resulting sample size (number of completed surveys) for these subgroups. For example, census data can be used to determine the proportion of specific underserved racial or socio-economic groups within the target location. Screeners could be employed to sample the target groups in relationship to the proportion of the total population (e.g., if the target audience represents 25% of the local population, the sample will consist of at least 25% of that population).

2. Describe the procedures for the collection of information, including:

1. Statistical methodology for stratification and sample selection,

The sample will consist of U.S. adults who are currently living in households in counties designated as eligible to receive individual and/or public assistance from FEMA as a result of a disaster. The sample will be geographically stratified into counties and sampling will be done independently within each county (region).

Telephone Survey

Using proportional sample allocation, the targeted number of surveys to be completed in each county is expected to be close to those proportions. It may be noted that the actual number of completed surveys for each county (and by landline and cell phone strata within each county) will depend on observed response rates and so they may not exactly match the corresponding targets. However, the goal will be to meet those targets to the extent possible by constant monitoring of the response rates and by optimally releasing the sample in a sequential manner throughout the data collection period. In addition, as possible, diversity of sampling will be increased through increased accessibility, such as availability of TTY technology.

For particular populations of interest, listed below, sampling will be conducted through options including listed samples, by employing a screener for specific demographics, or by connecting with programs and community leaders to gain access to specific groups. If a specific impact was observed in particular locations, sampling would focus on only those counties, and screen for experience with that impact. If the proportion of the sub population is large enough to be reached through random sampling, reaching a minimum of 100 participants of the 500 total participants, that method will be used for sampling. If the subpopulation is not large enough that random sampling is expected to reach 100 participants, an oversample will be applied. Either way, the responses from the subpopulation are included as part of the total population analysis, as well as specific comparative and descriptive sub-group analysis.

This collection may be used for follow-up surveys to track changes in attitudes and behaviors of disaster survivors over time. For these surveys, the sampling methods will be structured to match previous surveys of the same population samples. Survey items of particular interest over the long term will be repeated in research with other survivors from beyond the current hurricane season to track changes over time.

Within each county, the sampling of landline and cell phones will be carried out separately from the respective sampling frames. For both landline and cell phones, the geographic location of the respondent will be determined based on respondent’s self-reported response to a question on location (e.g., “what is your zip-code?”). For the cell phone sample, data will be collected from all respondents regardless of whether they also have access to a landline.

All cell numbers are de-duped against the most up to date available cell data bases when received and before dialing starts and are thereafter also de-duped against updated ported number data bases daily to capture any cell numbers that may have been reassigned in the prior 24 hours. It may be noted that due to continuous porting of numbers from landline to cell and cell to landline, some numbers from landline exchanges may turn out to be cell phones and conversely, some numbers sampled from the cell phone exchanges may actually be landline numbers. However, such numbers will be relatively rare and the vast majority of landline and cell phone numbers will be from the corresponding frames. The survey will also find out from the respondents if the number called is actually a landline or a cell phone number. The physical location of respondents will therefore be based on their self-reported location information (for example, based on their self-reported zip-code information) and will not be determined based on their telephone exchange.

Focus group and Interview Sampling

As focus groups and interviews allow for in-depth topic exploration, they will be used to target particular underserved audience to better understand their needs related to disaster preparedness, by measuring the following:

• Attitudes/Perceptions

• Awareness

• Knowledge

• Actions taken

• Barriers to action

• Motivation

• Self-efficacy

For the underserved audiences, listed below, sampling will be conducted through options including listed samples, by employing a screener for specific demographics, or by connecting with programs and community leaders to gain access to specific groups. If a specific impact was observed in particular locations, sampling would focus on only those counties, and screen for experience with that impact. A focus group or set of interviews will focus on one to two audiences per implementation, as mixing audiences would not allow for the necessary data analyses.

Demographics of affected respondents may focus on the six historically underserved populations, depending on the regions affected and the specific goals of collection:

• Socio-economically Disadvantaged: Populations that are disadvantaged due to low levels of income, community influence, and/or status.

• People with Access and Functional Needs: Populations that experience difficulty seeing, hearing, speaking, walking, taking care of daily needs, and/or living independently.

• Ethnic Minorities: Populations that may live in geographically and/or socially isolated communities, feel distrustful of police and emergency personnel, and/or those with limited English proficiency.

• The Very Young and Very Old: Populations that may have mobility constraints or concerns, and may rely on others for safety and preparedness.

• Sex and Gender: Populations that have been historically underserved based on sex, gender, and/or preference.

• Tribal Communities: Tribes or groups that are federally recognized and eligible for funding and services from the Bureau of Indian Affairs (BIA), there are currently 566 federally recognized tribes.

• Members of religious minorities.

• Persons who live in rural areas.

We will recruit a diverse audience within the underserved group, either with guidance from local partners or through random sampling. The specific metrics of diversity will be determined by the goals of the research. For example, for tribal populations, this could include recruiting members of a few tribes, or those who live on reservations and those who do not. Or, for another example, interviews would include individuals with a diverse set of access and functional needs, such as those who are legally blind or require the use of a wheelchair. The focus groups will draw from a single geographic area, as participants will need to gather to a single location to participate. The interviews will focus on one to three areas, as appropriate for the target audience and the impact zone of the disaster. Interviews can be conducted in person or on the phone, as appropriate for the audience, allowing for a greater diversity of locations if desired. As possible, disabilities, access, and functional needs will be accounted for during each specific implementation of the focus groups and cognitive interviews to allow for diverse sampling through accessibility for all. Focus group facilities will be chosen that are easily accessible for participants, considering the building (e.g., ramps), transportation (e.g., proximity of public transit), language (e.g., conducting focus groups in Spanish), and other needs as appropriate.

2. Estimation procedures

If needed, sample data will be weighted to generate unbiased estimates. Within each stratum (county), weighting could be carried out to adjust for (i) unequal probability of selection in the sample and (ii) nonresponse.

In studies dealing with both landline and cell phone samples, one approach is to screen for “cell only” respondents by asking respondents reached on the cell phones whether or not they also have access to a landline and then interviewing all eligible persons from the landline sample whereas interviewing only “cell only” persons from the cell phone sample. The samples from such designs are stratified, with each frame constituting its own stratum. A dual-frame design will be used where dual users (those with access to both landline and cell phones) can be interviewed in either sample. This will result in two estimates for the dual users based on the two samples (landline and cell). The two estimates for the dual users will then be combined and added to the estimates based on landline-only and cell-only population to generate the estimate for the whole population.

Composite pre-weight: As needed, the states will be used as weighting adjustment classes. The composite pre-weight could be generated within each weighting class. The weight assigned to the i^th respondent in the h^th weighting class (h=1, 2, 3, 4) could be calculated as follows:

W_{(landline,hi)} = (N_hl/n_hl)(1/RR_hl)(n_cwa/n_ll)(λ^IDual) for landline sample cases (1)

W_(Cell,hi) = (N_hc/n_hc)(1/RR_hc)(1 – λ)^IDual for cellular sample cases (2)

where

N_hl: size of the landline RDD frame in weighting class h

n_hl: sample size from landline frame in weighting class h

RR_hl: response rate in weighting class h associated with landline frame

n_cwa: number of adults in the sampled household

n_ll: number of residential telephone landlines in sampled household

I^Dual: indicator variable with value 1 if the respondent is a dual user and value 0 otherwise

N_hc: size of the Cell RDD frame in weighting class h

n_hc: sample size from Cell frame in weighting class h

RR_hc: response rate in weighting class h associated with Cell frame

‘λ’ is the “mixing parameter” with a value between 0 and 1. If roughly the same number of dual users is interviewed from both samples (landline and cell) within each state, then 0.5 will serve as a reasonable approximation to the optimal value for λ. This adjustment of the weights for the dual users based on the value of the mixing parameter ‘λ’ will be carried out within each state. For this study, the plan is to use a value of ‘λ’ equal to the ratio of the number of dual users interviewed from the landline frame and the total number dual users interviewed from both frames within each state. One or two additional values of the mixing parameter may be tested to see the impact on survey estimates. It is anticipated that the value of the mixing parameter will be close to 0.5.

It may be noted that equation (2) above for cellular sample cases doesn’t include weighting adjustments for (i) number of adults and (ii) telephone lines. For cellular sample cases, as mentioned before, there is no within-household random selection. The random selection can be made from all persons sharing a cell phone but the percentage of those sharing a cell phone is rather small and it would also require additional questionnaire time to try to capture such information. The person answering the call will be selected as the respondent if he or she is otherwise found eligible and hence no adjustment based on “number of eligible adults in the household” will be necessary. The information on the number of cell phones owned by a respondent could also be asked to make adjustments based on number of cell phones. However, the percentage of respondents owning more than one cell phone is expected to be too low to have any significant impact on sampling weights. For landline sample cases, the values for (i) number of eligible adults (n_cwa) and (ii) number of residential telephone lines (n_ll) may have to be truncated to avoid extreme weights. The cutoff value for truncation will be determined after examining the distribution of these variables in the sample. It is anticipated that these values may be capped at 2 or 3.

Response rate: If needed, the response rates (RR_hl and RR_hc mentioned above in equations (1) and (2)), could be measured using the AAPOR (3) definition of response rate within each weighting class and calculated as follows:

RR = (number of completed interviews) / (estimated number of eligibles)

= (number of completed interviews) / (known eligibles + presumed eligibles)

It will be straightforward to find the number of completed interviews and the number of known eligibles. The estimation of the number of “presumed eligibles” will be done in the following way: In terms of eligibility, all sample records (irrespective of whether any contact/interview was obtained) may be divided into three groups: i) known eligibles (i.e., cases where the respondents, based on their responses to screening questions, were found eligible for the survey), ii) known ineligibles (i.e., cases where the respondents, based on their responses to screening questions, were found ineligible for the survey), and iii) eligibility unknown (i.e., cases where all screening questions could not be asked, as there was never any human contact or cases where respondents answered the screening questions with a “Don’t Know” or “Refused” response and hence the eligibility is unknown).

Based on cases where the eligibility status is known (known eligible or known ineligible), the eligibility rate (ER) is computed as:

ER = (known eligibles) / (known eligibles + known ineligibles)

Thus, the ER is the proportion of eligibles found in the group of respondents for whom the eligibility could be established.

At the next step, the number of presumed eligibles will be calculated as:

Presumed eligibles = ER × number of respondents in the eligibility unknown group

The basic assumption is that the eligibility rate among cases where eligibility could not be established is the same as the eligibility rate among cases where eligibility status was known. The response rate formula presented above is based on standard guidelines on definitions and calculations of Response Rates provided by AAPOR (American Association for Public Opinion Research).

Post-stratification weight—If weighted samples are required, once the two samples are combined using the composite weight (equations (1) and (2) above), a post-stratification weighting step will be carried out, to simultaneously rake the combined sample to (i) known characteristics of the target population (adults living in the designated counties, age, gender, and race/ethnicity ) and (ii) an estimated parameter for relative telephone usage (landline-only, cell only, cell mostly, other dual users). The target numbers for post-stratification weighting will be obtained from the latest available county data available.

The target numbers for the relative telephone usage parameter will be based on the latest estimates from NHIS (National Health Interview Survey). For the purpose of identifying the “cell mostly” respondents among the group of dual users, the following question will be included in the survey.

DC

QID:103424 Of all the telephone calls your household receives (read 1-3)?

1 All or almost all calls are received on cell phones

2 Some are received on cell phones and some on regular phones, OR

3 Very few or none are received on cell phones

4 (DK)

5 (Refused)

Respondents choosing response category 1 (all or almost all calls are received on cell phones) will be identified as “cell mostly” respondents.

After post-stratification weighting, the distribution of the final weights will be examined and trimming of extreme weights, if any, will be carried out if necessary to minimize the effect of large weights on variance of estimates.

3. Degree of accuracy needed for the purpose described in the justification

The margin of error (MOE) for estimating the unknown population proportion ‘P’ at the 95% confidence level could be derived based on the following formula:

MOE = 1.96 * where “n” is the sample size (i.e. the number of completed surveys).

In a dual frame household-based survey, some design effect is expected but the precision for survey-based estimates for most subgroups of interest are likely to have reasonable precision. For example, the sampling error associated with an estimate based on a sample size of 1,000 with a design effect of 1.25 will still be below ±3.5 points. Hence, the accuracy and reliability of the information collected in this study will be adequate for its intended uses. The sampling error of estimates for this survey will be computed using special software (like SUDAAN) that calculates standard errors of estimates by taking into account the complexity, if any, in the sample design and the resulting set of unequal sample weights.

4. Unusual problems requiring specialized sampling procedures

Unusual problems requiring specialized sampling procedures are not anticipated at this time. If response rates fall below the expected levels, additional sample will be released to generate the targeted number of surveys. However, all necessary steps to maximize response rates will be taken throughout the data collection period and hence such situations are not anticipated.

5. Any use of periodic (less frequent than annual) data collection cycles to reduce burden

The data will be collected on an “as needed” basis after an applicable disaster, and therefore would not be conducted on an annual basis.

3. Describe methods to maximize response rates and to deal with issues of non‑response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Survey based estimates for this study will be weighted to minimize any potential bias that may be associated with unit level non-response, If found necessary, the respondents may be split into two groups: (i) early or ‘easy to reach’ and (ii) late or ‘difficult to reach’ respondents. The total number of calls required to complete an interview will be used to identify these groups. This comparison will be based on selected key questions from the main survey and on the assumption that the latter group may in some ways resemble the population of non-respondents. The goal of the analysis plan will be to assess the nature of non-response pattern in the main survey.

Methods to maximize quality and response rates

During data collection, ISA’s Quality Assurance (QA) department runs through an extensive list of checks for trackers, and Action Research supervises to ensure quality of data. The following outlines these checks.

Systemized Data Collection

In addition, the sample will be dialed in a systemized manner. The first replicate in each area will be released by ISA’s CATI system for initial attempts. The system manages the callback times as well. When a respondent asks for a specific callback day and time the system will bring that up at that time for another call. When a respondent doesn’t ask for a specific time to be called back or when no respondent was reached on any given attempt, the system brings their number up randomly at a different time of day and a different day than the original call. This ensures that all potential respondents will be called on several different occasions in ISA’s attempt to complete an interview. Sample records are given up to eight call attempts before a number is resolved as “call limit reached.” Good numbers are also resolved after second refusals or as a request from a respondent.

Run dummy data and check marginals

The initial dummy marginal is primarily used to check logic - such as skip patterns, selection processes, etc.

Run a marginal after 1st night's interviewing

This marginal allows QA to double check skip patterns, Don’t Know (DK) ratios as well as odd responses which are carefully reviewed and reported to the project managers, so that they can de-brief the interviewers.

Run survey length by interviewer

The length is normally compared to the overall avg. length. If the length is a + or - 10% off, then the interviewer is automatically flagged and will be monitored closely. The survey will also be validated to ensure qualification questions as well as major skip pattern questions were correctly coded. In addition, ISA will run a marginal for that specific interviewer to review certain responses that may have driven the length to increase/decrease.

Internal Monitoring

30-50% of ISA’s Data Collectors are monitored daily. Each Data Collector is monitored, on average, three times per week – 30-50% of the shifts they work. Data Collectors receive feedback and coaching based on performance at all times. Monitors use a report form to evaluate Data Collectors – from reading verbatim, to not biasing respondents, articulation, flow and confidence. In addition, if for some reason, an interviewer was flagged by CATI's QA as having high ratios of DK's, shorter surveys or a higher production, this interviewer will be monitored more often. In addition to verifying proper qualification, monitors review probing and clarification methods.

Client Monitoring

ISA offers clients sophisticated remote and in-house monitoring. Up to 6 remote clients can monitor a project at one time. Remote monitors are facilitated by a live Quality Assurance Supervisor so that immediate client feedback can be relayed on to data collectors. Action Research will periodically monitor surveys to assure quality.

Validation

ISA validates 15% of a specific Interviewer’s overall work. Questions normally asked are qualifying questions, quota questions (Age, Income, and Ethnicity) as well as major skip pattern questions.

Run don't know %'s by interviewer

If for some reason, it becomes apparent that an interviewer has a high ratio of DK's, he/she will be re-trained and re-briefed. In addition, the interviewer will be heavily monitored. If for some reason, the numbers don't improve, the interviewer will be removed from this study.

Review the open ends daily

This method allows ISA to verify that the interviewers and probing and clarifying properly. For the responses that are not fully probed and/or clarified, ISA will re-contact these respondents and get a more accurate response.

Run a marginal at the 1/3 of the way point, 2/3 of the way point and at the end

These marginals are compared to previous waves in terms of key metrics and DK levels.

If these metrics don't match, the question #'s in dispute will be reported to the Project Manager who addresses it with supervisors, monitors, interviewers as well as clients.

On-going interviewer training

All Data Collectors are required to attend regular continuing education seminars to improve their interviewing skills. Trainers teach “refresher” workshops – from communication, to Probing & Clarifying, Refusal Prevention and Introduction.

Issues of Non-Response

Survey based estimates for this study will be weighted to minimize any potential bias, including any bias that may be associated with unit level nonresponse. For any subgroup of interest, the sampling error will depend on the sample size. All estimates will be weighted to reduce bias and it will be possible to calculate the sampling error associated with any subgroup estimate in order to ensure that the accuracy and reliability is adequate for intended uses of any such estimate. Based on experience from conducting similar surveys previously and given that the mode of data collection for the proposed survey is telephone, the extent of missing data at the item level is expected to be minimal. We, therefore, do not anticipate using any imputation procedure to handle item-level missing data.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of test may be submitted for approval separately or in combination with the main collection of information.

The CATI surveys will be tested with fewer than ten (10) respondents prior to fielding.

5. Provide the name, affiliation (company, agency, or organization), and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The representative of the contractor who will collect and/or analyze the information for the agency is:

Name	Agency/Company/Organization	Number Telephone
Kelly Walter	Deloitte	212-492-4000

Page 7 of 7