1660-NW131 - Supporting Statement B 2023 07 06 clean

July 6, 2023

SUPPORTING STATEMENT PART B FOR

Generic Clearance for the Multi-Modal Mixed Methods Collection of Information to Inform Agency Marketing and Outreach

OMB Control No.: 1660-NW131

COLLECTION INSTRUMENT(S): None

B. STATISTICAL METHODS

Data collection methods and procedures will vary. When generalizable information is desired, statistical methods, including appropriate sampling methods, will be used to limit burden on the general public while supporting accurate information gathering for FEMA’s use. The primary purpose of these collections will be to inform marketing and outreach of a FEMA program. Both generalizable and non-generalizable information is suitable for this purpose, and this supporting statement describes both. Publishing or otherwise releasing this information will be for the purpose of deepening engagement of industry, other stakeholders, the FEMA regions, and the general public. All information released to the public will be appropriately aggregated and/or anonymized to protect participant privacy.

1. Universe and Respondent Selection

Non-Generalizable Quantitative and Qualitative Data

Many activities under this generic clearance will involve the collection of novel qualitative and quantitative non-generalizable data from self-selected members of the general public using convenience and/or quota sampling approaches. Qualitative data will be collected through a variety of methods (e.g., focus groups, surveys, key informant interviews, cognitive interviews). Specifics related to the sampling universe, collection methods, data collection instruments, and scope for each individual collection will be described fully in future specific information collection requests submitted under this generic clearance. The samples associated with these collections are not subject to the same scrutiny as scientifically drawn samples where estimates are published in peer-reviewed journals or relied upon for longitudinal comparisons. Information gained from these collections will be used internally to strengthen marketing and outreach efforts to support a FEMA program or programs. Information may also be used to engage external stakeholders.

Generalizable Quantitative Survey Data

In certain circumstances the program will collect statistically validated generalizable samples through surveys. Topics anticipated to be covered include perceptions of the program specially, as well as general knowledge and attitudes relevant to the program, among members of the U.S. population or a subset of the U.S. population. Specifics related to sampling strategies, cadence of collection, investigatory methods, the data collection instrument, and the proposed sampling universe will be provided in the specific information collection requests submitted under this generic clearance. Information from these surveys will be used to strengthen program awareness, marketing and outreach campaigns, community outreach, and strategic planning consistent with the program’s stated goals and objectives.

In general, the program anticipates that future surveys will be stratified by FEMA region, and the program plans to sample a total of 5,000 persons (500 from each of the 10 FEMA regions). Region-specific surveys may also be conducted with stratification at the state level. All surveys will be collected using web-based methods accessible through laptop computers, smartphones, and tablets. The sampling universe for these collections will include the entire civilian non-institutionalized U.S. adult population (aged 18 years and older) who have access to internet-based surveys administered through a cell phone, computer, or tablet. This excludes adults in penal or mental institutions, those without internet access or access to the above-mentioned devices, and/or adults who do not read and/or speak English or Spanish well enough to understand surveys.

The program will employ a third-party independent polling firm who has experience collecting nationally representative samples of this type. Based on previous experience the program anticipates the overall response rate for each web-based survey to be between 3%-5%.

2. Procedures for Collecting Information

Statistical methodology for stratification and sample selection

Non-Generalizable Quantitative and Qualitative Data

Data collection methods, sample size estimates, and procedures will vary for non-generalizability investigatory efforts, the specifics of each collection will be provided in the specific information collection requests submitted under this generic clearance. In general, the program and/or its contractors will rely on a reputable marketing research or survey/polling firm to recruit participants for qualitative studies such as focus groups and cognitive interviews. For more limited data collections, a snowball sampling approach may be used with existing networks of the program and/or its contractors. The program does anticipate that some collections will be specific to particular groups by factors such as geography (e.g., FEMA region), participatory status (e.g., program customers or participants), stakeholder group (e.g., stakeholders in the program). For these the program will outline any specialized stratification and selection techniques as necessary, as well as mitigation strategies or offsetting low responses with the specific information collection requests submitted under this generic clearance.

Generalizable Quantitative Survey Data

For surveys intended to be generalizable, the following procedures will be used to ensure accurate and transparent statistical methods are applied. For sampling related to nationally representative surveys the program will create 10 separate mutually exclusive sampling strata based on the 10 FEMA regions covering the United States and its territories. To ensure sufficient representation across states and territories a proportional geographic stratification approach may be implemented. As a guide the program anticipates enrolling approximately 500 respondents per strata, but this number may be adjusted slightly to account for any differential population distribution between FEMA regions. The same approach would be applied for regional surveys with stratification by state.

To recruit participants for web-based surveys the program will acquire statistically representative panels provided through a reputable market research or survey/polling firm. Panel recruitment utilizes probability-based recruitment and comes from a variety of sources. The online panels have been recruited through a diversified network rather than through a single source to avoid 'professional' panelists. To ensure that the recruitment is as broad, diversified, and as exhaustive as possible, a wide range of different methods and sources are used for recruitment. The program will rely on random sampling techniques to recruit participants from panels within each stratum. Surveys will be administered using a web-based application programming interface (API). Invitations to participate will be shared digitally with potential participants. Screening questions will confirm eligibility (e.g., residency status) to ensure accurate weighing.

Estimation procedure

Non-Generalizable Quantitative and Qualitative Data

Special estimation procedures are not applicable for non-generalizable sampling procedures.

Generalizable Quantitative Survey Data

For generalizable quantitative surveys the program will rely on post-collection weighting estimation procedures to improve the accuracy and generalizability of results to the desired sampling universe. Inverse probability weighting procedures (where the inverse probability of being selected based on the sampling frame) will be used to construct weights.¹ As each stratum will contain a mutually exclusive sampling frame, this procedure will be repeated to generate stratum specific weights.

Table 2. Example Base Weight Construction for National Sample

	Sample Size	Selection Probability	Weight
FEMA Region 1	X1	X1 /Y = p1	1/p1== w1
FEMA Region 2	X2	X2 /Y = p2	1/p2== w2
…	…
Sampling Universe	Xall	1	Wall

Additionally, post stratification techniques such as raking and imputation may be used to account for known bias inherent in surveys of this type.

• Raking is an iterative process used to ensure that weights are consistent with population estimates with regards to known socio-demographic factors such as age, education level, and race/ethnicity. Population distribution estimates will be based on the Census 2020. The program anticipates using the anesrake procedure² in R 3.6.3 since the algorithm performs capping, adjusting reduce outlier weights that would be larger than anticipated. The algorithm default of 5 was used to as the weight cap. Additionally the anesrake package includes a function to quality check that weighting adjustments were appropriately made to match target population proportions.

• Multiple Imputation techniques may be employed to control for item non-response bias by accounting for missing data.^3,4 Also, it can help reduce complete case analysis bias if regression methods are used to analyze data.⁵ In conducting imputation, the program will ensure that survey skip patterns are upheld and imputed responses fall within the natural bounds of survey responses.

Degree of accuracy needed for the purpose described in the justification

Non-Generalizable Quantitative and Qualitative Data

These procedures are not applicable for non-generalizable sampling.

Generalizable Quantitative Survey Data

For generalizable quantitative surveys the anticipated degree of accuracy for a 5,000 person (500 survey/FEMA region) is a standard margin of error (MoE) of ±1.4 percentage points based on the following equation:

MoE = 1.96 *

Where n=5,000 (number of completed surveys) and P=0.5 (sampling proportion). Estimates will be calculated using a standard statistical software package (e.g., R, SAS, SPSS, Stata). Using the above calculation the MoE for a 500 persons survey would be ±4.4 percentage points.

Unusual problems requiring specialized sampling procedures

For both non-generalizable and generalizable collections the program does not anticipate any problems requiring specialized sampling procedures. The program will keep recruitment open until sampling targets are achieved. The Agency may use skilled contractor support with experience in data collection, statistical analysis, and sampling to efficiently and effectively achieve sampling targets.

Any use of periodic (less frequent than annual) data collection cycles to reduce burden

For both non-generalizable and generalizable collections sampling will be cross-sectional in nature meaning that the program will not conduct any follow-up/repeated measure or cohort-based sampling. Given the large sampling universe the probability of selecting the same individual for multiple collections is extremely low.

3. Methods to Maximize Response

Non-Generalizable Quantitative and Qualitative Data

These procedures are not applicable for non-generalizable sampling.

Generalizable Quantitative Survey Data

For generalizable quantitative web-based surveys the program will use standard techniques to maximize response rates. These include:

• Potential respondents will receive a pre-notification letter detailing the purpose of the survey, followed by a digital survey notification containing an embedded hyperlink or other simple way to locate where the survey can be completed.

• The survey questionnaire will provide clear instructions for completing the survey and reiterate the purpose of the survey and how results will be used.

• The survey will be open a sufficient amount of time to achieve sampling targets.

• A limited number (n=up to 6) of reminder digital notification(s) will be sent to non-respondents, providing the survey hyperlink and encouraging response.

• The survey will allow respondents to resume completing the survey where they left off.

• A web-based graphic user interface (GUI) will be utilized to facilitate navigation and completion of the survey across multiple devices.

• Web-based participants may receive points for participation from the online panels they are recruited from (if the survey firm provides this for panel participants).

Please refer to the Estimation Procedures section above for a detailed accounting of anticipated weighting, raking, and imputation procedures the program will use to increase the accuracy and reliability of survey data.

4. Testing of Procedures

Non-Generalizable Quantitative and Qualitative Data

Limited cognitive interviews or other forms of pretesting may be used for non-generalizable testing. When applicable, pretesting will be conducted on no more than nine individuals to ensure accuracy in procedures. This pretesting may be done with internal staff, a limited number of external colleagues, and/or customers who are familiar with the programs and products. If the number of pretest respondents exceeds nine members of the public, the Agency will submit the pretest instruments for review under this generic clearance. Data from the pretest will be analyzed to make necessary revisions to improve usability of the instrument.

Generalizable Quantitative Survey Data

For generalizable quantitative web-based surveys, cognitive interviews or other forms of pretesting will be conducted on no more than nine individuals to ensure accuracy in procedures and instrument skip pattern logic are upheld. This pretesting may be done with internal staff, a limited number of external colleagues, and/or customers who are familiar with the programs and products. If the number of pretest respondents exceeds nine members of the public, the Agency will submit the pretest instruments for review under this generic clearance. Data from the pretest will be analyzed to make necessary revisions to improve usability of the instrument.

4. Contacts for Statistical Aspects and Data Collection

Team PCI, which includes members of Yes& and Deloitte Consulting LLP, has been contracted by the FEMA NFIP to conduct all generalizable and non-generalizable data collection and analyze procedures. The individuals consulted on the statistical aspects of design are:

Ruxi Giura, MA

Digital Insurance Communications Specialist

Marketing & Outreach | Federal Insurance | Federal Insurance and Mitigation Administration

Office: 202-316-5766

[email protected]

Rhodri Dierst-Davies, PhD, MPH

Evaluation Subject Matter Expert & Principal Investigator

Deloitte Consulting LLP

555 Mission Street

San Francisco, CA 94105

(415) 336-8935

[email protected]

Andrew Teie, BS

Vice President of Brand Strategy and Customer Experience

Yes& Agency

1700 Diagonal Road, #450

Alexandria, VA 22314

[email protected]

For other FEMA programs using this generic clearance, the program will obtain information from statisticians in the development, design, conduct, and analysis of customer/partner service surveys, when appropriate. This statistical expertise will be available from agency statisticians or from contractors and the Agency will include the names and contact information of persons consulted in the specific information collection requests submitted under this generic clearance.

1 Mansournia M A, Altman D G. Inverse probability weighting BMJ 2016; 352 :i189 doi:10.1136/bmj.i189

2 Anesrake Package Documentation. https://cran.r-project.org/web/packages/anesrake/anesrake.pdf

3 Kontopantelis E, White IR, Sperrin M, Buchan I. Outcome-sensitive multiple imputation: a simulation study. BMC Med Res Methodol. 2017;17(1):2. Published 2017 Jan 9. doi:10.1186/s12874-016-0281-5

4 Yuan, Yang C. (2010). "Multiple imputation for missing data: Concepts and new development" (PDF). SAS Institute Inc., Rockville, MD. 49: 1–11.

5 Mukaka M, White SA, Terlouw DJ, Mwapasa V, Kalilani-Phiri L, Faragher EB. Is using multiple imputation better than complete case analysis for estimating a prevalence (risk) difference in randomized controlled trials when binary outcome observations are missing? Trials. 2016;17:341. doi:10.1186/s13063-016-1473-3

7