Analysis Plan - record of change

N ational Preparedness Task Force

Directorate for Preparedness

Department of Homeland Security

Analysis Plan: Survey of State and Local Government Emergency Officials

The purpose of this survey is to determine the effectiveness and user-friendliness of Department of Homeland Security (DHS) interactions with State and local government emergency officials. Research questions address whether additional emphasis is needed to improve communication and coordination with emergency officials across either disciplines (e.g., fire and law enforcement) or jurisdictions. For all research questions, independent variables include the associated demographic information of the respondent, i.e., their homeland security function, experience, frequency of interaction with DHS and jurisdictional characteristics. Respondents’ indications of the user-friendliness and effectiveness of relevant Federal programs constitute the dependent variables for this study. The data will be analyzed using accepted statistical theory and methods at the 5% significance level. Relevant literature is detailed in the Survey of State and Local Government Emergency Officials (SSLGEO) Interim Report. These documents and studies informed the design of the survey instrument, sampling strategy and analysis plan for the SSLGEO.

Sampling Design

Congress has specified that the survey have an adequate representational response from State and local homeland security officials across functional areas and jurisdictional levels. To satisfy this requirement, the survey sample design was constructed to account for the relative proportions of the subgroups of interest (such as law enforcement, fire, medical, and emergency planners). The goal is to generate a sample that will allow valid and defensible inferences to be made between and across the relevant subgroups of the sample, so that the results of the survey can be generalized to the population of State and local homeland security officials at large. The sample design will ensure that the survey results can drive comprehensive and effective policy modeling when addressing any problems relating to the effectiveness and user-friendliness of DHS programs that are identified by the survey.

Lessons Learned Information Sharing (LLIS.gov) users are the population from which the sample will be drawn because they constitute the best available representation of the national community of emergency response providers and State and local homeland security personnel. With tens of thousands of registered users from across the country, LLIS.gov provides a potential respondent pool that is large enough to capture all relevant subgroups of interest. The user population is sufficiently diversified across functional areas and jurisdictional levels.

A stratified random sampling strategy will ensure the sample meets compositional requirements. The proportion of the sample that falls into a category (e.g. fire, law enforcement, or county, municipal) will mirror the percentage of the actual population within LLIS that falls into the same category.

The sample size will be large enough to ensure that comparisons can be made across categories and that the entire sample is representative of the community at large. DHS anticipates disseminating approximately 1,500 surveys. Based on the population’s size and composition, DHS believes that this number of surveys will provide an adequate number of responses that can be analyzed and used to draw statistically significant conclusions.

Several strategies have been identified to assure adequate response rates. The survey distribution will be reinforced with phone calls, and preexisting relationships with LLIS users will be leveraged to encourage participation. These follow-up strategies can also be tailored to compensate for uneven response rates across subgroups, and to ensure that the analysis provides meaningful results.

Survey Dissemination and Collection

As a preliminary step to validate the contact information of the respondent pool and to potentially increase response rates in the survey, a letter of introduction will be sent out via email to the respondent population one week prior to survey distribution. The letter introduces the survey, explains its purpose, provides respondents with an estimated time commitment, assures confidentiality, and outlines the way the results of the survey will be used. The letter of introduction is shown below.

National Preparedness Task Force

Preparedness Directorate

Washington, DC 20531

Dear [Insert Applicable Name]:

In the next week, the Department of Homeland Security (DHS) will begin surveying homeland security officials and first responders to help assess the effectiveness and user-friendliness of DHS programs. The results of the survey will guide ongoing efforts to improve DHS programs. Feedback from stakeholders across the country will be critical to this initiative.

You have been identified as an active member of the homeland security community and DHS needs your input to understand how we can more effectively meet the needs of those with whom we work most closely. Within the next week, you will receive an email invitation to participate in this survey. The survey should take no more than twenty minutes of your time to complete. If you have any questions at any time during this process please contact Brad Wolvin at (202) 416-0193 or via email at [email protected].

Your responses will be aggregated and reported to Congress in December 2006. No names will be used in the report. The results of the survey will be drawn upon to gauge the usability and effectiveness of DHS programs and to identify areas for potential improvements. We appreciate your support and willingness to participate in this important effort.

Sincerely,

George Foresman

Under Secretary for Preparedness

Department of Homeland Security

The actual distribution of the survey will occur one week later as respondents are invited to visit a dedicated website to fill out the survey. The online collection mechanism will facilitate the collection and standardization of the data and provide immediate feedback on the response rates in each of the subgroups of the sample population. Response rates will be monitored throughout the data collection phase of this process. Surveys collected by phone and other follow-up methods will be used to boost response rates when deemed necessary to maintain the proportionality that is required for the sample to be representative of the entire population and each of its subgroups. All data collection activities will maintain the integrity of the dataset and ensure that validity of the study.

Inference and Comparisons

General measurements of effectiveness and user-friendliness will be relatively easy to produce once all the survey responses have been collected; however, additional analysis is needed to be able to focus policy choices and to maximize the value of the survey. Data will be analyzed on several additional levels: first, sentiments within in the national community of responders and State and local homeland security officials; second, statistically significant differences in the responses of the different categories included in this study (fire, law enforcement, medical, and emergency planners); and third, differences that emerge with the various jurisdictional subgroups that have been identified (municipal, State, county, regional). The goal of the analysis is to provide a clear picture of the effectiveness and user-friendliness of DHS programs and to identify specific opportunities for improvement.

Data analysis will utilize accepted hypothesis testing methodology at the 5% level of significance ( p=.05 ). Statistical tests (e.g., Chi-squared tests and t-tests) will be used to determine if typical responses differ across population subgroups. Estimates of the standard error and covariance (in the event that variables are not independent) will inform the choice of methods for comparisons. Analyses will also account for the reliability of the survey instrument.

Findings at the National Level

In recognition of the broad mission and many different types of DHS programs, the survey has been designed to identify specific opportunities for improvement in each relevant program area (grant management, intelligence sharing, etc.). In the event that the variance of the responses is significant, the data can be further analyzed to search for correlations that may explain variations according to region of the country, population density, or other demographic variables. General findings regarding the effectiveness and usability will be identified to help focus recommendations for improvement, thus reinforcing the link between survey results and policy development.

Analysis of Functional User Subgroups

Analysis of the functional user subgroups will seek to identify any significant differences in perception of DHS programs across homeland security disciplines (fire, law enforcement, medical, emergency planning). The analysis strategy will mirror that of the national analysis, but may also identify significant differences that exist across homeland security disciplines. Similar inquiries will be performed in the event that there is significant variance across or within groups and to determine if the responses that were most frequently offered are truly representative of the sentiments of the whole. Other dependent variables include jurisdiction, time in current position, and DHS workload.

Analysis of Jurisdictional-Level Subgroups

Similar analysis will be performed to determine if there is a correlation between the user’s perception of effectiveness, usability, and the jurisdictional level on which they interact with DHS programs. This analysis will help to determine the level at which policy reforms and initiatives must be executed in order to assure that improvement strategies are applied effectively.

All levels of analysis will use accepted statistical analysis techniques such as analysis of variances, calculations of confidence intervals, and chi-squared tests for statistical significance. Data will be presented in a simple and straightforward manner to ensure the transparency of the analysis.

Survey Analysis Output

The results of this survey will provide actionable insight into how State and local emergency officials interact with DHS and pave the way for programmatic improvements. The statistical analysis plan is consistent with OMB best practices and will produce for DHS statistically defensible conclusions.

4

A

nalysis Plan: Survey of State and Local Govenrment Emergency Officials |