Supporting_Statement_519-0-1__11.8.11_PART_B 1660-0044 11-8-11

3/1/11

Supporting Statement for

Paperwork Reduction Act Submissions

OMB Control Number: 1660 - 0044

Title: Emergency Management Institute Follow-up Evaluation Survey

Form Number(s): FEMA Form 519-0-1 (Presently FEMA Form 95-56)



B. Collections of Information Employing Statistical Methods.



When Item 17 on the Form OMB 83-I is checked “Yes”, the following documentation should be included in the Supporting Statement to the extent it applies to the methods proposed:

1. Describe (including numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection has been conducted previously, include the actual response rate achieved during the last collection.

 

The potential number of respondents is approximately 14,000 a year.  All EMI resident course participants receive the FEMA Form 519-0-1 form, along with a letter from the EMI Superintendent 90 days after the completion of an EMI resident course.  Based on the response rate from FY2010, it is expected that approximately 3,800 questionnaires will be returned. 

Respondent’s Occupational Category	Total number of Entities
Individuals and Households	12,600
State, Local and Tribal Government	1,400
	14,000

Approximately ten percent of the questionnaires are sent to state, local and tribal government officials.  The remaining ninety percent are sent to individuals and households. 

In FY2010 24,839 questionnaires were distributed and 6,674 were received, resulting in an approximate response rate of 27%. 

 

2. Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection:

No stratification or sample selection is used because this study is a full census of all who have completed the course, and this method is what we have used in the past. The questionnaires are distributed back to the respective course instructors once received. The questionnaire seeks their qualitative feedback in regard to the course. This is a course evaluation, and the method is simply all students who have taken an EMI resident course receive the letter and questionnaire 90 days after the course is completed. It is not given immediately after course completion because we are asking if they have been able to apply what they learned in training to their jobs. We have to wait to allow them to implement it. We do, however, emphasize to them at course end that this questionnaire will be coming in 90 days and strongly encourage them to return it. We provide a postage paid envelope and an across the board follow-up reminder to return it three weeks after the initial mailing. Students’ most recent addresses are collected at the time of course end to ensue that they will successfully receive the questionnaire in three months. The addresses are provided to us by the NETC Admissions Office, where all their course information and demographical data is stored.  If a student moves in the 90-day time period, they would need to contact the NETC Admissions office to change their record.  Therefore, we are assured that we have fairly accurate contact data for our respondents.

• Estimation procedure:

The questionnaire seeks their qualitative feedback for the purpose of gaining an in-depth understanding. Therefore, no estimation procedures are used. Estimation procedures are using a sample statistic such as a mean to determine a probable value of a population parameter. Since this study is qualitative in nature, no such estimation procedures will be needed.

• Degree of accuracy needed for the purpose described in the justification:

Because the feedback we receive from this questionnaire is primarily qualitative in nature, there is no established degree of accuracy that is required. The qualitative results are reviewed on an individual course level by different course managers. The questionnaire simply seeks in depth understanding regarding the level of preparedness the student feels after taking the course. The data provided back is qualitative, so any input regarding the effectiveness of the course would be considered fulfilling the purpose of this initiative.

• Unusual problems requiring specialized sampling procedures:

There are no unusual problems requiring specialized sampling procedures.

• Use of periodic (less frequent than annual) data collection cycles to reduce burden:

No periodic data collection cycles are implemented to reduce burden.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

Our current response rate tracks with the industry standard of 20-23% response rates on surveys or questionnaires. As EMI reviews the entire Level 3 Evaluation process, we will be considering how best to provide this form and collect the information in an electronic format. An electronic format could increase our response rates by providing a more convenient and easier way for individuals to respond. To increase response rates we intend to send reminders and another copy of the questionnaire across the board three weeks after the questionnaire has been given. In the event of response rates are below 80%, a non-response analysis will be performed on the group(s) that may be in question. These analyses, if need be conducted, will be conducted by using the “SPSS Analysis of Missing Data” module of the general SPSS software package and the findings of the analysis will be addressed accordingly.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

No pilot testing procedures have been undertaken. The questionnaire has been revised based on previous lessons learned and feedback. This is primarily because it is the same questionnaire used in the past with only one primary question which was very basic and we have had very little variation in the changes that have been made in the questionnaire. We have not experienced any major questions about clarification needed in answering the questions in the questionnaire over the duration that we have used it.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Name	Title	Organization	Phone number
Dana Moat	Training Specialist	DHS/FEMA	301-447-1922
Matthew Prager	DL Section Chief	DHS/FEMA	301-447-1148
Nicole Bouchet	Statistician	FEMA	202-646-2814

4