Supporting Statement Part B_1660-0039

September 11, 2014

Supporting Statement for

Paperwork Reduction Act Submissions

OMB Control Number: 1660-0039

Title: National Fire Academy Long-term Evaluation Form for Supervisors and National Fire Academy Long-term Evaluation Form for Students/Trainees

Form Number(s): FEMA Form 078-0-2 and 078-0-2A

B. Collections of Information Employing Statistical Methods.



When Item 17 on the Form OMB 83-I is checked “Yes”, the following documentation should be included in the Supporting Statement to the extent it applies to the methods proposed:

1. (a) Describe (including numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. (b) Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. (c) Indicate expected response rates for the collection as a whole. If the collection has been conducted previously, include the actual response rate achieved during the last collection.

Based on the participant sign up rate from the 3 previous years, the potential respondent universe is anticipated to total 3,000 annually (approximately 1,500 students and 1,500 supervisors). The potential respondent universe is made up of those NFA students who sign up both their supervisor and themselves to complete the LTE.

2. Describe the procedures for the collection of information including:

Summary descriptive statistics are routinely calculated and reported for review by training and instructional systems specialists and USFA/NFA managers.

Training specialists, instructional systems specialists, and managers will receive course specific reports. The reports present descriptive statistics, which deal with program effectiveness. The program effectiveness results provide NFA Training Specialists and Management with information useful for instruction and curriculum revision.

These statistics include percentage distributions of the following:

(1) Students and supervisors who judge that the student’s training has:

• Improved the student’s job performance;

• Contributed to the student’s professional development;

• Improved the performance of the department;

• Led to a reduction in all hazards risks within the community, and generally

• Has been worth recommending to others.

(3) Students who report specific applications of training materials, i.e., the percent of students who indicate that they have applied specific training objectives in terms of the following frequencies:

• Used a great deal

• Used somewhat

• Not used at all

• No Opportunity to Use

These data will permit an immediate assessment of NFA training in terms of both individual training classes and curriculum-wide applications. For example, NFA will be able to know what percent of training objectives are being applied by what percent of its students. That is, for any given curriculum, NFA will be able to report that “X” percent of students are actually applying “Y” percent of the training objectives provided in/by the training classes. Additionally, NFA will have a snap- shot picture of which classes within the curriculum are having the widest applications within the student’s local training organization.

(4) Students and supervisors who judge that the student’s training has:

• Made the student more safety conscious;

• Provided prevention ideas that can be incorporated into the department’s public education programs;

• Helped with policy development; and

• Met department expectations.

(5) Students reporting their experience of training diffusion, i.e., the percent of students who have shared training knowledge and information:

• Formally in a training class;

• Informally in a group; or

• Informally, one on one.

Comments are also requested from students and supervisors in the areas of:

• Needed improvements in NFA training;

• Descriptions of incidents in which students have applied NFA training; and

• Topics for future training classes.

In addition to the descriptive statistics and narrative types of data provided in regular LTE reports, special reports will be run from LTE data for other internal and external audiences as required.

There are several ways in which the data from this data collection are used. For example,

(1) Training specialist staff determine the effectiveness of NFA training in advancing the skills of trainees and the resulting benefit to his/her organization;

(2) Instructional staff determine appropriate methods and delivery modes for NFA training; and

(3) U.S. Fire Administration/NFA management staff examines the success of NFA training in achieving its strategic goals to support the Agency’s performance measurement process – serving as a budget performance measure.

• Statistical methodology for stratification and sample selection,

All students attending courses on the campus of the NFA are provided with the opportunity to sign up for the LTE procedure. All students who sign up and their supervisors are provided with access to the LTE online forms. It is a full census. Every individual student has an opportunity to participate. The return of the electronic forms is fully anonymous and automated. However, if the provided email address is not functional, paper letters are generated and sent by the United States Postal Service with the LTE Web site address and an anonymous password.

• Estimation procedure,

The anonymous character of the LTE procedure does not permit respondents and non-respondents to be compared using the student’s personal demographic information gathered by the NFA through other means. The selective non-response to individual items is not investigated and no attempt is made to use statistical regression procedures to estimate missing values. Descriptive results are reported with the number (n) of the respondents for that question.

• Degree of accuracy needed for the purpose described in the justification,

Degree of accuracy (margin of error) is irrelevant to this information collection because there is no sampling process involved. Additionally, estimates of true scores are not needed in this case since (1) data will not be used for inferential purposes and (2) the homogeneity of the target population and interest in the subject ensure satisfactory levels of useful and consistent information.

• Unusual problems requiring specialized sampling procedures, and

There are no unusual problems requiring specialized sampling procedures.

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

There is no use of periodic data collection cycles to reduce burden.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

In order to maximize response rates, the National Fire Academy has placed informational posters within each of their resident classrooms to encourage student participation. In addition, clear instructions are provided to both students and supervisors via e-mail. The NFA has also established an on-line evaluation comments web-page and direct mailbox to assist respondents with questions and to provide feedback.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

Lessons learned from previous usability testing for the NFA’s Level I end-of-course form have been incorporated into the LTE instruments. Further testing of minor changes to forms is not deemed necessary.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Ms. Dawn Long, (301) 447-1488/U.S. Fire Administration, National Fire Academy.

4