Supporting Statement Part A_1660-0039

September 11, 2014

Supporting Statement for

Paperwork Reduction Act Submissions

OMB Control Number: 1660-0039

Title: National Fire Academy Long-Term Evaluation Form for Supervisors and National Fire Academy Long-Term Evaluation Form for Students/Trainees

Form Number(s): FEMA Form 078-0-2 and 078-0-2A.

General Instructions

A Supporting Statement, including the text of the notice to the public required by 5 CFR 1320.5(a)(i)(iv) and its actual or estimated date of publication in the Federal Register, must accompany each request for approval of a collection of information. The Supporting Statement must be prepared in the format described below, and must contain the information specified in Section A below. If an item is not applicable, provide a brief explanation. When Item 17 or the OMB Form 83-I is checked “Yes,” Section B of the Supporting Statement must be completed. OMB reserves the right to require the submission of additional information with respect to any request for approval.

Specific Instructions

A. Justification

 1. Explain the circumstances that make the collection of information necessary (give details as to why this information is being collected). Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information. Provide a detailed description of the nature and source of the information to be collected.

The National Fire Academy (NFA) is mandated under the Fire Prevention and Control Act of 1974 (Public Law 93-498) to provide training and education to the Nation’s fire service and emergency service personnel. The state-of-the-art programs offered by the NFA serve as models of excellence and state and local fire service agencies rely heavily on the curriculum to train their personnel. To maintain the quality of these training programs, it is critical that courses be evaluated after students have had the opportunity to apply the knowledge and skills gained from their training.

 2. Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection. If applicable, provide a detailed description of how the information will be shared and for what programmatic purpose.

Information collected with these evaluation forms enables U.S. Fire Administration (USFA) /NFA staff to monitor and recommend changes in course materials, individual subject selection criteria, and to make curriculum-wide reviews and assessments. The online electronic forms are processed through a Microsoft Structured Query Language (MS SQL) database and reports are made available electronically to USFA/NFA management and staff. Reports of longer term outcomes are annually posted online and disseminated in presentations by the NFA.

Below is a detailed explanation of how each form will be utilized:

1. FEMA Form 078-0-2, Long Term Evaluation for Supervisors – This form asks supervisors of students who have taken NFA training to judge if the training has improved the student’s job performance, contributed to their professional development, improved the department’s performance, led to a reduction in fire related risks within the community, and is the training worth recommending to others. Responses assist in maintaining quality of NFA training programs and respondent’s ability to apply the knowledge and skills gained in their department or community.

2. FEMA Form 078-0-2A, Long Term Evaluation for Students – This form asks students who have taken NFA training to judge if the training has improved the respondents job performance, contributed to their professional development, improved their department’s performance, led to a reduction in fire related risks within the community, and is the training worth recommending to others. Student’s report on specifics of training materials and experience of training diffusion. Responses assist in maintaining quality of NFA training programs and respondent’s ability to apply the knowledge and skills gained in their department or community.

Descriptive statistics obtained from each student and their supervisor, concerning training uses and effectiveness, are compiled for review by training and instructional systems specialists and USFA/NFA managers.

In particular, training specialists, instructional systems specialists and managers receive course specific reports which provide two types of study data. The two types of information include descriptive statistics, described below in section A, and narrative responses, described below in section B.

Section A. The first type of reported information is comprised of descriptive statistics, which deal with program effectiveness. The program effectiveness results provide NFA’s training specialists and management with information useful for consideration of curriculum and content revision.

The reports include percentage distributions of the following descriptive statistics:

(1) Supervisors who judge that the student’s training has:

• Improved the student’s job performance;

• Contributed to the student’s professional development;

• Improved the performance of the department;

• Led to a reduction in fire related risks within the community, and

• Has been worth recommending to others.

(2) Students who report specific applications of training materials, i.e., the proportion of students who indicate having applied specific training objectives.

These results permit an immediate assessment of NFA training in terms of both individual training classes and curriculum-wide applications. For example, NFA is able to assess what percent of training objectives are being applied by what percent of its students. In other words, NFA can report that “X” percent of students are actually applying “Y” percent of the training objectives presented in the training classes. Additionally, the results present a snapshot picture of which classes within the curriculum are having the widest application within the student’s local training organization.

(3) Students who judge that the student’s training has:

• Helped the department address all-hazard problems in community high-risk areas;

• Helped with policy development;

• Provided opportunities to collaborate and/or network with other NFA students.

(4) Students reporting their experience of training diffusion, i.e., the percent of students who have shared training knowledge and information:

• Formally in a training class;

• Informally in a group; or

• Informally, one on one.

Section B. The second type of information provided by the Long Term Evaluation (LTE) reports is based on narrative data and is prepared for training specialists and NFA managers. Narrative data vary but generally include comments provided by the students and supervisors about:

• Needed improvements in the NFA training:

• Descriptions of incidents in which students have applied NFA training: and

• Topics for future training classes.

In addition, special reports are run from LTE data for other internal and external audiences as requested.

There are several ways in which the narrative data are used. For example:

(1) Training specialists can determine the effectiveness of NFA training at advancing the skills of trainees and the resulting benefit to his/her organization,

(2) Instructional staff can determine appropriate methodologies and delivery modes for future NFA training, and

(3) U.S. Fire Administration/NFA management can evaluate the training on achievement of strategic goals that support the Agency’s performance measurement process.

The state-of-the-art programs offered by the NFA serve as models of excellence. State and local fire service agencies rely heavily on the curriculum to train their personnel. To maintain the high standards of these programs, it is critical that courses be evaluated after students have had the opportunity to apply the knowledge and skills gained.

This request includes information collection activity targeting NFA 2-day, 6-day, and 10-day training courses. This request seeks approval for two electronically based long-term evaluation (LTE) forms which will collect information from both course graduates and their respective supervisors.

The information to be collected addresses the student’s applications of specified training objectives upon returning to his or her local department or professional organization.

The data collection will be managed by NFA staff with expertise and experience in data collection, analysis and reporting.

 3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden.

The survey activity includes both fully electronic and automated methods. Students volunteer to complete the LTE process while completing their end of course evaluations or by clicking on the LTE icon at a networked campus workstation. Automated email letters, which contain the secure LTE URL, http://www.usfa.dhs.gov/applications/nfaeval/lte/login.do, are computer generated and sent to the signed-up students and their supervisors inviting them to complete the online form. For email messages which are returned undeliverable due to incorrect email addresses, physical letters containing the secure LTE URL, http://www.usfa.dhs.gov/applications/nfaeval/lte/login.do, and a personal password are generated and mailed by US Postal Service. The objectives of the course set by the NFA Training Specialists determine when the survey activity occurs. This will range from 120 to 240 days following course participation. Participants have a 60-day turnaround time for completion.

The LTE can only be completed online and the data is immediately stored in a MS SQL database. Standard reports are generated directly from the MS SQL database and targeted searches can be specified and data downloaded for further analysis using MS Excel macros and SPSS syntax files.

The electronic format of the LTE reduces the paper burden of students and supervisors. All students are requested to sign-up both themselves and their supervisor for the LTE when completing the End of Course (EOC) evaluation during their time at the NFA. Those students and supervisors receive the LTE link as scheduled.

 4. Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Item 2 above.



This information is not collected in any form, and therefore is not duplicated elsewhere.

5. If the collection of information impacts small businesses or other small entities (Item 5 of OMB Form 83-I), describe any methods used to minimize.

This information collection does not have an impact on small businesses or other small entities.

 6. Describe the consequence to Federal/FEMA program or policy activities if the collection of information is not conducted, or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

If the data collection is not conducted, the USFA/NFA will not be able to determine its effectiveness in carrying out its mission to provide effective training and education for the Nation’s fire and emergency response community.

 7. Explain any special circumstances that would cause an information collection to be conducted in a manner:

The special circumstances contained in item 7(a) thru (h) of the supporting statement are not applicable to this information collection.

 (a) Requiring respondents to report information to the agency more often than quarterly.

  (b) Requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it.



 (c) Requiring respondents to submit more than an original and two copies of any document.



 (d) Requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records for more than three years.



 (e) In connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study.



 (f) Requiring the use of a statistical data classification that has not been reviewed and approved by OMB.

 (g) That includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use.

 (h) Requiring respondents to submit proprietary trade secret, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information’s confidentiality to the extent permitted by law.



8. Federal Register Notice:



 a. Provide a copy and identify the date and page number of publication in the Federal Register of the agency’s notice soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments. Specifically address comments received on cost and hour burden.



A 60-day Federal Register Notice inviting public comments was published August 4, 2014, 79 FR 45208. There were no comments received for this collection of information.

A 30-day Federal Register Notice inviting public comments was published on October 16, 2014, 79 FR 62169. There were no comments received for this collection of information.

 b. Describe efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or reported.

Contracted personnel provide expert assistance in the operation of the data collection software/system and will be consulted in the availability and operation of such.

c. Describe consultations with representatives of those from whom information is to be obtained or those who must compile records. Consultation should occur at least once every three years, even if the collection of information activities is the same as in prior periods. There may be circumstances that may preclude consultation in a specific situation. These circumstances should be explained.

The NFA consults with its residential student population on a regular basis regarding course evaluations. The Superintendent holds a “Superintendent’s Lunch” with class representatives during every resident class delivery cycle. This luncheon provides the opportunity for the Superintendent and class representatives to discuss issues concerning training development, course delivery and course evaluation. The Superintendent also visits each class during every delivery cycle and provides an open forum for any issues the students and instructors want to discuss. Therefore, consultations occur at least once every three years.

 9. Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.

FEMA does not provide payments or gifts to respondents in exchange for a benefit sought.

 

10. Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy. Provide details on:

The electronic letter inviting student and supervisor participation in the LTE program indicates that responses to the questionnaire will remain anonymous.  In specific terms, the relevant portion of this letter reads:

“Please know that we appreciate you taking the time to complete this important evaluation.  Please know as well:  Your responses to the questions contain no names and will remain anonymous.  Additionally, individual data are never shared or reported out apart from aggregate government summaries.

This assurance to respondents does not hamper sharing data with program managers involved in this information collection.  The NFA’s LTE system is covered under the DHS Wide Contact List PIA and was added to the Appendix B of DHS/ALL/PIA-006 - DHS General Contacts on September 17, 2009. The DHS Department-wide PIA, DHS/ALL/PIA-006 - DHS General Contacts List, and the associated Department-wide SORN, DHS/ALL-002 - Department of Homeland Security (DHS) Mailing and Other Lists System (73 FR 71659, November 25, 2008,) are available at

http://www.dhs.gov/privacy-documents-department-wide-programs. Appendix B of DHS/ALL/PIA-006 - DHS General Contacts List provides a list of qualifying programs or projects which includes FEMA National Fire Academy Long-Term Evaluation and FEMA USFA WebForm.

 11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

There are no questions of a sensitive nature.

 12. Provide estimates of the hour burden of the collection of information. The statement should:



 a. Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. Unless directed to do so, agencies should not conduct special surveys to obtain information on which to base hour burden estimates. Consultation with a sample (fewer than 10) of potential respondents is desired. If the hour burden on respondents is expected to vary widely because of differences in activity, size, or complexity, show the range of estimated hour burden, and explain the reasons for the variance. Generally, estimates should not include burden hours for customary and usual business practices.

It is anticipated that 1,500 students will complete FEMA Form 078-0-2A. Each respondent will only complete the form once and each response will require .17 hours to complete the form. The total annual hour burden is 1,500 x .17 hours = 255 annual hours.

It is anticipated that 1,500 supervisors will complete FEMA Form 078-0-2. Each respondent will only complete the form once and each response will require .1 hours to complete the form. The total annual hour burden is 1,500 x .1 hours = 150 annual hours.

 b. If this request for approval covers more than one form, provide separate hour burden estimates for each form and aggregate the hour burdens in Item 13 of OMB Form 83-I.

c. Provide an estimate of annualized cost to respondents for the hour burdens for collections of information, identifying and using appropriate wage rate categories. NOTE: The wage-rate category for each respondent must be multiplied by 1.4 and this total should be entered in the cell for “Avg. Hourly Wage Rate”. The cost to the respondents of contracting out or paying outside parties for information collection activities should not be included here. Instead this cost should be included in Item 13.

Note: The “Avg. Hourly Wage Rate” for each respondent includes a 1.4 multiplier to reflect a fully-loaded wage rate.

According to the U.S. Department of Labor, Bureau of Labor Statistics website (www.bls.gov) the wage rate category for Fire Fighters (representing all firefighters) is estimated to be ($23.00 x 1.4=) $32.20 per hour including the wage rate multiplier, therefore, the estimated burden hour cost to respondents Fire Fighter is estimated to be 255 hours x $32.20 = $8,211 annually.

According to the U.S. Department of Labor, Bureau of Labor Statistics website (www.bls.gov) the wage rate category for Fire Fighters (representing all first-line supervisors/managers of fire fighting and prevention workers) is estimated to be ($34.23 x 1.4=) $47.92 per hour including the wage rate multiplier, therefore, the estimated burden hour cost to respondents Fire Fighter is estimated to be 150 hours x $47.92 = $7,188 annually.

13. Provide an estimate of the total annual cost burden to respondents or record keepers resulting from the collection of information. (Do not include the cost of any hour burden shown in Items 12 and 14.)

There are no record keeping, capital, start-up or maintenance costs associated with this information collection.

14. Provide estimates of annualized cost to the Federal Government. Also, provide a description of the method used to estimate cost, which should include quantification of hours, operational expenses (such as equipment, overhead, printing and support staff), and any other expense that would have been incurred without this collection of information. You may also aggregate cost estimates for Items 12, 13, and 14 in a single table.



*Note: The “Salary Rate” includes a 1.4 multiplier to reflect a fully-loaded wage rate.

15. Explain the reasons for any program changes or adjustments reported in Items 13 or 14 of the OMB Form 83-I in a narrative form. Present the itemized changes in hour burden and cost burden according to program changes or adjustments in Table 5. Denote a program increase as a positive number, and a program decrease as a negative number.

For FEMA Form 078-0-2A (students/trainees), the previously approved burden hours were 450, the current estimated annual hour burden is 255, resulting in a decrease (program change) of 195 hours. The decrease in burden hours is based on the actual sign-up rate and the removal of demographic questions from the form for this survey over the past 3 years. The previous estimate was based on an anticipated rate in moving to an electronic format.

For FEMA Form 078-0-2 (supervisors), the previously approved burden hours were 247.5, the current estimated annual hour burden is 150, resulting in a decrease (program change) of 97.5 hours. The decrease in burden hours is based on the actual sign-up rate and the removal of demographic questions from the form for this survey over the past 3 years. The previous estimate was based on an anticipated rate in moving to an electronic format.

For FEMA Form 078-0-2A (students/trainees), the previously approved cost burden on the OMB inventory was $13,644.00; the current estimate is $8,211resulting in a decrease (program change) of $5,433. The decrease is due to eliminating demographic questions on the form and using an estimated response rate based on past response rates using the same sign-up process.

For FEMA Form 078-0-2 (supervisors), the previously approved cost burden was $11,367.68; the current estimate is $7,188 resulting in a decrease (program change) of $4,179.68. The decrease is due to eliminating demographic questions on the form and using an estimated response rate based on past response rates using the same sign-up process.



16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.



The Long Term Evaluation Report is published annually for each Fiscal Year on the NFA Web site. The annual LTE report contains graphic representations of (1) percent agreement by students and supervisors with course satisfaction statements, (2) the relative frequency and utility-rating of training experiences at the NFA and other training locations, (3) estimates for the amount of training diffusion on the job and to work colleagues, and (4) relative frequency of various suggestions for improvements and new training courses. These results are system generated with minimal demand on the time (2 hours) of an NFA staff member.

 17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain reasons that display would be inappropriate.



FEMA will display the expiration date for OMB approval of this information collection.

 18. Explain each exception to the certification statement identified in Item 19 “Certification for Paperwork Reduction Act Submissions,” of OMB Form 83-I.



FEMA does not request an exception to the certification of this information collection.

13