Revised11-2-2011 -- 2011 Supporting Statement OMB1660-0039 FINAL A

November 2, 2011

Supporting Statement for

Paperwork Reduction Act Submissions

OMB Control Number: 1660-0039

Title: National Fire Academy Long-term Evaluation Form for Supervisors and National Fire Academy Long-term Evaluation Form for Students/Trainees

Form Number(s): FEMA Form 078-0-2 (Presently FEMA Form 95-58) and 078-0-2A (Presently FEMA Form 95-59)

General Instructions

A Supporting Statement, including the text of the notice to the public required by 5 CFR 1320.5(a)(i)(iv) and its actual or estimated date of publication in the Federal Register, must accompany each request for approval of a collection of information. The Supporting Statement must be prepared in the format described below, and must contain the information specified in Section A below. If an item is not applicable, provide a brief explanation. When Item 17 or the OMB Form 83-I is checked “Yes,” Section B of the Supporting Statement must be completed. OMB reserves the right to require the submission of additional information with respect to any request for approval.

Specific Instructions

A. Justification

 1. Explain the circumstances that make the collection of information necessary (give details as to why this information is being collected). Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information. Provide a detailed description of the nature and source of the information to be collected.

The National Fire Academy (NFA) is mandated under the Fire Prevention and Control Act of 1974 (Public Law 93-498) to provide training and education to the Nation’s fire service and emergency service personnel. The state-of-the-art programs offered by the NFA serve as models of excellence and state and local fire service agencies rely heavily on the curriculum to train their personnel. To maintain the quality of these training programs, it is critical that courses be evaluated after students have had the opportunity to apply the knowledge and skills gained from their training.

 2. Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection. If applicable, provide a detailed description of how the information will be shared and for what programmatic purpose.

Information collected with these evaluation forms enables U.S. Fire Administration (USFA) /NFA staff to monitor and recommend changes in course materials, individual subject selection criteria, and to make curriculum-wide reviews and assessments. The online electronic forms are processed through a Microsoft Structured Query Language (MS SQL) database and reports are made available electronically to USFA/NFA management and staff. Reports of longer term outcomes are annually posted online and disseminated in presentations by the NFA.

Below is a detailed explanation of how each form will be utilized:

1. FEMA Form 078-0-2, Long Term Evaluation for Supervisors – This form asks supervisors of students who have taken NFA training to judge if the training has improved the student’s job performance, contributed to their professional development, improved the department’s performance, led to a reduction in fire related risks within the community, and is the training worth recommending to others. Responses assist in maintaining quality of NFA training programs and respondent’s ability to apply the knowledge and skills gained in their department or community.

2. FEMA Form 078-0-2A, Long Term Evaluation for Students – This form asks students who have taken NFA training to judge if the training has improved the respondents job performance, contributed to their professional development, improved their department’s performance, led to a reduction in fire related risks within the community, and is the training worth recommending to others. Student’s report on specifics of training materials and experience of training diffusion. Responses assist in maintaining quality of NFA training programs and respondent’s ability to apply the knowledge and skills gained in their department or community.

Descriptive statistics obtained from each student and their supervisor, concerning respondent demographics and training uses and effectiveness, are compiled for review by training and instructional systems specialists and USFA/NFA managers.

In particular, training specialists, instructional systems specialists and managers receive course specific reports which provide two types of study data. The two types of information include descriptive statistics, described below in section A, and narrative responses, described below in section B.

Section A. The first type of reported information is comprised of descriptive statistics, which deal with both student demographics and program effectiveness. The demographic statistics are used to confirm empirically that the NFA is reaching the audiences it has identified for training deliveries. The program effectiveness results provide NFA’s training specialists and management with information useful for consideration of curriculum and content revision.

The reports include percentage distributions of the following descriptive statistics:

(1) Student background characteristics (i.e., gender, age, educational levels, NFA field course experience, years in fire and/or emergency services, responsibilities, department size, and percent of career personnel in the student’s department).

(2) Students and supervisors who judge that the student’s training has:

• Improved the student’s job performance;

• Contributed to the student’s professional development;

• Improved the performance of the department;

• Led to a reduction in fire related risks within the community, and

• Has been worth recommending to others.

(3) Students who report specific applications of training materials, i.e., the proportion of students who indicate having applied specific training objectives.

These results permit an immediate assessment of NFA training in terms of both individual training classes and curriculum-wide applications. For example, NFA is able to assess what percent of training objectives are being applied by what percent of its students. In other words, NFA can report that “X” percent of students are actually applying “Y” percent of the training objectives presented in the training classes. Additionally, the results present a snapshot picture of which classes within the curriculum are having the widest application within the student’s local training organization.

(4) Students and supervisors who judge that the student’s training has:

• Made the student more safety conscious;

• Provided prevention ideas that can be incorporated into the department’s public education programs;

• Helped with policy development; and

• Met department expectations.

(5) Students reporting their experience of training diffusion, i.e., the percent of students who have shared training knowledge and information:

• Formally in a training class;

• Informally in a group; or

• Informally, one on one.

Section B. The second type of information provided by the Long Term Evaluation (LTE) reports is based on narrative data and is prepared for training specialists and NFA managers. Narrative data vary but generally include comments provided by the students and supervisors about:

• Needed improvements in the NFA training:

• Descriptions of incidents in which students have applied NFA training: and

• Topics for future training classes.

In addition, special reports are run from LTE data for other internal and external audiences as requested.

There are several ways in which the narrative data are used. For example:

(1) Training specialists can determine the effectiveness of NFA training at advancing the skills of trainees and the resulting benefit to his/her organization,

(2) Instructional staff can determine appropriate methodologies and delivery modes for future NFA training, and

(3) U.S. Fire Administration/NFA management can evaluate the training on achievement of strategic goals that support the Agency’s performance measurement process.

The state-of-the-art programs offered by the NFA serve as models of excellence. State and local fire service agencies rely heavily on the curriculum to train their personnel. To maintain the high standards of these programs, it is critical that courses be evaluated after students have had the opportunity to apply the knowledge and skills gained.

This request includes information collection activity targeting NFA 2-day, 6-day, and 10-day training courses. This request seeks approval for two electronically based long-term evaluation (LTE) forms which will collect information from both course graduates and their respective supervisors.

The information to be collected addresses the student’s applications of specified training objectives upon returning to his or her local department or professional organization.

The data collection will be managed under the overall guidance of the NFA project officer, but it will be implemented by contracted staff with expertise and experience in data collection, analysis and reporting.

 3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden.

The survey activity includes both fully electronic and automated methods. Students volunteer to complete the LTE process while completing their end of course evaluations or by clicking on the LTE icon at a networked campus workstation. Automated email letters, which contain the secure LTE URL, http://www.usfa.dhs.gov/applications/nfaeval/lte/login.do, are computer generated and sent to the signed-up students and their supervisors inviting them to complete the online form. For email messages which are returned undeliverable due to incorrect email addresses, physical letters containing the secure LTE URL, http://www.usfa.dhs.gov/applications/nfaeval/lte/login.do, and a personal password are generated and mailed by US Postal Service. The objectives of the course set by the NFA Training Specialists determine when the survey activity occurs. This will range from 120 to 240 days following course participation. Participants have a 60-day turnaround time for completion.

The LTE can only be completed online and the data is immediately stored in a MS SQL database. Standard reports are generated directly from the MS SQL database and targeted searches can be specified and data downloaded for further analysis using MS Excel macros and SPSS syntax files.

The electronic format of the LTE reduces the paper burden of students and supervisors. All students are requested to sign-up both themselves and their supervisor for the LTE when completing the End of Course (EOC) evaluation during their time at the NFA. Those students and supervisors receive the LTE link as scheduled.

 4. Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Item 2 above.



This information is not collected in any form, and therefore is not duplicated elsewhere.

5. If the collection of information impacts small businesses or other small entities (Item 5 of OMB Form 83-I), describe any methods used to minimize.

This information collection does not have an impact on small businesses or other small entities.

 6. Describe the consequence to Federal/FEMA program or policy activities if the collection of information is not conducted, or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

If the data collection is not conducted, the USFA/NFA will not be able to determine its effectiveness in carrying out its mission to provide effective training and education for the Nation’s fire and emergency response community.

 7. Explain any special circumstances that would cause an information collection to be conducted in a manner:

 (a) Requiring respondents to report information to the agency more often than quarterly.

  (b) Requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it.



 (c) Requiring respondents to submit more than an original and two copies of any document.



 (d) Requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records for more than three years.



 (e) In connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study.



 (f) Requiring the use of a statistical data classification that has not been reviewed and approved by OMB.

 (g) That includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use.



 (h) Requiring respondents to submit proprietary trade secret, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information’s confidentiality to the extent permitted by law.



The special circumstances contained in item 7 of the supporting statement are not applicable to this information collection.

8. Federal Register Notice:



 a. Provide a copy and identify the date and page number of publication in the Federal Register of the agency’s notice soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments. Specifically address comments received on cost and hour burden.



A 60-day Federal Register Notice inviting public comments was published on February, 2011, Volume 75 pp. No comments were received. See attached copy of the published notice included in this package.

A 30-day Federal Register Notice inviting public comments was published on , 2011, Volume 75 pp. No comments were received. See attached copy of the published notice included in this package.

 b. Describe efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or reported.

Contracted personnel provide expert assistance in the formulation of questionnaires/data collection instruments which the Academy and other divisions within the USFA use to evaluate their training and education programs. The Evaluation Center staff also develops reporting formats used to display the results of the data analysis and routinely prepares reports in conjunction with individual training courses.

c. Describe consultations with representatives of those from whom information is to be obtained or those who must compile records. Consultation should occur at least once every three years, even if the collection of information activities is the same as in prior periods. There may be circumstances that may preclude consultation in a specific situation. These circumstances should be explained.

The NFA consults with its residential student population on a regular basis regarding course evaluations. The Superintendent holds a “Superintendent’s Lunch” with class representatives during every resident class delivery cycle. This luncheon provides the opportunity for the Superintendent and class representatives to discuss issues concerning training development, course delivery and course evaluation. The Superintendent also visits each class during every delivery cycle and provides an open forum for any issues the students and instructors want to discuss. Therefore, consultations occur at least once every three years.

 9. Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.

FEMA does not provide payments or gifts to respondents in exchange for a benefit sought.

 

10. Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy. Provide details on:

The electronic letter inviting student and supervisor participation in the LTE program indicates that responses to the questionnaire will remain anonymous.  In specific terms, the relevant portion of this letter reads:

“Please know that we appreciate you taking the time to complete this important evaluation.  Please know as well:  Your responses to the questions contain no names and will remain anonymous.  Additionally, individual data are never shared or reported out apart from aggregate government summaries.

This assurance to respondents does not hamper sharing data with program managers involved in this information collection.  The NFA’s LTE system is covered under the DHS Wide Contact List PIA and was added to the Appendix of the DHS PIA on June 19, 2008.

A Privacy Threshold Analysis (PTA) was approved on 6/19/2008 and is valid until 6/19/2011. Currently, this PTA is associated by the DHS General Contact List PIA and the DHS Mailing and other lists the System of Records Notice SORN – DHS-002, as per the adjudication on the PTA. The NFA’s LTE system is covered under the DHS Wide Contact List Privacy Impact Analysis (PIA) and was added to the Appendix of the DHS PIA on June 19, 2008.

 11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

There are no questions of a sensitive nature.

 12. Provide estimates of the hour burden of the collection of information. The statement should:



 a. Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. Unless directed to do so, agencies should not conduct special surveys to obtain information on which to base hour burden estimates. Consultation with a sample (fewer than 10) of potential respondents is desired. If the hour burden on respondents is expected to vary widely because of differences in activity, size, or complexity, show the range of estimated hour burden, and explain the reasons for the variance. Generally, estimates should not include burden hours for customary and usual business practices.

It is anticipated that 2,250 students will complete FEMA Form 078-0-2 and 2,250 supervisors will complete FEMA Form 078-0-2A. Each respondent will only complete the form once and each response will require .2 hours to complete the form. The total annual hour burden is 2,250 x .2 hours = 450 annual hours.

It is anticipated that 2,250 supervisors will complete FEMA Form 078-0-2. Each respondent will only complete the form once and each response will require .11 hours to complete the form. The total annual hour burden is 2,250 x .11 hours = 247.5 annual hours.

 b. If this request for approval covers more than one form, provide separate hour burden estimates for each form and aggregate the hour burdens in Item 13 of OMB Form 83-I.

c. Provide an estimate of annualized cost to respondents for the hour burdens for collections of information, identifying and using appropriate wage rate categories. NOTE: The wage-rate category for each respondent must be multiplied by 1.4 and this total should be entered in the cell for “Avg. Hourly Wage Rate”. The cost to the respondents of contracting out or paying outside parties for information collection activities should not be included here. Instead this cost should be included in Item 13.

Table A.12: Estimated Annualized Burden Hours and Costs
Type of Respondent	Form Name / Form Number	No. of Respondents	No. of Responses per Respondent	Avg. Burden per Response (in hours)	Total Annual Burden (in hours)	Avg. Hourly Wage Rate	Total Annual Respondent Cost
State, local or Tribal government	NFA Long Term Evaluation Students/Trainees/ FEMA Form 078-0-2A (Formerly FEMA Form 95-59)	2,250	1	.20 (12 minutes)	450	$30.32	$13,644.00
State, local or Tribal government	NFA Long Term Evaluation Supervisors / FEMA Form 078-0-2 (Formerly FEMA Form 95-58)	2,250	1	.11 (6.5 minutes)	247.5	$45.93	$11,367.68
Total		4,500			697.5 hours		$25,011.68

• Note: The “Avg. Hourly Wage Rate” for each respondent includes a 1.4 multiplier to reflect a fully-loaded wage rate.

According to the U.S. Department of Labor, Bureau of Labor Statistics website (www.bls.gov) the wage rate category for Fire Fighters (representing all firefighters) is estimated to be ($21.66 x 1.4=) $30.32 per hour including the wage rate multiplier, therefore, the estimated burden hour cost to respondents Fire Fighter is estimated to be 450 hours x $30.32 = $13,644.00 annually.

According to the U.S. Department of Labor, Bureau of Labor Statistics website (www.bls.gov) the wage rate category for Fire Fighters (representing all first-line supervisors/managers of firefighters) is estimated to be ($32.81 x 1.4=) $45.93 per hour including the wage rate multiplier, therefore, the estimated burden hour cost to respondents Fire Fighter is estimated to be 247.5 hours x $45.93 = $11,367.68 annually.

13. Provide an estimate of the total annual cost burden to respondents or record keepers resulting from the collection of information. (Do not include the cost of any hour burden shown in Items 12 and 14.)

There are no record keeping, capital, start-up or maintenance costs associated with this information collection.

14. Provide estimates of annualized cost to the Federal Government. Also, provide a description of the method used to estimate cost, which should include quantification of hours, operational expenses (such as equipment, overhead, printing and support staff), and any other expense that would have been incurred without this collection of information. You may also aggregate cost estimates for Items 12, 13, and 14 in a single table.



Annual Cost to the Federal Government

Item	Cost ($)
Contract Costs: a.) Funding for Web Farm Services utilized in maintaining the Evaluation Center application $10,000.00. Fixed price contract with Synthesis Professional Services providing onsite services for LTE Level 3 Evaluations of on-campus resident courses. b.) Contractor cost LTE FEMA Form 078-0-2/FEMA Form 078-0-2A processing (501.33 hrs. x $47.09) + (306.67 hrs. x $71.51) + (156.67 hrs x $30.52) = $50,319.17. Therefore 10,000 + 50,319.17= $ 60,319.17.	$60,319.17
Staff Salaries (one GS 12/ step 5 employee spending approximately 1840 hours of time annually to provide contractual oversight for this data collection x 1.4. [105,101 x .01 = 1051.01 x 1.4 = 1,471.41.]	$1,471.41
Facilities [cost for renting, overhead, etc. for data collection activity]	$0.00
Computer Hardware and Software (SPSS PASW Renewal)	$2,200.00
Equipment Maintenance Scantron scanner	$950.00
Travel	$0.00
Printing: Online format	$0.00
Postage [annual number of data collection instruments x postage]	$0.00
Other:	$0.00
Total	$64,940.58

*Note: The “Salary Rate” includes a 1.4 multiplier to reflect a fully-loaded wage rate.

15. Explain the reasons for any program changes or adjustments reported in Items 13 or 14 of the OMB Form 83-I in a narrative form. Present the itemized changes in hour burden and cost burden according to program changes or adjustments in Table 5. Denote a program increase as a positive number, and a program decrease as a negative number.

Itemized Changes in Annual Burden Hours
Data collection Activity/Instrument	Program Change (hours currently on OMB Inventory)	Program Change (New)	Difference	Adjustment (hours currently on OMB Inventory)	Adjustment (New)	Difference
FEMA Form 078-0-2A (Formerly FEMA Form 95-59) NFA Long Term Evaluation Students / Trainees	743.5	450.0	-293.5
FEMA Form 078-0-2 (Formerly FEMA Form 95-58) NFA Long Term Evaluation Supervisors	382.5	247.5	-135.0
Total(s)	1,126	697.5	-428.5

For FEMA Form 078-0-2A (students/trainees), the previously approved burden hours were 742.5, the current estimated annual hour burden is 450, resulting in a decrease (program change) of 293.5 hours. The decrease in burden hour’s results from moving away from a paper based form to an electronic online format. As a result of these changes, students/trainees spend less time completing and submitting the form.

For FEMA Form 078-0-2 (supervisors), the previously approved burden hours were 382.5, the current estimated annual hour burden is 247.5, resulting in a decrease (program change) of 135.0 hours. The decrease in burden hour’s results from moving away from a paper based form to an electronic online format. As a result of these changes, supervisors spend less time completing and submitting the form.

Itemized Changes in Annual Cost Burden
Data collection Activity/Instrument	Program Change (hours currently on OMB Inventory)	Program Change (New)	Difference	Adjustment (hours currently on OMB Inventory)	Adjustment (New)	Difference
FEMA Form 078-0-2A (Formerly FEMA Form 95-59) NFA Long Term Evaluation Students / Trainees	0	$13,644.00	+$13,644.00
FEMA Form 078-0-2 (Formerly FEMA Form 95-58) NFA Long Term Evaluation Supervisors	0	$11,367.68	+$11,367.68
Total(s)				0	$25,011.68	+$25,011.68

For FEMA Form 078-0-2A (students/trainees), the previously approved cost burden on the OMB inventory was 0; the current estimated annual cost burden is $13,644.00, resulting in an increase (program change) of $13,644.00. The increase is due to the cost burden not being previously recorded on the OMB inventory.

For FEMA Form 078-0-2 (supervisors), the previously approved cost burden was 0; the current estimated annual cost burden is $11,367.68, resulting in an increase (program change) of +$11,367.68. The increase is due to the cost burden not being previously recorded on the OMB inventory.

 16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.



The Long Term Evaluation Report is published annually for each Fiscal Year on the NFA Web site http://www.usfa.dhs.gov/nfa/evaluation/eval_report.shtm. The annual LTE report contains graphic representations of (1) percent agreement by students and supervisors with course satisfaction statements, (2) the relative frequency and utility-rating of training experiences at the NFA and other training locations, (3) estimates for the amount of training diffusion on the job and to work colleagues, and (4) relative frequency of various suggestions for improvements and new training courses. These results are system generated with minimal demand on the time (2 hours) of an Evaluation Center staff member.

 17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain reasons that display would be inappropriate.



FEMA will display the expiration date for OMB approval of this information collection.

 18. Explain each exception to the certification statement identified in Item 19 “Certification for Paperwork Reduction Act Submissions,” of OMB Form 83-I.



FEMA does not request an exception to the certification of this information collection.

13