1660-0039 Supporting Statement 7-24-08 Part A

Supporting Statement for Paperwork Reduction Act Submissions (7/23/08)

OMB Control Number: 1660-0039

Title: National Fire Academy Long-term Evaluation Form for Supervisors and National Fire Academy Long-term Evaluation Form for Students

Form Number(s): FF 95-58 and FF 95-59

General Instructions

A Supporting Statement, including the text of the notice to the public required by 5 CFR 1320.5(a)(i)(iv) and its actual or estimated date of publication in the Federal Register, must accompany each request for approval of a collection of information. The Supporting Statement must be prepared in the format described below, and must contain the information specified in Section A below. If an item is not applicable, provide a brief explanation. When Item 17 or the OMB Form 83-I is checked “Yes,” Section B of the Supporting Statement must be completed. OMB reserves the right to require the submission of additional information with respect to any request for approval.

To complete the supporting statement, type in your responses in the white space below each question. Your responses should be full and complete and provide sufficient information to help the OMB desk officer to understand what you are planning to do and why and how the Agency/Federal Government will benefit from and use the information you will be obtaining or soliciting.

Specific Instructions

A. Justification

 1. Explain the circumstances that make the collection of information necessary (give details as to why this information is being collected). Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information. Provide a detailed description of the nature and source of the information to be collected.

The National Fire Academy (NFA) is congressionally mandated to provide training and education to the Nation’s fire service and emergency response personnel. The state-of-the-art programs offered by the NFA serve as models of excellence and state, and local fire service agencies rely heavily on the curriculum to train their personnel. To maintain the high standards of these programs, it is critical that courses be evaluated after students have had the opportunity to apply the knowledge and skills gained.

Although this collection does not expire until March 31, 2009, it is being resubmitted because the paper-based format is being converted to an electronic on-line submission that will allow for the inclusion of additional NFA courses. The previous collection was based on a stratified random sample of students attending 10-day on-campus residential training courses. Since the program’s inception more than ten years ago, the NFA has expanded its curriculum base to include more training courses that are shorter-duration in length. This request includes information collection activity targeting additional NFA 2-day and 6-day training courses. This request seeks approval for two electronically based long-term evaluation (LTE) forms which will collect information from both course graduates and their respective supervisors.

As with past LTE data collections, the information to be collected electronically addresses the student’s applications of specified training objectives upon returning to his or her local department or professional organization. In the electronic version of the forms all students attending the selected courses [together with their respective supervisors] will be sent an electronic link to a new web-based application that will provide direct access to the appropriate form. These links will be sent to the respective student-supervisor pairs between four and six months after the course ends (depending upon the training objectives of the specific course).

The data collection will be managed under the overall guidance of the NFA project officer, but it will be implemented by contracted staff with considerable training and experience in data collection, analysis and reporting.

There are several advantages to be gained by this shift to an electronic data collection strategy. First, the shift to an electronic version of the LTE forms reduces the paper burden of students and supervisors having to complete and return mailed-out questionnaires. Second, this shift permits an immediate opportunity to survey the training applications of not only students who have attended NFA 10-day courses, but it expands the information collection effort to include other NFA courses that are shorter-duration in length, e.g., 2-day and 6-day training courses. Thus, it provides a more accurate index into the changing demographics of this population and its anticipated curriculum needs (as evidenced by the survey’s specific open-ended items), and it also facilitates the potential for full curriculum updates as well.

When LTE evaluation first began the NFA’s sampling strategy presumed a three year cycle to encompass the full review of the curriculum which only included 10-day courses. This framework was sufficient because it facilitated the sampling and review of all NFA resident courses within a three year window which allowed for updates and revisions to curriculum within the Academy’s five-year curriculum review cycle. However since that time, the Academy has increased its delivery capacity and added 6-day courses and is now including the 2-day State Weekend Program. As a result, a stratified random sample of all 10-day, 6-day and 2-day courses is now needed.

 2. (1) Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, (2) indicate the actual use the agency has made of the information received from the current collection. If applicable, provide a detailed description of how the information will be shared and for what programmatic purpose.

(1) The data collected by NFA for its LTE forms will be used by:

(a) Instructional development/training specialist staff, and

(b) U.S. Fire Administration/NFA management staff

Information collected with these evaluation forms will enable NFA staff to monitor and recommend changes in course materials, individual subject selection criteria, and to make curriculum-wide reviews and assessments. The online electronic forms will process the data through an Oracle database, and reports will be made available electronically to USFA/NFA management and staff.

There are several ways in which the data from this data collection are used. For example,

(1) Training specialist staff determine the effectiveness of NFA training in advancing the skills of trainees and the resulting benefit to his/her organization;

(2) Instructional staff determine appropriate methodologies and delivery modes for NFA training; and

(3) U.S. Fire Administration/NFA management staff evaluate the applications of NFA training on achieving its strategic goals to support the Agency’s performance measurement process – serving as a budget performance measure (see Exhibit #2/Performance Measures Template).

Standardized descriptive statistics and narrative types of data provided in regular LTE reports to training and instructional systems specialists and NFA managers, special reports will be run from LTE data for other internal and external audiences as required.

 3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden.

The survey activity will include electronic and automated methods and therefore is a change from paper forms to using information technology to reduce burden. The data collection will be electronic and automated. The electronically fillable forms will be made available online for use by NFA students and their supervisors 4-6 months after the course ends. The data from these submissions will be processed using an Oracle database and both the data and subsequent reports resulting from these submissions will be generated.

 4. Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Item 2 above.



No other systematic collection of data regarding such information as NFA student demographics, course content and effectiveness of the training on student work performance, or the benefit to an individual student’s department performance exists. The students remain anonymous when they fill out the form. Thus, it is necessary to collect both the demographic data and opinions of the students/supervisors within the same form in order to correlate the information in all data fields without defying their anonymity.

5. If the collection of information impacts small businesses or other small entities (Item 5 of OMB Form 83-I), describe any methods used to minimize.

The information collection has no additional burden on small entities. Only individual students who attend NFA training and their supervisors are asked to complete this evaluation form. Small businesses and/or other small entities are not required to complete this course evaluation form.

 6. Describe the consequence to Federal/FEMA program or policy activities if the collection of information is not conducted, or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

If the data collection is not conducted, the USFA/NFA will not be able to determine its effectiveness in carrying out its mission to provide effective training and education for the Nation’s fire and emergency response community.

 7. Explain any special circumstances that would cause an information collection to be conducted in a manner:

 (a) Requiring respondents to report information to the agency more often than quarterly.

 

Respondents are only required to provide the information once, after completion of each training course. There are no special circumstances that require the collection to be conducted in a manner that is inconsistent with the general information collection guidelines in 5 CFR 1320.6.

 (b) Requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it.



There are no requirements for respondents to prepare a written response to this collection of information in fewer than 30 days after receipt of it. The long-term evaluation process elicits Level III training data which measures the student’s transfer of training skills to the job at the student’s work and/or organization. In order to capture the transfer of skills, the evaluation forms are administered to students and their supervisors 4-6 months after the conclusion of the training course, allowing students enough time to apply what they have learned and allowing their supervisors time to evaluate the results. Completion of the forms is voluntary, and does not require respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it.

 (c) Requiring respondents to submit more than an original and two copies of any document.



There are no requirements for a respondent to submit more than an original and two copies of any document. Respondents are required to submit only the original evaluation form.

 (d) Requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records for more than three years.



There is no recordkeeping requirement involved in this collection.

 

 (e) In connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study.



There is a survey involved with this collection. Refer to Part B for details on the proposed statistical methodology. The survey methodology used in this information collection follows accepted standards of data quality and reliability. The proposed survey will be designed to produce valid and reliable results that can be generalized to the universe of study.

 (f) Requiring the use of a statistical data classification that has not been reviewed and approved by OMB.

The proposed information collection does not require any use of a statistical data classification that has not been reviewed and approved by OMB. In addition, the information collected is for internal use for program planning, management and evaluation and is not intended as general statistical information for public dissemination.

 (g) That includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use.



There is no pledge of confidentiality that is not supported by authority established in statue or regulation for this data collection.

 (h) Requiring respondents to submit proprietary trade secret, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information’s confidentiality to the extent permitted by law.



There are no requirements for respondents to submit proprietary trade secret, or other confidential information for this data collection.

8. Federal Register Notice:



 a. Provide a copy and identify the date and page number of publication in the Federal Register of the agency’s notice soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments. Specifically address comments received on cost and hour burden.



A 60-day Federal Register Notice inviting public comments was published on September 21^st, 2007, page 54055, and a 30-Day Federal Register Notice was published on December 28^th, 2007, page 73861. No comments were received for this information collection.

 b. Describe efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or reported.

The USFA’s National Fire Academy is currently in the third year of a five-year contract with Synthesis, Inc., to provide on-site evaluation services. As such, contracted personnel provide expert assistance in the formulation of questionnaires/data collection instruments which the Academy and other divisions within the USFA use to evaluate their training and education programs. Evaluation Center staff also develops reporting formats used to display the results of the data analysis and routinely prepares reports in conjunction with individual training courses.

c. Describe consultations with representatives of those from whom information is to be obtained or those who must compile records. Consultation should occur at least once every three years, even if the collection of information activities is the same as in prior periods. There may be circumstances that may preclude consultation in a specific situation. These circumstances should be explained.

The NFA consults with its residential student population on a regular basis regarding course evaluations. The Superintendent holds a “Superintendent’s Lunch” with class representatives during every resident class delivery cycle. This luncheon provides the opportunity for the Superintendent and class representatives to discuss issues concerning training development, course delivery and course evaluation. The Superintendent also visits each class during every delivery cycle and provides an open forum for any issues the students and instructors want to discuss. Therefore, consultations occur more often than once every three years.

 9. Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.

There is no remuneration to respondents for their participation in this collection.

 

10. Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy. Provide details on:

Responses are strictly voluntary; therefore the decision to answer this questionnaire comes from respondents’ own volition. Respondents can refuse participation at any time by terminating or refusing to fill out the questionnaire.

Respondents are informed that the survey is a voluntary way of providing information. Applicants have the opportunity to decline participation by simply refusing to complete or submit the questionnaire. Since respondents are informed of the voluntary participation in this survey, by choosing to complete and submit the questionnaires respondents are granting such consent.

The information collected from the survey instruments is entered into a secured database.

The system is protected by multiple layers of physical and electronic security under DHS control. Direct database access is provided only to DHS system administrators. A firewall will be in place to separate the web server from the application and database server. The database is password-protected. Data is not released to external requestors. The online web based form is part of the USFA web farm application and has received the necessary certification and accreditation/privacy impact and has the authority to operate.

A privacy Threshold Analysis and a Privacy Impact Assessment have been completed for this collection, and are currently under review by the agency Privacy Officer. All information collected will be kept private to the extent allowable by law.

Data will be analyzed and reported in an aggregate format to prevent a disclosure of any individual respondent’s information. Demographic data elements are used exclusively for analytical purposes. There is no intent to identify individual respondents in conjunction with any other data elements. Individual responses will not be used in conjunction with demographic data elements. Collective/summary demographic data for courses and/or broad curriculum areas will be used only to analyze fire service training needs overall.



 11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

There are no questions of a sensitive nature.

 12. Provide estimates of the hour burden of the collection of information. The statement should:



 a. Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. Unless directed to do so, agencies should not conduct special surveys to obtain information on which to base hour burden estimates. Consultation with a sample (fewer than 10) of potential respondents is desired. If the hour burden on respondents is expected to vary widely because of differences in activity, size, or complexity, show the range of estimated hour burden, and explain the reasons for the variance. Generally, estimates should not include burden hours for customary and usual business practices.

Due to the cost associated with processing the paper-based version of these forms, the NFA had initially begun its LTE program with a 10 percent stratified random sample of 30-35 annual course offerings that included 750 students and their 750 supervisors for a total sample of 1,500 respondents. In both FY2004 and FY2006, however, some sampling loss occurred because of cancelled offerings, such that the LTE sample size was limited to 1280 (640 students and their 640 supervisors) in FY2004 and to 800 (400 students and their 400 supervisors) in FY2006.

Through the use of the online electronic forms and automated processing of the LTE instrument, we intend to expand our sample to cover the fact that we are using more class offerings now as well as using stratified random sampling now. Also because we are now using a stratified random sample, we will need a larger sample to get the returns we expect. The LTE program will be expanded to include the Academy’s shorter duration 2-day and 6-day courses. This revised approach will add to the stratified random sampling of the Academy’s curriculum by including thirty 2-day course offerings and thirty 6-day course offerings to the existing 30 10-day sample. All told the sample size will increase from 750 students and 750 supervisors to 2,250 students and 2,250 supervisors. The individual hour burden per response will remain the same: 20 minutes or .33 hours for students and 10 minutes or .17 hours for supervisors.

 b. If this request for approval covers more than one form, provide separate hour burden estimates for each form and aggregate the hour burdens in Item 13 of OMB Form 83-I.

This request covers two forms, FF 95-99 and FF 95-58. See Table A. 12.

c. Provide estimates of annualized cost to respondents for the hour burdens for collections of information, identifying and using appropriate wage rate categories. The cost of contracting out or paying outside parties for information collection activities should not be included here. Instead this cost should be included in Item 13.

The annualized cost to respondents totals $26,268.30.

Table A.12: Estimated Annualized Burden Hours and Costs
Type of Respondent	Form Name / Form Number	No. of Respondents	No. of Responses per Respondent	Avg. Burden per Response (in hours)	Total Annual Burden (in hours)	Avg. Hourly Wage Rate	Total Annual Respondent Cost
Individual or household	FF 95-59	2,250	1	.33	742.50	$19.80 (1)	$14,701.50
Individual or household	FF 95-58	2,250	1	.17	382.50	$30.24 (2)	$11,566.80
Total		4,500			1,125		$26,268.30

^{(1) Median hourly rate for Firefighters, Bureau of Labor Statistics, Occupational Employment and Wages , 2006.}

^{(2) Median hourly rate for First-Line Supervisors/Managers of Firefighting and Prevention Workers, Bureau of Labor Statistics, Occupational Employment and Wages 2006}

13. Provide an estimate of the total annual cost burden to respondents or record keepers resulting from the collection of information. The cost of purchasing or contracting out information collection services should be a part of this cost burden estimate. (Do not include the cost of any hour burden shown in Items 12 and 14.)

The cost estimates should be split into two components:

a. Operation and Maintenance and purchase of services component. These estimates should take into account cost associated with generating, maintaining, and disclosing or providing information. Include descriptions of methods used to estimate major cost factors including system and technology acquisition, expected useful life of capital equipment, the discount rate(s), and the time period over which costs will be incurred.

There is no start-up, maintenance or operational cost to respondents involved in this collection.

b. Capital and Start-up-Cost should include, among other items, preparations for collecting information such as purchasing computers and software, monitoring sampling, drilling and testing equipment, and record storage facilities.

There is no capital and start-up cost to respondents involved in this collection.

Annual Cost Burden to Respondents or Record-keepers

Data Collection Activity/Instrument	*Annual Capital Start-Up Cost (investments in overhead, equipment and other one-time expenditures)	*Annual Operations and Maintenance Cost (such as recordkeeping, technical/professional services, ect.)	Annual Non-Labor Cost (expenditures on training, travel and other resources)	Total Annual Cost to Respondents
Total	---------------	---------------	---------------	---------------

14. Provide estimates of annualized cost to the Federal Government. Also, provide a description of the method used to estimate cost, which should include quantification of hours, operational expenses (such as equipment, overhead, printing and support staff), and any other expense that would have been incurred without this collection of information. You may also aggregate cost estimates for Items 12, 13, and 14 in a single table.



The following provides details about the approximate cost to the Government of this data collection:

 

Contract costs/operational costs are estimated at $105,000.00 per year. They include staffing, maintaining the electronic database, and report generation. The Academy's Evaluation Center is staffed with three full-time contractor staff who oversee the database management and report generation under the guidance of the National Fire Academy Project Officer.

Staff salaries are estimated at $9,000.00 per year. They include approximately 1 hour per day of the Project Officer’s time calculated at the GS-12 level.

IT maintenance for the online web farm application is estimated at $15,000.00 per year.

Annual Cost to the Federal Government

Item	Cost ($)
Contract Costs [Survey development, implementation, analysis and reporting]	105,000.00
Staff Salaries [ 1 GS-12 employee spending approximately 1 hour per day of the Project Officer’s time for this data collection]	9,000.00
Facilities [cost for renting, overhead, ect. for data collection activity]
Computer Hardware and Software [cost of equipment annual lifecycle]
Equipment Maintenance [IT maintenance]	15,000.00
Travel
Printing [number of data collection instruments annually]
Postage [annual number of data collection instruments x postage]
Other
Total	$129,000

 

15. Explain the reasons for any program changes or adjustments reported in Items 13 or 14 of the OMB Form 83-I in a narrative form. Present the itemized changes in hour burden and cost burden according to program changes or adjustments in Table 5. Denote a program increase as a positive number, and a program decrease as a negative number.

Definitions

Program changes should not be confused with adjustments.

i) Program change

A "Program increase" is an additional burden resulting from an action or directive of a branch of the Federal government (e.g., an increase in sample size or coverage, amount of information, reporting frequency, or expanded use of an existing form). This also includes previously in-use and unapproved information collections discovered during the ICB process, or during the fiscal year, which will be in use during the next fiscal year.

A "Program decrease", is a reduction in burden because of: (1) the discontinuation of an information collection; or (2) a change in an existing information collection by a Federal agency (e.g., the use of sampling (or smaller samples), a decrease in the amount of information requested (fewer questions), or a decrease in reporting frequency).

ii) An "Adjustment" denotes a change in burden hours due to factors over which the government has no control, such as population growth, or in factors which do not affect what information the government collects or how (e.g., changes in the methods used to estimate burden or correction of errors in burden estimates).

Itemized Changes in Annual Burden Hours
Data collection Activity/Instrument	Program Change (hours currently on OMB Inventory)	Program Change (New)	Difference	Adjustment (hours currently on OMB Inventory)	Adjustment (New)	Difference
FF 95-59: (Student/Trainee)	248	742.50	+494.5	+494.5
FF 95-58: (Supervisor)	127	382.50	+255.5	+255.5
Total(s)	375	1,125	+750	+750

Explain: This information collection retains similar statistical methodology approved for the same information collection, 1660-0039, in the current OMB Inventory aside from the increase in sample size and the stratified random sample selection now added. The current burden hours are 375 and we are proposing 1125. With the electronic conversion from the paper-based form to the electronic online form, the NFA will no longer be constrained by workload processing issues and will be able to extend the evaluation process to include a broader swath of NFA’s courses, namely its shorter duration 2-day and 6-day training courses.

Itemized Changes in Annual Cost Burden
Data collection Activity/Instrument	Program Change (cost currently on OMB Inventory)	Program Change (New)	Difference	Adjustment (cost currently on OMB Inventory)	Adjustment (New)	Difference
FF 95-59	0	$14,701.50	+$14,701.50
FF 95-58	0	$11,566.80	+$11,566.80
Total(s)	0	$26,268.30	+$26,268.30

Explain: Cost burden was not included in the last submission. The cost burden now reflects the actual cost burden to respondents.

 16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.



There is no plan to publish the results of the data collection at this time.

 17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain reasons that display would be inappropriate.



A valid OMB Control Number and expiration date will be displayed in the survey forms.

 18. Explain each exception to the certification statement identified in Item 19 “Certification for Paperwork Reduction Act Submissions,” of OMB Form 83-I.



No exceptions referenced above are sought for this collection.

13