0032_Supporting Statement_Part B

March 21, 2007

Supporting Statement Part B. for Paperwork Reduction Act Submission

OMB Control Number: 1660-0032

Title: National Fire Academy End of Course Evaluation Form

Form Number(s): FF 95-20

General Instructions

A Supporting Statement, including the text of the notice to the public required by 5 CFR 1320.5(a)(i)(iv) and its actual or estimated date of publication in the Federal Register, must accompany each request for approval of a collection of information. The Supporting Statement must be prepared in the format described below, and must contain the information specified in Section A below. If an item is not applicable, provide a brief explanation. When Item 17 or the OMB Form 83-I is checked “Yes”, Section B of the Supporting Statement must be completed. OMB reserves the right to require the submission of additional information with respect to any request for approval.

To complete the supporting statement, type in your responses in the white space below each question. Your responses should be full and complete and provide sufficient information to help the OMB desk officer to understand what you are planning to do and why and how the Agency/Federal Government will benefit from and use the information you will be obtaining or soliciting.

Specific Instructions

B. Collections of Information Employing Statistical Methods.



When Item 17 on the Form OMB 83-I is checked “Yes”, the following documentation should be included in the Supporting Statement to the extent it applies to the methods proposed:

1. (a) Describe (including numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. (b) Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. (c) Indicate expected response rates for the collection as a whole. If the collection has been conducted previously, include the actual response rate achieved during the last collection.

(a) The target population (universe) of this data collection is ‘all the students enrolled in NFA-sponsoring training courses’. The data collection entails a census of all students attending NFA-sponsoring training both on-campus and off-campus. Hence, no sampling strategy is necessary.

(b) Number of entities in the universe and sample (there is no stratification).

	Universe		Sample = Universe
	Num of Entities [Description]	Size	Num of Entities [Description]	Size
NFA Sponsored Training courses	1 [Individual]	14,000	1 [Individual]	14,000
Total	1	14,000	1	14,000

(c) The actual response rate of 99.65 % percent was achieved during the last collection conducted in FY2005 for the resident (i.e., Emmitsburg) classes. A similar response rate of 98.57 % was achieved for the off-campus classes (i.e., NFA offerings provided off site away from Emmitsburg). Given both the stability of the NFA student population and the established pattern of course evaluation administration, it is anticipated that response rate(s) for the coming collections will remain in the 95-99% range. Many NFA students have attended other NFA sponsored training courses and are familiar with the evaluation process. Likewise, individual state fire training agencies routinely conduct NFA sponsored training courses and are experienced with the mechanics of the administration and collection of the evaluation instrument.

2. Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection,

The data collected represent a census of the entire population of students completing NFA-sponsored end-of-course course evaluations, so no sampling selection is necessary. There is no stratification.

• Estimation procedure,

The data collected represent a census of the entire population of students completing NFA-sponsored end-of-course course evaluations, so no estimation procedures are anticipated, since population parameters will be directly obtained.

In the event of missing data, however, SPSS diagnostic tests—available through the SPSS Missing Values Analysis component—will be employed. These diagnostic tests identify patterns of missing data through item by item analyses that test for differences between respondents and non-respondents. Thus, these diagnostics permit estimates of population parameters for all items in which missing values are recorded, and they allow the user a greater sense of accuracy for all items that include missing values. SPSS software provides an extended (75 pages) summary of its programming capabilities for identifying the kinds of missing data patterns that can occur, and Exhibit #2 summarizes the specific types of information provided by these tests.

• Degree of accuracy needed for the purpose described in the justification,

Degree of accuracy (margin of error) is irrelevant to this information collection because there is no sampling process involved (see B. 1. a.). In addition, extremely accurate estimates of true scores are not needed in this case since (1) data findings will not be used for inferential purposes and (2) the homogeneity of the target population and interest in the subject ensures satisfactory levels of validity and reliability based on respondents’ ability to provide useful and consistent information.

• Unusual problems requiring specialized sampling procedures, and

There are no unusual problems requiring specialized sampling procedures.

• Any use of periodic (less frequent than annual) data collection cycles to m reduce burden.

There is no use of periodic data collection cycles to reduce burden.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

In order to maximize response rates, clear instructions are provided to instructors or delivery personnel at the onset of the course delivery cycle and information collection stage. Written instructions are provided to students who complete both the paper-based form as well as the online web-based form. Detailed instructions are provided with adequate completion time. The NFA has also established an on-line evaluation web-page and direct mailbox to assist respondents with questions.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

A usability test was conducted for the online evaluation survey instrument to ensure that the web-based application was in working order for the currently approved NFA EOC evaluation form (Implementation of the Government Paperwork Elimination Act, Office of Management and Budget, PART I. Section 2. a.) (Exhibit #3). This test was conducted over a period of four months (December, 2005-March, 2006), with specific attention to (1) the ease of navigation within the form, (2) the clarity of the form’s items (3) the length of completion time and (4) the instructions for use. The test was conducted under the oversight of both the Project Officer and NFA WEB Master, and trained staff from the NFA’s Evaluation Center. Students volunteered assessments and comments and where applicable (e.g., in navigation directions), this feedback was incorporated into the online version of the currently approved NFA EOC evaluation form.

The tested online evaluation form will be revised to have exactly the same questionnaires as the proposed form in this ICR upon OMB approval.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Ms. Terry Gladhill, (301) 447-1239/U.S. Fire Administration, National Fire Academy. Dr. Mary-Paula Walsh, HSFEEM-06-C-0143 (301) 447- 1335.

Attachment

Exhibit #1: NFA EOC Sample Report

Exhibit #2: SPSS Missing Values Test Narrative

Exhibit #3: Screen Shots of the Online Electronic Form

Exhibit #4: Public Law 93-498

4