1660-0039 Supportin Statement 7-24-08 Part B

B. Collections of Information Employing Statistical Methods.



When Item 17 on the Form OMB 83-I is checked “Yes”, the following documentation should be included in the Supporting Statement to the extent it applies to the methods proposed:

1. (a) Describe (including numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. (b) Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. (c) Indicate expected response rates for the collection as a whole. If the collection has been conducted previously, include the actual response rate achieved during the last collection.

The potential respondent universe totals 4,500 annually which includes 2,250 students and 2,250 supervisors. The respondents will be randomly selected from approximately thirty 2-day courses, thirty 6-day course offerings and thirty 10-day course offerings. The respondent universe is based on a total population universe of approximately 10,000 fire and emergency response personnel that attend NFA on-campus residential training course each year.

	Universe		Sample
	Num of Entities [Description]	Size	Num of Entities [Description]	Size
NFA Sponsored Training/On-campus (FF 95-59)	1 [Individual - student]	2.250	1 [Individual]	2,250
NFA Sponsored Training/On-campus (FF 95-58)	1 [Individual - supervisor]	2,250	1 [Individual]	2,250
Total	1	4,500	1	4,500

Historical response rates for the paper based forms are as follows. These data are based on 10-day training courses and do not include a sampling of the additional types of courses we plan to add in this sample:

Fiscal Year	Response Rate – Students FF 95-59	Response Rate – Supervisors FF 95-58
FY2004	73% (470)	74% (474)
FY2005	57% (427)	70% (542)
FY2006	64% (342)	73% (530)

2. Describe the procedures for the collection of information including:

Analysis Plan

Descriptive statistics from each (student/supervisor) group concerning respondent demographics and training applications and effectiveness will be compiled for review by training and instructional systems specialists and USFA/NFA managers.

In particular, training specialists, instructional systems specialists and managers will receive course specific reports (Exhibit #1/sample report) which provide two types of study data. The first type of reported data presents five sets of descriptive statistics, which deal with both student demographics and program effectiveness. The student demographics provide NFA the opportunity to confirm empirically that it is reaching the audiences it has identified for training deliveries.

Similarly, the program effectiveness data are also descriptive and these provide NFA’s Training Specialists and Management information useful in consideration of curriculum revision in subsequent offerings of the training classes

These statistics include percentage distributions of the following:

(1) Student background characteristics (i.e., percent distributions of gender, age, educational levels, NFA field course experience, years in fire and/or emergency services, responsibilities, department size and percent of career personnel in the student’s department).

(2) Students and supervisors who judge that the student’s training has:

• Improved the student’s job performance;

• Contributed to the student’s professional development;

• Improved the performance of the department;

• Led to a reduction in fire related risks within the community, and generally

• Has been worth recommending to others.

(3) Students who report specific applications of training materials, i.e., the percent of students who indicate that they have applied specific training objectives in terms of the following frequencies:

• Used a great deal

• Used somewhat

• Not used at all

• No Opportunity to Use

These data will permit an immediate assessment of NFA training in terms of both individual training classes and curriculum-wide applications. For example, NFA will be able to know what percent of training objectives are being applied by what percent of its students. That is, for any given curriculum, NFA will be able to report that “X” percent of students are actually applying “Y” percent of the training objectives provided in/by the training classes. Additionally, NFA will have a snap- shot picture of which classes within the curriculum are having the widest applications within the student’s local training organization.

(4) Students and supervisors who judge that the student’s training has:

• Made student more safety conscious;

• Provided prevention ideas that can be incorporated into the department’s public education programs;

• Helped with policy development; and

• Met department expectations.

(5) Students reporting their experience of training diffusion, i.e., the percent of students who have shared training knowledge and information:

• Formally in a training class;

• Informally in a group; or

• Informally, one on one.

The second type of data provided by the LTE reports prepared for training and instructional systems specialists and NFA managers is narrative data. Narrative data vary but generally include comments provided by students and supervisors who indicate:

• Needed improvements in the NFA training;

• Descriptions of incidents in which students have applied NFA training; and

• Topics for future training classes.

In addition to the standardized descriptive statistics and narrative types of data provided in regular LTE reports to training and instructional systems specialists and NFA managers, special reports will be run from LTE data for other internal and external audiences as required.

There are several ways in which the data from this data collection are used. For example,

(1) Training specialist staff determine the effectiveness of NFA training in advancing the skills of trainees and the resulting benefit to his/her organization;

(2) Instructional staff determine appropriate methodologies and delivery modes for NFA training; and

(3) U.S. Fire Administration/NFA management staff examine the success of NFA training in achieving its strategic goals to support the Agency’s performance measurement process – serving as a budget performance measure (see Exhibit #2/Performance Measures Template).

• Statistical methodology for stratification and sample selection:

The data collected represent a stratified random sample of NFA students attending 2-day, 6-day and 10-day training courses. Both stratification and sample selection are based on the total population universe that encompasses NFA on-campus residential training venues. This includes approximately 10,000 students, of which 4,500 represent our anticipated annual sampling frame. This includes students (n=2,250) and their supervisors (n=2,250). The student samples will be drawn from thirty 2-day course offerings, thirty 6-day course offerings and thirty 10-day course offerings.

When the LTE first began the NFA’s sampling strategy was based on a three-year review cycle. While this process allowed enough time for the Academy to review all of its 10-day courses within the established five-year course review cycle, it is no longer valid given the type and number of courses the NFA now offers. In order to maintain currency, the NFA must increase its sample size to include its shorter duration courses.

• Estimation procedure:

The data collected represent a stratified random sample of the Academy’s on-campus residential curriculum. Thirty 2-day courses, thirty 6-day courses and thirty 10-day courses will be used each year in the sampling process. .

In the event of missing data, SPSS diagnostic tests—available through the SPSS Missing Values Analysis component—will be employed. These diagnostic tests identify patterns of missing data through item by item analyses that test for differences between respondents and non-respondents. Thus, these diagnostics permit estimates of population parameters for all items in which missing values are recorded, and they allow the user a greater sense of accuracy for all items that include missing values. SPSS software provides an extended summary of its programming capabilities for identifying the kinds of missing data patterns that can occur, and Exhibit #3 summarizes the specific types of information provided by these tests.

• Degree of accuracy needed for the purpose described in the justification:

The degree of accuracy is something we aim to increase. By selecting a larger sample size, we hope to decrease the margin of error. Also because we are now using a stratified random sample, we will need a larger sample to get the returns we expect. Extremely accurate estimates of true scores are not needed in this case since (1) data findings will not be used for inferential purposes and (2) the homogeneity of the target population and interest in the subject provide for more satisfactory levels of validity and reliability based on respondents’ ability to provide useful and consistent information.

• Unusual problems requiring specialized sampling procedures, and

There are no unusual problems requiring specialized sampling procedures.

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

There is no use of periodic data collection cycles to reduce burden.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

In order to maximize response rates, the National Fire Academy has placed informational posters within each of their resident classrooms to encourage student participation. In addition, clear instructions are provided to both students and supervisors via e-mail. The NFA has also established an on-line evaluation comments web-page and direct mailbox to assist respondents with questions and to provide feedback. It is expected that these measures will help to maintain sufficiently high response rates suitable to analysis, but in the event of response rates falling below 80%, a non-response analysis will be performed on the group(s) in question. These analyses will be conducted by using the “SPSS Analysis of Missing Data” module of the general SPSS software package and the findings of the analysis will be addressed accordingly.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

Lessons learned from previous usability testing for the NFA’s Level I end-of-course form have been incorporated into the LTE instruments. Usability tests on less than 10 individuals have been scheduled for late 2008, to ensure that the web-based application is in working order. This test will be conducted over a period of approximately six months, with specific attention to (1) the ease of navigation within the form, (2) the clarity of the form’s items (3) the length of completion time and (4) the instructions for use. The test will be conducted under the oversight of both the Project Officer and USFA Web Master, and trained staff from the NFA’s Evaluation Center. Students and supervisors who volunteer assessments and comments and where applicable (e.g., in navigation directions), this feedback will be incorporated into the final online application of FF 95-58 and FF 95-59.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Ms. Terry Gladhill, U.S. Fire Administration, National Fire Academy, (301) 447-1239.

Dr. Mary-Paula Walsh, HSFEEM-06-C-0143, (301) 447- 1335.

5