1110-0076 Supporting Statement B (05022023)

B. Collections of Information Employing

Statistical Methods

The agency should be prepared to justify its decision not to use statistical methods in any case where such methods might reduce burden or improve accuracy of results. When the question “Does this ICR contain surveys, censuses or employ statistical methods” is checked, "Yes," the following documentation should be included in the Supporting Statement to the extent that it applies to the methods proposed:

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The FBI invites all law enforcement respondents to complete the instructor or presenter and course questionnaires for all FBI training and/or education programs the respondents complete. These programs are made up of multiple courses and delivered by numerous instructors and/or presenters. In addition, all respondents are invited to complete a general related questionnaire to ensure the learning environment is sufficient. To help eliminate the burden, questionnaires are typically no more than fifteen questions with one or two being open ended responses.

Example, NA respondents only register for approximately seven courses out of approximately 54 offered. Respondents are only asked to respond for courses they complete. NEI, LEEDS, and other FBI training programs are only asked to respond to an end of course (single) questionnaire and general questionnaire at the completion of their training programs.

Questionnaires are typically accessed via a link in an email or located within each course on a course management system. Questionnaires are open for a period of one to two weeks prior to course completion which eliminates respondents from being burdened when they leave the FBI Academy at Quantico, VA. Each questionnaire includes a description of its aim and the purposes to why the data is being collected. The questionnaire will also explain to respondents that their participation is completely anonymous.

After completing the questionnaires, the respondents will be instructed to select “submit” in order to record their responses. Because all respondents of these FBI training programs will be offered the opportunity to participate in the questionnaires, the questionnaires do not involve any sampling, methods for stratification, imputations or similar statistical methods.

2. Describe the procedures for the collection of information including:

* Statistical methodology for stratification and sample selection,

* Estimation procedure,

* Degree of accuracy needed for the purpose described in the justification,

* Unusual problems requiring specialized sampling procedures, and

* Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

To ensure a high response rate, several measures have been put into place. For numerical data collection a software evaluation application tool will be used to calculate simple totals, percentages, and cumulative percentages. This questionnaire tool will also capture student comments entered in response to open-ended questions. Next, the questionnaire will focus strictly on those issues of great interest in order to keep it as short as possible while still accomplishing the goals of the evaluation. In addition, by offering the questionnaires online, we reduce the level of effort needed to complete it, as well as the steps involved in creating the database for analysis.

Respondents are also reminded about the questionnaires during the one to two week evaluation period and within their group meetings while at the FBI Academy. This face-to-face reminder is done by FBI Academy staff personnel. Finally, by including all respondents we feel confident that the resulting data will reflect the experience of a wide range of respondents, including the full variety of law enforcement agencies and personnel levels. It is with the understanding of the respondents that by completing these questionnaires they will help the FBI determine the strengths and weaknesses of the existing training program.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

No methods were developed to deal with issues of non-response.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of test may be submitted for approval separately or in combination with the main collection of information.

The questionnaires included as part of this package were developed following extensive interviews with program managers. After determining the desired content, items were developed by experienced evaluation experts. A thorough review of the content centered on several issues, including:

• Clarity of instructions

• Ease with which information requested can be provided

• Length of the instrument, and

• Quality of the resulting data in terms of how it can be used to answer the questions of interest.

These questionnaires identify the types of questions used to obtain feedback for FBI programs. FBI program managers will select questions from the ones provided in order to obtain data for their specific program. The Evaluation and Assessment Unit has documented successful results of the use of these types of questionnaires and their value for improving education and training programs.

Due to the nature of this collection, questionnaires will be designed and developed with questions that focus on the following areas:

Instructor or Presenter Questionnaire:

• The experience of the instructor on the topic

• The appropriate use of examples and scenarios

Course Questionnaire:

• The overall quality of the lesson, course, or program

• The applicability of activities or field trips associated with the program

• The application of the content to the respondents’ role

• The delivery methods used

General Questionnaire:

• The administrative requirements proceeding the attendance of the program

• The overall atmosphere of the Academy

• The resources and facilities (quality of food/cafeteria, maintenance and custodial support)

Open-Ended Questions will be used in various questionnaires to capture overall opinions about the topics:

• What did respondents like least/most about their instructor

• What was least/most valuable about the course or program

• Allow for any additional comments student want to make related to the program

Once questions are selected, developers will build collection instruments to support the different programs. Beta tests will be performed to ensure functionality of the questionnaires.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The following individuals were consulted regarding the statistical aspects of the project:

Unit Chief

Evaluation and Assessment Unit

Unit Chief

National Academy

Unit Chief

Leadership Programs Management Unit

Training Division

Dean of Academics

FBI Academy

Quantico, Virginia 22135

The persons who will be involved in collecting and analyzing the evaluation data:

Program Managers, Instructors, Instructional Systems Specialists (ISSs) and Management and Program Analysts (MAPAs) assigned to the:

Evaluation and Assessment Unit, FBI Academy, Quantico, Virginia 22135

National Academy, FBI Academy, Quantico, Virginia 22135

Leadership Programs Management Unit, FBI Academy, Quantico, Virginia 22135