1110-0021_Supporting_Statement

PAPERWORK REDUCTION ACT SUBMISSION

Supporting Statement

Agency: Federal Bureau of Investigation (FBI)

Title: FBI National Academy Level III Evaluation

Forms: FBI National Academy Post-Course Questionnaire for Graduates

FBI National Academy Post-Course Questionnaire for Supervisors of Graduates

OMB No: 1110-0021

Prepared by: Laleatha B. Goode

Management and Program Analyst

Federal Bureau of Investigation

FBI Academy

Training Division,

1 Range Road

Quantico, Virginia 22135

Telephone: (703) 632-3495

Fax: (703) 632-3111

The FBI Training Division's Curriculum Management Section (CMS), Curriculum Planning and Support Unit (CPSU) is requesting the Office of Management and Budget (OMB) approve the Paperwork Reduction Act submission for a reinstated data collection. The collection consists of one customer satisfaction questionnaire directed at the FBI National Academy (NA) graduates external customer base and one questionnaire directed at the supervisors identified by the graduates of FBI National Academy. CMS wants to ensure that the FBI National Academy is meeting the needs of both its graduates and sponsoring agencies and offer graduates and their supervisors an opportunity to offer suggestions for program enhancement.

Responses from the questionnaires will assist the CMS in identifying deficiencies and strengths in service delivery. From this information, training areas can be identified to foster development of best customer service practices to increase efficiency and quality of training.

The FBI National Academy Post-Course Questionnaire for Graduates will be distributed to all past graduates of eight sessions of FBI National Academy. The questions will target the graduates’ perception of the training they received at FBI National Academy. The graduates selected to take this questionnaire has completed training from six months to two and half years prior to the date of the questionnaire.

The FBI National Academy Post-Course Questionnaire for Supervisors of Graduates will be distributed to all of supervisors identified by graduates of the eight sessions. The questions will target the extent to which the supervisors believe the training provided by the FBI National Academy improved the performance of the graduates and positively impacted their agencies. The supervisors selected to take this questionnaire have been identified by graduates who have completed training at the FBI National Academy from six months to two and a half years prior to the date of the questionnaire.

A. JUSTIFICATION

1. Circumstances of the collection

Post Graduate Evaluation of the FBI National Academy

The Federal Bureau of Investigations (FBI) National Academy was created in response to a 1930 study released by the National Committee on Law Observation and Enforcement. This body was appointed by Hebert Hoover to study causes of criminal activity (largely related to prohibition) and make recommendations regarding appropriate national policy responses. One of those recommendations was regarding appropriate national policy responses. One of those recommendations was to establish as "FBI Police Training School" that would offer training to law enforcement professionals from around the country to bring about greater standardization of techniques and a higher level of professionalism.

Today, the mission of the Academy is to "support, promote, and enhance the personal and professional development of law enforcement leaders by preparing them for complex, dynamic and contemporary challenges through innovative techniques, facilitating excellence in education and research, and forging partnerships throughout the world.” To meet this mission, the FBI National Academy offers a quarterly 10-week program including undergraduate and graduate courses at their campus in Quantico, Virginia. The subject areas covered include law, behavioral science, forensic science, leadership development, communication, and health and fitness. In addition to class work, Academy attendance allows law enforcement professionals the opportunity to network, share ideas, and learn from their various experiences. Participation is by invitation only and is implemented though a supervisory nomination process. Attendees represent state and local police and sheriffs’ departments, military police organizations, and federal law enforcement agencies from the United States and over 150 foreign nations.

2. Purpose and Use of the Information

The purpose of the proposed data collection from both questionnaires is to gather information from FBI National Academy graduates and their identified supervisors to determine the impact of training on job performance and agency effectiveness. Under the most common models of training evaluations, four levels are defined. A Level I evaluation seeks information from students on the degree to which they felt the training experience was of value to them. Under Level II, resulting knowledge is assessed. Level III is concerned with the degree to which a training experience actually affected post training performance, which is the focus of this data collection. A Level IV evaluation seeks evidence that training has had the intended impact on the organization as a whole.

Clearly, in order to perform a Level III evaluation, information must be sought from trainees after they have returned to the workplace and have had sufficient time to apply the material taught. This data collection will also involve asking the trainees’ supervisors questions about the impact of the training provided.

FBI National Academy Post-Course Questionnaire for Graduates

The information collected from the FBI National Academy Post-Course Questionnaire for Graduates will determine the strengths and weaknesses of the existing program. This will be accomplished by: (a) collecting evidence regarding the personal impact of Academy attendance in regard to additional educational pursuits; (b) assessing graduate views of the program (e.g., quality of instruction, value of knowledge/skills taught); (c) assessing graduate views of the professional impact of Academy attendance (e.g., improved job performance enhanced prospects for promotion or career advancement); and (d) collecting suggestions for additional courses or topic areas to be added to the curriculum. Obtaining this information from graduates will provide a clear picture of the impact of the Academy program, as well as point to ways in which improvements or alterations can be instituted to strengthen that impact. No one other than graduates can more directly provide feedback on how their experience affected them and the work they do. This questionnaire represents the most cost-effective method for allowing them to provide such feedback and, as a result, create the opportunity for improvements to Academy programs to benefit future participants.

FBI National Academy Post-Course Questionnaire for Supervisors of Graduates

The information collected from the FBI National Academy Post-Course Questionnaire for Supervisors of Graduates will determine the strengths and weaknesses of the existing program. This will be accomplished by: (a) assessing supervisor views of the program (e.g., quality of instruction, value of knowledge/skills taught); (b) assessing supervisor views of the impact of training on their agencies (e.g., improved job performance, enhanced prospects for promotion or career advancement); and (c) collecting suggestions for additional courses or topic areas to be added to the curriculum. Obtaining this information from the graduates’ supervisors will determine the extent of the impact of the Academy’s program on participants and agencies, as well as point to ways in which improvements or alterations can be instituted to strengthen the program. No one other than the supervisors of FBI National Academy graduates can more directly provide feedback on how this experience affected their agencies and the officers’ job performance. This questionnaire represents the most cost-effective method for allowing the supervisors to provide such feedback and, as a result, create the opportunity for improvements to Academy programs.

This data collection is justified under 5 USC 410.601(f) (Office of Personnel Management, Training, Evaluation of Training). This section reads as follows:

Under provisions of chapter 41 of title 5, United States Code, and Executive Order 11348, the agency head shall evaluate training to determine how well it meets short and long-range program needs by occupations, organizations, or other appropriate groups. The agency head may conduct the evaluation in the manner or frequency he or she considers appropriate.

3. Use of Automated, Electronic, Mechanical or Technological Collection of Techniques

To the extent possible, all data collection will be conducted through the Internet. The data collection plan calls for FBI National Academy personnel to contact graduates and their supervisors via email and inform them of the existence and purpose of the evaluation. Therefore, all activities associated with this data collection involve information technology. The decision was made to use these methods to make it as easy as possible for all respondents to participate. Note that should a graduate or his supervisor prefer to complete a paper version of the questionnaire, he/she will be allowed to request one via email. We anticipate that this option will be selected infrequently.

4. Efforts to Identify Duplication

Given the unique nature of the FBI National Academy’s program, Level III evaluation efforts are possible only by questioning past graduates and their supervisors. No other sources are available or viable for this purpose.

5. Impact on Small Business

The collection of information will not have an impact on small businesses.

6. Consequences of not Collecting Data

Although the content of the National Academy’s program and the way in which it is conveyed have been carefully designed to address the institution’s mission, the changing nature of law enforcement activities over time presents the possibility that additional content may be needed or that content has become less worthwhile to professionals in the field. For instance, the widespread introduction of new technologies may require new lessons or modules on their most effective implementation. FBI National Academy personnel and instructors strive to remain up to date on practices in the field, and input from practitioners on the ground is an essential part of that process. Furthermore, subsequent experiences of participants may suggest to them ways in which the information could be more effectively conveyed (e.g., by providing critical incidents covering actual events.) Finally, post-Academy experience may reveal to graduates local or regional variations in practices that could be effectively transmitted during training. Failure to conduct follow-up evaluations will severely hamper FBI National Academy personnel in their efforts to ensure that the program provided is current, comprehensive, and maximally effective in its implementation.

The evaluations will be asked to be completed every other year. In order to have data that is meaningful, it must be collected every other year to establish a baseline and comparison for subsequent years to be able to adequately measure progress and trends.

7. Special Circumstances

(a) Respondents will not be required to report information more than quarterly. (b) Although open-ended responses will be included in the questionnaire, most questions are in multiple-choice format. Those questions that do require a written response are intended to elicit brief comments or recommendations related to the FBI National Academy program content. We will be asking graduates to respond in a period of two weeks or less (which could be extended), but feel the nature of the written responses is such that this will not inflict undue burden. This is especially true since all input will be gathered via the Internet. (c) Respondents will not be required to submit documentation of any kind with their completed questionnaires. (d) Respondents will not be required to retain records of any kind as part of this data collection. (e) All graduates of the FBI National Academy from six months to two and a half years prior to the date of the questionnaire and their identified supervisors, approximately 4,000 law enforcement professionals, will be questioned. Given the comprehensiveness of this coverage, the results will be generalized to the cadre of recent participants. (f) No statistical data classification is required. (g) The proposed questionnaire does not include a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use. (h) Respondents are not required to submit proprietary trade secrets or other confidential information.

8. Outside Consultation

Human Resources Research Organization (HumRRO)

66 Canal Center Plaza, Suite 400

Alexandria, VA 22314

(703) 706-5681

9. Payment to Respondents

No payment or gifts will be offered to respondents

10. Assurance of Confidentiality

All respondents will be assured of confidentiality under Public Law 93-573 (Privacy Act of 1974). They will be assured that individual data will not be released, and that only aggregate data will be reported. Limited demographic information will be collected, allowing no possibility of identifying individual respondents.

11. Justification for Sensitive Questions

This data collection includes no questions of a sensitive nature.

12. Estimates of Hour Burden

Based on the target population we estimate the maximum response time for the questionnaire to be 30 minutes. With a sample size of 2,300 and a 70% response rate, we anticipate the final database will include 1610 respondents. Thus, the total burden for the data collection will be 2,088 hours. Wage rate data was obtained from the Department of Labor.¹ Because respondents occupy a variety of positions within law enforcement agencies, we took the average of the upper limit salaries from the following classifications to represent wage rate: detectives/criminal investigators, police and detective supervisors, grade 14/15 FBI special agents, police chiefs, deputy police chiefs, and police captains. The average annual salary came to $93,227. Given a yearly total of 2,088 hours on the job, the average hourly salary comes to $40.04. With a 30 minute response time, the costs associated with this effort come to $20.02 per respondent, or a total of $32,232.

.

13. Estimates of Cost Burden

No equipment, software, systems, or technology will be purchased to support this effort. Completing the both evaluations will have no monetary cost on the respondents.

14. Estimates of Annualized Cost to the Federal Government

This is a one-time data collection. Costs outlined in 12/13 represent the total cost of this project.

15. Reasons for Program Changes

There are not program changes or adjustment reported in Items 13 or 14 of OMB Form 83I.

16. Plans for Tabulation and Publication

The Evaluation Program of the CPSU, CMS, will provide a report summarizing the background, methodology, and results of the effort to the FBI National Academy to internally provide information on program impact and potential enhancements. The report will not be released to the public but maintained in a database for use by FBI employees or FBI contract employees. At this time it is anticipated that the following data summaries will be provided:

Frequencies summarizing the responses to each of the close-ended response items.

Content analyses will also be conducted to form categories of barriers that have prevented application of Academy-gained skills and knowledge, and suggested topics for inclusion in the program.

At this time, it is not anticipated that complex statistical issues will emerge. The time schedule for the project is provided in the table below.

Event	Completion Date
Questionnaire approval by OMB
Questionnaire programming
Initial contact with graduates
Reminder notifications
Completion of data collection
Content analysis
Final questionnaire database
Data analysis
Delivery of draft report
Delivery of final report

17. Request for Approval not to display OMB Expiration Date

The expiration date for OMB approval number will be displayed.

18. Exceptions to the Certification Statement

No exceptions to the certification statement are requested.

1 U.S. Department of Labor (2010-11). Occupational Outlook Handbook. Retrieved February 22, 2012, from http://www.bls.gov/oco/ocos160.htm

8