Supporting Statement A-MFO-v3.3.4

Supporting Statement A

Survey of Industry’s Response to Safety Alert for Operators (SAFO) 17007

1. Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection.

The Federal Aviation Administration (FAA), through human factors specialists in the Office of Aviation Safety (AVS), develop regulations, guidance, and procedures to support the certification, production approval, and continued airworthiness of aircraft; and certification of pilots, mechanics, and others in safety-related positions. In the accomplishment of this responsibility, AVS, the Next Generation Air Transportation System (NextGen) Human Factors Division (ANG-C1), and MITRE Corporation have designed a one-time electronic survey to collect anonymous information from operator employees on manual flight operation policies and procedures developed, retained, or updated after the FAA published Safety Alert for Operators (SAFO) 17007 “Manual Flight Operations Proficiency” in 2017. The collection of this information will support important safety functions performed by the FAA. This information will enhance the Agency’s ability to evaluate in-part whether operators:

• sufficiently address manual flying skills (2016 DOT OIG Report);

• evaluate how air carriers have incorporated recommendations from SAFO 17007 into training policies and line-operation policies so pilots can develop and maintain manual flight operation proficiency (2017 FAA SAFO 17007); and

• evaluate strategies used by industry to address causes of increased pilot reliance on automated aircraft systems and the effect of this reliance on manual flight operation proficiency (2019 FAA Working Paper ICAO 40^th General Assembly).

An IATA worldwide survey of pilots found manual flight opportunities were limited by organization culture and policy, and airline policies around use of automation (2020 IATA Aircraft Handling and Manual Flying Skills Report). However, the number of respondents was weighted towards non-North American pilots. From the IATA survey, there were only 224 North American respondents and limited insights in how domestic US operators may have addressed SAFO 17007.

Information learned from this survey will inform the FAA’s decision, in part, to retain or potentially update as appropriate the human factors aspects of 14 CFR Part 121 guidance materials (14 CFR Part 121 Subparts N, O, Y). This information will also inform the development of a planned FAA Flightpath Management (FPM) Advisory Circular (AC), specifically the manual flight operations section.

2. Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.

An anonymous one-time electronic survey will be used to collect information from aviation professionals responsible for the development, maintenance, and use of manual flight operation training policies and line operation policies. All electronic survey information will be collected, protected, deidentified, analyzed, and aggregated by the MITRE Corporation on behalf of the FAA NextGen Human Factors Division (ANG-C1) and Office of Aviation Safety (AVS) Chief Scientific and Technical Advisory (CSTA) for Flight Deck Human Factors. The MITRE Corporation will safeguard all data from improper access, modification, and destruction consistent with FAA standards for confidentiality, privacy, and electronic information. The information collection is designed to yield data that meet all applicable information quality guidelines.

1. Response to the collection of information is voluntary.

2. Entities who are asked to respond include employees of 14 CFR Part 121 (air carriers) and 14 CFR Part 135 (air taxi) operators responsible for the development, oversight, management, and use of company manual flight operation standards, procedures, and policies for training, line operations, and safety.

3. An electronic survey will be used to collect information.

4. The collection of information frequency is one-time.

5. Non-personally identifiable information (PII) will be collected from respondents on operators manual flight operation policies and procedures.

6. Information from this survey will be briefed by the MITRE Corporation to the ANG-C1 Program Manager and AVS Chief Scientific and Technical Advisor for Flight Deck Human Factors. Information from this survey may be used by the FAA to support publicly disseminated information in the form of data to characterize findings or recommendations in a technical report. Any information provided to the public will be presented in aggregate form.

7. Information is needed to evaluate how operators have incorporated safety critical recommendations from SAFO 17007 (Manual Flight Operations Proficiency) into training policies and line operation policies so can pilots develop and maintain manual flight operation proficiency. The FAA will use this data to:

   1. Understand the impact of SAFO 17007 to operator policies and procedures

   2. Identify the current state of manual flight operation policies and procedures

   3. Identify the effect of future technology and procedure changes to current-day opportunities used by Part 121 and Part 135 pilots to develop and maintain manual flight operation proficiency

The information collected will not be disseminated directly to the public; however, results may be used to inform scientific, management, technical or general informational publications.

3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology.

• Invitation to participate in an electronic survey will be delivered via email to Part 121 and Part 135 operators. The email will contain a weblink to direct a respondent to a survey. In lieu of a unique identifier or password, each survey will have a specific URL directed to a participant. All responses are anonymous. No personally identifying information (PII) will be collected or stored.

• The electronic survey is estimated to an average of 20 minutes for each respondent to complete. It is expected that respondents will complete the survey in one session, however incomplete surveys can be saved and modified by the respondent. Once submitted, respondents will no longer have access to their submitted information.

• The electronic survey can be completed online 24/7 for 60 days after an invitation to participate is sent. Up to 3 electronic reminders to complete the survey will be provided to respondents who have not completed the electronic survey after 7 days, 14 days, and 21 days.

• Electronic survey results will be used to inform updates as appropriate to FAA rules, guidance, and other safety related publications on manual flight operations. The information collected will not be disseminated directly to the public; however, results may be used to inform scientific, management, technical or general informational publications in aggregate form.

4. Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Item 2 above.

This is the only data collection effort to solicit feedback regarding industry’s response to SAFO 17007. ANG-C1 and the AVS Chief Scientific and Technical Advisor for Flight Deck Human Factors are the only known parties who would request this information. No other information sources have been identified which would provide the required information. Operator policies and procedures are not publicly shared; therefore, this is the only reliable method to gather anonymous information from a representative industry sample.

5. If the collection of information involves small businesses or other small entities, describe the methods used to minimize burden.

This collection will not involve small business or small entities.

6. Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

Safety Alerts for Operators (SAFOs) are published by the FAA to convey time sensitive information to the aviation community, especially air carriers. It is critical the FAA understand industry’s policies and procedures in response to SAFO 17007 “Manual Flight Operations Proficiency”. Failure to collect this information will limit important safety functions performed by the Agency. If industry’s policies and procedures do not encourage pilots to develop and maintain proficiency in manual flight operations, then pilots may not successfully revert to manual flight when an automated aircraft system fails, or an emergency occurs. This survey critical to the FAA mission and the safety of the flying public.

7. Explain any special circumstances that would cause an information collection to be conducted in a manner:

No special circumstances exist.

8. Provide information on the PRA Federal Register Notice that solicited public comments on the information collection prior to this submission. Summarize the public comments received in response to that notice and describe the actions taken by the agency in response to those comments. Describe the efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or reported.

A Federal Register Notice, published on October, 8, 2020, solicited public comment. No comments were received

The MITRE Principal Investigator (PI) consulted with commercial Pilot Subject Matter Experts (SMEs) who have familiarity with Operations and Standards, Safety Offices, and Training Programs. The pilot SMEs informed the determination the data we are seeking via the SAFO 17007 surveys are not publically available. Pilot SMEs informed the survey questions asked, and reviewed the instructions and surveys.

9. Explain any decisions to provide payments or gifts to respondents, other than remuneration of contractors or grantees.

No payments or gifts will be provided to respondents

10. Describe any assurance of confidentiality provided to respondents and the basis for assurance in statute, regulation, or agency policy.

All data provided will be kept private to the extent possible by law. To preclude the identification of individual responses, all respondents will be given a participant code that does not identify them or their organization. Only the MITRE project leaders will have access to the coding key, which will be destroyed after data analyses are complete. Only analyses and reports of aggregate data will be produced and released.

Free text responses will be analyzed by creating generalization categories of similar comments and then counting the number of textual responses of a similar nature. The results will be aggregated and reported using the counts and general categories. None of the actual free text will be reported, but may be paraphrased in such a way to avoid identification of operator or respondent.

For records management purposes, the FAA will retain a record of all survey responses however there are no plans to utilize the raw data.

11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private.

No questions of a sensitive nature relating to sexual behavior, religious attitudes, or other matters commonly considered private, will be asked.

12. Provide estimates of the hour burden of the collection of information. The statement should:

The survey instruments are presented separately as each of the surveys are designed to reflect the individual’s responsibilities (i.e. Operations & Standards, Safety, or Training) within the Part 121 operators who are responsible for understanding and implementing changes associated with SAFO 17007. Respondents are salaried, but an estimate cost of response is included.

OMB Statement B details how the universe of respondents was established. In summary, 972 potential respondents were identified from a list of Part 121 operators, and a list of Part 135 operators filtered to those operating three or more turbo-jet powered aircraft. Subject matter expert opinion from pilots who have flown for Part 121 or Part 135 operators identified that there may be up two individuals per department that may serve as leadership for the operator’s Operations & Standards, Safety, or Training departments. Thus, 324 respondents are expected for each of the 3 surveys corresponding to the 3 offices (see Part B for details).

Zeroes were included for record keeping and disclosure as the data will not be linked to an individual or an organization and respondents are not expected to keep records. As respondents are assumed to be knowledgeable about their operator policies, SOPs, and roles an average of 20 minutes (0.33 hours) was assumed for the survey response time.

Costs: The cost estimate used in the tables below were obtained from the Bureau of Labor Statistics’ Occupational Outlook Handbook. For the ‘SAFO Survey for Training survey, the median hourly wage of $54.50 was estimated for Training and Development Managers¹. For the surveys on Operations and Standards, and Safety, a median salary of $60.72 ($121,430 divided by 2000 hours = $60.72) was calculated for Commercial Airline Pilots².

Burden Calculation: Per the Department of Health and Human Services³, a factor of 2 of the base wage to calculate full burden. On page 30, HHS states, “As an interim default, while HHS conducts more research, analysts should assume overhead costs (including benefits) are equal to 100 percent of pretax wages…” To isolate the overhead rate, the Department subtracted the benefits rate of 69 percent from the recommended rate of 100 percent.

Table 1: SAFO Survey for Operations and Standards

Summary (Annual numbers)	Reporting	Recordkeeping	Disclosure
# of Respondents	324	0	0
# of Responses per respondent	1	0	0
Time per Response	20 min ( )	0	0
Hourly Wage (1/3)	($60.72 * ) = $20.04	0	0
Hourly Wage (Burdened)	($60.72 * 2 * ) = $40.48	0	0
Total # of responses	324	0	0
Total burden (hours)	(324 * . ) = 108	0	0
Total Unburdened Cost (adjusted)	(324* $20.04) = $6,492.96	0	0
Total Burdened Costs (adjusted)	(324* $40.08) = $12,985.92	0	0

Table 2: SAFO Survey for Safety Office

Summary (Annual numbers)	Reporting	Recordkeeping	Disclosure
# of Respondents	324	0	0
# of Responses per respondent	1	0	0
Time per Response	20 min ( )	0	0
Hourly Wage (1/3)	($60.72 * ) = $20.04	0	0
Hourly Wage (Burdened)	($60.72 * 2 * ) = $40.08	0	0
Total # of responses	324	0	0
Total burden (hours)	(324 * ) = 108	0	0
Total Unburdened Cost (adjusted by 1/3 wage)	(324* $20.04) = $6,492.96	0	0
Total Burdened Costs (adjusted by 1/3 wage)	(324* $40.08) = $12,985.92	0	0

Table 3: SAFO Survey for Training

Summary (Annual numbers)	Reporting	Recordkeeping	Disclosure
# of Respondents	324	0	0
# of Responses per respondent	1	0	0
Time per Response	20 min ( )	0	0
Hourly Wage (1/3)	($54.50 * ) = $17.99	0	0
Hourly Wage (Burdened)	($54.50 * 2 * ) = $35.98	0	0
Total # of responses	324	0	0
Total burden (hours)	(324 * ) = 108	0	0
Total Unburdened Cost (adjusted by 1/3 wage)	(324 * $17.99) = $5,828.76	0	0
Total Burdened Costs (adjusted by 1/3 wage)	(324 * $35.98) = $11,657.52	0	0

Table 4: Total across three surveys.

Summary (Annual numbers)	Reporting	Recordkeeping	Disclosure
Total # of Respondents	972	0	0
# of Responses per respondent	1	0	0
Time per Response	20 min ( )	0	0
Total burden (hours)	(972 * ) = 324	0	0
Total Unburdened Cost (adjusted by 1/3 wage)	($6,492.96 + $6,492.96 + $5,828.76) = $18,814.68	0	0
Total Burdened Costs (adjusted by 1/3 wage)	($12,985.92 + $12,985.92+ $11,657.52) = $37,629.36	0	0

Thus the estimated total burdened cost for the one-time data collection, across the three surveys, is $37,629.36 from 972 respondents reflecting 324 total hours of burden annually. Per respondent, the fully burdened costs range from $35.98 to $40.08 per respondent depending on his/her role in the organization, at an estimated 0.33 hours per response.

13. Provide an estimate for the total annual cost burden to respondents or record keepers resulting from the collection of information.

There are no cost other than those explained in the answer to question 12

14. Provide estimates of annualized costs to the Federal government. Also, provide a description of the method used to estimate cost, which should include quantification of hours, operational expenses (such as equipment, overhead, printing, and support staff), and any other expense that would not have been incurred without this collection of information.

The data collection activity is a piece of a larger research and development project performed by the MITRE Corporation on behalf of the FAA. The FAA provides technical and programmatic oversight of the project regardless if the survey activity is undertaken. The Federal Government incurs personal compensation and benefits (PC&B) whether or not this project is undertaken. Nevertheless, an estimate of government hourly rate is provide to estimate the hours providing feedback and coordinating the survey.

Note: Projected Costs = Hourly Rate ($) x Estimated Hours

Product/Delivery	Hourly Rate	Estimated Hours	Projected Cost
Background and content development	54.111	10	$541.10
Online survey development	0	0	0
Database build and analyses	0	0	0
Online survey distribution, reminders, technical support	0	0	0
Item Report development, presentations	0	0	0
TOTAL		10	$541.10

1 – GSA rates for a GS-13 Step 3 in the Washington D.C. area were used (LINK to OPM Hourly Table). Rate reflects an average of hourly rates of the FAA Program Manager.

Researchers at the MITRE Corporation will develop, conduct, and process results from the survey. It is assumed they will spend no more than 10% of the total project cost on activities related to the survey. The tasks they will perform with full cost estimates as follows:

Product/Delivery	Hourly Rate1	Estimated Hours	Projected Cost
Background and content development	150	160	$24,000.00 $23,200.00
Online survey development	150	20	$3,000.00
Database build and analyses	150	40	$6,000.00
Online survey distribution, reminders, technical support	150	40	$6,000.00
Item Report development, presentations	150	80	$12,000.00 $11,600.00$
TOTAL		340	$51,000.00

1 – An hourly rate was created from the following calculation. One MTS unit, including PC&B is $300,000. $300k divided by 2000 work hours = $150.

Therefore, the total estimated costs to the government for this data collection is $51,000.00.

15. Explain the reasons for any program changes or adjustments.

This is a new collection.

16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.

A summary of the survey results will be made available approximately 6 months after data collection completion.

Raw data from the survey will not be published. However, aggregate responses may be included in a MITRE Technical Report to the FAA or included in scientific papers.

The information collected will not be disseminated directly to the public; however, results may be used to inform scientific, management, technical or general informational publications.

17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons why display would be inappropriate.

Not applicable

18. Explain each exception to the topics of the certification statement identified in “Certification for Paperwork Reduction Act Submissions.”

No exceptions.

1 Source: Bureau of Labor Statistics. “Occupational Outlook Handbook”. September 2020. Training and Development Managers. https://www.bls.gov/ooh/management/training-and-development-managers.htm#tab-1

2 Source: Bureau of Labor Statistics. “Occupational Outlook Handbook”. September 2020. Airline and Commercial Pilots. https://www.bls.gov/ooh/transportation-and-material-moving/airline-and-commercial-pilots.htm

3 Source: U.S. Department of Health and Human Services, “Guidelines for Regulatory Impact Analysis” (2016), https://aspe.hhs.gov/system/files/pdf/242926/HHS_RIAGuidance.pdf.