2120-0060 2018 part B final V3

GENERAL AVIATION AND PART 135 ACTIVITY SURVEY

2120-0060

SUPPORTING STATEMENT FOR 2021 – 2023 SURVEY CYCLE

B. COLLECTION OF INFORMATION EMPLOYING STATISITCAL METHODS

1. Respondent Universe and Sample

Universe and Sample Size

The respondent universe is comprised of the Civil Aviation Registry. As of November 2020, it consists of owners-of-record of approximately 340,000 general aviation aircraft in the U.S. as well as Puerto Rico, Virgin Islands, America Samoa, and Guam. Also included are the U.S. owned territories of Palau, Micronesia, the Northern Mariana Islands, and the Marshall Islands. The source of aircraft owners’ addresses is the Aircraft Registration Master File. The survey is based on a probability sample of general aviation aircraft. The basic design and survey procedures remain the same as in previous surveys. Appendix A contains a detailed description of the sample frame and methodology.

The survey sample is stratified by 15 aircraft types,¹ nine FAA regions, whether the aircraft operates under a Part 135 certificate, and whether the aircraft was manufactured within the past 5 years. These strata are used to optimize the efficiency of the sample – classifying aircraft into groups with similar flying characteristics and conditions – and attain high levels precision for estimating hours flown, which is the key design variable.

Several types of aircraft are sampled at a rate of 1.0 (i.e., 100 percent sample). These include Alaska-based aircraft, rotorcraft, turbine-powered aircraft, FAR Part 135 aircraft, aircraft manufactured within the previous five years, and special light-sport aircraft. The survey sample includes all aircraft in these categories so that statistical estimates of flight activity will meet desired levels of precision. Underreporting among Alaska-based aircraft, rotorcraft, and turbine-powered aircraft make it necessary to sample these aircraft at 100 percent. The survey samples FAR Part 135 aircraft and recently manufactured aircraft at 100 percent to support the FAA’s need to understand changes in the fleet and activity among aircraft with high levels of activity. By sampling these aircraft at 100 percent, the FAA can evaluate whether new safety technologies provided in newer and high-end aircraft mitigate risk.

Aircraft not included in one of the sample strata described above comprise the remainder of the survey sample. They are selected randomly based on sampling fractions defined for each cell in the sample design matrix, with sampling fractions set to reach desired levels of precision for estimates of hours flown within the overall constraint of the maximum sample size.

Response Rates

The table below shows the response rates since 1993.

Year	Sample Size	Respondents	Response Rate%	PO returns*	Adj. Response Rate%**
1993	29,760	18,659	62.3	1,367 (4.6%)	65.9
1994	29,599	17,989	60.8	3,206 (10.8%)	68.2
1995	29,776	16,484	55.3	2,246 (7.5%)	59.9
1996	29,952	19,362	64.6	1,641 (5.8%)	68.4
1997	29,954	18,268	60.9	1,843 (6.2%)	65.0
1998	30,114	18,342	60.9	1,593 (5.3%)	64.3
1999***	30,064	15,553	52.5	1,480 (4.9%	54.4
2000	31,039	16,044	52.5	1,459 (4.7%)	54.2
2001	30,886	16,432	53.9	1251 (4.1%)	55.4
2002	30,817	15,254	51.7	1,277 (4.1%)	51.6
2003****	31,996	14,471	46.6	1,354 (4.2%)	47.2
2004	75,061	32,056	42.7%
2005	76,902	34,248	44.5%
2006	83,892	38,973	46.5%
2007	83,530	38,920	46.6%
2008	81,708	37,269	45.6%
2009+	84,411	36,222	42.9
2010	84,257	37,215	44.2%
2011*****	*****	*****	*****
2012	85,403	37,119	43.8%
2013	85.490	37,060	43.7%
2014	84,846	36,630	43.4%
2015	84,964	36,491	43.4%
2016	84,168	36,002	42.8%
2017	84,201	35,200	41%
2018	84,801	36,800	43%
2019	84,210	35,220	41.4%

* Questionnaires returned to the Post Office because they were not deliverable.

After 2003, PO returns were no longer used to re-calculate an adjusted response rate.

** Response rate after adjusting for the original sample size for Post Office returns.

*** In 1999 the method of calculating response rate was changed to include only valid

responses rather than forms returned.

**** Beginning in 2003, the sample frame included registry records listed as Postmaster return

for 10 years of less. Removing Postmaster returns from the 2003 sample frame yields a

response rate of 50.8%.

***** Unreliable response results

⁺ Slight decrease in the response rate is the result of a shortened field period due to

contracting

The response rate is calculated conservatively following guidelines published by the American Association for Public Opinion Research (AAPOR), a professional association that establishes standards, “best practice” guidelines, and a code of ethics for professional survey researchers and research firms.² Specifically, the response rate is computed as the number of completed and partial surveys returned divided by the total number of eligible aircraft in the sample using the following formula.

RR = (C + P) / (C + P) + (NR + INS + REF + PMR + UNK)

Where:

RR = Response Rate

C = Completed survey

P = Partial survey

NR = No response

INS = Insufficient complete; a partial survey that is not sufficient to count

as a complete

REF = Refused

PMR = Post Master Returned, no new address

UNK = Unknown eligibility

The numerator is comprised of completed surveys and partial surveys that provide enough information to be used for analysis. At a minimum, respondents must provide information on hours flown to be included in the numerator.

In addition to completed and partial surveys, the denominator includes cases for which no response was received, insufficient survey returns (i.e., no data reported for hours flown), refusals, surveys returned as undeliverable by the USPS, and cases of unknown eligibility. The last category includes aircraft in which the owners cannot be identified or cannot report (e.g., owner is deceased and the survivors cannot report on the aircraft activity, survey recipient does not own the aircraft listed).

The denominator includes aircraft that were sold or destroyed during the survey year. The survey collects data on flight activity for the portion of the year the aircraft was eligible to fly, and data collection efforts attempt to identify and mail surveys to new owners.

The denominator excludes aircraft known not to be part of the general aviation fleet or known not to be eligible to fly during the survey year. These are aircraft that were destroyed prior to the survey year, displayed in a museum, operated primarily as an air carrier or air cargo carrier, registered outside the US, exported overseas, or owned and operated by the military.

2. Information Collection Procedures

Methodology for Stratification and Sample Selection

The survey will sample 100 percent of Alaska-based aircraft, rotorcraft, turbine-powered aircraft, FAR Part 135 aircraft, and special light-sport aircraft. The remaining survey sample design is a stratified, random sample. The sample is selected from a three-way stratified matrix. The three stratification criteria are: region of aircraft registration, “aircraft type” designating 15 categories, and those operating under FAR Part 135, and whether the aircraft was manufactured in the past 5 years.

Since the FAA’s primary requirement is for annual hours flown, optimal determination of sample size is based on flight time variation by state and aircraft type, and a sampling fraction is determined for each cell with a no-zero population. Sample units are selected randomly within each stratum.

Estimation Procedure

Each aircraft in the sample is given a weight which is the inverse of the cell’s sampling fraction, and which corresponds to the number of aircraft in the sample frame represented by that aircraft. When all responses to the survey are tallied, each weight is adjusted according to the response rate for the aircraft cell, counting an aircraft for which no survey questions are answered as a non-respondent and an aircraft for which at least one question was answered as a respondent. The method of weight adjustment actually incorporates the response rates into the final weight and simplifies estimation procedures.

Data from completed surveys are weighted to reflect population characteristics. The weights reflect the proportion of aircraft sampled from the population in each sample strata and differential response as well as a small adjustment for aircraft that are not part of the survey population.

Initially, each aircraft for which we receive a completed survey is given a weight that reflects sampling fraction and differential response. That is:

WEIGHT = (Population N_ijkl/Sample N_ijkl) * (N Respondents_ijkl/Sample N_ijkl)

where i, j, k, and l represent the four sample strata of aircraft type, FAA region, Part 135 status, and whether an aircraft was manufactured in the past 5 years.

The weight is subsequently adjusted to reflect new information about non-general aviation aircraft. That is, survey responses that identify an aircraft as not being part of the survey population are used to remove aircraft proportionally from the sample and from the population. This adjustment is done at the level of the 14 aircraft types. The procedure assumes that non-GA aircraft occur in the same proportion among survey respondents and non-respondents. To the extent that non-GA aircraft are less likely to receive and complete a survey, this approach will underestimate the adjustment for aircraft that are not part of the general aviation population.

Statistical estimates of fleet size and activity are obtained by summing weighted counts across the desired categories (e.g., by aircraft type, region, type of use) for characteristics or activities of interest (e.g., public use, fractional ownership, landing gear, flight conditions). Percent standard errors are computed from unweighted survey counts to ensure that levels of precision are not overestimated.

Degree of Accuracy

The survey is designed to provide estimates of annual flight hours at the region level and by 15 different aircraft types with percent standard errors less than 20%. Beginning with the 2004 survey, aircraft in several key groups were sampled at 100%: Part 135 aircraft, Alaska-based aircraft, rotorcraft, and turbine aircraft. In subsequent years, newly manufactured, and light-sport aircraft were added to the 100% sample. In 2008, the 100% sample of light-sport aircraft was limited to special light-sport aircraft. Other light-sport aircraft are sampled at lower rates. The remaining aircraft are sampled in proportions required to achieve precisions for hour estimates for each region and for each of the 15 aircraft types.

To facilitate the comparison of estimates and error, tables of standard errors for all estimated quantities are printed in the survey report.

The survey sample is designed to maximize the precision with which we can estimate flight activity (hours flown) for general aviation aircraft. As described above in section B.1, the sample is stratified by aircraft type, FAA region, Part 135 status, and recently manufactured status. Several aircraft are sampled at rates of 1.0 to ensure desired levels of precision. Aircraft that are sampled at rates of less than 1.0 are subject to random selection based on sampling fractions defined for each cell in the sample design matrix. Average annual flight hours is the primary measure needed by the FAA to address survey goals. Sample fractions for each sample strata are defined to optimize sample size to obtain a desired level of precision for an estimate of flight activity within the constraint of maintaining a total sample size of approximately 85,000. Data from the previous survey year on average hours flown, variability in hours flown by region and aircraft type, and response rates are used to set precision levels and identify the optimal sample size for each strata.

Using a 95 percent confidence interval, this can be written as³

N = (1.96)² * (Estimate of Standard Error of Hours Flown)² / (Estimate Average Hours Flown * Desired Precision)²

where data from the previous survey year are used to estimate the standard error and average hours flown for each strata.

The result of this calculation (N) is further adjusted upwards after accounting for nonresponse and bad address information. Data from the previous survey year are used to estimate nonresponse for each aircraft type. Data from the current survey year sample information are used to estimate the number of records with incorrect or undeliverable addresses.

To facilitate the comparison of estimates, the survey report publishes percent standard errors for all estimated quantities. The “percent standard error” is the ratio of the standard error to its estimate multiplied by 100. Reporting percent standard errors makes it possible to compare the precision of estimates from year to year and across categories.

More information on the accuracy of estimates and efforts to minimize error in the survey is provided in Appendix A.

Unusual Problems Requiring Specialized Sampling Procedures

Additional procedures are required to select the 100 percent samples described in B.1. The 100 percent samples of Alaska-based aircraft, rotorcraft, turbine-powered aircraft, light-sport aircraft, and aircraft manufactured within the past 5 years are identified by using information from the Civil Aviation Registry. The 100 percent sample of aircraft certificated to fly Part 135 is selected by drawing upon information from the FAA’s Operations Specifications Subsystem database which is merged with the Registry by N Number.

Frequency of the Survey

This survey is conducted on an annual basis to meet the minimum data currency requirements.

3. Response Rate Improvement Efforts

Current activities to improve the GA Survey response rate and minimize bias due to nonresponse fall under four categories: improving the sample frame, designing and implementing a stratified sample, employing data collection methodologies to maximize response rate, and performing data imputations. Each of these activities is described below.

The Sample Frame and Defining the GA Universe

The contractor works with the FAA and with staff at the Civil Aviation Registry to improve their understanding of the Registry, identify, and remove non-general aviation aircraft prior to selecting the sample for the GA Survey. These activities also include working with supplemental databases to identify aircraft that fly Part 121, aircraft that have been exported overseas, destroyed prior to the survey year, or displayed in a museum. These efforts help to minimize bias by eliminating, to the extent possible, non-GA aircraft from the sample. If these activities were not performed, the survey likely would have a much lower response rate, many responses would reflect the activity of non-GA aircraft, and the extent of nonresponse bias might be estimated to be much greater.

Stratified Sample Design

The contractor implements a sample stratified by FAA region, aircraft type, on-demand Part 135 status, and recent year of manufacture. These strata are defined to identify aircraft with similar patterns of aviation activity, thus minimizing sampling error and support the quality of weighting adjustments for differential response rates. Survey weights, constructed by sample strata, adjust counts for varying rates of sample selection, differential response, and reported non-GA activity help to minimize the impact of nonresponse. By contrast, if the sample design was not stratified, it would assume that all aircraft had roughly similar aviation activity, and survey weights would treat high-end/high-use aircraft flying over 2,000 hours a year in the same way as an aircraft that flew 100 hours. Weighting procedures and steps taken to adjust the sample and the population counts for non-GA activity are discussed in Appendix A.

Maximize Response Rate

Efforts are taken throughout the field period to maximize the response rate. These efforts include data collection protocols, use of supplemental data sources, and outreach with stakeholder organizations to encourage participation. Specifically:

• Two modes of administration are offered to facilitate access to the survey ­– a postcard invitation to complete the survey on the Internet followed by a mail survey to be completed by pen or pencil.

• The survey is mailed three times periodically over the field period to individuals who have not yet responded. A reminder/thank-you postcard is also sent.

• Cover letters accompanying each survey mailing clearly explain the purpose of the survey as well as the endorsement (organizational logos) of several aviation associations. A special insert, with the endorsement of Alaska-based aviation associations, is included in all surveys mailed to Alaska in an effort to encourage participation among this population, which has traditionally had lower response rate than other geographies.

• Cover letters assure owners of the confidentiality of their responses and inform them: “Names of individuals are never associated with responses. There is an identification number on your survey only so [survey contractor] knows who should receive the survey.”

• A dedicated toll-free 800 telephone number and email address is made available to respond to questions, and the Internet survey includes a hyperlink to the email address.

• Collaboration with aviation organizations and industry groups throughout the field period increase awareness of the survey and encourage cooperation of owners or operators of multiple aircraft.

• Telephone and email contact with owners or operators of large fleets of aircraft facilitate reporting and encourage participation.

• Poor contact information is a significant source of nonresponse. To help ensure that mail surveys reach sample members, additional sources are used to update address information and maximize response (e.g., National Change of Address, updates from aviation associations)

• An extended field period allows maximum opportunity to participate.

Imputation for Item Non-Response

The contractor uses procedures to minimize the effects of item non-response on statistical estimates. Respondents must provide valid responses to select survey questions to meet a minimum standard for a completed survey; nevertheless, there is not 100 percent completion of all survey items. Following data processing and data cleaning procedures, they use nearest neighbor substitution methods to impute missing data. Cases with imputed values are flagged so that statistical estimates can be checked for sensitivity to the procedure.

They also work with the FAA and with stakeholders to minimize the need for imputation by regularly reviewing the design and content of the survey. These efforts are directed towards ensuring that respondents can provide accurate and reliable answers to survey questions. The percentage of item nonresponse dropped noticeably following this work.

Non-Response Bias Report

The unit response rate calculation for the 2019 GA Survey is 41.4 percent. A survey response rate is not weighted.  It’s the (Number of completed surveys) / (Number of aircraft sampled) as shown in Table A.9 of Appendix A.

Because the response rate on the GA Survey is less than 80 percent, an analysis of potential non-response bias was conducted to measure, adjust for, report, and analyze unit and item non-response to assess their effects on data quality. The analysis used four different approaches to examine the potential for non-response bias from the 2019 GA Survey: examination of subgroup response rates, comparison of sample and frame estimates, comparison of sample-weighted and fully-weighted estimates, and a comparison with external data sources. No method gave a strong indication that the GA Survey estimates are subject to substantial non-response bias. Some areas of potential bias were identified for piston rotorcraft as this aircraft type had lower response rates but relatively high levels of activity.

4. Test of Survey Procedures

The survey procedures to be used in the years 2021-2023 are similar to those used in the previous surveys. No pretest is needed.

5. Survey Consultation

The survey undergoes annual continuous improvement in order to meet users’ needs. These improvements are the result of consultations primarily with National Transportation Safety Board (NTSB), Aircraft Owners and Pilots Association (AOPA), General Aviation Manufacturers Association (GAMA), National Business Aviation Association (NBAA), Helicopter Association International (HAI), as well as offices and services within FAA.

The survey operations, including survey sample, selection, data collection, data processing, and data analyses and survey report generation will be performed by the winner of that competition.

1 The sample design classifies aircraft into 15 major types, but this does not preclude providing statistical estimates in greater detail. The published tables report for 17 aircraft types, with the finer detail achieved by disaggregated some aircraft types by number of seats.

2 The American Association for Public Opinion Research. 2000. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. Ann Arbor, MI: AAPOR.

3 See Hubert M. Blalock, Jr. Social Statistics, 2^nd edition (1972), New York: McGraw-Hill, page 215.

18