Response Rate Calculation and Analysis of Non-Response

M EM OR ANDUM
TO:

Brad Wacker, FAA Work Order Monitor
Jeff R. Brister, Interim FAA Work Order Monitor
Warren Randolph
Randolph, FAA-AVP Branch Manager

FROM:

Bonnie Brandreth, Peg Krecker, Tetra Tech

SUBJECT:

General Aviation and Part 135 Activity Survey
Survey:
Response
e Rate Calculation and Analysis of Non
Non-response

DATE:

15 December 2011

REF:

Contract GS-10F
10F-0017K/Order DTFAWA-07-F-00017
00017 Option Year 4 (Modification 0015)
001

All surveys experience some degree of non
non-response. Because non-respondents
respondents may differ from respondents
re
in terms of the variables collected in the survey, the occurrence of non
non-response
response gives rise to concerns about
bias in the survey results. One approach to understanding the relationship is to conduct non-response
non
bias
studies. This memo includes assessments of the potential for both unit and item non
non-response
response bias in the
2010 General Aviation and Part 135 Activity Survey (GA Survey). The goal of the research was to measure,
adjust for, report, and analyze unit and item non
non-response to assess their effects on data quality to inform
end-users
users of the GA Survey. The analysis focuses on the potential for non
non-response
response bias in the key variable
of interest from the survey: estimates of hours flown.

Procedures used to Maximize Survey Response Rates
The best approach to minimize non--response
response is to plan and implement data collection procedures aimed at
achieving high response rates. Listed below are techniques established in the survey research literature that
have been adopted by the 2010 GA Survey
Survey.
•
•
•
•
•
•
•
•
•
•
•

Pre-notify participants
Publicize the survey
Design survey instruments carefully
Manage survey length
Use reminder notes
Provide multiple response opportunities (Internet, mail, telephone)
Monitor survey completion during the data collection period
Establish surveyy importance
Foster survey commitment
Provide survey feedback
Extensive training of interviewers for telephone follow
follow-up

For owners and operators of single aircraft, survey data were collected through two venues—the
venues
Internet and
mailings of the questionnaire.
re. We first sent the owners/operators of sampled aircraft a postcard inviting them
to complete the survey on the Internet. For aircraft that had no response by Internet, we mailed survey

Tetra Tech, Inc.
6410 Enterprise Lane Suite 300, Madison, WI 53719
Tel 608.316.3700 Fax 608.661.5181 www.tetratech.com

questionnaires to owners/operators three times during the field period as well as a reminder/thank you
postcard between the first and second mailings.
A slightly modified data collection procedure was used for respondents who own/operate three or more
aircraft to reduce respondent burden and improve representation of activity among high-end and high-use
aircraft. The form, developed in cooperation with several aircraft operators and aviation associations, allows
an operator to report a summary of activity for a group of aircraft of a similar type instead of requiring the
operator to complete a separate and longer questionnaire for each individual aircraft. Data collection for
multiple-aircraft owners/operators followed the same timing as that for owners/operators of single aircraft and,
like the single-aircraft owners/operators, three survey mailings were conducted as well as a reminder/thank
you letter between the first and second mailing. To maximize survey response, we placed follow-up telephone
calls to all multiple-aircraft owners/operators who had not responded previously by Internet or Mail. Telephone
staff verify the survey reached the appropriate individual, encourage participation, and offer technical
assistance. Staff will also collect essential data by telephone (e.g., number of aircraft by type, active status,
and hours flown). This alternative data collection track for owners/operators of multiple aircraft accounts for
22.6 percent of all aircraft responding to the survey.
All survey mailings include cover letters explaining the purpose of the survey, how the data will be used, and
how aircraft were selected into the sample. Recipients are assured their responses are confidential and
participation is voluntary. Answers to frequently asked questions as well as a toll-free telephone number and
an email address to contact are provided. The letter is printed on FAA letterhead and signed by the FAA
Administrator. Nine national aviation associations endorse the survey effort and encourage aircraft owners to
participate; their association logos are printed in the footer of the letter. Surveys sent to aircraft based in
1
Alaska include an insert with a statement of support by three Alaska aviation associations.

Unit Response Rate Calculations
The survey population for the 2010 GA Survey includes all civil aviation aircraft registered with the FAA that
are based in the US or US territories and that were in existence and potentially active between January 1 and
December 31, 2010. The Aircraft Registration Master File, maintained by the FAA’s Mike Monroney
Aeronautical Center in Oklahoma City, Oklahoma, serves as the sample frame or list of cases from which a
sample of civil aircraft is selected. The Registry’s list of aircraft as of December 31, 2010 is used to define the
survey population. The Registry, like many sample frames, is an imperfect representation of the survey
population. While it may exclude a small number of aircraft that operate under the FAA regulations governing
the operation of general aviation and on-demand Part 135 aircraft, it also includes aircraft that are not part of
the survey population. Prior to sample selection several steps are taken to identify and remove ineligible
2
aircraft from the sample frame. After excluding aircraft identified as ineligible for the survey 304,334 records
remain, which is 81.4 percent of the Registry as of December 31, 2010.
The 2010 GA Survey sample included 84,982 aircraft. The sample is stratified by aircraft type, FAA region in
which the aircraft is registered, whether the aircraft operates under a FAR Part 135 certificate, and whether
the aircraft was manufactured within the past five years. To support analysis needs, the survey design
includes a 100 percent sample of turbine aircraft, rotorcraft, special light-sport aircraft, aircraft certified to
operate under Part 135, Alaska-based aircraft.
The unweighted unit response rate (RRU) for the 2010 GA Survey is 44.2 percent. The RRU is computed
as the number of completed and partial surveys returned divided by the total number of eligible aircraft in the
sample using the following formula:
RR = (C + P) / (C + P) + (NR + INS + REF + PMR + UNK)

1

Aviation associations that support the survey include the Aircraft Owners and Pilots Association, Experimental Aircraft
Association, General Aviation Manufacturers Association, Helicopter Association International, Light Aircraft
Manufacturers Association, National Agricultural Aviation Association, National Air Transportation Association, National
Business Aviation Association, and Regional Air Cargo Carrier Association. The Alaska Airmen’s Association, Alaska Air
Carriers Association, and the Medallion Foundation are also listed in survey forms sent to Alaska.
2
Appendix A of The General Aviation and Part 135 Activity Survey describes ineligible aircraft on the sample frame.
TETRA TECH

Page 2

Where
RR = Response Rate
C = Completed survey
P = Partial survey
NR = No response
INS = Insufficient complete; a partial survey that is not sufficient to count as a complete
REF = Refused
PMR = Post Master Returned, no new address
UNK = Unknown eligibility
The numerator is comprised of completed surveys and partial surveys that provide enough information to be
used for analysis. Partial surveys must include information on hours flown to be included in the numerator.
For aircraft that are included in the data collection procedures based on the abbreviated survey form, the
number of hours flown is reported as the average across the fleet for up to six aircraft types.
In addition to completed and partial surveys, the denominator includes cases for which no response was
received, insufficiently completed surveys (i.e., no data reported for hours flown), refusals, surveys returned
as undeliverable by the USPS, and cases of unknown eligibility. The last category includes aircraft in which
the owners cannot be identified or cannot report about aircraft activity (e.g., owner is deceased and the
survivors cannot report on the aircraft activity, survey recipient does not own the aircraft listed).
The denominator includes aircraft that were sold or destroyed during the survey year. The survey collects
data on flight activity for the portion of the year the aircraft was eligible to fly, and data collection efforts
attempt to identify and mail surveys to new owners.
The denominator excludes aircraft known not to be part of the general aviation fleet or known not to be
eligible to fly during the survey year. These are aircraft that were destroyed prior to the survey year, displayed
in a museum, operated primarily as an air carrier, operated primarily outside of the US, or exported overseas.
The weighted unit response rate (RRW) for the 2010 GA Survey is 49.5 percent. RRW is computed using
the same basic formula as the unweighted response rate but takes into account the different probabilities of
selection of sample units given the stratified sample design.
Item non-response rates (RRI) is calculated for each item on the GA Survey as the ratio of the number of
respondents for which an eligible response was not obtained to the number of aircraft for which that item was
presented. The number presented for an item is the number of eligible aircraft less the number of aircraft with
a valid skip for the item. When an abbreviated questionnaire is used to obtain data from owners/operators of
multiple aircraft that are part of a large fleet, the eliminated questions are treated as item non-response. The
abbreviated form collects data on key variable for major classes of aircraft (e.g., hours flown, how flown, fuel
consumption, fractional ownership, and number of landings). The form does not collect data on flight
conditions, fuel type, landing gear, or avionics.
Imputation of missing data is very important for stabilizing the estimates of aircraft activity and equipment.
Values are imputed for variables if the survey response is incomplete, the survey form did not include the
question, or the Registry data field is blank. To further reduce the bias introduced by item non-response, a
replacement value is selected from another aircraft in the survey that is similar to the non-respondent aircraft.
For most variables, a nearest-neighbor imputation procedure is used so that missing data are replaced with
values based on an aircraft with otherwise similar characteristics. Data are sorted by aircraft characteristics
and starting values are selected randomly within that sorted sequence.
Table 1 lists the variables for which values are imputed, describes the imputation procedure, and shows the
percentage of cases with imputed data. The table shows rates of imputation among aircraft that received the
full survey form (first column of numbers) as well as rates of imputation for all survey responses, including
TETRA TECH

Page 3

those that returned a short form (last column). It is important to recognize that the latter figures will have
inflated imputation rates: data for many items are structurally missing because the questions were not asked
on the short form.
Table 1. Imputation Rates for Item Non-Response in the 2010 GA Survey

Variable

Hours by use (e.g.,
personal, business
transport)
Fractional ownership
hours
Hours rented/leased *
Public use hours
Hours by flight
plans/flight conditions *
Airframe hours *
Number of landings
Landing gear *
Fuel type *
Fuel burn rate
Avionics equipment *
State primarily flown

Imputation Procedure

Mean values by aircraft type
Nearest neighbor by aircraft type by
engine manufacture model
Nearest neighbor by aircraft type by
engine manufacture model
Nearest neighbor by aircraft type by
engine manufacture model
Mean values by aircraft type
Nearest neighbor by aircraft type by
age
Nearest neighbor by aircraft type by
engine manufacture model by age
Nearest neighbor by aircraft type by
engine manufacture model
Nearest neighbor by aircraft type by
engine manufacture model
Nearest neighbor by aircraft type by
engine manufacture model
Nearest neighbor by aircraft type by
engine manufacture model by age
Assign state of registration from
Registry Master

Percent
Imputed
(full
survey
form only)

Percent
Imputed
(incl. short
form)

1.1

2.0

0.5

0.9

1.9

23.8

1.9

2.6

1.8

23.8

2.5

24.3

3.0

4.3

2.2

24.1

2.4

24.2

2.1

3.7

3.5

29.5

22.9

25.1

Percentages are based on unweighted survey responses (total 37,215).
* Question not asked on the abbreviated survey form administered to owners/operators of
multiple aircraft.

In 2010, rates of imputations are typically less than two percent for sampled aircraft that completed the full
survey form. Item non-response on key activity variables are consistently low—hours flown by use (1.1
percent), fractional ownership hours (0.5 percent), rented or leased hours (1.9 percent), public use hours (1.9
percent), and hours by flight conditions (1.8 percent). Other variables have slightly higher imputation rates but
are still well below four percent (airframe hours, landings, fuel consumptions, and avionics equipment). The
state in which an aircraft is primarily flown is the only variable with markedly higher rates of imputation (23
percent). In fact, data on this variable are seldom missing, but many answers cannot be coded to a single
state because respondents list more than one state, describe a region, or simply indicate “US.”
Over the last 10 years several changes have been made to the survey to reduce item non-response bias. (1)
The layout of the questionnaire was made more user-friendly by increasing font size and space between
questions; (2) existing instructions were simplified and new instructions were added based on pretest
respondent feedback after completing the survey; (3) the confidentiality of survey results has been
emphasized to reduce respondent concerns that data for specific items they report will be used
inappropriately; (4) respondents have been encouraged to report their best guess if they do not have exact
TETRA TECH

Page 4

information; (5) questions were revised to simplify the computations performed by respondents and eliminate
the need for them to refer to previous answers; and (6) instructions to enter a zero, rather than leave an item
blank, has minimized the frequency of ambiguous answers.

Analysis of Potential Non-response Bias
The non-response bias analysis conducted on the 2010 GA Survey data examines the potential for bias in the
estimates of the key survey design variable, number of hours flown in the calendar year, and looks at the
effect of the non-response weighting adjustments that were made to reduce the bias.
Examination of Subgroup Response Rates. While the level of non-response does not necessarily translate
to bias, large differences in the response rates of subgroups serve as indicators that potential biases may
exist. For example, if the response rates for high-use and low-use aircraft were very different, any difference
between the means of the respondents and non-respondents would result in a bias in the estimate of hours
flown. A limitation of this approach is that response rates can only be calculated for those subgroups where
the subgroup characteristics are known for both the respondents and non-respondents. In the GA Survey, this
information is taken from the data on the sampling frame.
Table 2 presents a comparison of response rates by aircraft type. Overall, the analysis does not show notable
differences in response rates across the seven major aircraft types.
Table 2. Response Rates by Aircraft Type (Unweighted)
Aircraft Type

Sample

Completes

Response
Rate

Fixed Wing - Piston
1 engine, 1-3 seats

6,761

3,075

45.5%

1 engine, 4+ seats

15,707

7,086

45.1%

2 engines, 1-6 seats

5,176

2,158

41.7%

2 engines, 7+ seats

2,749

1,090

39.7%

1 engine

4,485

2,046

45.6%

2 engines, 1-12 seats

4,530

1,734

38.3%

2 engines, 13+ seats

1,157

375

32.4%

12,409

5,070

40.9%

5,082

1,630

32.1%

Turbine: 1 engine

5,757

2,605

45.2%

Turbine: Multi-engine

1,640

780

47.6%

Glider

1,746

900

51.5%

Lighter-than-air

2,472

985

39.8%

Amateur

6,495

3,926

60.4%

Exhibition

1,965

902

45.9%

Experimental: Other

1,958

704

36.0%

4,168

2,149

51.6%

84,257

37,215

44.2%

Fixed Wing - Turboprop

Fixed Wing - Turbojet
Rotorcraft
Piston

Other Aircraft

Experimental

Light-sport
Total

TETRA TECH

Page 5

Subgroups within major categories do show a few aircraft types with lower response rates. Among fixed wing
turboprop aircraft with a twin-engine and 13 or more seats, the response rate is lower (32.4 percent). These
aircraft are more often part of a multi-aircraft fleet and certificated to operate Part 135 so it is more difficult to
identify and obtain the cooperation of a knowledgeable respondent for the aircraft. Because this type of
aircraft generally fly more hours than other fixed wing turboprop aircraft, the lower response rate may reflect a
downward bias on the estimates of aircraft activity.
Piston rotorcraft are less likely to respond to the survey than turbine rotorcraft. Piston rotorcraft tend to fly
fewer hours than turbine rotorcraft so the lower response rate likely reflects an upward bias on the estimates
of aircraft activity.
This approach does not account for any sampling weight adjustments. In the GA Survey, the sampling weight
adjusts for variation based on aircraft type, FAA region, age of aircraft, and Part 135 status. Table 3 presents
response rates weighted for sampling and shows the effect of the adjustment is to reduce non-response bias.
However, the response rate for piston rotorcraft does not follow this pattern as the weighted and unweighted
response rates are similar.
Table 3. Response Rates by Aircraft Type (Sample-Weighted)
Aircraft Type

Population

Completes

Response
Rate

Fixed Wing - Piston
1 engine, 1-3 seats

61,423

31,737

51.7%

1 engine, 4+ seats

122,700

63,029

51.4%

2 engines, 1-6 seats

14,571

5,975

41.0%

2 engines, 7+ seats

6,010

2,520

41.9%

1 engine

4,449

2,046

46.0%

2 engines, 1-12 seats

4,571

1,732

37.9%

2 engines, 13+ seats

1,008

377

37.4%

12,088

5,070

41.9%

Piston

4,985

1,630

32.7%

Turbine: 1 engine

5,682

2,605

45.8%

Turbine: Multi-engine

1,610

780

48.4%

Fixed Wing - Turboprop

Fixed Wing - Turbojet
Rotorcraft

Other Aircraft
Glider

3,014

Lighter-than-air

6,238

1,566
2,515

52.0%
40.3%

Experimental
Amateur

35,717

Exhibition

3,025

Experimental: Other

2,176

Light-sport
Total

19,820
1,405
812

55.5%
46.4%
37.3%

10,237

4,632

45.2%

299,505

148,251

49.5%

TETRA TECH

Page 6

Comparison of Sample and Frame Estimates. The second approach for examining the potential for nonresponse bias in statistical estimates based on the GA Survey involves comparing sample estimates from the
responding aircraft to the population values computed from the sampling frame. Clearly, only variables on the
sampling frame can be used in such comparisons. The weights used in this comparison are based on the
probability of selection, with no non-response adjustments. The strength of this approach is that any
differences are due solely to non-response error.
Age of aircraft can be computed for all aircraft based on sample frame information (year of manufacture).
Table 4 presents estimates of mean age of aircraft (in years) for the frame and the sample by aircraft type.
The sample estimates are weighted by the sample weight. This analysis shows little evidence of nonresponse bias as the difference in mean age of aircraft is consistently very small. Of the 18 detailed aircraft
types, 16 of the 18 estimates differ by less than 2 years. Only one estimate differs by more than 4 years
(experimental-other).
Table 4. Mean Age of Aircraft by Aircraft Type: Frame and Survey Estimates
Survey Estimates
Aircraft Type

Sample
Frame

Sample
Weighted

Sample and
Non-response
Weighted

Fixed Wing - Piston
1 engine, 1-3 seats

49.8

51.6

51.4

1 engine, 4+ seats

37.6

38.5

38.2

2 engines, 1-6 seats

39.0

39.0

39.0

2 engines, 7+ seats

37.7

37.1

37.2

1 engine

14.3

14.6

14.2

2 engines, 1-12 seats

28.2

27.5

27.5

2 engines, 13+ seats

23.3

24.6

24.7

Fixed Wing - Turbojet

16.1

13.7

14.4

Piston

23.1

21.8

22.3

Turbine (1 engine)

20.8

20.1

20.5

16.1

16.0

15.9

34.6

35.0

34.9

18.1

16.4

16.7

16.5

14.9

15.2

Fixed Wing - Turboprop

Rotorcraft

Turbine (multi-engine)
Other Aircraft
Glider
Lighter-than-air
Experimental
Amateur
Exhibition

38.3

37.1

37.3

Other
Light-sport

31.0

35.3

32.9

Experimental

8.4

8.3

8.6

Special

3.0

3.0

3.0

34.6

35.4

35.1

Total

TETRA TECH

Page 7

This approach has the same limitation as the response rate analysis presented above by not accounting for
non-response adjustments. If the differences between subgroups are associated with characteristics that are
used in the non-response adjustment process, then this approach does not reflect that fact.
Comparison of Sample-Weight Adjusted and Fully-Adjusted Estimates. The third approach to evaluating
bias in the GA Survey is to compare estimates of hours flown in the calendar year that include the adjustment
for non-response to estimates based on weights that do not have any non-response adjustments (base
weights that take into account only the probability of selection). The main goal of the approach is to examine
the effect of the non-response adjustments on the estimates. Large differences between the sample-weight
adjusted and fully-adjusted estimates may indicate the potential for non-response bias on the key design
variable of the GA Survey.
Table 5 presents a comparison of sample-weight adjusted and fully-adjusted estimates for number of hours
flown by aircraft type. Estimates differ by fewer than 20 hours for 15 of the 18 categories suggesting that
effect of non-response bias is relatively small. The three detailed aircraft types with larger differences in hours
flown between sample-weighted- and fully-weighted-data include single-engine fixed wing turboprops (23.7
hours), fixed wing turbojets (42.6 hours), and single-engine turbine rotorcraft (40.9 hours).
Table 5. Annual Average Hours Flown by Aircraft Type,
Sample-Weighted and Fully-Adjusted Estimates

Aircraft Type

Weighted
for
Sampling

Weighted for
Sampling and
Non-response

Fixed Wing - Piston
1 engine, 1-3 seats

72.2

73.0

1 engine, 4+ seats

91.8

92.4

2 engines, 1-6 seats

108.3

104.6

2 engines, 7+ seats

146.1

137.5

1 engine

281.5

257.8

2 engines, 1-12 seats

238.8

231.6

Fixed Wing - Turboprop

2 engines, 13+ seats

295.1

281.6

Fixed Wing - Turbojet

336.5

293.9

225.5

221.4

Rotorcraft
Piston
Turbine (1 engine)

442.1

401.2

Turbine (multi-engine)

417.6

399.5

Glider

47.7

48.4

Lighter-than-air

23.2

23.4

42.5

42.9

Other Aircraft

Experimental
Amateur
Exhibition
Other

48.2

48.3

127.4

146.0

35.5

35.4

Light-sport
Light-sport - Experimental
Light-sport - Special
Total

TETRA TECH

84.4

83.9

110.0

111.0

Page 8

Comparison with External Data Sources. The fourth approach used to investigate the potential for nonresponse bias is to compare estimates from the GA Survey to estimates from other surveys with similar items.
Large differences may indicate potential bias and the need for further study. However, differences cannot be
solely attributed to non-response bias because there are many other possible sources of the differences. For
example, estimates from the different surveys may not be comparable because of coverage disparities, time
periods that are not the same, differences in question wording, context effects, and a host of other sources of
non-sampling error. Despite these severe limitations, differences in estimates serve to alert users to potential
concerns and may facilitate uncovering important issues.
There are no other survey data with similar measures against which to evaluate estimates from the GA
Survey. As a result, several steps are undertaken annually to review the estimates and benchmark the
results. These steps include review of activity estimates by experts in the FAA and industry that can speak
knowledgeably about the relative activity of aircraft in different regions, among different aircraft categories,
and for different uses (e.g., Part 135, air medical, business transportation). Estimates of selected aircraft
types, such as rotorcraft, are compared with industry data available from associations such as HAI or
manufacturers. Estimates of landings are compared with tower data and flight hours among high-use
categories are examined in light of industry trends—e.g., growth in Part 135 operating organizations, pilot
layoffs among large Part 135 operators during economic downturns.

TETRA TECH

Page 9