Survey Design and Instrument Supplement

Survey Design Supplement

Survey design and instrument changes, tests, and studies

The Office of Management and Budget (OMB) and Bureau of Transportation Statistics (BTS) would like FAA to conduct additional pre-testing to ensure the survey methodology is robust and following best practices. The following is an overview of areas for further analysis:

1. Low response rates in pilot surveys;

2. Considering using a sample of registered UAS operators instead of conducting a census

3. Risks of double sampling respondents if operators self-report and are registrant present in both the part 107 registry and the recreational registry

4. Asking operators with multiple UAS questions about flight behavior without conditioning the answer on the type of UAS could lead to misleading data

5. Not differentiating between individuals, organizations, and firms of different sizes and structures could lead to oversampling of one of the groups or challenges with the validity of certain subgroup’s reporting

6. Ensuring that the questions asked might provide useful information for rulemaking

To address these concerns, APO will assess the following changes to the survey:

• Change I: Clear registry databases and update invalid email addresses, which includes removing registrant duplication in both registries (Addresses concern #1)

• Change II: Switch from a census of the registries to a stratified random sample of the registries and stratify the sample geographic area (county) and operator type (see test III) (Concern #2)

• Change III: Separate the questionnaires for recreational & part 107 registrants and condition the self-identity questions on whether the registrant is from the recreational or part 107 registry (Concern #3)

• Change IV: Ask respondent to list their most flown aircraft and their associated flight behavior with that aircraft and the number of aircraft ask to report will be determined by pre-test II (Concern #4)

In addition, APO will fine tune the survey design and instrument by evaluating the following studies:

• Quantitative Test

  • Test I: Test which mode of invitation has the greatest response rate: email, push-to-web mailer, text/phone call, or mailed paper questionnaire (Concern #1)

  • Test II: Test what is the optimal length of questionnaire by varying the number and detail of the questions (Concern #4)

  • Test III: Test whether ex-anti categories developed from registry data accurately predict actual category as revealed in self-identifying questions (Concern #5)

  • Test IV: Test whether using the registries as the sample frame introduces sample bias by taking a convenience sample through an invitation posted on FAA social media and compare those answer to the registry sample (Concern #2)

• Qualitative Studies

  • Study I: Determine is the structure and substance of the questions asked are appropriate by asking respondents to comment on the questionnaire after completion (Concern #4)

  • Study II: Asking respondents to participate in either a focus group or a structured interview about the structure, substance, and flow of questions in the questionnaire (Concern #4&5)

Pending the outcome of pre-testing and the above described studies, FAA may modify the survey instruments and/or methodology. Prior to conducting the full survey, FAA will seek approval from OMB via a non-substantive change request.