Supporting Statement B - VANGARD

Supporting Statement - Part B

Collections of information employing statistical methods

Section B must be completed if the information collected from covered members of the public will be used to statistical purposes.

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used.

Approximately 360 to 400 respondents are sought to be recruited for the Varied Advanced Air Mobility (AAM) Noise and Geographic Area Response Difference (VANGARD) test. Respondents will take the VANGARD test through the NASA Remote Psychoacoustic Testing Platform. Development and the first exercise of the NASA Remote Psychoacoustic Testing Platform is detailed here in Ref. [1].

There are three target geographic locations from which the respondents will be recruited. They are the metropolitan areas of New York City, Los Angeles, and Dallas/Fort Worth. These metropolitan areas were selected based on AAM/Urban Air Mobility (UAM) operations research that determined cities where initial AAM/UAM vehicle operations are most likely to occur [2]. More than three metropolitan areas would be preferred for the VANGARD test, but recruiting respondents from more metropolitan areas would require more respondents to be recruited to adequately cover all metropolitan areas. Having more respondents would require additional resources for test subject recruitment. A minimum of three metropolitan areas was selected as a compromise between having respondents from a variety of metropolitan areas likely to see initial AAM/UAM operations and limiting the number of respondents.

The United States Postal ZIP Codes in each of the three metropolitan areas were categorized as having either “high” ambient noise or “low” ambient noise. Before the ambient noise categorization can be made, the ZIP Codes that comprised the metropolitan areas needed to be determined. In each of these three metropolitan areas, all United States Postal ZIP Codes that were within one degree latitude and one degree longitude (approximately within 69 miles) of commercial airports in the metropolitan area were found. Airports were considered as focus points because initial AAM/UAM operations are likely to have a starting point or at least one destination in a trip be a commercial airport. For New York City, the commercial airports around which to find United States Postal ZIP Codes were LaGuardia Airport, John F. Kennedy International Airport, Newark Liberty International Airport, Westchester County Airport, and Long Island MacArthur Airport. For Los Angeles, the commercial airports considered were Los Angeles International Airport, John Wayne Airport, and Hollywood Burbank Airport. For Dallas/Fort Worth, the commercial airports considered were Dallas Fort Worth International Airport and Dallas Love Field. The L50 value for each ZIP Code was then determined from the United States National Park Service DataStore: https://irma.nps.gov/DataStore/. The L50 values are the A-weighted sound level measurement that is exceeded 50% of the time for a measurement period. After analyzing the distribution of L50 values, 50 dBA was approximately the average L50 value over all ZIP Codes in the metropolitan areas. ZIP Codes that had L50 values greater than or equal to 50 dBA were categorized as “high” ambient, and ZIP Codes that had L50 values less than 50 dBA were categorized as “low” ambient.

The ambient noise categorization of ZIP codes that has been described is not the only method of approximating the ambient noise environment of respondents. Other metrics, like L90, which is the A-weighted sound level measurement that is exceeded 90% of the time for a measurement period, may also be used. However, accessible databases of ambient noise metrics other than L50 for United States ZIP Codes were not able to be acquired for the VANGARD test. The United States National Park Service DataStore, https://irma.nps.gov/DataStore/, was the most accessible resource to the test investigators. Responses to test stimuli from VANGARD respondents will be made publicly available by NASA. Personally identifiable information on the respondents will not be included with responses, but the United States Postal ZIP Code of each respondent will be included with their responses. From these data, other researchers may reanalyze the VANGARD test results using ambient noise metrics that may be accessible to them for each ZIP code.

Approximately 60 respondents from each of the ambient regions (low or high) in each of the three metropolitan areas will be aimed to be recruited for a total of 360 respondents. The goal of 60 respondents from each ambient region of each metropolitan area was selected based on respondent participation and response standard deviation in the previous exercise of the NASA Remote Psychoacoustic Testing Platform, which was called a Feasibility Test, that is described in Ref. [1].

To arrive at the total number of respondents needing to be recruited, the minimum number of respondents needed is first determined. Responses from 26 Feasibility Test respondents answered the research question on whether the online Feasibility Test could replicate results from a previous in-person experiment. The Feasibility Test had its respondents use a continuous 11-point annoyance rating scale. The VANGARD test will use also use an 11-point annoyance rating scale, but a discrete set of 11 points will be used as recommended by the standard from Ref. [3]. A numerical rating of “0” means “Not at all annoyed,” and a numerical rating of “10” means “Extremely Annoyed.” Power analysis was performed to determine the number of VANGARD respondents needed to have a 90% probability (power) of detecting a mean response difference of one numerical annoyance rating between two sound stimuli. Being able to differentiate between one numerical mean annoyance rating between two stimuli will increase the possibility that the sign test and linear regression techniques, which will be discussed in Statement B4, will also be able to determine statistically significant results. To perform the Power analysis, the standard deviation of responses to one stimulus, whose mean annoyance response serves as the null hypothesis, is assumed to have the same value as the average standard deviation of responses to all Feasibility Test stimuli that were determined to have normally distributed responses. The result of the Power analysis is shown in Figure 1. Thirty-four respondents are needed for a power of 0.90. Combining the fact that 26 Feasibility test respondents provided data to obtain statistically significant results, and 34 respondents being the number needed for a power of 0.90, it is preferred that 30 respondents in each ambient region of each metropolitan area are available for analyses.

Figure 1. Power versus sample size for obtaining annoyance response difference of one rating based on standard deviation of Feasibility Test responses.

A reason for aiming for results from at least 30 respondents in each ambient region of each metropolitan area is to perform separate analyses from the main research objectives in Statement A1. The separate analyses are to “de-couple” the potential geographic differences in the data that are not simply due to ambient noise differences. Comparisons would need to be made to verify that there are no significant differences between metropolitan areas. For example, a categorical linear regression of mean annoyance responses to a certain metric, like A-Weighted Sound Exposure Level of each test stimulus, between respondents in low ambient ZIP Codes of New York City and respondents in low ambient ZIP Codes of Los Angles can be checked to see there is a statistically significant difference in linear regression trend lines between the two groups. If a statistically significant difference is found, it may help explain the variation in the mean annoyance to the A-Weighted Sound Exposure Level of each stimulus. While these separate analyses are not main test objectives, they can help provide context and improved understanding of results for the Statement A1 main test objective.

Variation in audio equipment is another reason why no less than 30 respondents in each ambient region of each metropolitan area should complete the VANGARD test. Feasibility test respondents were restricted to wearing over-the-ear headphones. While VANGARD respondents will be encouraged to wear over-the-ear headphones, to allow respondents without access to over-the-ear headphones to take the test, VANGARD respondents will be allowed to use other audio equipment. More varied audio equipment in the VANGARD test may cause more variation in responses compared to the Feasibility Test. Therefore, aiming for results from less than 30 respondents in each ambient region of each metropolitan area may be insufficient for obtaining statistically significant results.

To arrive at the total number of respondents needing to be recruited, participation rate is next taken into consideration. In the Feasibility Test, 86 of the 146 recruited respondents, which is 58% of respondents, set up accounts to receive the test application, although only 48 respondents fully completed the test, which gives a 33% completion rate. Feasibility test respondents did not receive monetary compensation. From this experience, it is estimated that 50% of VANGARD respondents will create test accounts. With monetary incentives, it is assumed that most of the VANGARD respondents who create NASA Launchpad Guest accounts will complete the VANGARD test. A test completion rate of 50% is assumed instead of 58% (the percentage of accounts setup for the Feasibility Test) to allow for an incorrect assumption that all respondents who set up accounts to take the test will complete the test. Assuming a 50% test completion rate, 60 respondents will need to be recruited from each ambient region of each metropolitan area for a total of 360 respondents. The test subject recruitment contractor has rounded the total number of respondents to recruit to be 400 to add an additional cushion of 40 respondents in case the test completion rate is less than 50%.

2. Describe the procedures for the collection of information.

Annoyance responses to stimuli will be collected electronically as described in the document, “TestApplicationProcedure_VANGARD.pdf.” Responses will be returned to a NASA cloud service (the NASA Amazon Web Services) as they are entered by test subjects through the remote test application, which is accessed by test subjects on web browsers from their computers. Test subject response data from the NASA cloud service will be transferred to other NASA-managed computers for analyses after the VANGARD test finishes. Data will not be retained on the NASA cloud service.

Before the main portion of the test, respondents are requested to provide the make and model of the computer they are using to take the test and the make and model of the audio device that they are using. Respondents have the option to indicate “I don’t know” if they are unable or unwilling to provide the make and model of their devices. They are also asked to perform a manual sound calibration where they adjust the volume on their computer and provide the test application the final volume level.

3. Describe methods to maximize response rates and to deal with issues of non-response.

As described in Statement B1, there is an aim of recruiting at least 360 respondents for the VANGARD test. In case the number of respondents who take and complete the test is less than 50%, the test subject recruitment contractor is aiming to recruit 400 respondents from identified low and high ambient noise ZIP Codes of New York City, Los Angeles, and Dallas/Fort Worth to take the VANGARD test using the NASA Remote Psychoacoustic Test Application. As explained in more detail in Statement B1, this recruitment goal should allow for sufficient test results if approximately 50% or respondents take and complete the test.

Respondents for the previous Feasibility Test were not compensated monetarily. Respondents for the VANGARD test will be compensated with a $25 gift card if they fully complete the VANGARD test. The monetary incentive to completing the test should help with issues of non-response.

4. Describe any tests of procedures or methods to be undertaken.

As described in Statement A16, Research Question 1, given in Statement A1, will be answered using the Sign Test method when analyzing test results. Research Question 1 is regarding whether respondents in high and low ambient noise environments respond differently to AAM/UAM noise. The Sign Test is described in Ref. [1] The Sign Test can compare the median annoyance between two groups of respondents. The Sign Test is nonparametric, and hence, it does not require any distributional assumptions on the annoyance responses. To answer Research Question 1 in the VANGARD test, the two groups will be the responses to test stimuli from test subjects in the “high” and “low” ambient environments. If the p-value obtained from the Sign Test is less than a significance level of 0.05, then test results will provide evidence that respondents in “high” ambient environments respond differently to AAM/UAM noise stimuli than respondents in “low” ambient environments.

Research Question 2, given in Statement A1, will also be answered using the Sign Test method. Research Question 2 is regarding whether there is an annoyance response difference between different phases of AAM/UAM aircraft flight. When using the Sign Test for Research Question 2 in the VANGARD test, the two groups will be the responses to test stimuli representing takeoff operations and the responses to test stimuli representing landing operations for a particular AAM/UAM aircraft. If the p-value obtained from the Sign Test is less than a significance level of 0.05, then test results will provide evidence that respondents respond differently to takeoff and landing operations for that aircraft. This comparison between landing and takeoff operations will be repeated for each AAM/UAM aircraft for which both landing and takeoff test sound stimuli exist. Another comparison between the annoyance responses to all landing sound stimuli and the annoyance responses to all takeoff stimuli may will also be performed.

As mentioned in Statement A16, a t-test and Analysis of Variance (ANOVA) may also be used to answer Research Questions 1 and 2 if it can be shown that annoyance responses of two groups of test subjects are jointly Gaussian distributed. The 1954 method by H. Chernoff and E.L. Lehmann is one method to determine if responses are Gaussian distributed [4].

As described in Statement A16, Research Question 3, given in Statement A1, will be answered using linear regression. Research Question 3 is regarding how annoyance changes with distance to the aircraft. Linear regression can analyze the mean annoyance response to different stimuli against a range of metric values for the stimuli. For Research Question 3, the metric will be distance to aircraft, which is approximated by gaining the sound level of an aircraft sound stimulus to different levels. The coefficient of determination from the linear regression indicates how well mean annoyance linearly varies with distance to aircraft, and the slope of the trend line indicates how much annoyance changes with distance. If a nonlinear relationship between mean annoyance and distance is observed, then a nonlinear trend fit of mean annoyance to aircraft distance will be attempted.

Research Question 4, given in Statement A1, will also be answered using the linear regression. Research Question 4 is regarding how well objective parameters like sound quality metrics and sound exposure level describe the annoyance response to AAM/UAM noise stimuli. For Research Question 4, separate linear regressions will be run on between the mean annoyance responses and each objective parameter of interest. The coefficient of determination from the linear regression will indicate how well mean annoyance linearly varies with each objective parameter, which is an indication of the parameter’s descriptive capability towards the mean annoyance. If a nonlinear relationship between mean annoyance and objective parameters are observed, then nonlinear trend fits of mean annoyance to the objective parameters will be attempted.

As stated in Statement A1, correlation coefficients will be used to answer Research Question 5 on how noise sensitivity affects annoyance. There are 26 noise sensitivity questions that appear in post-test survey, and they are given in “PostTestQuestions.pdf.” Except for noise sensitivity Question 5, all noise sensitivity questions ask respondents to provide a numerical rating. Noise sensitivity Questions 1-4 are exploratory. For each of the noise sensitivity Questions 1-4, a correlation coefficient will be found between the numerical ratings from all respondents to the noise sensitivity question and the mean annoyance responses over all test sounds from all respondents. Correlation coefficients closer to 1 or -1 indicate that answers to these four noise sensitivity questions may be a good predictor of the annoyance response. Questions 6-26 comprise the Weinstein Noise Sensitivity Scale [5]. The numerical ratings to questions 6-26 are summed according to the method given in Ref. [5]. A correlation coefficient will be found between the summed score for Questions 6-26 for all the respondents and the mean annoyance responses over all test sounds from all respondents.

Noise sensitivity Question 5 is also an exploratory question with a “Yes” or “No” response. The Sign Test will determine if the median annoyance response to test sounds is significantly different between respondents who answered “Yes” and those who answered “No” to Question 5.

Self-reported computer make and model, audio device make and model, and computer volume level provided by respondents during the manual sound calibration may be used in subsequent analyses to better understand test responses. For example, if there are respondents who appear to have outlier responses, and if those respondents provided their audio device make and model, the spectral characteristics of the audio devices may be investigated to understand if they can explain the outlier responses. If investigators can obtain copies of the devices used by a respondent, they may be able to measure the audio output using the self-reported volume level. The performance of the manual sound calibration method can be assessed by comparing the audio output level to an intended level. These subsequent analyses using self-reported device and volume information are not primary VANGARD test objectives. They are expected to be performed after and separate from analyzing test results to answer Research Questions 1-5 from Statement A1.

The VANGARD test, by itself, is not designed to address how much respondent computer variation, audio device variation, and manual sound calibration affect the variation in response to test stimuli. Subsequent testing, which is not the focus of this information collection, is being planned with a smaller number of respondents who are provided kiosk computers with known and unmodifiable audio settings. Response variation among respondents who take the test on kiosks will be compared to response variation among respondents who take the test using their own computer and audio devices.

5. Provide the name and email, or telephone number, of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Durand Begault (durand.r.begault@nasa.gov, 650-604-3920), NASA Ames Research Center, Human Systems Integration Division, was consulted on test design, stimuli generation, and how to properly divide respondents among geographic locations.

Nathan Cruz (nathan.b.cruze@nasa.gov, 757-864-5388), NASA Langley Research Center, Engineering Integration Branch, was consulted to understand how to recruit respondents for the VANGARD test so that statistically significant results can be obtained.

Tyler Tracy (tyler.d.tracy@nasa.gov, 757-864-6072), NASA Langley Research Center, Applied Acoustics Branch, helped determine United States ZIP Codes in the metropolitan areas of interest for the VANGARD test.

Analyses of test subject responses will be performed by the principal investigator, Siddhartha Krishnamurthy (siddhartha.krishnamurthy@nasa.gov, 757-864-7656), NASA Langley Research Center, Applied Acoustics Branch.

References

[1]	S. Krishnamurthy, S. Rizzi, R. Biziorek, J. Czech, J. Berg, D. Tannler, D. Bean, A. Ayrapetyan, J. Wivagg and J. Wivagg, "Remotely Administered Psychoacoustic Test for sUAS Noise to Gauge Feasibility of Remote UAM Noise Study," Advances and Current Practices in Mobility, vol. 6, no. 2, pp. 986-999, 2024.
[2]	A. L. Garrow, J. B. German and E. C. Leonard, "Urban air mobility: A comprehensive review and comparative analysis with autonomous and electric ground transportation for informing future research," Transportation Research Part C: Emerging Technologies, vol. 132, p. 103377, 2021.
[3]	Standardization, International Organization for, "Technical Specification, ISO/TS 15666, Second Edition, Acoustics - Assessment of noise annoyance by means of social and socio-acoustic surveys," International Organization for Standarization (ISO) 2021, 2021.
[4]	M. H. DeGroot and M. J. Schervish, "Testing Whether a Distribution Is Normal," in Probability and Statistics, D. Lynch, Ed., Pearson Education, Inc., 2012, pp. 636 - 637.
[5]	G. A. Luz, "Noise Sensitivity Rating of Individuals," Sound and Vibration, vol. 116, no. 6, pp. 3471-3479, 2005.