Part B - statistical methods_final version_December

Enhanced Geo-Targeted 3060-XXXX

Wireless Emergency Alerts December 2019

Part B. Collections of Information Employing Statistical Methods:

1. A description follows:

Geographic Area #1

Preliminary Survey

The respondent universe for the preliminary survey in geographic area #1 is the control group #1, i.e., employees and other members of the Commission’s alerting partner #1 that elect to participate in the test. The Commission estimates this to be 1,000 individuals spread across four providers (Sprint, Verizon, AT&T and T-Mobile), two operating systems (Android and iOS) and four geographic sub-areas within or in proximity of the target geographic polygon, leading to 4 × 2 × 4 = 32 strata.

Sampling estimates for each strata of the control group are provided below:¹

	Potential respondents in universe	Expected response rate	Estimated persons in the sample
Provider 1, Operating System 1, Sub-Area 1	31	100%	31
Provider 2, Operating System 1, Sub-Area 1	31	100%	31
. . .	. . .	. . .	. . .
Provider 4, Operating System 2, Sub-Area 4	31	100%	31
Total:	992	--	992

Live Test Survey

The respondent universe for the live test survey in geographic area #1 consists of the estimated 1,000 control group #1 members (described immediately above) and non-control group members (i.e., all individual members of the public with an eligible mobile device that are located within or around the prescribed alerting area, that have opted-in to receiving the WEA Test category of alerts that receive the Test alert and elect to respond). The Commission estimates the non-control group #1 population to number 5,000 individuals. This is based on the following sub-estimates:

• Total number of individuals within a 0.1-mile overshoot of the geo-targeted area: 530,000

• Percentage of individuals with a mobile device: 70%

• Percentage of devices that are eligible for geo-targeting: 30%

• Percentage of devices affirmatively opted-in to received WEA Test alerts: 15%

• Voluntary response rate to the live test survey: 30%

Sampling estimates for each strata of the non-control group are provided below:²

	Potential respondents in universe	Expected response rate	Estimated persons in the sample
Provider 1, Operating System 1, Sub-Area	5213	30%	156
Provider 2, Operating System 1, Sub-Area 1	521	30%	156
...	…	…	…
Provider 4, Operating System 2, Sub-Area 4	521	30%	156
Total:	16,672	30%	4,992 (rounded)

Thus, sampling estimates for the live test survey in geographic area #1, adding numbers from the control and non-control groups, are:

	Estimated persons in the sample
Provider 1, Operating System 1, Sub-Area	187
Provider 2, Operating System 1, Sub-Area 1	187
...	…
Provider 4, Operating System 2, Sub-Area 4	187
Total:	5,9844

Geographic Area #2

The Commission’s analysis for geographic area #2, involving alerting originator #2 and control group #2, is identical to that of geographic area #1.

2. The Commission will use a number of procedures for its proposed data collection. As noted above, the control group individuals will be asked to complete a preliminary survey delivered via e-mail. This preliminary survey will improve the statistical reliability of information received in the subsequent live test survey which the control group and other members of the public will receive via a hyperlink embedded in a WEA test alert. Control group and non-control respondents will complete these surveys directly from their electronic mobile devices.

The Commission designed strata along three parameters reflecting areas of potential divergence in performance for enhanced geo-targeting performance. The strata and control group, for each of geographic areas #1 and #2, are estimated to encompass four major providers, two major device operating systems and four distinct geographic subareas, leading to 4 × 2 × 4 = 32 strata. The Commission intends to assess (i) the probability of receiving the test message within each target subarea with a 95% confidence interval, (ii) the probability of receiving the test message outside each target subarea with a 95% confidence interval and (iii) any large differences in these probabilities (roughly ±0.10) based on operating system and/or providers at a 0.05 level of significance. These values represent minimal measures of accuracy that the Commission is targeting based on the information collection.

3. The Commission will employ several methods to maximize response rates and deal with potential issues of non-response. First, the Commission will utilize the control group of respondents, described above. The control group consists of employees and other members of the Commission’s alert originator partners and these individuals will be instructed on how to respond to the preliminary test and live test surveys, how to opt-in to receive WEA Test alerts on their mobile devices and will be able to participate as part of their normal job duties to their employer. Control group individuals are thus expected to be reliable respondents to both information collections, even if members of the non-control group provide a low response rate.

Further, the Commission is undertaking efforts to maximize the response rate of the non-control group as well. The Commission will work jointly with its alert originating partners to prepare training materials and conduct public and stakeholder outreach efforts directed at the non-control group, including by issuing a Public Notice announcing the planned live test in each geographic area and encouraging the public to opt-in to State/Local WEA Tests. These efforts will take place in the months preceding each live test.

The proposed information collection is expected to yield reliable data, both overall and for each stratum described above. Based on the assumptions above and its preliminary statistical modeling, the Commission estimates the accuracy and reliability of the information collected for in each geographic area to be:

Based only on the control group responses only (31 respondents per stratum):

• a 95% confidence interval for the probability of receiving the test message within target geographic areas with a margin of error ±0.0381;

• a 95% confidence interval for the probability of receiving the test message outside the target geographic areas with a margin of error ±0.0418; and

• the ability to detect large differences (roughly ±0.10) in the probability of receiving test messages across operating and providers at the 0.05 level of significance.

Based only on responses from the control group and non-control group (187 respondents per stratum):

• a 95% confidence interval for the probability of receiving the test message within target geographic areas with a margin of error ±0.0154;

• a 95% confidence interval for the probability of receiving the test message outside the target geographic areas with a margin of error ± a 95% confidence interval for the probability of receiving the test message outside the target geographic areas with a margin of error ±0.0170; and

• the ability to detect large differences (roughly ±0.05) in the probability of receiving test messages across operating and providers at the 0.05 level of significance.

4. As noted above, the Commission seeks to perform a preliminary test in each geographic area as a means of collecting information to refine its subsequent live test in the area. The preliminary test information collection will allow the Commission to further estimate the number of participants who would make up the control group, their physical locations, the providers involved and their respective proportions, and the types of eligible devices present. The Commission will use this information together with its partnering alert originator to make necessary adjustments to the control group to ensure the results of the subsequent live test minimize burden while still providing reliable results.

5. The following individuals have been consulted on statistical aspects of the design:

• Cha-Chi Fan, Mathematical Statistician, Office of Economics and Analytics, Federal Communications Commission, Tel. (202) 418-1554.

• Jay Bennett, Statistician, Public Safety and Homeland Security Bureau, Federal Communications Commission, Tel. (202) 418-2761.

• John Healy, Supervisory Telecommunications Systems Specialist, Public Safety and Homeland Security Bureau, Federal Communications Commission, Tel. (202) 418-2448.

The Information Technology Center of the FCC will actually collect information. The information will be analyzed by the individuals listed above for the Commission.

1 The above reflects 992 participants, rather than 1,000, due to fractional rounding and the possibility that a small number of respondents may have characteristics that do not fall within one of the 32 strata (e.g., may use an operating system that is neither iOS or Android).

2 The above reflects 4,992 participants, rather than 5,000, due to fractional rounding and the possibility that a small number of respondents may have characteristics that do not fall within one of the 32 strata (e.g., may use an operating system that is neither iOS or Android).

3 This number was determined as follows 530,000 (individuals within or near geo-targeted area) × 0.70 (mobile device users) × 0.30 (eligible devices) × 0.15 (opt-in rate) × 1/32 (single strata) ≈ 521 individuals.

4 The above reflects 5,984 participants, rather than 6,000, due to fractional rounding across the 32 identified strata.

4