Nonresponse Bias Analysis

LEMAS Nonresponse Bias Report
Introduction
We examine two features in this nonresponse bias report. First, we calculate the effect size as a
measure of the difference between the respondents and the population. Additionally, we compare nonresponse weighted and design weighted estimates for a few outcomes to empirically examine the effect
of nonresponse weighting.

Effect Size: Comparing Respondents to the Frame
We compare respondents to the frame on two characteristics – agency size (using data on frame) and
population served within each agency type. We use the categories of agency sized that were used in
strata construction and the population served as documented in the analysis file. Table 1 shows the
Cohen’s effect size for the two variables for each agency type. There were no separate agency size strata
for state agencies, so that analysis was not conducted.
Table 1: Nonresponse Effect Size 1
Agency Type
Local Police
Sheriff
State

Agency Size Effect Size Population Served Effect Size
0.08
0.08
0.07
0.07
N/A
0.03

Generally, an effect size is defined as small if it is less than 0.2. As shown in Table 1, all of the effect
sizes are lower than that threshold, meaning that they are all considered small, Thus, we do not have
evidence of a statistical difference between the respondents and the frame.

1

Detailed calculations are in the attached appendix.

1

Effect of Nonresponse Bias Weighting
When constructing analysis weights, nonresponse adjustment was used. If this adjustment affects
estimates significantly, it can be a sign of nonresponse bias. We compare the estimates for 5 outcomes
for estimates under design weights compared to estimates under analysis weights.
Table 2: Comparison of Estimates Using Design and Analysis Weights
Estimate
Average FTS
% Female Chiefs
Average number
hours required for
training
Operating Budget
Per Sworn Officer
Operating Budget
Per Resident

Weight
Design
Analysis
Design
Analysis
Design
Analysis
Design
Analysis
Design
Analysis

Local Police

41.2
38.2
2.9
2.9
984.5
968.0

1,268,334
1,300,040
5,038
5,331

Sheriff

59.9
57.9
1.0
0.9
895.4
890.5
2,574,389
2,608,018
1,889
1,933

State

1,138.4
1,138.4
6.8
6.8
1479.3
1479.3
201,175
201,175
43
43

There does not appear to be a significant impact on the estimates when using the non-response
adjusted analysis weights compared to the design weights. Thus, there is no evidence of significant
nonresponse bias.

2

Table 1. Nonresponse bias effect size for local police departments, LEMAS 2016

Variable

Agency Size
100+
50-99.5
25-49.5
10-24.5
5-9.5
2-4.5
1-1.5
Population Served
1,000,000 or more
500,000–999,999
250,000–499,999
100,000–249,999
50,000–99,999
25,000–49,999
10,000–24,999
2,500–9,999
2,499 or fewer

Released Eligible
Sample
NUM
PCT

Respondents
NUM

Nonrespondents

PCT

NUM

Diff in Percent (Resp
vs. Nonresp)

Diff in Percent (Resp
vs. Overall)

z(i)

Effect size

PCT

638
132
272
542
500
392
137

24.4
5.1
10.4
20.7
19.1
15.0
5.2

564
108
232
458
402
284
88

26.4
5.1
10.9
21.4
18.8
13.3
4.1

74
24
40
84
98
108
49

15.5
5.0
8.4
17.6
20.5
22.6
10.3

10.9
0.0
2.5
3.8
-1.7
-9.3
-6.2

2.0
0.0
0.5
0.7
-0.3
-1.7
-1.1

0.002
0.000
0.000
0.000
0.000
0.002
0.002

0.08

14
30
55
205
291
204
321
676
817

0.5
1.1
2.1
7.8
11.1
7.8
12.3
25.9
31.3

14
30
50
183
251
174
275
559
600

0.7
1.4
2.3
8.6
11.8
8.1
12.9
26.2
28.1

0
0
5
22
40
30
46
117
217

0.0
0.0
1.0
4.6
8.4
6.3
9.6
24.5
45.5

0.7
1.4
1.3
4.0
3.4
1.9
3.2
1.6
-17.4

0.1
0.3
0.2
0.7
0.6
0.3
0.6
0.3
-3.2

0.000
0.001
0.000
0.001
0.000
0.000
0.000
0.000
0.003

0.08

Table 2. Nonresponse bias effect size for sheriff's departments, LEMAS 2016

Variable

Agency Size
100+
50-99.5
25-49.5
10-24.5
5-9.5
2-4.5
1-1.5
Population Served
1,000,000 or more
500,000–999,999
250,000–499,999
100,000–249,999
50,000–99,999
25,000–49,999
10,000–24,999
2,500–9,999
2,499 or fewer

Released Eligible
Sample
NUM
PCT

Respondents
NUM

Nonrespondents

PCT

NUM

Diff in Percent (Resp
vs. Nonresp)

Diff in Percent (Resp
vs. Overall)

z(i)

Effect size

PCT

358
62
99
159
90
38
4

44.2
7.7
12.2
19.6
11.1
4.7
0.5

271
49
79
106
69
24
2

45.2
8.2
13.2
17.7
11.5
4.0
0.3

87
13
20
53
21
14
2

41.4
6.2
9.5
25.2
10.0
6.7
1.0

3.7
2.0
3.6
-7.6
1.5
-2.7
-0.6

1.0
0.5
0.9
-2.0
0.4
-0.7
-0.2

0.000
0.000
0.001
0.002
0.000
0.001
0.001

0.07

33
60
97
161
83
112
136
100
28

4.1
7.4
12.0
19.9
10.2
13.8
16.8
12.3
3.5

26
43
76
126
67
78
99
68
17

4.3
7.2
12.7
21.0
11.2
13.0
16.5
11.3
2.8

7
17
21
35
16
34
37
32
11

3.3
8.1
10.0
16.7
7.6
16.2
17.6
15.2
5.2

1.0
-0.9
2.7
4.3
3.5
-3.2
-1.1
-3.9
-2.4

0.3
-0.2
0.7
1.1
0.9
-0.8
-0.3
-1.0
-0.6

0.000
0.000
0.000
0.001
0.001
0.000
0.000
0.001
0.001

0.07

Table 3. Nonresponse bias effect size for state police agencies, LEMAS 2016

Variable

Population Served
1,000,000 or more
500,000–999,999
250,000–499,999
100,000–249,999
50,000–99,999
25,000–49,999
10,000–24,999
2,500–9,999
2,499 or fewer

Released Eligible
Sample
NUM
PCT

43
6
0
0
0
0
0
0
0

87.8
12.2
0.0
0.0
0.0
0.0
0.0
0.0
0.0

Respondents
NUM

Nonrespondents

PCT

39
5
0
0
0
0
0
0
0

88.6
11.4
0.0
0.0
0.0
0.0
0.0
0.0
0.0

NUM

Diff in Percent (Resp
vs. Nonresp)

Diff in Percent (Resp
vs. Overall)

z(i)

Effect size

PCT

4
1
0
0
0
0
0
0
0

80.0
20.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0

8.6
-8.6
0.0
0.0
0.0
0.0
0.0
0.0
0.0

0.9
-0.9
0.0
0.0
0.0
0.0
0.0
0.0
0.0

0.000
0.001
0.000
0.000
0.000
0.000
0.000
0.000
0.000

0.03