Attachment O -- Non-response Bias Analysis

Attachment P -- Non-response Bias Analysis of Private Establishments from the 2014 Medical Expenditures Panel Survey – Insurance Component (MEPS-IC)

Introduction:

When an expected unit response rate is below 80 percent, OMB Standards & Guidelines for Statistical
Surveys recommends conducting a nonresponse bias analysis. Of the 42,055 sample units selected for the 2014 MEPS-IC, 27,226 (64.7%) responded, 11,776 (28.0%) did not respond, and 3,053 (7.3%) were out of sample or out of business. Removing the out of sample and out of business units from the response rate calculation results in an unweighted response rate of 69.8 percent. As shown in the formula below, nonresponse bias is a function of both the nonresponse rate and the difference between the respondent mean and the nonrespondent mean on the variable of interest:

OR

Respondent Mean = Full Sample Mean + (Nonresponse Rate)*(Respondent Mean –Nonrespondent Mean)

In the MEPS-IC we are most concerned about nonresponse bias in our key estimates- the percent of establishments offering health insurance, the percent of employees offered health insurance and the percent of employees enrolled in health insurance, among other important estimates. Unfortunately, since we do not have these estimates for the nonresponding establishments, we cannot directly measure the potential nonresponse bias in these estimates. However, from the sampling frame we have data for both responding and nonresponding establishments that are correlated with, or vary by, many of our key estimates. These variables include the size of the firm the establishment is in (number of employees), the industry group the establishment belongs to and the region of the country where the establishment is located (Census division). This analysis will compare the responding establishments to the nonresponding establishments on these sampling frame variables, using both a chi-square test of independence and a t-test to test differences in means and percentages.

The rest of this memo includes three sections where the differences between responding and nonresponding establishments will be tested and discussed, followed by a discussion of the weighting adjustments for nonresponse bias and a conclusion section.

Firm Size:

Firm size is highly correlated with at least one of our key measures, the percentage of establishments that offer health insurance. In 2014, 25.7 percent of private sector establishments in firms with less than 10 employees offered health insurance and this percentage increased to establishments in firms with 1,000 or more employees where 99.2 percent offered insurance. Table 1 presents the results of a chi-square test of the relationship between firm size and response. The test shows that response to the MEPS-IC is not independent of firm size and this may be a source of nonresponse bias.

To identify which firms size categories are possibly the source of this bias, table 2 shows the percent distribution of responding and nonresponding establishments across the firm size categories and the results of testing the difference in these percentages. The results show those establishments in firms with less than 10 employees, and those with 25 to 99 employees may be a source of nonresponse bias.

Industry Group:

Most of the MEPS-IC key estimates vary by industry group. For example, in 2014 the percent of establishments that offered health insurance to their employees ranged from 23.6 percent for establishments in agriculture, fishing and forestry to 61.8 percent for those in mining and manufacturing. Table 3 presents the results of a chi-square test of the relationship between industry group and response. The test shows that response to the MEPS-IC is not independent of industry category and this may be a source of nonresponse bias.

To identify which industry category is possibly the source of this bias, table 4 shows the percent distribution of responding and nonresponding establishments across the industry categories and the results of testing the difference in these percentages. The results show those establishments in agriculture, fishing and forestry, mining and manufacturing, construction, and professional services may be a source of nonresponse bias.

Census Division:

Many of the MEPS-IC key estimates vary by Census division. For example, in 2013 the percent of employees in establishments that offered health insurance ranged from 82.0 percent for establishments located in West South Central to 88.6 percent for those located in New England. Table 5 presents the results of a chi-square test of the relationship between Census region and response. The test shows that response to the MEPS-IC is not independent of Census division and this may be a source of nonresponse bias.

To identify which Census division is possibly the source of this bias, table 6 shows the percent distribution of responding and nonresponding establishments across divisions and the results of testing the difference in these percentages. The results show those establishments located in 6 of the 9 divisions may be a source of nonresponse bias.

Weighting adjustments for nonresponse bias:

The base sampling weights of the respondents to the MEPS-IC are adjusted so that the respondents also represent the nonrespondents while minimizing the bias associated with nonresponse. The adjustment is made by controlling firm size, establishment size, industry group, type of firm, and state. Thus, a nonresponding establishment is represented by a responding establishment with characteristics similar to the extent possible in terms of these variables. A raking procedure is applied to adjust the weights of the respondents to represent all eligible establishments on the frame (i.e., both respondents and nonrespondents) while controlling for the marginal distributions of all these variables. The raking adjustment is expected to reduce any bias due to nonresponse to the extent the MEPS-IC estimates are associated with the characteristics used in the raking procedure. Since the MEPS-IC estimates are generally highly correlated with these characteristics, the weighting adjustment is expected to minimize the nonresponse bias to a large extent.

Conclusion:

The results of this analysis show that there is the potential for nonresponse bias in the MEPS-IC. Although we never really know the extent of any bias in the survey estimates, since the distributions of responding and nonresponding establishments are close, and since the weighting adjustment takes into account the important variables by which MEPS-IC estimates mostly vary, we can be fairly confident that, to the extent possible, nonresponse bias has been addressed in the MEPS-IC.

		Table 1. Chi-Square of Response by Firm Size, 2014 MEPS-IC
		Firm Size		Responding (N)			Nonresponding (N)				Total
Frequency		Less than 10		2653849			1263299				3917148
Expected				2685550			1231598
Percent				39.96			19.02				58.98
Row Pct				67.75			32.25
Col Pct				58.29			60.5
		10 to 24		550138			240744				790882
				542219			248663
				8.28			3.63				11.91
				69.56			30.44
				12.08			11.53
		25 to 99		377294			146313				523607
				358979			164628
				5.68			2.2				7.88
				72.06			27.94
				8.29			7.01
		100 to 999		298031			138730				436762
				299439			137323
				4.49			2.09				6.58
				68.24			31.76
				6.55			6.64
		1,000 or more		673818			298987				972805
				666944			305861
				10.15			4.5				14.65
				69.27			30.73
				14.8			14.32
		Total		4553130			2088073				6641204
				68.56			31.44				100
		Statistic		DF			Value				Prob
		Chi-Square		4			4776.6482				<.0001
		Cramer's V					0.0268
Table 2. T-test of Response by Firm Size, 2014 MEPS_IC
Firm Size			Responding (%)		Nonresponding (%)				DF	t Value		Pr > |t|
Less than 10			58.31		60.47				22377	-4		<.0001
10 to 24			12.1		11.48				39000	1.76		0.0776
25 to 99			8.28		7.01				23839	4.4		<.0001
100 to 999			6.54		6.65				27276	-0.4		0.6883
1,000 or more			14.77		14.39				22425	0.98		0.328
Total			100		100
	Table 3. Chi-Square of Response by Industry, 2014 MEPS-IC
	Industry					Responding (N)		Nonresponding (N)					Total
Frequency	Agriculture,					118102		63347.5					181450
Expected	Fishing, and Forestry					124400		57050
Percent						1.78		0.95					2.73
Row Pct						65.09		34.91
Col Pct						2.59		3.03
	Mining and					206529		83937.3					290466
	Manufacturing					199140		91326
						3.11		1.26					4.37
						71.1		28.9
						4.54		4.02
	Construction					362858		201415					564273
						386859		177414
						5.46		3.03					8.5
						64.31		35.69
						7.97		9.65
	Utilities and					137421		61984					199405
	Transportation					136710		62695
						2.07		0.93					3
						68.92		31.08
						3.02		2.97
	Wholesale Trade					241713		114359					356072
						244119		111953
						3.64		1.72					5.36
						67.88		32.12
						5.31		5.48
	Financial Services					528676		237594					766270
	and Real Estate					525346		240924
						7.96		3.58					11.54
						68.99		31.01
						11.61		11.38
	Retail Trade					643184		286023					929207
						637053		292154
						9.68		4.31					13.99
						69.22		30.78
						14.13		13.7
	Professional Services					1178993		507821					1686814
						1156460		530354
						17.75		7.65					25.4
						69.89		30.11
						25.89		24.32
	Other					1135655		531592					1667246
						1143044		524202
						17.1		8					25.1
						68.12		31.88
						24.94		25.46
	Total					4553130		2088073					6641204
						68.56		31.44					100
	Statistic					DF		Value					Prob
	Chi-Square					8		8512.2193					<.0001
	Cramer's V							0.0358

Table 4. T-test of Response by Industry, 2014 MEPS_IC
Industry	Responding (%)	Nonresponding (%)	DF	t Value	Pr > |t|
Agriculture, Fishing, and Forestry	2.59	3.04	20723	-2.44	0.0146
Mining and Manufacturing	4.54	4.01	23455	2.4	0.0165
Construction	7.99	9.61	20628	-5.11	<.0001
Utilities and Transportation	3.01	2.98	22297	0.15	0.8811
Wholesale Trade	5.31	5.48	21899	-0.7	0.4823
Financial Real Estate	11.59	11.42	22334	0.48	0.6341
Retail Trade	14.11	13.74	22434	0.98	0.3284
Professional Services	25.92	24.25	39000	3.51	0.0005
Other	24.94	25.47	22070	-1.07	0.2863
Total	100	100

	Table 5. Chi-Square of Response by Division, 2014 MEPS-IC
	Division	Responding (Y)	Nonresponding (N)	Total
Frequency	New	240544	102128	342672
Expected	England	234932	107740
Percent		3.62	1.54	5.16
Row Pct		70.2	29.8
Col Pct		5.28	4.89
	Middle	587908	335436	923344
	Atlantic	633034	290310
		8.85	5.05	13.9
		63.67	36.33
		12.91	16.06
	East	670924	289825	960749
	North	658678	302071
	Central	10.1	4.36	14.47
		69.83	30.17
		14.74	13.88
	West	388453	137034	525487
	North	360268	165219
	Central	5.85	2.06	7.91
		73.92	26.08
		8.53	6.56
	South	861378	416445	1277823
	Atlantic	876060	401763
		12.97	6.27	19.24
		67.41	32.59
		18.92	19.94
	East	242474	92219.5	334693
	South	229462	105231
	Central	3.65	1.39	5.04
		72.45	27.55
		5.33	4.42
	West	480372	227581	707952
	South	485364	222589
	Central	7.23	3.43	10.66
		67.85	32.15
		10.55	10.9
	Mountain	355633	140220	495853
		339951	155902
		5.35	2.11	7.47
		71.72	28.28
		7.81	6.72
	Pacific	725446	347185	1072630
		735383	337248
		10.92	5.23	16.15
		67.63	32.37
		15.93	16.63
	Total	4553130	2088073	6641204
		68.56	31.44	100
	Statistic	DF	Value	Prob
	Chi-Square	8	24415.1746	<.0001
	Cramer's V		0.0606

Table 6. T-test of Response by Geographic Division, 2014 MEPS_IC
Division	Responding (%)	Nonresponding (%)	DF	t Value	Pr > |t|
New England	5.3	4.85	39000	1.84	0.066
Middle Atlantic	12.92	16.07	20505	-7.97	<.0001
East North Central	14.74	13.87	39000	2.27	0.0231
West North Central	8.52	6.59	24796	6.77	<.0001
South Atlantic	18.92	19.93	21814	-2.3	0.0216
East South Central	5.33	4.41	24142	3.94	<.0001
West South Central	10.54	10.92	21901	-1.09	0.2777
Mountain	7.8	6.74	23623	3.72	0.0002
Pacific	15.93	16.62	21868	-1.68	0.0939
Total	100	100