Att F_Nonresponse Follow-up Study Report

This report evaluates the non-response follow-up (NRFU) used on the Feasibility Survey (FS) for the National Intimate Partner and Sexual Violence Survey (NISVS). The primary research questions the report addresses are:

How does the NRFU affect the response rate?
How does the NRFU affect the representation of the population for the survey?
How do the prevalence estimates differ between the early respondents who are not part of the NRFU (hereafter referred to as ‘Early respondents’) and the NRFU group?
What are the survey costs and benefits of doing the NRFU?
What are recommendations with respect to doing the NRFU in the future?

In the rest of this chapter the design of the FS and the methods used for weighting the data that are used in the analysis are described. The second chapter provides data on the contribution the NRFU added to the response rate of the FS. The third chapter compares the demographic and health indicators for the Early responders to the NRFU responders. Chapter four compares the primary prevalence estimates between the Early and NRFU responders. Chapter five presents a cost-benefit analysis of the NRFU. The final chapter summarizes the results and presents recommendations.

The Design of the Feasibility Survey

The procedures for the FS are shown in Figure 1.

Initial Protocol (Phase 1, Early Responders)

The first step was to send a letter asking an adult to complete the screening survey on the web (Figure 2-1 – Box A). If the person selected for the extended interview was the screener respondent, then that person was instructed to proceed directly to the extended interview. If the screener respondent was not selected for the extended survey, the screener respondent was instructed to ask the selected adult to log in to the website and complete the survey.

If the screener was not completed, a letter was sent by express delivery asking the respondent to either fill out the screener on the web or complete a paper version of the screener (Figure 2-1 – Box C). If the web screener was still not completed and the paper screener was not returned after this mailing, the household was considered for subsampling for the nonresponse follow-up
(Figure 2-1 – Box G, described below).

If the web screener was completed but no extended survey was returned, a follow-up letter was sent inviting the selected individual to complete the survey on the web (Figure 2-1 – Box D). If there was no response, a letter was sent express delivery to the selected respondent (Figure 2-1 – Box E), asking the person to complete the extended survey by web or by an alternative mode. Half of these were given the choice between the web and a paper version of the questionnaire. The other half were given the choice between the web or to call in and complete the survey over the phone.

As a result of the FS, the option giving the web and telephone was chosen to move forward for the national survey (the Pilot Study). The analysis in this report includes those who responded with this option. Those who were assigned to the web-paper option are excluded from this analysis.

NRFU (Phase 2)

Once all attempts above were exhausted, a fifty percent subsample of the nonrespondents was selected for NRFU. Contacts using the same procedures as described above were attempted, but offered a larger incentive. Screener non-respondents received a letter encouraging them to go to the web to complete the screener or complete the enclosed paper screener. Additionally, 50 percent of the extended survey non-respondents were selected for NRFU¹.

Weighting and Statistical Significance

The analysis required the development of weights for a subset of the completes from the full NISVS FS. Two subsets were defined: The Early cases consist of the 3,009 completed cases; The NRFU group consists of 182 cases. We developed 3 weights to expand to the full population: one weight for the Early cases, one weight for the NRFU cases, and one weight for the combined (Both) early and NRFU cases.

For all three sets of weights, we began with the weights developed for the full ABS sample that accounted for sampling (base weights), unknown eligibility and NRFU subsampling. These were the base weights for this special weighting process.

Next, we adjusted the weights for nonresponse using the same nonresponse adjustment methodology as described in the FS report but restricted to the subset of cases for Early, NRFU and Both for the appropriate weight. The adjustment cells were created by Census division and own/rent status. Hispanic status was used for the weights employed for the full FS, but had to be dropped due to small cell sizes. In addition, Census division had to be dropped when doing the NRFU weight, again due to small cell sizes.

The person-level weights were created exactly as described for the larger FS by adjusting the weight by the number of adults in the household, capped at four. The final step was raking and trimming each set of weights to the same control totals. The raking dimensions were age (18-29, 30-44, 45-59, 65+), marital status (never married, married, other), education (high school or below, some college, Bachelor’s degree or higher), and gender (female, male). Race was not used because of small sample sizes. The same imputed values used in the FS were retained for this process if any of the demographics was missing. Minor modifications in the raking process were implemented to obtain convergence. For example, the trimming cutoff was increased from 4.5 times the median weight to 5 times, the number of iterations was increased from 15 to 50, and the minimum raking dimension sample size was reduced from 85 to 20.

For variance estimation, the replicate weighting procedure was done exactly as done for the FS.² To test for differences a z-test comparing two proportions was used. When the significance of the difference was at least at the five percent level, the difference is considered statistically significant. Throughout the report, differences that do not meet this standard are discussed, but are described as not being statistically significant. The significance level is shown when it is below 20 percent. This is done to provide the reader with an idea of differences that are approaching statistical significance.

^{Figure
1. Mailing sequence for ABS frame}

2. Completed Interviews and Response to NRFU

The data collection design included a number of stages, some of which contained experiments. Table 1 provides the number of completed surveys for those who participated in the survey as part of the web-CATI choice experiment. There were a total of 3191 surveys used in the analyses in this report. This includes all those who completed the survey before they were given a choice of modes or as part of the NRFU (n=2804). There were 205 surveys completed when given a choice of mode. There were 103 individuals who were part of the NRFU but were never given a choice of mode and 79 completes after being selected for the NRFU and given a choice of mode. The 182 NRFU respondents (103 + 79 = 182) make up the group analyzed below as the NRFU group.

To assess how much the NRFU contributes to the response rate, it is necessary to weight the experimental groups by the subsampling rates. Half of the eligible sample was assigned to the web‑CATI mode choice group. The second column reflects this by multiplying the 205 completes by 2 (410). Similarly, half of those eligible for NRFU were sampled. Those in the NRFU group before being given the mode choice are multiplied by 2 to account for this (103 x 2 = 206). Those eligible for both NRFU and the web-CATI mode choice are multiplied by 4 to reflect being in both groups (4 x 79 = 316).

Approximately 14 percent of the overall response to the survey came from the NRFU. Since the overall response rate to the survey was 33 percent, the response rate before the NRFU was 28%. Alternatively, the NRFU added approximately five percentage points to the overall response rate.

Table 1. Number of Completed Surveys and Proportion of Response Collected by the NRFU

	Unweighted	Weighted for selection⁺
Number returned before mode choice	2804	2804
Number returned at Mode Choice	205	410
NRFU returned before Mode Choice	103	206
NRFU returned After Mode Choice	79	316
Total	3191	3736
Percent NRFU responses	6%	14%

⁺ Weighted column accounts for the subsampling done for the mode selection experiment and for the NRFU.

3. Representation of the Sample

Comparison of Demographics

The demographic distributions were calculated for several different socio-demographic characteristics and are shown in Tables 2-4. These distributions were calculated using the selection weights, which reflect the probability of being selected into the sample.

Detailed Results

For all those that responded to the FS (Table 2), there are a number of significant differences in the demographic composition of the population. These differences include:

Age. There are more NRFU respondents in the 18-29 group (18.6% Early vs. 34.1%; p<001). There are fewer NRFU in the 65+ age group (30.5% Early vs. 15.2%, p<.001).
Marital Status. There are significantly fewer married respondents in the NRFU group (58.5% Early vs. 47.1% NRFU; p<.001). There are significantly more never married in the NRFU group (19.6% Early vs. 28.4% NRFU; p<.001).
Race/Ethnicity. There are significantly more Hispanics in the NRFU group (11.6% Early vs. 19.9% NRFU; p<.001) and significantly fewer Non-Hispanic Whites (71.9% Early vs. 64.4% NRFU; p<.002).
Education. There are significantly more NRFU respondents in the High School or less group (19.5% Early vs. 25.0% NRFU; p<.026).
Income. There are significantly more NRFU respondents in the low-income group of less than $25,000 (15.6% Early vs. 22.2% NRFU; p<.011).
Internet Access. NRFU respondents were more likely to have internet access with no cellphone data plan (95.1% Early vs. 97.6% NRFU; p<.004). They were also more likely to generally have access (99.1% Early vs. 99.7% NRFU; p<.001). The percentages for internet measures are extremely high, which is largely because most respondents are using the web to participate. But because the percentages are so high, especially the second measure, the estimates of standard errors are unreliable. Neither of the differences are large or substantively meaningful.
Home Ownership. Fewer NRFU respondents own their home than Early respondents (74.3% Early vs. 66.1%; p<.002), while more NRFU respondents are renters (24.8% Early vs. 32.8% NRFU; p<.001).

The distributions by sex are shown in Tables 3 and 4. The patterns are similar for each sex, with several exceptions.

There are several significant differences for females which are not significant for males.

Age. For females, the difference between the NRFU and Early responders is significant for all of the age groups. There is a much higher percentage of NRFU respondents in the 18-29-year-old age group for females (19.1% for Early vs. 45.3% for NRFU; p<.001). There are more Early responders in the other age groups. For males, only the 65+ age group differs with more in the Early group (32.0% Early vs 14.5% NRFU; p<.001).
Marital Status. There are more Early respondents who are married (56.3% Early vs. 41.1 NRFU; p<.001) and fewer who are not married (18.3% Early vs. 31.8% NRFU; p<.008). For males, only the married group difference was significant (61.1% Early vs. 53.3% NRFU; p<.012).
Income. There are more NRFU respondents in the less than $25k income group (17.7% Early vs. 29.2% NRFU; p<.001) and fewer in the $75k or greater group (45.4% Early vs. 31.6% NRFU; p<.001). For males, there are fewer NRFU respondents in the $25k to less than $50k category (16.9% Early vs. 6.8% NRFU; p<.000).

For males, there are also several differences that are significant which are not for females:

There are significantly more male NRFU respondents who are Hispanic (11.8% Early vs. 28.9% NRFU; p<.001) and fewer who are white (72.8% Early vs. 59.4% NRFU; p<.001). There are no significant differences for females.
There are significantly fewer male NRFU respondents who were born in the US (88.4% Early vs. 80.6% NRFU; p<.025). There is no difference for this characteristic for females.

Discussion

A summary of the above comparisons is shown in Table 5 below. The NRFU population does differ markedly from the Early responders with respect to several important characteristics, including age, marital status, race-ethnicity, and education. In all but one case (income), the direction of these differences is that the NRFU group has a higher proportion of respondents who are under-represented in the sample. For example, those 18-29, those not married, Hispanics and those with lower education are all under-represented when comparing the total sample to the ACS benchmarks. In all these cases, therefore, the NRFU is bringing the sample closer to the general population.

With respect to variables that might be related to risk of victimization, it is notable that for females the NRFU brings in a large proportion of those in the 18-29 age group. As noted in the Feasibility report, this group has particularly high estimates of victimization, although it should be noted that the Early responders for females is close to the ACS benchmark (19.1% vs. 20.5%). Once incorporating the NRFU group into the total sample, there is an over-representation of the youngest group (21.9% vs. 20.5%), although the differences are very small and none of the differences with the ACS are statistically significant.³

Comparison of Health Indicators

Tables 6-8 compare the distribution of selected health characteristics for the Early responders, those in the NRFU group and the combination of the two. These estimates use the full sample weight for each of the three groups. Each of these are adjusted for non-response and raked to national socio-demographic totals as described in Chapter 1. The last column of the table is the comparable measure from the National Health Interview Survey (NHIS).

Detailed Results

Only a few of the differences between the Early and NRFU responders are statistically significant. However, this is, in part, a function of the high standard errors from the NRFU group. This is related to a combination of the relatively small sample sizes and a large design effect associated with the final weights. For the estimates that combine the two sex groups together:

Physical, mental or emotional problem: More NRFU respondents report having a physical, mental or emotion problem than the Early respondents (14.4% Early vs. 20.6% NRFU; not significant), although the difference is not statistically significant. The Early group’s estimate is closest to the NHIS benchmark of 12.8 percent.
Hospitalized: There more Early respondents who say they had been hospitalized overnight in the last 12 months (14.2% Early vs. 10.9%; not significant), although the difference is not statistically significant. The estimate when combining the Early and NRFU samples (13.9%) is higher than the ACS estimate (p<.000). The NRFU estimate is closest to the NHIS estimate.
Asthma: There are more NRFU respondents who report having asthma (11.9% Early vs. 13.2%; not significant), although the difference is not statistically significant. The survey estimates are all below the NHIS estimate of 15.0 percent, with the NRFU estimate being closest.
Depression: The NRFU group is significantly higher than the Early group (23.7% Early vs. 31.5% NRFU; p<.035). Both estimates are well above the NHIS estimate of 15.7 percent.

The distributions by sex are very similar to when combining across sex.

For males, the distributions are in the same direction as when combining across sex. However, none of the differences are statistically significant, including the comparison for depression.
For females:

The direction of the differences between the NRFU and Early group is the same as when combining across sex and for males, with one exception. For Asthma, the NRFU group is slightly lower than the Early group (14.3% Early vs. 12.9% NRFU; not significant), although as with males, the difference is not statistically significant.
For physical, mental and emotional problems, the NRFU group is substantially lower, but not statistically significant, than the Early group (13.5% Early vs. 24.7% NRFU; p<.112).
For Depression, the NRFU is higher than the Early group but is not quite statistically significant (29.4% Early vs. 41.0% NRFU; p<.051).

Discussion

The two measures related to mental health (physical, mental or emotional problems; depression) were nominally higher in the NRFU group compared to the Early group for females. Only one of the two measures was significantly different (depression), but this is partly because of the large design effect and small sample sizes associated with the NRFU group. The direction of these differences was similar for males, but not nearly as large and not statistically significant. There were smaller differences between the groups for reports of hospitalization and asthma.

Table 2. Comparison of Weighted Demographic Distributions for Early, NRFU Respondents, and American Community Survey (ACS) – Selection Weights^x

	Estimates (percent)				p-value*
	Early	NRFU	Both	ACS	Both-ACS	Early-NRFU
Age
18-29	18.6	34.1	20.3	21.5	0.159	<0.001
30-44	26.7	27.4	26.8	25.1	0.046	0.801
45-64	24.2	23.3	24.1	25.7	0.040	0.734
65+	30.5	15.2	28.9	27.7	0.156	<0.001
Sex
Male	46.6	48.4	46.8	48.7	0.039	0.493
Female	53.4	51.6	53.2	51.3	0.039	0.493
Marital status
Married	58.5	47.1	57.3	50.4	<0.001	<0.001
Never married	19.6	28.4	20.5	30.0	<0.001	<0.001
Other	21.9	24.5	22.2	19.5	0.001	0.367
Race
Hispanic	11.6	20.0	12.5	15.7	<0.001	<0.001
NH-White	71.9	64.4	71.1	64.1	<0.001	0.002
NH-Black	6.3	6.2	6.2	12.0	<0.001	0.978
NH-Multiracial	2.7	3.2	2.7	1.7	0.001	0.613
NH-Other	7.6	6.2	7.4	6.6	0.082	0.327
Education
High school or less	19.5	25.0	20.1	36.4	<0.001	0.026
Some college	34.5	28.6	33.9	34.8	0.303	0.066
Bachelor’s or higher	46.0	46.4	46.0	28.9	<0.001	0.921
Income
Less than $25,000	15.6	22.2	16.3	15.7	0.373	0.011
$25,000 – $49,999	18.6	14.1	18.2	20.0	0.007	0.057
$50,000 – $74,999	16.2	17.6	16.4	18.3	0.005	0.428
$75,000+	49.6	46.0	49.2	46.0	<0.001	0.277
Born in United States
Yes	88.1	84.3	87.7	81.6	<0.001	0.124
No	11.9	15.7	12.3	18.4	<0.001	0.124
Access to internet not including through cellphone
Yes	95.1	97.6	95.4	75.6	<0.001	0.004
No	4.9	2.4	4.7	24.4	<0.001	0.004
Any access to internet
Yes	99.1	99.7	99.1	87.4	<0.001	0.001
No	0.9	0.3	0.9	12.5	<0.001	0.001
Home ownership
Owned	74.	66.1	73.4	66.8	<0.001	0.002
Rented	24.8	32.8	25.7	31.6	<0.001	0.001
Other	0.9	1.1	0.9	1.6	<0.001	0.678

^X Combines males and females

* Significance of a difference of proportions using a z-test. Significance tests with ACS assume the ACS has no sampling error.

Table 3. Comparison of Weighted Demographic Distributions for Early, NRFU Respondents, and American Community Survey (ACS) – Males – Selection Weights

	Estimates (percent)				p-value
	Early	NRFU	Both	ACS	Both-ACS	Early-NRFU
Age
18-29	18.0	22.2	18.5	22.6	0.002	0.137
30-44	26.4	33.2	27.1	25.8	0.318	0.087
45-64	23.6	30.1	24.3	25.9	0.204	0.119
65+	32.0	14.5	30.1	25.7	<0.001	<0.001
Marital status
Married	61.1	53.3	60.2	52.3	<0.001	0.012
Never married	21.1	24.9	21.5	33.2	<0.001	0.14
Other	17.9	21.7	18.3	14.5	<0.001	0.169
Race
Hispanic	11.9	28.9	13.8	16.2	0.029	<0.001
NH-White	72.9	59.4	71.3	64.3	<0.001	<0.001
NH-Black	5.0	5.6	5.0	11.5	<0.001	0.794
NH-Multiracial	2.9	2.2	2.8	1.7	0.015	0.605
NH-Other	7.4	3.9	7.0	6.3	0.296	0.12
Education
High school or less	19.7	22.9	20.2	37.8	<0.001	0.301
Some college	32.5	27.7	31.9	34.1	0.115	0.241
Bachelor’s or higher	47.8	49.4	48.0	28.1	<0.001	0.708
Income
Less than $25,000	13.2	14.7	13.3	13.4	0.944	0.471
$25,000 – $49,999	16.9	6.8	15.7	19.4	<0.001	<0.001
$50,000 – $74,999	15.6	16.9	15.7	18.7	0.008	0.63
$75,000+	54.4	61.6	55.2	48.4	<0.001	0.12
Born in United States
Yes	88.4	80.6	87.5	81.6	<0.001	0.025
No	11.6	19.4	12.5	18.4	<0.001	0.025
Access to internet not including through cellphone
Yes	95.5	96.7	95.6	76.1	<0.001	0.352
No	4.5	3.3	4.4	23.9	<0.001	0.352
Any access to internet
Yes	98.9	100.0	99.1	87.8	<0.001	0.002
No	1.1		0.9	12.2	<0.001
Home ownership
Owned	74.8	70.6	74.3	67.2	<0.001	0.183
Rented	24.1	29.4	24.7	31.2	<0.001	0.095
Other	1.1		1.0	1.6	0.006

* Significance of a difference of proportions using a z-test. Significance tests with ACS assume the ACS has no sampling error.

Table 4. Comparison of Weighted Demographic Distributions for Early, NRFU Respondents, and American Community Survey (ACS) – Females – Selection weights

	Estimates (percent)				p-value*
	Early	NRFU	Both	ACS	Both-ACS	Early-NRFU
Age
18-29	19.1	45.3	21.9	20.5	0.215	<0.001
30-44	27.0	21.9	26.5	24.5	0.056	0.036
45-64	24.7	16.8	23.9	25.5	0.121	0.002
65+	29.2	15.9	27.8	29.5	0.138	<0.001
Marital status
Married	56.3	41.1	54.7	48.7	<0.001	<0.001
Never married	18.3	31.8	19.7	27.0	<0.001	<0.001
Other	25.5	27.1	25.7	24.2	0.219	0.57
Race
Hispanic	11.3	11.6	11.4	15.2	<0.001	0.921
NH-White	71.1	69.1	70.9	63.9	<0.001	0.573
NH-Black	7.4	6.8	7.3	12.4	<0.001	0.644
NH-Multiracial	2.5	4.2	2.7	1.7	0.024	0.287
NH-Other	7.7	8.4	7.8	6.8	0.151	0.481
Education
High school or less	19.3	27.0	20.2	35.0	<0.001	0.008
Some college	36.3	29.4	35.6	35.4	0.909	0.094
Bachelor’s or higher	44.4	43.6	44.3	29.6	<0.001	0.865
Income
Less than $25,000	17.7	29.2	19.0	17.9	0.277	<0.001
$25,000 – $49,999	20.1	20.9	20.2	20.5	0.76	0.789
$50,000 – $74,999	16.8	18.3	17.0	17.8	0.401	0.272
$75,000+	45.4	31.6	43.9	43.8	0.945	<0.001
Born in United States
Yes	87.8	88.0	87.8	81.6	<0.001	0.954
No	12.2	12.0	12.2	18.4	<0.001	0.954
Access to internet not including through cellphone
Yes	94.7	98.5	95.1	75.2	<0.001	0.002
No	5.3	1.5	4.9	24.8	<0.001	0.002
Any access to internet
Yes	99.2	99.5	99.2	87.1	<0.001	0.159
No	0.8	0.5	0.8	12.9	<0.001	0.159
Home ownership
Owned	73.9	61.8	72.6	66.4	<0.001	<0.001
Rented	25.5	36.1	26.6	32.1	<0.001	<0.001
Other	0.7	2.1	0.8	1.6	<0.001	0.162

* Significance of a difference of proportions using a z-test. Significance tests with ACS assume the ACS has no sampling error.

Table 5. Significant Differences for NRFU Group Compared to Early Group by Demographic Characteristic*

	Both	Male	Female
Age	More 18-29; Fewer 65+	Fewer 65+	More 18-29; Fewer in All Other Ages
Sex	No Differences	—	—
Marital status	Fewer Married; More Not Married	Fewer Married	Fewer Married; More Not Married
Race-Ethnicity	More Hispanics, Fewer Whites	More Hispanics, Fewer Whites	No Differences
Education	More High School or Less	More High School or Less	More High School or Less
Income	More 25K or less	Fewer in $25K to less than 50k	More less than $25K; Fewer 75k or more
Born in the US	No Differences	Fewer Born in US	No Differences
Internet Access data plan	More with Plan	No Differences	More with Plan
Internet Access	More Access	More Access	No Differences
Home Ownership	Fewer Owners; More Renters	No Differences	Fewer Owners; More Renters

*Cell values indicate how NRFU compares to Early. For example, “More 18-29” = there are more 18-29-year-old in the NRFU group relative to Early group.

Table 6. Comparison of final Weighted Health Indicators for Early, NRFU Respondent and National Health Interview Survey (NHIS)^x

	Estimates (percent)				p-value*
	Early	NRFU	Both	NHIS	Both-NHIS	Early-NRFU
Any adult in the household have physical, mental, or emotional problem preventing from working
Yes	14.4	20.6	14.7	12.8	0.042	0.18
No	85.6	79.4	85.3	87.2	0.042	0.18
Any adult in the household been hospitalized overnight in last 12 months
Yes	14.2	10.9	13.9	10.9	<0.001	0.312
No	85.8	89.1	86.1	89.1	<0.001	0.312
Doctor, nurse, or other health professional told you that you have asthma
Yes	12.0	13.2	12.2	15.0	<0.001	0.686
No	88.0	86.8	87.8	85.0	<0.001	0.686
Doctor, nurse, or other health professional told you that you have any type of depression
Yes	23.7	31.5	24.7	15.7	<0.001	0.035
No	76.3	68.5	75.3	84.2	<0.001	0.035

^X Combines males and females

* Significance of a difference of proportions using a z-test.

Table 7. Comparison of Final Weighted Health Indicators for Early, NRFU Respondents and National Health Interview Survey (NHIS) – Males

	Estimates (percent)				p-value*
	Early	NRFU	Both	NHIS	Both-NHIS	Early-NRFU
Any adult in the household have physical, mental, or emotional problem preventing from working
Yes	15.5	16.3	15.2	NA	NA	0.881
No	84.5	83.7	84.8	NA	NA	0.881
Any adult in the household been hospitalized overnight in last 12 months
Yes	13.7	11.0	13.5	NA	NA	0.537
No	86.3	89.0	86.5	NA	NA	0.537
Doctor, nurse, or other health professional told you that you have asthma
Yes	9.4	13.5	9.8	12.3	0.008	0.396
No	90.6	86.5	90.2	87.7	0.008	0.396
Doctor, nurse, or other health professional told you that you have any type of depression
Yes	17.6	21.6	18.1	11.0	<0.001	0.468
No	82.4	78.4	81.9	89.0	<0.001	0.468

* Significance of a difference of proportions using a z-test.

Table 8. Comparison of Final Weighted Health Indicators for Early, NRFU Respondents, and National Health Interview Survey (NHIS) – Females

	Estimates (percent)				p-value*
	Early	NRFU	Both	NHIS	Both-NHIS	Early-NRFU
Any adult in the household have physical, mental, or emotional problem preventing from working
Yes	13.5	24.7	14.2	NA	NA	0.112
No	86.5	75.3	85.8	NA	NA	0.112
Any adult in the household been hospitalized overnight in last 12 months
Yes	14.6	10.9	14.3	NA	NA	0.377
No	85.4	89.1	85.7	NA	NA	0.377
Doctor, nurse, or other health professional told you that you have asthma
Yes	14.3	13.0	14.5	14.4	0.967	0.741
No	85.7	87.0	85.6	85.6	0.967	0.741
Doctor, nurse, or other health professional told you that you have any type of depression
Yes	29.4	41.0	30.9	20.2	<0.001	0.051
No	70.6	59.0	69.1	79.8	<0.001	0.051

* Significance of a difference of proportions using a z-test.

4. Comparison of Prevalence Estimates: Early Versus NRFU Responders

Prevalence estimates were calculated for the Early and NRFU groups using the final weights with respect to each group. The discussion below describes results for the lifetime and 12-month estimates.

Lifetime Prevalence Estimates

Comparison of lifetime prevalence estimates between the Early and NRFU groups are provided in Tables 9 and 10.

For the sample that combines males and females (Table 9), the estimates for the NRFU group were consistently higher than the Early group. However, only one of these differences was statistically significant.

The overall estimates for contact sexual violence were higher for the NRFU group (29.7% Early vs. 33.0% NRFU; not significant) but was not statistically significant. Among the components of Contact Sexual Violence, rape had the largest difference (11.1% Early vs. 16.24% NRFU; p<.097). The only difference, among the components, that is not in this direction is made to penetrate, where the Early group is slightly higher than the NRFU group (2.5% Early vs. 1.6% NRFU; not significant).
Stalking is higher for the NRFU group, although the difference is not significant (8.9% Early vs. 10.8% NRFU; not significant).
For Intimate Partner Violence (IPV) the NRFU group is significantly higher than the Early group (27.6% Early vs. 36.8% NRFU; p<.046). All of the component types of IPV are in this same direction, although none are statistically significant. Stalking by an intimate partner has the highest proportionate difference among the component types of IPV (4.5% Early vs. 7.1% NRFU; not significant). In terms of percentage points, intimate partner physical violence has the biggest difference (22.3% Early vs. 28.5% NRFU; p<.111).

The results are very different by sex. For males, the differences are in both directions (Figures 2-4 and Table 10). In addition, the size of most of the differences are not large.

For contact sexual violence, the Early group has a higher estimate than the NRFU group, although this difference is not significant (17.4% Early vs. 11.7 NRFU; p<.144). For the components of this type of victimization the direction of this difference is mixed.

For unwanted touching (12.3% vs. 11.1%; not significant) and made to penetrate (5.2% Early vs. 3.3%; not significant) the Early group is higher than the NRFU. None of these differences are significant.
For Sexual Coercion there the two estimates are virtually the same (5.3% Early vs 5.5% NRFU; not significant).
For Rape, the NRFU group is higher than the Early group (2.8% Early vs. 5.3% NRFU; not significant), although not significant.

For stalking, the Early group is higher than that of the NRFU group but is not significant at the five percent level (5.2% Early vs. 2.4% NRFU; p<.081).
For the estimate of IPV, the NRFU group is slightly higher than the Early group (20.4% Early vs. 22.1% NRFU; not significant). However, there is a mix of direction and significance among the components of this type of victimization.

For Contact Sexual Violence by an Intimate Partner, the Early group is significantly higher than the NRFU group (5.0% Early vs. 2.1% NRFU; p<.050). For Stalking by an intimate partner, the Early group is also higher than the NRFU group, although not significant (1.8% Early vs. 1.4 NRFU; not significant).
For Intimate Partner Physical Violence the NRFU group is higher than the Early group, but not significantly so (18.2% Early vs. 20.2% NRFU; not significant).

For Intimate Partner Psychological Aggression, the NRFU group is higher than the Early group, but the difference is not significant (29.5% Early vs. 34.6% NRFU; not significant).

^{Figure
2. Male Lifetime Prevalence Estimates for Early and NRFU
Respondents for Contact Sexual Violence, Stalking, Intimate
Partner Violence, and Intimate Partner Psychological Aggression+}

+ Error bars indicate 95% confidence interval.

# p<.10 using a difference of proportion z test.

^{Figure
3. Male Lifetime Prevalence Estimates for Early and NRFU
Respondents for Types of Contact Sexual Violence+}

+ Error bars indicate 95% confidence interval.

^{Figure
4. Male Lifetime Prevalence Estimates for Early and NRFU
Respondents for Types of Intimate Partner Violence+}

+ Error bars indicate 95% confidence interval.

* p<.05 using a difference of proportion z test.

For females (Figures 5-7, Table 10), the NRFU group is higher than the Early group for all of the comparisons, with several being significant. The differences are generally larger than that for males.

For Contact Sexual Violence, the NRFU group is higher than the Early group and almost reaches statistical significance at the five percent level (41.2% Early vs. 53.4% NRFU; p<.052). All of the differences for the component types of victimization are between 5 and 9 percentage points, although none are significant at the 5 percent level.

Unwanted touching (34.3% Early vs. 43.2 NRFU; p<.122)
Sexual Coercion (17.0% Early vs. 22.1% NRFU; not significant)
Rape (19.0% Early vs. 26.7% NRFU; p<.118)

For stalking, the NRFU group is 6 percentage points higher than the Early group (12.4% vs. 18.8%; not significant) but not statistically significant.
For Intimate Partner Violence, the NRFU group is significantly higher than the Early group (34.3% Early vs. 51.0% NRFU; p<.018). The differences for the component types of victimization are in the same direction and range from 5 to 10 percentage points, although these differences are not statistically significant.

Contact Sexual Violence by Intimate Partner (17.1% Early vs. 24.6% NRFU; p<.099)
Stalking by Intimate Partner (7.1% Early vs. 12.6% NRFU; not significant)
Intimate Partner Physical Violence (26.2% Early vs. 36.5% NRFU; p<.090)

For Intimate Partner Psychological Aggression, the NRFU group is 9 percentage points higher than the Early group (35.9% vs. 44.7%; p<.133) but not statistically significant.

^{Figure
5. Female Lifetime Prevalence Estimates for Early and NRFU
Respondents for Contact Sexual Violence, Stalking, Intimate
Partner Violence, and Intimate Partner Psychological Aggression+}

+ Error bars indicate 95% confidence interval.

* p<.05; ** p<.01; Statistical tests use a difference of proportion z test.

^{Figure
6. Female Lifetime Prevalence Estimates for Early and NRFU
Respondents for Types of Contact Sexual Violence+}

+ Error bars indicate 95% confidence interval.

^{Figure
7. Female Lifetime Prevalence Estimates for Early and NRFU
Respondents for Types of Intimate Partner Violence+}

+ Error bars indicate 95% confidence interval.

# p<.10 using a difference of proportion z test.

12-Month Prevalence Estimates

The comparisons of 12-month prevalence estimates between the Early and NRFU groups is provided Tables 11 and 12.

In most cases the NRFU estimate is larger than the estimate for the early responders for the sample that combines males and females. However only one difference is significant at the five percent level.

Contact sexual violence: The NRFU group is higher than the Early group (4.4 Early vs. 6.4% NRFU; not significant), but this is not significant. The NRFU group is higher for all of the component types of victimization, except for Rape. None of the differences are significant.

Unwanted touching (2.0% Early vs. 3.5% NRFU; not significant)
Sexual Coercion (2.2% Early vs. 5.2% NRFU; p<.154)
Rape (1.1% Early vs. 1.1% NRFU; not significant)
Made to Penetrate (0.2% Early vs. 0.4% NRFU; not significant)

Stalking: The NRFU group is higher than the Early group (2.8% Early vs. 3.2% NRFU; not significant), but this is not significant.
Intimate Partner Violence: The NRFU group is higher than the Early group (4.3% Early vs. 8.8% NRFU; p<.062). It is not significant at the five percent level. The NRFU group is higher than the Early group for all of the component types of victimization, but none of these differences are significant at the five percent level.

Contact Sexual Violence (2.1% Early vs. 4.4% NRFU; not significant)
Stalking by Intimate Partner (0.7% Early vs. 2.6% NRFU; not significant)
Intimate Partner Physical Violence (2.1% Early vs. 6.0% NRFU; p< .063)

Intimate Partner Psychological Aggression: The NRFU group is higher than the Early group (4.0% Early vs. 12.3% NRFU; p<.011) and the difference is significant.

As with the lifetime prevalence estimates, the results are very different by sex. For males the differences are in both directions (Figure 2; Table 10).

For contact sexual violence, the Early group has a higher estimate than the NRFU group (2.0% Early vs. 0.7% NRFU; p<.107) and the difference is not significant at the five percent level. However, for the components of this type of victimization the direction of this difference is mixed.

For unwanted touching, the Early group is higher than the NRFU (1.1% Early vs. 0.7% NRFU; not significant). The difference is not significant.
Made to Penetrate: The NRFU group is higher than the Early group (0.5% Early vs. 0.7% NRFU; not significant). The difference is not significant.
For Sexual Coercion (0.5% Early vs. .7% NRFU; not significant) Rape (0.3% Early vs. 0.7% NRFU; not significant) and Made to Penetrate (0.5% Early vs. 0.7% NRFU; not significant) the NRFU group is higher than the Early group. None of these are significant.

Stalking: The Early group is higher than that NRFU group (2.3% Early vs 0.7% NRFU; p<.072) and it is not significant at the five percent level.
For the overall estimate of IPV, the Early group is higher than the NRFU group (2.9% Early vs. 1.7% NRFU; p<.182) but is not significant. The estimates for the component parts are very small. For Contact Sexual Violence by Intimate Partner and Stalking by Intimate Partner, no respondent reported an incident in the NRFU group.

Intimate Partner Physical Violence (2.0% Early vs. 1.7% NRFU; not significant)

For Intimate Partner Psychological Aggression, the NRFU group is higher than the Early group, but the difference is not significant (3.6% Early vs. 5.8% NRFU; not significant).

For females, the NRFU group is higher than the Early group for all but one of the comparisons, with several being significant.

For Contact Sexual Violence, the NRFU group is higher than the Early group (6.6% Early vs. 11.9% NRFU; p<.183), but the difference is not significant. All but one of the differences for the component types of victimization are in the same direction, although none are significant.

Unwanted touching (2.8% Early vs. 6.1% NRFU; not significant)
Sexual Coercion (3.9% Early vs. 9.4% NRFU; p<.161)
Rape (1.9% Early vs. 1.4% NRFU; not significant)

For stalking, the NRFU group is higher than the Early group (3.3% Early vs. 5.5% NRFU; not significant) although not significant.
For Intimate Partner Violence, the NRFU group is significantly higher than the Early group (5.5% Early vs. 15.8% NRFU; p<.032). The difference for the component types of victimization are in the same direction with one being significant.

Contact Sexual Violence by Intimate Partner (3.5% Early vs. 8.6% NRFU; p<.169)
Stalking by Intimate Partner (1.1% Early vs. 5.0% NRFU; p<.179)
Intimate Partner Physical Violence (2.1% Early vs. 10.3% NRFU; p<.048)

For Intimate Partner Psychological Aggression, the NRFU group is significantly higher than the Early group (4.4% Early vs. 18.6% NRFU; p<.009).

Summary of Comparisons of Prevalence Estimates

The results of the above analysis of prevalence estimates are summarized in Table 13. The table displays which of the two groups is greater for each sex for the lifetime and 12-month estimates. It also provides an indication if the difference is statistically significant. For males, the differences are generally very small, none of the 22 comparisons being significant at least at the five percent level. The direction of the differences is also not consistent. For some types of victimization, the Early group is greater than the NRFU group, while the opposite is the case for others. For females, the NRFU group is nominally greater than the Early group for all but one of the 22 differences. Several are significant at the 1 percent level. For lifetime victimization, the differences range from five to ten percentage points for the various types of victimizations. For 12‑month victimization, the differences are somewhat smaller, reflecting the lower estimates related to the shorter reference period. The lack of evidence showing large differences to be statistically significant is, in part, due to the very large standard errors for the NRFU group. When split by sex, there are fewer than 100 unweighted cases for each group. This is indicated by the very large confidence bars shown in the figures above.

The higher estimates for the NRFU may reflect a disproportionate number of females age 18-29 who made up the NRFU group. As noted in the previous section, almost half of the female NRFU respondents were in this age group. The weighting for the NRFU was done after combining the two sex groups together. Consequently, even after weighting, there is a disproportionate number of young people in the NRFU group for females. To check whether this might be driving the significance of the effects of the NRFU, we estimated a series of logistic regressions that predicted each of the prevalence estimates with an indicator of the sample group (Early vs. NRFU) and age. In all cases the effect of sample group remained statistically significant.

The above analysis indicates that the NRFU brings in female respondents who have significantly higher victimization estimates. Interpreted as an indicator of non-response bias, this suggests that the estimates for females from the FS may be biased in a negative direction (i.e., they are too low). This assumes that the NRFU group is similar to those individuals who did not respond at all. This assumption is not testable and has been shown to not hold in some cases.⁴

The differences between the Early group and the final estimate including the NRFU range from .5 to 1.5 percentage points (Tables 9 and 10). For example, for Intimate Partner Violence, the estimate for females that is restricted to the Early group is 34.3 percent. The estimate that includes both the Early and the NRFU group is 35.8 percent; a difference of 1.5 percentage points (Table 10). The effect of the NRFU group on the final estimate is limited because it only makes up 14 percent of the weighted population totals.

Table 9. Lifetime Prevalence Estimates for Selected Measures of Sexual, Physical, and Emotional Abuse, Comparing Early and NRFU Groups*

	Prevalence Estimate			Sig. Test: p-value^X
	Early	NRFU	Both	Early – NRFU
Contact sexual violence	29.7	33.0	30.24	0.269
Unwanted touching	23.6	27.5	24.39	0.239
Sexual coercion	11.3	14.0	11.66	0.241
Rape	11.1	16.2	11.65	0.097
Made to penetrate+	2.5	1.6	2.50	0.286
Stalking	8.9	10.8	9.12	0.323
Intimate Partner Violence	27.6	36.8	28.61	0.046
Contact Sexual Violence by Intimate Partner	11.3	13.5	11.53	0.265
Stalking by Intimate Partner	4.5	7.1	4.90	0.240
Intimate Partner Physical Violence	22.3	28.5	23.06	0.111
Intimate Partner Psychological Aggression	32.8	39.7	33.66	0.067

* Combines estimates for males and females; ^X p-value is the result from a two-sample difference of means z-test.

+ Made to penetrate only asked of males.

Table 10. Lifetime Prevalence Estimates for Selected Measures of Sexual, Physical, and Emotional Abuse, Comparing Early and NRFU Groups by Sex

	Male				Female
	Early	NRFU	Both	Early – NRFU Sig. Test⁺ p-value	Early	NRFU	Both	Early – NRFU Sig. Test⁺ p-value
Contact sexual violence	17.4	11.7	17.1	0.144	41.2	53.4	42.6	0.052
Unwanted touching	12.3	11.1	12.5	0.379	34.3	43.2	35.6	0.122
Sexual coercion	5.3	5.5	5.4	0.398	17.0	22.1	17.5	0.234
Rape	2.8	5.3	3.2	0.271	19.0	26.7	19.6	0.118
Made to penetrate	5.2	3.3	5.2	0.279	NA	NA	NA	NA
Stalking	5.2	2.4	5.0	0.081	12.4	18.8	13.0	0.204
Intimate Partner Violence	20.4	22.1	20.9	0.381	34.3	51.0	35.8	0.018
Contact Sexual Violence by Intimate Partner	5.0	2.1	4.9	0.050	17.1	24.6	17.8	0.099
Stalking by Intimate Partner	1.8	1.4	1.9	0.377	7.1	12.6	7.7	0.213
Intimate Partner Physical Violence	18.2	20.2	18.7	0.368	26.1	36.5	27.2	0.090
Intimate Partner Psychological Aggression	29.5	34.6	30.1	0.269	35.9	44.7	37.1	0.133

NA – Not applicable. Not asked of female respondents.

+ p-value is the result from a two-sample difference of proportions z-test.

Table 11. 12-month prevalence estimates for selected measures of sexual, physical, and emotional abuse, comparing Early and NRFU Groups#

	Prevalence Estimate			Sig. Test⁺: p-value
	Early	NRFU	Both	Early – NRFU
Contact sexual violence	4.4	6.4	4.6	0.263
Unwanted touching	2.0	3.5	2.1	0.291
Sexual coercion	2.2	5.2	2.5	0.154
Rape	1.1	1.1	1.1	0.398
Made to penetrate*	0.2	0.4	0.3	0.372
Stalking	2.8	3.2	2.9	0.389
Intimate Partner Violence	4.3	8.8	4.9	0.062
Contact Sexual Violence by Intimate Partner	2.1	4.4	2.4	0.212
Stalking by Intimate Partner	0.8	2.6	1.0	0.208
Intimate Partner Physical Violence	2.1	6.0	2.6	0.063
Intimate Partner Psychological Aggression	4.0	12.2	4.9	0.011

# Estimates combine male and females.

+ p-value is the result from a two-sample difference of z-test.

* Made to penetrate only asked of males.

Table 12. 12-month prevalence estimates for selected measures of sexual, physical, and emotional abuse, comparing Early and NRFU Groups by sex

	Male				Female
	Early	NRFU	Both	Early – NRFU Sig. Test p-value⁺	Early	NRFU	Both	Early – NRFU Sig. Test p-value+
Contact sexual violence	2.0	0.7	1.9	0.107	6.6	11.9	7.1	0.183
Unwanted touching	1.1	0.7	1.0	0.363	2.8	6.1	3.2	0.267
Sexual coercion	0.5	0.7	0.5	0.370	3.9	9.4	4.4	0.161
Rape	0.3	0.7	0.4	0.330	1.9	1.4	1.8	0.363
Made to penetrate	0.5	0.7	0.5	0.373	NA	NA	NA	NA
Stalking	2.3	0.7	2.2	0.072	3.3	5.5	3.6	0.312
Intimate Partner Violence	2.9	1.6	2.8	0.182	5.5	15.8	6.8	0.032
Contact Sexual Violence by Intimate Partner	0.7	0	0.6		3.5	8.6	4.0	0.169
Stalking by Intimate Partner	0.4	0	0.4		1.1	5.0	1.6	0.179
Intimate Partner Physical Violence	2.0	1.6	2.0	0.378	2.1	10.3	3.2	0.048
Intimate Partner Psychological Aggression	3.5	5.8	3.6	0.319	4.4	18.6	6.2	0.009

NA – Not applicable. Not asked of female respondents.

+ p-value is the result from a two-sample difference of proportion z-test.

Table 13. Summary of Differences Between the Early and NRFU Lifetime and 12-Month Prevalence estimates by sex+

	Males		Females
	Lifetime	12-month	Lifetime	12-month
Contact sexual violence	Early	Early	NRFU*	NRFU
Unwanted touching	Early	Early	NRFU	NRFU
Sexual coercion	NRFU	NRFU	NRFU	NRFU
Rape	NRFU	NRFU	NRFU	Early
Made to penetrate	Early	NRFU	NA	NA
Stalking	Early	Early	NRFU	NRFU
Intimate Partner Violence	NRFU	Early	NRFU**	NRFU*
Contact Sexual Violence by Intimate Partner	Early*	—	NRFU	NRFU
Stalking by Intimate Partner	Early	—	NRFU	NRFU
Intimate Partner Physical Violence	NRFU	Early	NRFU	NRFU*
Intimate Partner Psychological Aggression	NRFU	NRFU	NRFU	NRFU**

+ The entry in a cell indicates which group’s estimate has a larger numeric number

* P<.05; ** p<.01;

NA – this is not asked of women

---- No comparison because of no reports of victimization

5. Costs of the NRFU

The NRFU component has both non-monetary and monetary costs associated with it. First, sampling non‑responders introduces a design effect related to the subsampling of non-respondents. A second non-monetary cost is that it lengthens the field period by approximately one month. This delays the final release of the estimates to the public. Third, there is a greater monetary cost to implement the NRFU because of the additional effort to reach the respondents, as well as the increased incentive used to motivate them to respond. This chapter describes these three different costs.

Design Effects with and Without the NRFU

The design effect (Deff) represents the ratio of the sampling variance for a particular statistic for the design that is implemented divided by the sampling variance that what would have been achieved by a simple random sample (SRS) with the same sample size. A ratio of 1 means the sampling variance is the same as one that implements a SRS. A Deff of greater than one provides the reduction in effective sample size that is due to the design.

Two important design features that affect the Deff are the amount of clustering in the sample design and the extent the weights are different across respondents. For the NISVS Feasibility Study, there was no clustering of the sample so this does not contribute to the Deff. However, the weights do differ across respondents for several reasons. First, the selection of a single person in a household leads to different selection probabilities across households of different sizes. The second source of variation are the weighting adjustments that are made to account for non-response and coverage. The NRFU adds a third source of unequal weighting. To create the NRFU sample, non-respondents are subsampled. This adds an extra adjustment for the NRFU group that gets folded into the weights when it is combined with the Early group.

Tables 14-15 provide the design effects for lifetime and 12-month prevalence estimates for males and females. The Deff for the Early group represents the effect of the design before the NRFU is implemented, while the Deff in the Both column represents the Deff that combines both the Early and the NRFU group. The final column is the ratio between the two.

For the lifetime estimates, 13 of the 21 Deff’s for the Early responders are less than the Both group. For example, the Deff for males for lifetime prevalence for Contact Sexual Violence is 1.75 for the Early group and 1.96 when including the NRFU group. For the 12-month estimates, 12 of the 21 Deff’s are lower for the Early group. One reason why there are not larger effects of the NRFU on the Combined Group Deff is that the NRFU did bring the sample closer to the national population for many of the characteristics that were used for the weighting. This reduces the effects associated with the raking stage of the weighting. There is also variance associated with the estimates of the standard errors, which we cannot quantify. This may also contribute to why more ratios are not less than 1. Nonetheless many of the ratios that are greater than one by only a small amount, many are less than 1.05, with only a few as high as 1.10. Whereas for those that are less than 1, there are a number of ratios .90 or less, with some as low as .58. In other words, the effect of the NRFU is to reduce the overall efficiency of the sample, as indicated by the values of less than one. It is difficult to point out which type of estimate is most affected given the small sample sizes and the variance associated with the estimates of the Deff.

Survey and Monetary Costs of the NRFU

The monetary cost to do the NRFU was computed by using estimated response rates from FS at each stage of the data collection protocol and estimating the amount of labor and other direct costs (postage, printing, incentives) with doing the survey with and without the NRFU. The costs assumed that 20,000 surveys will be completed by the Early group and 1,767 would be completed during the NRFU. The NRFU completes were computed by applying the Feasibility subsampling and NRFU response rate to the non-responders in the Early group.

The ratio between the unit cost with and without the NRFU is slightly less than 2.6. In other words, for the cost of each NRFU complete it is possible to collect slightly less than 2.6 completed surveys in the Early group. For example, rather than doing a survey with 20,000 Early completes and 1,767 NRFU completes (total of 21,767), it is possible to complete a survey with around 24,450 completes with no NRFU – or 2,683 more completes without the NRFU. The cost of the NRFU as a percentage of the Early costs is 22 percent.

The increase in precision of adding the 2,683 completes can be approximated by the ratio:

= .94

This means the standard errors are reduced by approximately 6 percent when eliminating the NRFU and using the money to collect more Early cases. This does not account for differential design effects, but the impact of this is relatively small, as shown in the previous section.

To make this more concrete, the standard errors for samples of 21,767 and 24,450 were approximated using the standard errors from the FS for lifetime prevalence estimates by sex (Table 10). The standard errors with NRFU and without NRFU (Early) for a sample of 21767 were approximated by:

SEE(NRFU)_ij = SEE(FEAS)_ij *

SEE(Early)_ij = .94 * SEE(NRFU)_ij

where: i = each of the 11 different prevalence estimates; j=male, female

SEE(NRFU)ij = the standard error for the ith estimate and the jth sex when including the NRFU assuming a sample size of 21767

SEE(Early)_ij= the standard error for the ith estimate and the jth sex without the NRFU and increasing the sample to 24,450

SEE(FEAS)_ij = The standard error for the ith estimate and the jth sex from the FS

n(FEAS)_j = The number of respondents for the jth sex for the FS

n(21767)_j = The number of respondents for the jth sex assuming 21767 responded to the survey

The changes in the standard errors for males range from .004 to .013 (Table 16). The changes for females range from .009 to .020 (Table 17). The changes for females are slightly larger because the prevalence estimates for females are larger. The final column in the two tables provide the difference between the prevalence estimates with and without the NRFU from the FS. As can be seen, the change in the standard error when eliminating the NRFU is much smaller than the change in the prevalence estimate. This is discussed further in the next section.

Table 14. Design Effects for Lifetime Prevalence Estimates for Early and Combined Samples by Sex

	Males			Females
	Early	Both	Ratio	Early	Both	Ratio
Contact sexual violence	1.75	1.96	0.90	1.42	1.48	0.96
Unwanted touching	1.38	1.62	0.86	1.57	1.68	0.94
Sexual coercion	1.12	1.04	1.08	1.71	1.56	1.10
Rape	1.23	1.42	0.86	1.29	1.63	0.79
Made to penetrate^*	1.04	1.17	0.88	.	.	.
Stalking	1.36	1.25	1.09	1.13	1.36	0.83
Intimate Partner Violence	1.49	1.40	1.06	1.42	1.37	1.04
Contact Sexual Violence by Intimate Partner	1.47	1.53	0.96	1.60	1.45	1.10
Stalking by Intimate Partner	1.81	1.82	1.00	1.09	1.39	0.78
Intimate Partner Physical Violence	1.52	1.48	1.03	1.60	1.79	0.89
Intimate Partner Psychological Aggression	1.17	1.19	0.98	1.14	1.19	0.96

* Applies to males only.

Table 15. Design Effects for 12-Month Prevalence Estimates for Early and Combined Samples by Sex

	Males			Females
	Early	Both	Ratio	Early	Both	Ratio
Contact sexual violence	1.17	1.13	1.04	1.98	1.81	1.10
Unwanted touching	1.13	1.13	1.00	1.25	1.59	0.79
Sexual coercion	0.92	1.01	0.92	2.00	2.04	0.98
Rape	1.03	1.22	0.85	1.66	1.55	1.07
Made to penetrate^*	0.95	1.09	0.88	.	.	.
Stalking	1.36	1.39	0.97	1.51	1.59	0.95
Intimate Partner Violence	1.03	1.01	1.02	1.74	2.16	0.80
Contact Sexual Violence by Intimate Partner	0.91	0.83	1.10	2.35	2.32	1.01
Stalking by Intimate Partner	1.28	1.19	1.08	1.01	1.75	0.58
Intimate Partner Physical Violence	1.39	1.36	1.02	1.27	1.97	0.65
Intimate Partner Psychological Aggression	1.18	1.26	0.94	1.78	2.17	0.82

* Applies to males only.

Table 16. Difference in Standard Error and Lifetime Prevalence Estimates Attributed to NRFU for Males

	Standard Error With NRFU+	Standard Error Without NRFU^x	Difference Between Standard Errors With and Without NRFU	Difference in Prevalence Estimates
Contact sexual violence	0.145	0.136	0.009	0.4
Unwanted touching	0.108	0.101	0.006	-0.2
Sexual coercion	0.072	0.068	0.004	-0.1
Rape	0.067	0.063	0.004	-0.4
Made to penetrate	0.076	0.072	0.005	0.0
Stalking	0.173	0.162	0.010	0.1
Intimate Partner Violence	0.168	0.158	0.010	-0.5
Contact Sexual Violence by Intimate Partner	0.072	0.068	0.004	0.2
Stalking by Intimate Partner	0.066	0.062	0.004	-0.1
Intimate Partner Physical Violence	0.165	0.155	0.010	-0.5
Intimate Partner Psychological Aggression	0.212	0.199	0.013	-0.6

+Assumes sample size of 21,767; x Assumes sample size of 24,450.

Table 17. Difference in Standard Error and Lifetime Prevalence Estimates Attributed to NRFU for Females

	Standard Error With NRFU+	Standard Error Without NRFU^x	Difference Between Standard Errors With and Without NRFU	Difference in Prevalence Estimates
Contact sexual violence	0.317	0.298	0.019	-1.5
Unwanted touching	0.203	0.191	0.012	-1.4
Sexual coercion	0.270	0.253	0.016	-0.6
Rape	0.153	0.144	0.009	-0.7
Made to penetrate
Stalking	0.216	0.203	0.013	-0.6
Intimate Partner Violence	0.340	0.319	0.020	-1.6
Contact Sexual Violence by Intimate Partner	0.275	0.258	0.016	-0.7
Stalking by Intimate Partner	0.151	0.142	0.009	-0.6
Intimate Partner Physical Violence	0.230	0.216	0.014	-1.0
Intimate Partner Psychological Aggression	0.328	0.308	0.020	-1.2

+Assumes sample size of 21,767; ^x Assumes sample size of 24,450.

6. Conclusions and Recommendations

This report has assessed the NRFU from three different perspectives. First, whether the NRFU improves the overall representation of the sample The NRFU improved the representation of the sample along several characteristics, including those in the 18-29 age group, those not married, with lower education and more renters. The representation along selected health characteristics was more mixed. None of the differences between the Early and NRFU groups were statistically significant. For two indicators, being hospitalized and having asthma, the NRFU brought the population closer to the national benchmark. For two other measures, physical/mental/emotional problems and depression, the NRFU was further away from the national benchmarks. Despite the latter result, overall, the NRFU does seem to improve the overall representation in selected demographic characteristics of the combined sample.

The second perspective was how the NRFU affected the estimates of victimization. When comparing the prevalence estimates between the Early and NRFU responders there were different patterns between males and females. Overall, there were smaller differences for males than females. There were fewer significant differences for males and they were in different directions. For males, the NRFU respondents had nominally higher estimates for some types of victimizations while for others they were lower. For females, the NRFU respondents consistently had higher estimates than the Early respondents. The differences were large in magnitude and several were significant at the five percent level. The effect on the estimates after combining the Early and NRFU respondents was between .5 and 1.5 percentage points. For estimates as high as 40 percentage points, this is around 3.75 percent of the estimate (1.5/40 = 3.75%).

The third perspective looked at the survey and monetary costs associated with the NRFU. The NRFU does increase the design effect of the survey, which reduces the efficiency of the overall sample size. Overall, the increase in the design effect was not very large. The analysis of the monetary costs found that for every one NRFU complete it would be possible to collect slightly less than 2.6 surveys in the Early group. This is based on a design that assumes the response rates from the FS and has the goal of completing 20,000 surveys in the Early group and 1,767 in the NRFU group. If the money spent on the NRFU was used to complete more Early interviews, the number of completes would be 24,450. Note that this calculation could change if different response rates were used. For example, all costs would go up if a significantly lower response rate was used.

The analysis approximated the gain in precision if the study did not do the NRFU and used the money to increase the sample size. The gains in precision were relatively small, many change the standard error at the third decimal. When compared to the effect on prevalence estimates with and without the NRFU, the change in precision is considerably smaller. One perspective is how the decision to keep the NRFU changes the root mean square error of the survey (RMSE). The RMSE is often used when considering the total survey error (TSE) of the survey and is computed by:

where: V=variance of the estimate; B² = Bias squared.

The analysis found that increasing the sample size for the estimates contributes between .009 to .020 to the RMSE. This is the change in the standard errors with and without the NRFU (Tables 16 and 17). In contrast, the change in the prevalence estimates, which can be interpreted as a change in the bias, contributes between .5 to 1.5 (Tables 16 and 17). From this perspective, it is best to keep the NRFU, rather than spending the money to increase the sample size.

The NRFU also increased the nominal response rate by approximately five percentage points. The response rate is the most visible indicator of data quality to those reviewing and using the survey. In the case of the FS, it raised the response rate enough to cross 30% (28% vs. 33%).

The other advantage of the NRFU is that it provides the survey with some indication of the non-response bias associated with the survey. The NRFU can be useful for tracking trends in non-response over time.

Besides the additional monetary costs associated with doing the NRFU, it also adds to the time needed to complete the survey. For the FS, it added one month to follow-up on the additional mailings to the non-respondents. Implementation of the NRFU necessarily has to wait until all of the multiple mailings have been completed, which for those invited to do the extended interview can be up to four mailings (not including the reminder postcards).

Continuing with the NRFU depends, in part, on what CDC would do with the savings from eliminating it. If the savings would be used to collect more interviews, the NRFU should be retained. The NRFU has very clear advantages with respect to reducing the bias of the survey and the added data would only marginally improve the precision of the estimates. If the CDC has other uses for the money or if it is desirable to reduce the cost of the survey for other reasons, then it might be worth eliminating it. While the NRFU does reduce the bias in the estimates, the effect is relatively small, since it only makes up around 14 percent of the completed interviews. For the largest differences discussed above, the effect size related to implementing the NRFU was around 3.75 percent of an estimate that is around 40 percent (e.g., Intimate Partner Violence for females).

A compromise position would be to do the NRFU on an occasional basis to assess trends in non-response bias. This compromise would provide users with information as response rates declined over time. However, doing the NRFU on an occasional basis would restrict its use to non-response bias analyses. Because the NRFU does have an impact on the estimates, prevalence estimates that include the occasional NRFU would interrupt the trends in years when it is not available. That is, estimates in a year with the NRFU would not be comparable to a year without the NRFU. This would require releasing two estimates for the year the NRFU was implemented. One with and one without the NRFU.

1Paper screener returns from the initial screener nonresponse mailing were not included in phase 2 subsampling since they had not fully completed the extended phase 1 protocol at the time of phase 2 subsampling.

2 For the Taylor series variance stratum and unit, the NRFU data set had too few observations to retain the same variables as used in the FS (this was retained for the Early data set). Instead, the Varstrat and Varunit variables were redefined with the same sort order used in the FS but only 91 Varstrata were created, each with a single Varunit=1 and Varunit=2.

3 Significance test completed assuming the ACS has no sampling error.

4 Lin, I-F, and Schaeffer, N.C. (1995). Using survey participants to estimate the impact of nonparticipation. Public Opinion Quarterly, 59(2), 236-58.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Atiba Hendrickson
File Modified	0000-00-00
File Created	2023-11-10

Att F_Nonresponse Follow-up Study Report