APPENDIX I 2021 NSCG Prenotice Experiment Research Questions and Design Rationale

APPENDIX I

2021 NSCG Prenotice Experiment

Research Questions and

Design Rationale

2021 NSCG Prenotice Experiment

Research Questions and Design Rationale

Background Information on the Prenotice

The 2017 NSCG survey cycle included a contact strategy experiment that altered the appearance and number of mailing materials sent to sample cases. The experiment reduced the number of reminder letters sent later in the data collection period and eliminated the prenotice. The decision to remove the prenotice from the NSCG contact strategy was based on research by Dillman and his colleagues (2014), who questioned the overall effectiveness of the prenotice letter, as it contains no call to action. Rather than sending a prenotice, they recommended replacing that contact with an additional reminder sent later in the contact strategy. Additionally, both the 2012 National Census Test and the ACS experimented with this new strategy in the presence of an internet push approach. The 2012 National Census Test found that sending an additional reminder in place of a prenotice had a slightly lower internet response rate (0.9 percent), but the difference was not statistically significant. The ACS also saw a small, significant, negative effect from removing the prenotification of 1.7 percent for internet response. However, after the data collection activities across all modes (paper, phone, and personal visit) were completed, there was no significant difference in the overall response rate. Both the National Census Test and the ACS opted to eliminate the prenotification due to the cost savings (Murphy and Roberts, 2015; Reiser et al., 2013).

Overall, the new letters and mailing schedule tested in the 2017 NSCG showed no significant differences in response rates, R-indicators (measures of sample representativeness), or key estimates, and they reduced costs compared to the standard mailings (Horwitz et al., 2018). Therefore, the new strategy was incorporated into the 2019 NSCG data collection. However, given all the factors that were altered in the 2017 experiment, it was not possible to measure the effect of any single change so we could not determine the impact of removing the prenotice.

There are benefits to sending the prenotice other than response rates. Specifically, we can identify bad addresses earlier in data collection so an effort can be made to locate these cases. Additionally, cases that are determined to be deceased or otherwise out-of-scope can be identified and removed from data collection, thereby reducing costs. With response rates declining, it is more important than ever to understand the impact achieved by each component of the NSCG data collection strategy. Isolating the effect of the prenotice will help determine the degree to which it influences response rates and costs.

Proposed Treatments for the 2021 NSCG Prenotice Experiment

The 2021 NSCG prenotice experiment proposes to test the inclusion and exclusion of the prenotice to measure its effectiveness within the NSCG data collection effort. The 2021 NSCG data collection strategy will include a prenotice and the treatment group will not be sent that mailing. All other factors in the contact strategy will remain consistent between the two groups including subsequent mail contacts, incentives, phone calls, emails, and mode preference. Because we were not able to measure the effect of the prenotice in the 2017 experiment, this will allow us to isolate its effect and examine how it contributes to the overall data collection effort.

2021 NSCG Prenotice Experimental Groups

There will be two experimental groups included in this experiment. The treatment group will not receive a prenotice one week prior to the beginning of data collection, while the control group will receive a prenotice. After the prenotice, the initial letter does not reference the prenotice, so all other mail contacts will remain the same across the experimental groups. Table I.1 provides a description of the experimental design. This design will apply to both the new and old cohorts.

Table I.1. 2021 NSCG Prenotice Experimental Groups

Prenotice	Treatment
Sent Week 0	Prenotice (PN)*
Not sent	No Prenotice (NPN)

*Control Group

Research Questions for the 2021 NSCG Prenotice Experiment

The experimental groups allow for the evaluation of the following research questions:

1. Relative to the control group, does eliminating the prenotice lead to lower response rates?

   1. Early in data collection?

   2. At the end of data collection?

2. Relative to the control group, does eliminating the prenotice lead to lower R-indicators?

3. Relative to the control group, does eliminating the prenotice lead to higher costs?

4. Relative to the control group, does eliminating the prenotice have any impact on key estimates?

References:

Dillman, D., Smyth, J., & Christian, L. (2014). Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method (4th Edition). New York: Wiley & Sons.

Horwitz, R., Newman, B., & Misra, J. (2018). Contact Strategy Experiment Results for the 2017 National Survey of College Graduates.

Murphy, P., Roberts, A. (2015). 2014 Pre-Notice Test. Available at https://www.census.gov/content/dam/Census/library/working-papers/2015/acs/2015_Murphy_01.pdf

Reiser, C., Barron, S., Bentley, M., Hill, J., Meier, A., Pape, T. (2013), “2012 National Census Test Contact Strategy Results,” Final Draft Report December 19, 2013.

I-1