2025 NSCG Methodological Experiments: Research Questions and Design Rationale

APPENDIX H

2025 NSCG Methodological Experiments:

Research Questions and Design Rationale

2025 NSCG Methodological Experiments:

Research Questions and Design Rationale

1. Plain Language Letters Experiment

Background Information on Plain Language Letters

The Plain Writing Act of 2010 requires federal agencies to communicate clearly in a way the public can understand. The Act states that plain writing means "writing that is clear, concise, well-organized, and follows other best practices appropriate to the subject or field and intended audience." In 2017, the American Community Survey (ACS) embarked on a strategic framework project that had five phases. One of these phases was to develop new mail materials based on plain language principles to improve the readability, lower the reading level and make the letters more visually appealing. In 2021, the ACS tested their new mail materials in an experiment. Their “Minimalist Treatment”, which used as few words as possible and only included the most important information needed to respond to the survey, yielded a significantly higher response rate than the production letters.

Proposed Treatments for the 2025 NSCG Plain Language Letters Experiment

This experiment will be limited to new sample members in an effort to minimize the operational impact on the National Processing Center. All new sample members are eligible to be in the experiment. Both experimental groups will follow the same mailout schedule and receive the same total number of contacts. The control group will use the same letters that were used in the 2023 NSCG, which follow the updated style tested in 2017 and fully implemented in 2019. These letters use a callout box with information on how to respond, including the URL, UserID and password. They also implement bullets to reduce the amount of block text and draw attention to important information. The treatment group will use plain language letters. These letters simplify the text, communicating only the information that sample cases need to know in concise language with a lower reading level. As a result, the letters are shorter and have more white space, making it easier for sample cases to find the information they need.

2025 NSCG Plain Language Letters Experimental Groups

There are only two experimental groups for the plain language letters experiment: a control group that receives the same letters used in 2023 and a treatment group that uses new letters developed using plain language principles.

Experimental Group	Count
Production Letters (Control)	49,200
Plain Language Letters (Treatment)	10,000

Research Questions for the 2025 NSCG Plain Language Letters Experiment

The experimental groups allow for the evaluation of the following research questions:

1. Relative to the control group, do plain language letters lead to a higher response rate?

2. Relative to the control group, does a plain language letter change the demographic makeup of respondents?

3. Relative to the control group, does sending a plain language letter lead to earlier response, hence reducing the number of follow-up contacts?

4. Compared to the control group, what is the cost per completed interview for those that receive plain language letters?

5. Is there a difference in inbound respondent contact rates (TQA calls or emails from respondents) between the control group and plain language treatments?

2. Text Message Experiment

Background Information on Text Message Experiment

The NSCG has experienced declining response rates in the younger cohort for several cycles. In an effort to connect with this group, research suggests that the younger generation’s preferred communication mode is text messaging (June, 2021) and text invitations have increased response rates when combined with other research modes (Kanticar & Marlar, 2017; De Bruijne & Wijnant, 2014; Mavletova & Couper 2014). Therefore, in 2021, the NSCG added a check box to the Contact screen that offered respondents the opportunity to opt in to receive text messages in future cycles. Approximately 35% of respondents selected the opt-in box. In 2023, a text message experiment was conducted among cases that had opted in in 2021. Results showed no differences in response rates or demographic distributions. However, text messages were not sent until week 12 and all were sent at 5pm eastern time.

This experiment will evaluate the impact of sending text messages earlier in data collection, in an effort to reduce the number of cases receiving a paper questionnaire, as well as vary the time of day the text is sent.

Proposed Treatments for the 2025 NSCG Text Message Experiment

This experiment will be limited to old cohort sample members who have previously opted in to receive text messages, provided a valid phone number, and do not live in Alaska or Hawaii¹. Both treatment groups will receive two text messages and the content of the text messages will be the same for both groups. The text messages will introduce the survey, provide the URL and login information, and information to opt-out and receive help. The only difference between the treatment groups is the timing of the text messages. One group will be sent earlier in the day (between 9am and 12pm) while the other will be sent later in the day (between 2pm and 5pm). Since the control group will not receive any text messages, we can also compare the overall impact of sending text messages to not sending them.

2025 NSCG Text Message Experimental Groups

There are three experimental groups for the text message experiment: a control group that does not receive any text messages, one treatment group that receives two text messages (Week 3 and Week 6) that are sent at 12pm eastern time, and another treatment group that receives two text messages (Week 3 and Week 6) that are sent at 5pm eastern time.

Research Questions for the 2025 NSCG Text Message Experiment

The experimental groups allow for the evaluation of the following research questions:

1. What was the impact on response?

   1. Was the change in response rate after a message was sent different across experimental groups?

   2. Did the demographic makeup of respondents change when text messages were sent?

   3. Was the demographic makeup of respondents different across experimental groups?

   4. Was the final mode of response different across experimental groups? Do people who received a text message respond on mobile devices at a higher rate than those that did not receive a text?

2. What were the operational workload impacts throughout the text message period?

   1. After each text message, what proportion of sample cases opted out of receiving additional text messages?²

      1. Did this vary by the time of day the text message was sent?

   2. Did the cases that opted out of receiving text messages ultimately respond to the NSCG?

      1. Did this vary by the time of day the text message was sent?

   3. What proportion of texts were undeliverable?

Experimental Group	Count3
No Text Message (Control)	10,000
Early Text Message (Treatment 1)	10,000
Late Text Message (Treatment 2)	10,000

References

Broderick Oliver, Lauren Contard & Dorothy Barth https://www.census.gov/content/dam/Census/library/working-papers/2023/acs/2023_Oliver_01.pdf

De Bruijne, M., & Wijnant, A. (2014). Improving response rates and questionnaire design for mobile web surveys. Public Opinion Quarterly, 78(4), 951-962.

June, S. (2021). Could Gen Z Free the World From Email? The New York TImes.

Kanticar, K., & Marlar, J. (2017). Novelty of Text Messages as Reminders for Web Surveys: Does it Last? Presented at the 2017 American Association for Public Opinion Research annual meeting. New Orleans, LA, May 18-21, 2017.

Mavletova, A., & Couper, M. P. (2014). Mobile web survey design: scrolling versus paging, SMS versus e-mail invitations. Journal of Survey Statistics and Methodology, 2(4), 498-518.

1 Due to the time difference in Alaska and Hawaii, text messages in the early treatment group will be sent too early in the morning. If this experiment is successful, we can consider sending to these states separately.

2² Sample cases could opt out of receiving additional text messages by replying STOP to the text reminder.

3 This number is an estimate. The exact number of old cohort members eligible to receive a text message will not be known until the 2025 sample is selected.

  

H-1