SUPPORTING STATEMENT
U.S. Department of Commerce
U.S. Census Bureau
American Community Survey Methods Panel Tests
OMB Control Number 0607-0936
Part B. Collections of Information Employing Statistical Methods
Sample Design
Summer 2015 Mail Messaging Test
Universe: The sample universe for the Summer 2015 Mail Messaging Test consists of all residential addresses in the United States from the Census Bureau’s Master Address File sampled for the September 2015 production ACS. No addresses can be in sample (production and methods panel tests) more than once in a 5-year period. This excludes addresses in remote Alaska. Additional information about the production ACS sample can be found under OMB Control Number 0607-0810.
Sample Selection and Sample Size: We have divided the monthly production sample of 295,000 addresses in 24 nationally representative groups of approximately 12,000 addresses each. For this test, planned for the September 2015 ACS panel, we will use two randomly assigned groups for each of the experimental treatment groups yielding approximately 24,000 addresses per panel and a total experimental sample size of 120,000 addresses.
The sample size provides us with enough statistical power (.80) to detect a statistical difference in the self-response rates of at least 1.25 percentage points between the control treatment and a test treatment. This assumes a 50 percent self-response rate. Given the sample size and cost structure of ACS data collection, even a one percentage point difference in response can have important cost impacts to the program, therefore test designs must be able to identify even minor differences in self-response rates.
Fall 2015 Mail Messaging Test
Universe: The sample universe for the Fall 2015 Mail Messaging Test consists of all residential addresses in the United States from the Census Bureau’s Master Address File sampled for the November 2015 production ACS. No addresses can be in sample (production and methods panel tests) more than once in a 5-year period. This excludes addresses in remote Alaska. Additional information about the production ACS sample can be found under OMB Control Number 0607-0810.
Sample Selection and Sample Size: We have divided the monthly production sample of 295,000 addresses in 24 nationally representative groups of approximately 12,000 addresses each. For this test, planned for the November 2015 ACS panel, we will use two randomly assigned groups for each of the treatment groups yielding approximately 24,000 addresses per panel and a total experimental sample size of 72,000 addresses.
The sample size provides us with enough statistical power (.80) to detect a statistical difference in the self-response rates of at least 1.25 percentage points between the control treatment and a test treatment. This assumes a 50 percent self-response rate. Given the sample size and cost structure of ACS data collection, even a one percentage point difference in response can have important cost impacts to the program, therefore test designs must be able to identify even minor differences in self-response rates.
2016 ACS Content Test
Universe: The sample universe for the 2016 ACS Content Test consists of all residential addresses in the United States from the Census Bureau’s Master Address File (excluding remote Alaska, Puerto Rico, and U.S. territories). Note that addresses selected to participate in the 2016 production ACS will be out-of-scope for the Content Test. Also, no addresses can be in sample (production and methods panel tests) more than once in a 5-year period.
Sample Selection: The sample design will be based on the ACS production sample design (multi-stage sample) modified to meet the test objectives. Modifications to the ACS sample design include adding an additional level of stratification by stratifying units into high and low ACS self-administered (Internet/Mail) response areas; selecting units with equal probabilities of selection within the high/low response strata; and sampling units as pairs. The high and low response strata are defined using ACS self-administered response rates at the census tract-level. Note that units within the low response strata will be sampled at a higher rate to ensure an approximately equal number of responses from both strata. Sample units will be selected in pairs by first systematically sampling an address within the defined sampling strata and then pairing that address with the address listed next in the geographically sorted list. Note that the pair may not be neighboring addresses. Each member of the pair will then be randomly assigned to either the control or the test group. As with ACS production, this test will include subsampling at the CAPI stage. The Content Follow-Up (CFU) Reinterview component of this test will not involve any additional sampling. All housing units that respond to the Content Test for whom we have a telephone number (including telephone numbers identified through a vendor look-up) will be included in the CFU Reinterview.
Sample Size: The 2016 ACS Content Test will consist of a national sample of 70,000 residential addresses in the United States (the sample universe does not include Puerto Rico, remote Alaska, or U.S. territories) with half in the control and the other half in the test group. The sample size provides us with enough statistical power (.80) to detect a difference in the gross difference rates of at least 2 percentage points between the control and test groups. The sample size and power requirements for this test were established based on the retirement income question, because it has the smallest universe of respondents among the topics being tested. We expect statistical testing on other topics with larger universes of respondents will be able to detect smaller differences in the gross difference rates. Gross difference rates for most of the topics included in the ACS have historically been small (ranging from less than one percent to five percent for most items studied). Given that the rates are small and we would not expect large differences in rates between the control and test treatments, to be able to detect improvements in the questions we need to be able to detect small differences in the gross difference rate. Some question changes that are being considered for the future ACS are also being considered for the 2020 Census (specifically, Race, Hispanic Origin, and Relationship). Preparation for the 2016 ACS Content Test in connection with these items has involved close collaboration with preparation for the 2015 National Content Test. The 2015 National Content Test will be the primary source of data for recommendations regarding future content due to the larger sample size designed to detect differences for subgroups that the ACS Content Test is not likely to be able to identify. However, the ACS Content Test provides valuable information regarding interviewer-administered modes and operational testing that will also inform recommendations for future content.
Other Potential Tests
Although plans for the following tests are not yet fully scoped, the following information describes current thoughts on the sample design for each.
2016 Mail Messaging Tests
Universe: The sample universe for the mail messaging tests consists of all mailable residential addresses in the United States from the Census Bureau’s Master Address File. Addresses selected to participate in production ACS will be out-of-scope for the tests. Also, no addresses can be in sample (production and methods panel tests) more than once in a 5-year period.
Sample Selection: The sample design will be based on the ACS production multi-stage sample design. Once the number of treatments is determined, methods will be developed to sample addresses and randomly assign them to the various experimental panels.
Sample Size: For planning and budget purposes, we are assuming that this mail messaging test will have a national sample of 60,000 addresses in the United States divided into treatments. The number of treatments will be determined as the details of the tests are defined, and the total sample size may therefore be revised.
2017 and 2018 Self-Response Tests
Universe: The sample universe for the self-response tests consists of all mailable residential addresses in the United States from the Census Bureau’s Master Address File. Addresses selected to participate in production ACS will be out-of-scope for the tests. Also, no addresses can be in sample (production and methods panel tests) more than once in a 5-year period.
Sample Selection: The sample design will be based on the ACS production multi-stage sample design. Once the number of treatments is determined, methods will be developed to sample addresses and randomly assign them to the various experimental panels.
Sample Size: For planning and budget purposes, we are assuming that each test will have a national sample of 60,000 addresses in the United States divided into treatments. The number of treatments will be determined as the details of the tests are defined, and the total sample sizes may therefore be revised. An additional 35,000 addresses (from the 60,000) will be included in a content followup interview.
2017 and 2018 Internet Tests
Universe: The sample universe for the Internet Tests consists of all mailable residential addresses in the United States from the Census Bureau’s Master Address File. Addresses selected to participate in production ACS will be out-of-scope for the tests. Also, no addresses can be in sample (production and methods panel tests) more than once in a 5-year period.
Sample Selection: The sample design will be based on the ACS production multi-stage sample design. Once the number of treatments is determined, methods will be developed to sample addresses and randomly assign them to the various experimental panels.
Sample Size: For planning and budget purposes, we are assuming that each test will have a national sample of 108,000 addresses in the United States divided into treatments. The number of treatments will be determined as the details of the tests are defined, and the total sample sizes may therefore be revised.
Procedures for Collecting Information
The Methods Panel Tests typically follow the data collection procedures for the production ACS mailing strategy, with modifications based on the design and purpose of the test. The production ACS uses multiple modes for data collection – Internet, mail, telephone, and personal visit.
In the production ACS, the Census Bureau mails ACS survey materials to about 286,000 housing unit (HU) addresses each month. The mailing operations are conducted through the U.S. Postal Service and use first-class postage rates for all pieces. For households eligible to receive survey materials by mail, the first contact includes a letter (Attachment A1 – ACS-13) and instruction card (Attachment A2 – ACS-34IM) explaining how to complete the survey online (in English and Spanish) and how the ACS data are used. The letter also lets respondents know that if they are unable to complete the survey online a paper questionnaire will be sent later.. Enclosed with the letter, a brochure (Attachment A4 – ACS-9) provides basic information about the survey in English, Spanish, Russian, Chinese, and Korean, and provides a phone number to call for assistance in each language. A Frequently Asked Questions (FAQ) brochure (Attachment A3 – ACS-10SM) is also included and answers questions about confidentiality, the benefits and uses of the information, as well as the legal requirements related to the survey. The Internet version of the questionnaire (Attachment B) is available in English and Spanish and includes questions about the HU and the people living in the HU. The Internet questionnaire has space to collect detailed information for twenty people in the household. The second mailing (Attachment C1 – ACS-20(L)) is a letter that reminds respondents to complete the survey online, thanks them if they have already done so, and informs them that a paper form will be sent later if we do not receive their response.
A third mailing is sent only to those sample addresses that have not completed the online questionnaire within two weeks. The content includes a cover letter (Attachment D2 – ACS-14), a paper copy of the questionnaire (Attachment D1 – ACS-1), an instruction guide for completing the paper form (Attachment D6 – ACS-30), an instruction card for completing the survey online (Attachment D3 – ACS-34RM), an FAQ brochure (Attachment D5 – ACS-10SM), and a return envelope (Attachment D4). The cover letter with this questionnaire package reminds the household of the importance of the ACS, and asks them to respond soon either by completing the survey online or by returning a completed paper questionnaire.
The fourth mailing is a postcard (Attachment E – ACS-29) that reminds respondents that “now is the time to complete the survey,” informs them that an interviewer may contact them if they do not complete the survey, and reminds them of the importance of the ACS.
A fifth mailing is sent to respondents who have not completed the survey within five weeks and are not eligible for telephone follow-up because we do not have a telephone number for the household. This postcard (Attachment F – ACS-23) also reminds these respondents to return their questionnaires and thanks them if they have already done so.
Each month all sample HUs are matched to commercial phone number lists in an attempt to find phone numbers. For addresses that were mailed survey materials but did not respond by Internet, mail, or by calling our telephone questionnaire assistance (TQA) line, we attempt Computer-Assisted Telephone Interviewing (CATI) interviews for all household addresses for which we obtained phone numbers. After the CATI operation is complete, we select a sub-sample of all households that still have not responded and assign them to the CAPI data collection mode. This subsample includes household addresses that are not mailable, that we do not have telephone numbers for, and are CATI non-interviews. Both CATI and CAPI instruments are available to interviewers in English and Spanish.
Methods panel tests that focus on self-response typically do not include the telephone and personal visit components of data collection, instead focusing on mail materials and data collection via Internet and paper questionnaires. The Fall 2015 Mail Messaging Test, 2016 Mail Messaging Test, 2017 Internet Test, and 2018 Internet Test will focus on self-response and will not include telephone or personal visit components of data collection.
Methods panel tests that include CATI and CAPI also typically follow production ACS procedures. Additionally, because the Summer 2015 Mail Messaging Test and the 2015 Fall Mail Messaging Test are using production sample for the test, all modes of data collection will be conducted.
Tests in 2017 and 2018 that include content changes, will also include a CATI interview to followup with all households that responded in the field test and for whom we have a telephone number. This reinterview will focus on particular questions that we are evaluating, and will not include every question asked in the field test. This followup interview is designed to measure either response reliability by re-asking the questions from the original interview or response bias by asking a set of questions deemed to elicit a “truth” measure.
2016 Content Test
The 2016 Content Test will follow the production ACS data collection process (Internet, paper, telephone, and personal visit) as outlined above. The ACS Content Test will differ from the production ACS in that the Content Test will not include data collected via the TQA program. The TQA operation provides direct assistance to respondents while answering the ACS survey, potentially influencing how they interpret and respond to questions. As a result, any cases that respond via TQA will be dropped from our analysis. For cost reasons, the Content Test will not be conducted in Alaska, Hawaii, Puerto Rico, or Group Quarters. The Internet instrument, CATI, and CAPI will be available in English and Spanish.
The 2016 ACS Content Test will also include a telephone followup reinterview, called the Content Followup (CFU), of all households who responded to the initial interview for whom we have a telephone number (either provided on the initial form or obtained from a telephone look-up). The CFU interview will be conducted two weeks after the original interview and be completed within three weeks. Only telephone interviews will be conducted.
At the first contact with a household in CFU, interviewers will ask to speak with the original respondent. If that person is not available, interviewers will schedule a callback at a time when the original respondent is expected to be home. If at the second contact we cannot reach the original respondent, the interview may be conducted with any other eligible household member (a household member who is 15 years or older).
The CFU will ask the basic demographic questions, a subset of the housing questions, and a subset of the detailed person questions. Questions will be asked in the CFU interview regardless of whether the question was left blank in the original interview. All questions will be asked as they are in the Content Test original interview CATI instrument (in order to measure response reliability), except for questions about retirement income. For retirement income, we will instead ask the series of retirement questions from the Current Population Survey in order to measure response bias. See Attachment L3 for the CATI wording that will be used in CFU as well as the specific CFU questions for retirement income.
Methods to Maximize Response
Similar to production ACS, the Methods Panel Tests will include TQA via a toll-free number in the mailings, which respondents may call to obtain help in completing the survey or to address questions regarding their participation in the ACS. Similarly, the mailing materials and online survey will provide links to additional information about the ACS as well as Census Bureau’s policies on privacy, security, and accessibility. The 2016 Content Test will also include the same CATI and CAPI operations used in the current ACS that allow for follow-up with those households that do not respond by Internet or mail.
Additional methods for maximizing response will be determined once the objectives of the tests are finalized.
Test Procedures
Summer 2015 Mail Messaging Test
The ACS Methods Panel provides opportunities for annual testing of content and/or survey methods to achieve efficiencies and improve data quality. This test will utilize the ACS Methods Panel to test the impact of softening or removing the mandatory messages in the mail materials as well as test other communication enhancements in the mail materials.
Fall 2015 Mail Messaging Test
The initial stages of the Fall 2015 Mail Messaging test consisted of developing an additional insert for the questionnaire package with input from subject matter experts and survey methodologists. For the field test, sampled addresses are placed into treatment groups with differing components included in the questionnaire package to evaluate their effectiveness.
2016 ACS Content Test
The initial stages of the Content Test consisted of content determination, cognitive laboratory pretesting, expert reviews and other pretesting methods for the purpose of developing alternate versions of question content identified for field testing. Representatives from numerous federal agencies, as well as other data users, contributed to these early pretesting efforts by providing their subject matter expertise.
For the field test, sampled addresses will be assigned to either the control or test group. Those addresses assigned to the test group will receive the revised ACS questions and the questions new to the ACS. The control group will receive the current questions on the production ACS as well as different versions of the new questions. In addition to the field test, the ACS Content Test will also include a CATI CFU as a method for collecting additional data quality measures. See Attachment L1 – ACS-1(X)CTC for the Content Test Control Treatment Questionnaire, Attachment L2 – ACS-1(X)CTT for the Content Test Test Treatment Questionnaire, and Attachment L3 – Content Test Question Wording for revised question wording and new questions in all modes, as well as CFU questions.
Other Potential Tests
Although plans for the following tests are not yet fully scoped, the following information describes current thoughts on the test procedures for each.
2016 Mail Messaging Tests
The initial stages of the test consists of design reviews by survey methodologists and subject matter experts. For the field tests, sampled addresses are placed into treatment groups with differing mailing strategies. Details of the sample design and treatments will be determined once the objectives of the tests are finalized.
2017 and 2018 Internet Tests
The initial stages of the tests consist of design reviews by survey methodologists, subject matter experts, and reviews of previous Internet test results. Cognitive or usability testing may be conducted, depending on the design of the test. Testing will focus on various topics and may change based on issues arising in ACS production or results of early testing. For the field tests, sampled addresses are placed into treatment groups. Details of the sample design and treatments will be determined once the objectives of the tests are finalized.
2017 and 2018 Self-Response Tests
The initial stages of the tests consist of design reviews by survey methodologists and subject matter experts. Cognitive or usability testing may be conducted, depending on the design of the test and the inclusion of content changes. Representatives from federal agencies will contribute to these early pretesting efforts by providing their subject matter expertise. For the field tests, sampled addresses are placed into treatment groups. Details of the sample design and treatments will be determined once the objectives of the tests are finalized.
Contacts for Statistical Aspects and Data Collection
The Census Bureau will collect and process these data as needed for each test. Within the Census Bureau, please consult the following individuals for further information.
Statistical Aspects
Anthony Tersine Assistant Division Chief, ACS Methods
Decennial Statistical Studies Division
Phone: (301) 763-1994
Overall Data Collection
Deborah Stempowski Chief, American Community Survey Office
Phone (301) 763-1417
Attachments
Note: Attachments A – F are the ACS production materials for 2015. Attachments G-J are materials for the Summer 2015 Mail Messaging Test. Attachment K is the mail insert for the Fall 2015 Mail Messaging Test. The question wording that will be used in the 2016 Content Test is outlined in Attachments L1 – L3. Any necessary modifications to attachments A – F for the other future tests will be provided to OMB as test plans mature.
ACS First Mailing (Initial Package)
Attachment A1 - ACS-13(L) (2015)(5-2015), ACS Introductory Letter
Attachment A2 - ACS-34IM(08-08-2013), Internet Response Instruction Card
Attachment A3 - ACS-10SM(2015)(6-2014), ACS Frequently Asked Question Brochure
Attachment A4 - ACS-9(2014)(8-2013), Multilingual Brochure
Attachment A5 - ACS-46IM(2013)(12-2012), ACS Stateside Outgoing Envelope
Internet Screen Capture Guide
Attachment B - American Community Survey Internet Screen Capture Guide (2015)
ACS Second Mailing (Reminder Letter)
Attachment C1 - ACS-20(L)(2015)(5-2015) Reminder Letter
Attachment C2 – ACS-40(2012) (6-2011) – ACS Reminder Envelope
ACS Third Mailing (Questionnaire Package)
Attachment D1 - ACS-1(2015)(06-17-2014), ACS Stateside Questionnaire
Attachment D2 - ACS-14(L)SM(2013)(6-2013), ACS Follow-up Letter
Attachment D3 - ACS-34RM(04-04-2014), ACS Instruction Card
Attachment D4 - 6385_47(2014)(10-2013), ACS Return Envelope
Attachment D5 - ACS-10SM(2015)(6-2014), ACS FAQ Brochure
Attachment D6 - ACS-30(2015)(5-2014), ACS Instruction Guide Booklet
Attachment D7 - ACS-46(2012)(5-2011), ACS Stateside Outgoing Envelope
ACS Fourth Mailing (Reminder Postcard)
Attachment E - ACS-29(2013)(5-2012), Reminder Postcard
ACS Fifth Mailing (Second Reminder Postcard)
Attachment F - ACS-23(2013)(5-2012), Second Reminder Postcard
Attachment G: Softened Mandatory Messaging Treatment (Summer 2015 Mail Messaging Test)
Summer 2015 Mail Messaging Design Treatment Materials
Attachment H1 - A3_Official Internet Invitation Letter
Attachment H2 - A1_Official Main Envelope
Attachment H3 - Rem2_Official Reminder Letter
Attachment H4 - C6_Official Choice Letter
Attachment H5 - C1_Official Main Envelope
Attachment H6 - D1_Official Second Reminder CardAttachment H7 - E1_Official Final Reminder Card
Attachment H8 - C4_Official FAQ
Attachment I: Summer 2015 Mail Messaging Test Wording Modifications
(Minimal Design Treatment, Softened Design Treatment and Design Treatment)
Attachment J: Minimal Design Treatment Initial Letter
Attachment K: ACS-8(X)(2015), “How Your Responses Help America” insert
2016 Content Test Question Wording
Attachment L1 – ACS-1(X)CTC, Content Test Control Treatment Questionnaire
Attachment L2 – ACS-1(X)CTT, Content Test Test Treatment Questionnaire
Attachment L3 – Content Test Question Wording
Attachment M: Secondary Metrics for the Analysis of the 2016 Content Test
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | SUMMARY STATEMENT |
| Author | hicks308 |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |