Department of Commerce
United States Census Bureau
OMB Information Collection Request
2018-2021 American Community Survey Methods Panel Testing
OMB Control Number 0607-0936
Part B – Collection of Information Employing Statistical Methods
The Mail Materials Test will use production American Community Survey (ACS) sample. The sample universe and sampling procedures for the production ACS consists of all mailable residential addresses in the United States from the Census Bureau’s Master Address File and employs a two-phase, two-stage sample design. Further information about the production ACS sample design can be found in the ACS Design and Methodology Report (U.S. Census Bureau, 2014).
The monthly ACS production sample of approximately 295,000 housing unit addresses is divided into 24 groups, called methods panel groups, where each group contains approximately 12,000 addresses. Each methods panel group is a representative subsample of the entire monthly sample, and each monthly sample is representative of the entire yearly sample and the country. The Census Bureau will use two randomly selected methods panel groups for each experimental treatment for this test. Hence, each treatment will have a sample size of approximately 24,000 addresses. In total, approximately 168,000 addresses will be used for the seven experimental treatments. The sample size will be able to detect differences of approximately 1.25 percentage points between the self-response return rates of the control and experimental treatments. The power of the test, which is 80 percent and where α=0.1, assumes a 50 percent response rate. The test will exclude sample in Remote Alaska since sampled addresses do not receive mail materials. Puerto Rico will also be out of scope for this test because of cost and operational complexities, including translation of materials. Additionally different mailing methods are used in Puerto Rico and low self-response makes it difficult to identify differences in experimental treatments.
Universe: The sample universe for the mail messaging tests consists of all mailable residential addresses in the United States from the Census Bureau’s Master Address File. Addresses selected to participate in production ACS will be out-of-scope for the tests. Also, no addresses can be in sample (production and methods panel tests) more than once in a 5-year period.
Sample Selection: The sample design will be based on the ACS production multi-stage sample design. Once the number of treatments is determined, methods will be developed to sample addresses and randomly assign them to the various experimental treatments. We are also considering using production sample, when possible, to conduct these tests; similar to the approach used for the Mail Materials Test.
Sample Size: Each mail messaging test will have a national sample of 60,000 addresses in the United States divided into treatments. The number of treatments will be determined as the details of the tests are defined. These tests will exclude Remote Alaska since sampled addresses do not receive mail materials. Similarly, Puerto Rico is out of scope for these tests because different mail methods are used.
Universe: The sample universe for the Multlingual Test consists of all mailable residential addresses in the United States from the Census Bureau’s Master Address File. Addresses selected to participate in production ACS will be out-of-scope for the test. Also, no addresses can be in sample (production and methods panel tests) more than once in a 5-year period.
Sample Selection: The sample design will be based on the ACS production multi-stage sample design but will also take into account the need to oversample areas of the county with a higher concentration of housing units needing materials in languages other than English. Once the number of treatments is determined, methods will be developed to sample addresses and randomly assign them to the experimental treatments.
Sample Size: This test is estimated to need a targeted national sample of 100,000 addresses, divided into treatments. The number of treatments will be determined as the details of the test are defined. These tests will exclude Remote Alaska since sampled addresses do not receive mail materials.
Universe: The sample universe for the Content Tests consists of all mailable residential addresses in the United States from the Census Bureau’s Master Address File. Addresses selected to participate in production ACS will be out-of-scope for the tests. Also, no addresses can be in sample (production and methods panel tests) more than once in a 5-year period.
Sample Selection: The 2019 Content Test will use production ACS sample for the test. The Census Bureau is currently planning for a control treatment which will contain the current ACS wording and a test treatment which would contain updated wording to be tested. Once the topics for the tests are determined, sample size calculations will be made and methods panel groups will be selected. Randomly selected methods panel groups will be assigned either the control treatment or test treatment.
The 2021 Content Test will use sample separate from the production ACS. The sample design will be based on the ACS production multi-stage sample design and on methods used in prior Content Tests. The sample size will be determined based on the topics being tested to ensure differences between treatments are detectable to a sufficient degree.
Sample Size: The 2019 Content Test is estimated to need a national sampel of 50,000 housing units, divided into treatments. A refinement of that estimate will be made once the topics for the test are determined. Puerto Rico and remote Alaska will be excluded from this test.
The 2021 Content Test is estimated to need a national sample of 100,000 addresses, divided into treatments. The number of treatments will be determined as the details of the test are defined. The past ACS Content Tests have excluded Puerto Rico, Alaska, and Hawaii for cost reasons. The Census Bureau will reconsider if Hawaii and non-remote portions of Alaska should continue to be excluded. These tests will continue to exclude Remote Alaska.
Universe: The sample universe for the Respondent Burden Questions Test consists of all mailable residential addresses in the United States from the Census Bureau’s Master Address File. Addresses selected to participate in production ACS will be out-of-scope for the test. Also, no addresses can be in sample (production and methods panel tests) more than once in a 5-year period.
Sample Selection: The sample design will be based on the ACS production multi-stage sample design. Once the number of treatments is determined, methods will be developed to sample addresses and randomly assign them to the experimental treatments.
Sample Size: This test is estimated to need a national sample of 100,000 addresses, divided into treatments. The number of treatments will be determined as the details of the test are defined. This test will focus on self-response modes and will exclude Remote Alaska since sampled addresses do not receive mail materials as well as Puerto Rico.
Universe: The sample universe for the Respondent Comment/Feedback Test consists of all mailable residential addresses in the United States from the Census Bureau’s Master Address File. Addresses selected to participate in production ACS will be out-of-scope for the test. Also, no addresses can be in sample (production and methods panel tests) more than once in a 5-year period.
Sample Selection: The sample design will be based on the ACS production multi-stage sample design. Once the number of treatments is determined, methods will be developed to sample addresses and randomly assign them to the experimental treatments.
Sample Size: This test is estimated to need a national sample of 100,000 addresses, divided into treatments. The number of treatments will be determined as the details of the test are defined. These tests will focus on self-response modes and will exclude Remote Alaska since sampled addresses do not receive mail materials.
Universe: The sample universe for the Administrative Data Use Test consists of all mailable residential addresses in the United States from the Census Bureau’s Master Address File as well as Group Quarters (GQ) facilities. Addresses and facilities selected to participate in production ACS will be out-of-scope for the test. Also, no housing unit addresses can be in sample (production and methods panel tests) more than once in a 5-year period.
Sample Selection: The sample design will be based on the ACS production multi-stage sample design. Additional sampling methodology may be incorporated in the design based on the availability of administrative data in an area and other considerations. Once the number of treatments is determined, methods will be developed to sample addresses and randomly assign them to the experimental treatments.
Sample Size: This test is estimated to need a national sample of 100,000 addresses and facilities, divided into treatments. The number of treatments will be determined as the details of the test are defined.
Universe: The sample universe for the Group Quarters Test consists of all non-institutational GQ facilities.
Sample Selection: The sample design will be based the availability of administrative data and other characteristics of the facilities eligible. Once the number of treatments is determined, methods will be developed to sample facilities and randomly assign them to the experimental treatments.
Sample Size: This test is estimated to need a national sample of 500 non-institutational GQ residents, divided into treatments. The number of treatments will be determined as the details of the test are defined.
The Methods Panel Tests typically follow the data collection procedures for the production ACS mailing strategy, with modifications based on the design and purpose of the test. For example, changing the wording in a letter mailed to the sampled address, changing the content of a question, or sending an additional reminder to a sampled address. Section 4 below, on Test Procedures, outlines specific changes to the data collection methodology for the proposed tests. A summary of data collection methodology for production ACS is below. More details about the data collection methodology for the ACS can be found in the ACS Design and Methodology report (U.S. Census Bureau, 2014).
Historically, the ACS had employed a tri-modal data collection strategy for household data collection—mail, telephone and personal visit. In 2011, the Census Bureau conducted two tests to assess the feasibility of providing an internet response option to housing units that receive survey materials by mail. Based on the results of these tests, the Census Bureau implemented an internet response option for the ACS for the start of the 2013 data collection. In October 2017, to ensure the ACS program was executed in the most efficient and cost-effective manner, the Census Bureau discontinued conducting nonresponse followup operations via computer-assisted telephone interviewing (CATI). Several ACS operations continue to make use of the telephone mode: Telephone Questionnaire Assistance (TQA), Failed Edit Followup (FEFU), and centralized reinterview operations.
For housing units eligible to receive survey materials by mail, the first contact includes a letter and an instruction card explaining how to complete the survey online. Also included are a Frequently Asked Questions (FAQ) brochure and a brochure that provides basic information about the survey in English, Spanish, Russian, Chinese, Vietnamese, and Korean, and provides a phone number to call for assistance in each language. The instruction card provides the information on how to respond in English and Spanish. The letter explains that if the respondent is unable to complete the survey online, a paper questionnaire will be sent later. The internet version of the questionnaire is available in English and Spanish and includes questions about the housing unit (HU) and the people living in the HU.
The second mailing is a letter that reminds respondents to complete the survey online, thanks them if they have already done so, and informs them that a paper form will be sent at a later date if we do not receive their response. This letter includes clear instructions to log in, including an explicit reference to the user identification number.
In a third mailing, the American Community Survey Housing Unit Questionnaire Package is sent only to those sample addresses that have not completed the online questionnaire. The content includes a follow up letter, a paper copy of the questionnaire, an instruction card for completing the survey online, an FAQ brochure, and a return envelope. The cover letter with this questionnaire package reminds the household of the importance of the ACS, and asks them to respond soon either by completing the survey online or by returning a completed paper questionnaire.
The fourth mailing is a postcard that reminds respondents that “now is the time to complete the survey,” informs them that an interviewer may contact them if they do not complete the survey, and reminds them of the importance of the ACS.
A fifth mailing is sent to respondents who have not completed the survey yet. This postcard reminds these respondents to return their questionnaires and thanks them if they have already done so. Production ACS mail materials can be found in Attachment G; additional production materials can be found in the OMB submission for the ACS (OMB control number 0607-0810).
The final mode of data collection is Computer-Assisted Personal Interviewing (CAPI) and is used to conduct personal interviews for a sample of addresses for which we have not obtained a self-response (paper or internet). CAPI instruments are available to interviewers in English and Spanish.
The Census Bureau provides TQA for respondents who need assistance with completing the paper or internet questionnaires, who have questions about the survey or who would like to complete the ACS interview over the telephone instead of by other modes. Respondents may call the ACS toll free TQA numbers listed on various ACS mail materials. The TQA staff answers respondent questions and/or completes the entire ACS interview using a computer-assisted telephone interview (CATI) instrument.
In addition to the ACS data collection from housing units, the data are also collected from a sample of GQ facilities and residents. At each sampled GQ facility, one GQ contact is interviewed to collect data about the GQ and to provide a list of residents in the GQ. This list is used to randomly select the sample of individuals to complete the ACS. An introductory letter is mailed to the sample GQ approximately two weeks prior to the period when an FR may begin making contact with the GQ. Resident-level personal interviews with sampled GQ residents are conducted using CAPI, but bilingual paper questionnaires can also be used for self-response. The GQ CAPI and paper questionnaires contain questions for one person.
Similar to production ACS, the Methods Panel Tests include a well researched mail contact strategy (as outlined in Section 2 above) to encourage self-response. TQA is available via a toll-free number in the mailings, which respondents may call to obtain help in completing the survey, to address questions regarding their participation in the ACS, or to complete the questionnaire over the phone. Similarly, the mailing materials and online survey will provide links to additional information about the ACS as well as Census Bureau’s policies on privacy, security, and accessibility.
Nonresponse followup operations are conducted to ensure a final high weighted response rate. The nonresponse followup operations are conducted via CAPI for a sample of addresses for which we have not obtained an internet or mail response. The HU CAPI instrument is available to interviewers in English and Spanish. We maintain high levels of data accuracy and response rates through interviewer instruction, training, and close monitoring of the data.
Additional methods for maximizing response are explored as part of several of the proposed tests. Once details of specific strategies are determined they will be provided as part of a non-substantive change request but in general, will likely include modified messaging in the mail materials to address concerns raised by respondents about the legitimacy of the survey, redesigned materials that are easier to read, multilingual materials to engage respondents with limited English proficiency, or an additional contact, for example.
The initial stages of the test designs involve reviews by survey methodologists, subject matter experts, and statisticians. Literature reviews are conducted as part of study plan development. Cognitive or usability testing may be conducted, depending the design of the test and prior research already conducted. Representatives from other federal agencies are included in development activities when the test relates to changes in content. Study plans are developed for each test and are reviewed internally by a cross-divisional experts in the areas related to the test. Reports are also reviewed in a similar manner.
The Census Bureau will collect and process these data as needed for each test. Within the Census Bureau, please consult the following individuals for further information on their area of expertise.
Statistical Aspects
Mr. Anthony Tersine Assistant Division Chief for Experiments and Evaluations
Decennial Statistical Studies Division
Phone: (301) 763-1994
Overall Data Collection
Dr. Victoria A. Velkoff Chief, American Community Survey Office
Phone: (301)-763-3609
U.S. Census Bureau. 2014. American Community Survey Design and Methodology. Washington, D.C.: U.S. Census Bureau. Accessed February 1, 2018. https://www.census.gov/programs-surveys/acs/methodology/design-and-methodology.html.
Attachment A: Description of Mail Materials for all Treatments in the Mail Materials Test
Attachment B: Treatment 1 Materials for the Mail Materials Test
Attachment C: Treatment 2 Materials for the Mail Materials Test
Attachment D: Treatment 3 Materials for the Mail Materials Test
Attachment E: Treatment 4 Materials for the Mail Materials Test
Attachment F: Treatment 5 Materials for the Mail Materials Test
Attachment G: Production ACS Mail Materials
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | 2018 E2E Census Test Part B |
| Author | Amy M Bishton (CENSUS/DCMD FED) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-15 |