SUPPORTING STATEMENT
U.S. Department of Commerce
U.S. Census Bureau
American Community Survey Methods Panel Tests
OMB Control Number 0607-0936
Part B. Collections of Information Employing Statistical Methods
Sample Design
2013 Questionnaire Design Test
Universe: The sample universe for the 2013 Questionnaire Design Test consists of all mailable residential addresses in the United States from the Census Bureau’s Master Address File. Addresses selected to participate in the 2013 production ACS will be out-of-scope for the Questionnaire Design Test. Also, no addresses can be in sample (production and methods panel tests) more than once in a 5-year period.
Sample Selection: The sample design will be based on the ACS production multi-stage sample design. We will select the sample by first systematically sampling an address and then grouping that address with the five addresses listed next in the geographically sorted list of eligible units. Each member of the group will be randomly assigned to the various experimental panels.
Sample Size: The Questionnaire Design Test will have a national sample of 50,000 addresses in the United States divided into five treatments. The sample size provides us with enough statistical power (.80) to detect a difference in the self-response rates of at least 2.2 percentage points between the treatments. We will also analyze item nonresponse rates as well as response distributions for any questions that experienced a substantial change in location on the questionnaire. We do not anticipate that the change in question position will affect response behavior, but we will use Chi-Square testing to confirm our hypothesis.
2015 ACS Content Test
Universe: The sample universe for the 2015 ACS Content Test consists of all residential addresses in the United States from the Census Bureau’s Master Address File (excluding remote Alaska). Note that addresses selected to participate in the 2015 production ACS will be out-of-scope for the Content Test. Also, no addresses can be in sample (production and methods panel tests) more than once in a 5-year period.
Sample Selection: The sample design will be based on the ACS production sample design (multi-stage sample) modified to meet the test objectives. Modifications to the ACS sample design include adding an additional level of stratification by stratifying units into high and low ACS self-administered (Mail/Internet) response areas; selecting units with equal probabilities of selection within the high/low response strata; and sampling units as pairs. The high and low response strata are defined using ACS self-administered response rates at the census tract-level. Note that units within the low response strata will be sampled at a higher rate to ensure an approximately equal number of responses from both strata. Sample units will be selected in pairs by first systematically sampling an address within the defined sampling strata and then pairing that address with the address listed next in the geographically sorted list. Note that the pair may not be neighboring addresses. One member of the pair will then be randomly assigned to either the control or the test group. As with ACS production, this test will include subsampling at the CAPI stage. The Content Follow-Up (CFU) Reinterview component will include all housing units that respond to the Content Test for whom we have a telephone number.
Sample Size: Because we do not know what questions will be proposed for content testing at this time, we estimate that the sample size using the size of the 2010 Content Test. The 2010 ACS Content Test consisted of a national sample of 70,000 residential addresses in the United States with half in the control and the other half in the test group. The sample size provided us with enough statistical power (.80) to detect a difference in the gross difference rates of at least 2 percent between the control and test groups. We will aim to detect the same difference level in the 2015 Content Test, and thus will submit a change request if the content selected for testing warrants a different sample size.
Internet Tests
Universe: The sample universe for the Internet Tests will include all mailable residential addresses in the United States (and Puerto Rico, where appropriate). We will select the samples using the Master Address File as the frame and excluding production ACS and Puerto Rico Community Survey (PRCS) sample cases from the universe. No addresses can be in sample (production and methods panel tests) more than once in a 5-year period.
Sample Selection: The sample will be selected from the universe of eligible addresses. We will incorporate stratification into the sample design as needed to meet test objectives. We will select the sample by first systematically sampling an address within the defined sampling strata and then grouping that address with the address(es) listed next in the geographically sorted list. Each member of the group will be randomly assigned to the various experimental panels.
Sample Size: The Internet Tests will range in size, but in total, will not exceed 200,000 residential addresses in the United States and 4,000 addresses in Puerto Rico across the six proposed tests.
Sample design for other testing will be determined once the objectives for those tests are finalized.
Procedures for Collecting Information
2013 Questionnaire Design Test
The field test portion of the Questionnaire Design Test will use some of the data collection methods used in the production ACS. We will mail all sampled addresses a pre-notice letter, a request to respond online, and a reminder postcard. Households that do not respond online in a timely manner will also be mailed a paper questionnaire and supplemental postcard urging response. We will not attempt CATI and CAPI contacts for addresses that do not respond online or return their mailed questionnaire; however, we will send the postcard used in production to households that did not respond and are ineligible for CATI because there is no phone number available for the address.
2015 ACS Content Test
The field test portion of the ACS Content Test will be based on the data collection methods used in the production ACS. We will mail all sampled addresses a pre-notice letter, a request to respond online, and a reminder postcard. Households that do not respond online in a timely manner will also be mailed a paper questionnaire and supplemental postcard urging response. For households that do not respond online or return their mailed questionnaire, we will attempt to collect their data through a follow-up CATI (or send another postcard if there is no phone number for the unit) or CAPI. During CATI and CAPI interviews, we will use Computer Audio Recorded Interviewing (CARI) technology to record portions of the interview related to the questions being tested for use in behavior coding. The purpose of the behavior coding is to identify interviewer or respondent behaviors that may contribute to response error.
For the CFU portion of the ACS Content Test, we will use a CATI interview to follow-up with all households that responded in the field test and for whom we have a telephone number. This reinterview will focus on particular questions that we are evaluating, and will not include every question asked in the field test.
The ACS Content Test and the production ACS will differ in that the Content Test will not include data collected via the TQA program. The TQA operation provides direct assistance to respondents while answering the ACS survey, potentially influencing how they interpret and respond to questions. As a result, any cases that respond via TQA will be dropped from our analysis.
The ACS Content Test will also differ from production ACS in that the Telephone Failed Edit Follow-up (FEFU) operation will not be included in the Content Test. The FEFU operation follows up households that return a mail form or respond on the Internet to collect data that are more complete. The ACS Content Test excludes this operation so that we can analyze data that most directly reflects the household’s initial response to the questions.
Internet Tests
We will use the same mailing strategy that is currently proposed to start in the ACS in January 2013 in testing. We will mail sampled addresses a pre-notice letter, a request to respond online, and a reminder postcard. Households that do not respond online in a timely manner will then be mailed a paper questionnaire and supplemental postcard urging response. We may test some supplemental contacts by email or text, depending on the availability of this information and compliance with privacy regulations, to attempt to increase response or data quality. Additionally, we may test some new features in the Internet instrument to help facilitate response or enhance the quality of the data collected. These tests will not include the CATI or CAPI operations like ACS production, since we are testing response by a self-administered mode.
The manner in which information will be collected for other tests will be determined once the objectives of those tests are finalized.
Methods to Maximize Response
2013 Questionnaire Design Test
Similar to production ACS, the Questionnaire Design Test will include TQA via a toll-free number in the mailings, which respondents may call to obtain help in completing the survey or to address questions regarding their participation in the ACS. Similarly, mail materials will include a URL where respondents can go to obtain additional information about the ACS.
2015 ACS Content Test
Similar to production ACS, the Content Test will include TQA via a toll-free number in the mailings, which respondents may call to obtain help in completing the survey or to address questions regarding their participation in the ACS. However, interviews completed via TQA will not be included in the analysis as stated in Section 2 above. Similarly, materials will include a URL where respondents can go to obtain additional information about the ACS. The test will also include the same CATI and CAPI operations used in the current ACS that allow for follow-up with those households that do not respond by mail or Internet.
Internet Tests
We will contact sampled households in the same manner we will be contacting households in ACS production starting in January 2013. We may test methods to increase response that alter some of the contacts. The TQA operation will also provide an opportunity for respondents to call with questions or do their interview over the phone. Moreover, the mailing materials and online survey will provide links to additional information about the ACS as well as Census Bureau’s policies on privacy, security, and accessibility.
Methods for maximizing response for the other tests will be determined once the objectives of those tests are finalized.
Test Procedures
2013 Questionnaire Design Test
The initial stages of the Questionnaire Design Test consists of reviews by survey methodologists and subject matter experts as well as cognitive laboratory pretesting for the purpose of developing alternate versions of the questionnaire for the field testing.
For the field test, sampled addresses from all treatments will be part of the same mailing strategy. The only differences will be in the second mailing, where the questionnaire will differ by treatment. The envelope will also be of a different size and shape depending on the dimensions and page length of the questionnaire.
2015 ACS Content Test
The initial stages of the Content Test consists of content determination, cognitive laboratory pretesting, expert reviews and other pretesting methods for the purpose of developing alternate versions of question content identified for the field testing. Representatives from numerous federal agencies, as well as other data users, will contribute to these early pretesting efforts by providing their subject matter expertise.
For the field test, sampled addresses will be assigned to either the control or test group. Those addresses assigned to the test group will receive the revised ACS questions and the questions new to the ACS. The control group will receive the current questions on the production ACS as well as different versions of the new questions. Recordings of the test and control questions captured via Computer Audio Recorded Interviewing (CARI) technology will be behavior coded to identify other issues with the administration of these questions in the interviewer-administered modes. In addition to conducting a field test, the ACS Content Test will also include a CATI CFU as a method for collecting additional data quality measures.
Internet Tests
We are planning several rounds of testing the design and implementation of the Internet mode. Across the three-year period, we will use a combination of cognitive testing, usability testing, and field testing to support research objectives. We will seek approval for the cognitive and usability testing under another OMB clearance once testing plans are finalized.
Testing will focus on various topics and may change based on issues arising in ACS production or results of early testing. The goals for the first tests are to reduce the number of cases that break-off the Internet survey to minimize the negative effects on item nonresponse. We are considering several ways to approach that goal: initiating contact using different mediums (email or text) for break-offs, using encouraging language to promote continuation in the Internet instrument, increasing the ease of survey re-entry, and changing some facets of the design to help encourage respondents to provide the information they know for their household.
Test procedures for other tests will be determined once the objectives of those tests are finalized.
Contacts for Statistical Aspects and Data Collection
The Census Bureau will collect and process these data as needed for each test. Within the Census Bureau, please consult the following individuals for further information.
Statistical Aspects
Anthony Tersine Assistant Division Chief, ACS Methods
Decennial Statistical Studies Division
Phone: (301) 763-1994
Overall Data Collection
James Treat Chief, American Community Survey Office
Phone (301) 763-3609
Attachments
Note: Attachments A – G are the ACS production materials for 2013. We will modify attachments A – E and G for the 2015 ACS Content Test. We will modify attachment C and possibly attachments E for the Questionnaire Design Test. We may make modifications to attachments A – D, and F-G for the Internet Test. We will provide the revised test documents to OMB once they are finalized.
A First Mailing: ACS-12(L)S (2013), ACS pre-notice letter; and ACS-40 (2013), ACS pre-notice envelope, ACS-9(2011) Multilingual Brochure
B Second Mailing: ACS-46(2012), ACS stateside outgoing envelope; ACS-13(L)SM (2013), ACS stateside introductory letter; ACS-10SM (2012), Stateside English ACS frequently asked question brochure; and ACS-34 (2013)IM Initial Mailing Instruction Card
C Internet Questionnaire Screenshots
D Third Mailing: ACS-20(2013) stateside reminder card
E Fourth Mailing, American Community Survey Household (HU) Questionnaire Package: ACS-46(2012), ACS stateside outgoing envelope; ACS-14(L)SM (2013), ACS replacement letter; ACS-1(2013), ACS HU questionnaire, stateside, English; ACS-30(2013), HU instruction guide booklet; ACS-34RM (2013) Replacement Instruction Card; ACS-10SM (2012), ACS frequently asked question brochure; and ACS-5385-47(2011), ACS business reply mail envelope
F Fifth Mailing: ACS-29 (2013), ACS HU new reminder postcard
G Sixth Mailing: ACS-23 (2013), additional mailing postcard, stateside, English
| File Type | application/msword |
| File Title | SUMMARY STATEMENT |
| Author | hicks308 |
| Last Modified By | Jennifer |
| File Modified | 2012-12-11 |
| File Created | 2012-12-11 |