SUPPORTING STATEMENT
U.S. Department of Commerce
U.S. Census Bureau
American Community Survey Methods Panel Tests
OMB Control Number 0607-0936
Part B. Collections of Information Employing Statistical Methods
Sample Design
2010 ACS Content Test
Universe: The sample universe for the 2010 ACS Content Test consists of all residential addresses in the contiguous United States (excludes Puerto Rico, Alaska, Hawaii, and U.S. territories) from the Census Bureau’s Master Address File. Note that addresses selected to participate in the 2010 ACS will be out-of-scope for the content test. Also no addresses can be in sample (production and methods panel tests) more than once in a 5-year period.
Sample Selection: The sample design will be largely based on the ACS production sample design (multi-stage sample) modified to meet the test objectives. Modifications to the ACS sample design include adding an additional level of stratification by stratifying units into high and low ACS mail response areas; selecting units with equal probabilities of selection within the high/low response strata; and sampling units as pairs. The high and low response strata are defined using ACS mail response rates at the census tract-level. Note that units within the low response strata will be sampled at a higher rate to ensure an approximately equal number of mail responses from both strata. Sample units will be selected in pairs by first systematically sampling an address within the defined sampling strata and then pairing that address with the address listed next in the geographically sorted list. Note that the pair may not be neighboring addresses. One member of the pair will then be randomly assigned to either the control or test group. As with ACS production, this test will implement sub-sampling at the CAPI stage in an effort to reduce data collection costs. The CFU Reinterview component of this test will not involve any sampling. All housing units that respond and for whom we have obtained a telephone number will go to the CFU Reinterview operation.
Sample Size: The 2010 ACS Content Test will consist of a national sample of 70,000 residential addresses in the contiguous United States (the sample universe does not include Puerto Rico, Alaska, and Hawaii) with half in the control and the other half in the test group. The sample size provides us with enough statistical power (.80) to detect a difference in the gross difference rates of at least 2 percent between the control and test groups.
Internet Mode Test
Universe: The sample universe for the Internet Test will include all residential addresses in the United States and Puerto Rico. We select the sample using the Master Address File as the frame and excluding production ACS and Puerto Rico Community Survey (PRCS) sample cases from the universe. Also no addresses can be in sample (production and methods panel tests) more than once in a 5-year period.
Sample Selection: The sample will be selected from the universe of eligible addresses using a stratified systematic design. The design is based on a combination of research done on the planning database for the 2010 Census Integrated Communications Plan and results from the Census Barriers, Attitudes, and Motivators Survey (CBAMS) that collected information from respondents about their Internet use. We plan to stratify tracts within our sample that are more likely to use the Internet, specifically those with a large proportion of advantaged homeowners or a large proportion of single, unattached, mobile households. We will analyze these tracts separately from the remaining tracts in our sample universe. There will not be any stratification in Puerto Rico since Internet access is not as prominent there as in the United States.
Sample Size: The Internet Mode Test will consist of a national sample of 120,000 residential addresses in the United States (15,000 addresses each in eight panels). As mentioned above, there are two strata – one containing households in tracts that we feel are more likely to use the Internet based on previous studies, and the remaining stratum containing the balance of tracts. For each stratum, there are four experimental treatments with different Internet notification strategies:
Not Prominent Offer - The Internet option will appear only on the questionnaire in a non-prominent place.
Prominent Offer (Choice) - Households will be given a choice of completing the ACS on paper or the Internet. The Internet option will appear in the mail materials and be prominently displayed on the questionnaire.
Push Internet on Regular Mailing Schedule - In the Initial Mailing, households will only be given instructions on how to complete the ACS on the Internet. They will not receive a questionnaire until the Replacement Mailing. The Internet messages will appear in the mail materials and be prominently displayed on the replacement questionnaire. The mailing sequence of the materials will follow the same schedule as ACS production.
Push Internet on Modified Mailing Schedule – Households will receive the same materials as the previous treatment except that the Replacement Mailing will be sent about two weeks after the Initial Mailing and before the Reminder Postcard.
The control for this test will consist of 30,000 ACS production cases – 15,000 in each stratum. The 120,000 sample size provides us with enough statistical power (.80) to detect a difference of about 2 percentage points in the overall response rates between the panels.
For Puerto Rico, we will select 2,000 residential addresses (1,000 addresses each in two panels). The two treatments will be a Not Prominent Offer and a Prominent Offer (Choice) as described above. This sample size provides us with enough statistical power (.80) to detect a difference of about 4.3 percentage points in the response rates during the mail data collection month between the panels.
ACS Content Reinterview Study
Universe: ACS production respondents
Sample Selection: Each month, we will select a sample of ACS production respondents with a phone number. The sample will be allocated in one of two ways: 1) to ensure equal numbers of respondents to each mode (mail, CATI, and CAPI) or 2) to ensure proportional allocation to the production respondent distribution by mode.
Sample Size: For the reinterview, we plan to break out the ACS interview questions into four subsets, called modules. Using an 80% power analysis, we determined that we need 15, 300 cases total for the year for one question module (or 5,100 per month) in order to detect a gross difference rate of 1.5% or higher. The gross difference rate measures the percentage of inconsistent responses between the original interview and the reinterview. Assuming an 85% response rate to the reinterview, we would need 18,000 cases for the year for one question module (6,000 cases per month). Since there are a total of 4 question modules, the total sample for the year is 72,000. This results in approximately a 3.8% sample of all ACS production respondents.
Procedures for Collecting Information
2010 ACS Content Test
The field test portion of the ACS content test will largely be based on the data collection methods currently used in the production ACS. All sampled addresses will be mailed a pre-notice letter, a self-administered paper questionnaire, and a reminder postcard. Households that do not return their initial questionnaire in a timely manner will also be mailed a replacement questionnaire. For households that do not return their mailed questionnaire, we will attempt to collect their data through a follow-up CATI or CAPI.
For the CFU portion of the ACS content test, we will attempt a follow-up CATI interview with all households that responded in the field test and for whom we have a telephone number. This reinterview will focus on particular questions which we are evaluating, and will not include every question asked in the field test.
The ACS Content Test and the current ACS will differ in that the content test will not include data collected via the TQA program. The TQA operation provides direct assistance to respondents while answering the mail form, potentially influencing how they interpret and respond to questions. As a result, any cases that respond via TQA will be dropped from our sample.
The ACS Content Test will also differ from the current ACS in that the Telephone Failed Edit Follow-up (FEFU) operation will not be included in the content test. The FEFU operation follows up households that return a mail form to collect more complete data. The 2010 ACS Content Test excludes this operation so that we can analyze data that most directly reflects the household’s response to the questions.
Internet Test
Currently, the ACS and the PRCS collect data using three modes: mailout/mailback of a paper questionnaire, CATI, and CAPI. In the proposed test we will offer a fourth response mode - an Internet self-response option - to respondents in the ACS and the PRCS.
Different notification strategies will be used to inform respondents of the Internet response option. In all strategies, the URL for the secure ACS Internet site and instructions for completing the survey online will be provided to respondents by mail.
The Census Bureau will design four versions of the ACS Internet instrument in-house - an English version and a Spanish version for both the ACS and the PRCS. The Internet instrument will be designed such that it presents stimulus to respondents consistent with the other data collection modes, while taking advantage of the technology to the extent possible to improve the quality of data. The Internet instruments will provide help on how to use the instrument as well as question-related help. We will still offer respondents the option of TQA. The current TQA operation provides direct assistance to respondents while answering the mail form, potentially influencing how they interpret and respond to questions.
This test will not include the CATI or CAPI operations like ACS production. Instead, a reinterview of Internet respondents will be conducted by CATI. The details about the content of the reinterview will follow. Also, a qualitative interview of Internet respondents, mail respondents, and nonrespondents will be conducted by CATI to ask about Internet use, privacy concerns, and the effectiveness of the mail materials.
ACS Content Reinterview Study
Cases that responded to our production ACS from all three ACS response modes (mail, CATI, and CAPI) will be included. Reinterview modules containing a subset of the questions will be created so that the entire ACS questionnaire can be tested over several months (so as not to overburden respondent households by reasking all of the ACS questions). Each question set or module of the CRS will require multiple data collection months to provide enough sample for analysis purposes.
Centralized CATI reinterviews will be conducted within 3 to 4 weeks of the original data collection. It is important that the reinterview is close enough timing-wise to the original data collection to minimize the possibility of changes in what you are trying to measure but far enough away so respondents do not exactly remember previous responses.
At the first contact with a household, the interviewer will ask to speak with the original respondent. If that person is not available, the interviewer will schedule a callback at a time when the original respondent is expected to be home. If, at the second contact, we cannot reach the original respondent, the interviewer will complete the interview with another adult household member.
Methods to Maximize Response
2010 ACS Content Test
The tests will follow the same general mailing strategy used by the ACS which has a stated goal of an overall weighted response rate of 92% or higher.
Similar to the current ACS, the content test will include TQA via a toll-free number on the questionnaires, which respondents may call to obtain help in completing the survey or to address more general questions regarding their participation in the ACS. However, interviews completed via TQA will not be included in the analysis as stated in Section 2 above. Similarly, materials for all tests will include a URL where respondents can go to obtain additional information about the ACS. The test will also include the same CATI and CAPI operations used in the current ACS that allow for follow-up with those households that do not respond by mail.
Internet Mode Test
Respondents will be contacted using multiple contact mailing strategies designed to increase response rate. At least one panel will use the current ACS mailing schedule and strategy. The production mailing pieces will be modified to include instructions for Internet response and messages encouraging response by Internet.
ACS Content Reinterview Study
Calls will be placed over a 2-week period within a few weeks of completing the ACS production interview. Attempts will be made at differing times across days to maximize response.
Test Procedures
2010 ACS Content Test
The initial stages of the Content Test consists of content determination, cognitive laboratory pretesting, expert reviews and other pretesting methods for the purpose of developing alternate versions of question content identified as eligible for the field testing. Representatives from numerous federal agencies, as well as other data users, have contributed to these early pretesting efforts by providing their subject matter expertise in the development of alternatives.
For the field test, sampled addresses are assigned to either the control or test group. Those addresses assigned to the test group will receive the revised ACS questions and the questions new to the ACS. The control group will receive the current questions on the production ACS as well as different versions of the new questions. Recordings of the test and control questions captured via Computer Audio Recorded Interviewing (CARI) technology will be behavior coded to identify other issues with the administration of these questions in the interviewer-administered modes. In addition to conducting a field test, the ACS content test will also include a CATI CFU as a method for collecting additional data quality measures.
Internet Test
Several rounds of testing are planned throughout the design and implementation of the Internet Mode Test. These include low-fidelity (paper) usability testing, cognitive testing, high-fidelity usability testing of the instruments, system testing of the integration and communication with current systems, and usability testing of respondent instructions in letters and on forms. The usability testing materials are approved under OMB clearance 0607-0725. Web survey design experts, survey methodologists, and subject matter experts will review the instruments during design and production. We will follow this pretesting with a field test to measure the impact on response from offering an Internet response option.
For the test, each sampled address will be assigned to one of the notification strategy panels. Addresses in all sample panels will receive information about the Internet response option to some degree and will have the opportunity to complete the ACS questionnaire using the Internet.
ACS Content Reinterview Study
System testing, including testing communication with the production system, will be done. Additionally, prior to doing a full year implementation, a one-month field test of the reinterview system will be done to ensure that it is working as designed.
Contacts for Statistical Aspects and Data Collection
The Census Bureau will collect and process these data as needed for each test. Within the Census Bureau, please consult the following individuals for further information.
Statistical Aspects
Anthony Tersine Assistant Division Chief, ACS Methods
Decennial Statistical Studies Division
Phone: (301) 763-1994
Overall Data Collection
Susan Schechter Chief, American Community Survey Office
Phone (301) 763-8950
Attachments
Note: Attachments A – G are the ACS production materials for 2010. We will modify attachments A – E and G for the 2010 ACS Content Test. We will make separate modifications to attachments A – D, F, and G for the Internet Test. We will provide these documents for the tests later as a change request to OMB.
Attachment A: ACS-12(L)S, Prenotice Letter
Attachment B: ACS-13(L)SM, Introductory Letter
Attachment C: ACS-1, Questionnaire
Attachment D: ACS-10SM, Frequently Asked Questions Brochure
Attachment E: ACS-30, Instruction Guide Booklet
Attachment F: ACS-20S, Reminder Card
Attachment G: ACS-14(L)SM, Followup Letter
Attachment H: 2010 ACS Content Test Cognitive Interviewing Final Report
Attachment I: 2010 ACS Content Test Proposed Field Test Question Wording
Attachment J: Internet Mode Test: Proposed Sample Screen Shots
| File Type | application/msword |
| File Title | SUMMARY STATEMENT |
| Author | hicks308 |
| Last Modified By | hoste003 |
| File Modified | 2009-12-29 |
| File Created | 2009-12-29 |