Collections of Information Employing Statistical Methods
Sample Design
As with the ACS, the 2007 Methods Panel tests will use the most recent version of the Census Bureau’s Master Address File for the sampling frame, and will only sample from the ACS 2007 sub-frame. Addresses selected to participate in the 2007 ACS will be out-of-scope for these tests, as will addresses in Puerto Rico. The test of new content will also exclude Alaska and Hawaii from the sample in order to contain field data collection costs. For each test, analysis will occur at a national level only, so the sample will reflect a national level design.
The test measuring the impact on response rates of using standard versus first-class postage for the prenotice letter and initial questionnaire packet will select a sample using the same basic process as the ACS production sample, though the sampling rate will be set to obtain a total sample of 20,000. All of the sampled addresses will receive a prenotice letter mailed using standard class postage since the analytical comparison group will come from the ACS February panel for which we send prenotice letters using first-class postage.
The test comparing the grid versus the sequential person design of the basic demographic section of the questionnaire will stratify the 2006 sub-frame into high and low response strata at the tract level. Addresses in the low response strata will be sampled at a higher rate in order to obtain a roughly equal number of responses from the two strata at a national level. Over sampling in the low response strata increases the number of responses obtained from minority populations. One of the differences between the grid and sequential design occurs in the specific layout of the response categories for the race question. By increasing the minority responses to the survey, we can better measure differences in response to the race question between the two layouts. The total sample size for this study is 40,000, with about 20,000 receiving the grid format and about 20,000 receiving the sequential person format questionnaires.
If funded, the third test assessing data quality of the proposed new field of degree question will select a sample using the same basic process as the ACS production sample, though the sampling rate will be set to obtain a total sample of 40,000. Half the sample will receive one version of a field of degree question, and the other half will receive a second version of the question. This test will include CATI and CAPI data collection. As with ACS production, this test will implement sub-sampling at the CAPI stage in an effort to reduce data collection costs. The CFU Reinterview component of this test will not involve any sampling. All housing units that respond and for whom we have obtained a telephone number will go to the CFU Reinterview operation.
Procedures for Collecting Information
In the test comparing standard versus first-class postage as well as the comparison of the two questionnaire layouts, we will only collect data by mail since a main goal of each test is to measure the impact on mail response. The U.S. Postal Service will conduct all mail operations. The test comparing standard versus first-class postage on the prenotice letter will follow the mailing strategy and timing used by the February panel of ACS production with the exception that we will mail the prenotice letter using standard postage a few days earlier to account for potentially slower mail delivery. By mailing the standard postage prenotice letter a little earlier, we anticipate that addresses will receive these letters about the same time as the addresses in the ACS February panel receive the prenotice letters mailed with first-class postage.
The test comparing the grid versus sequential layout of the basic demographic section of the ACS will follow the same mailing strategy as the March panel of ACS production, without any exceptions. About half the sample will receive the grid version of the questionnaire in both the initial and replacement mailing package, if the address requires a replacement package. The other half will receive the initial and replacement questionnaire, if necessary, with the sequential person design.
The test assessing data quality of the field of degree question will collect data using all the same modes as ACS production; mail, CATI and CAPI. Additionally, all housing units that respond and for whom we have a phone number will continue to the CFU Reinterview, also a CATI operation. Each phase of data collection will mirror the start and end dates as the March panel of ACS production. The CFU Reinterview operation will begin about two weeks after mail-out of the initial questionnaire package and will continue until approximately two weeks after the close of the CAPI operation. Cases will continue to the CFU Reinterview on a flow basis, with each case going to CFU Reinterview in about two weeks after the receipt of the original interview.
Methods to Maximize Response
The tests will follow the same mailing strategy used by the ACS which consistently results in overall response rates of 94% or higher, with about 50% of total response coming from mail. The obvious exception to the mailing strategy applies to the test comparing standard verses first-class versus postage in which we will slightly alter the timing of the first mailing to account for potentially slower mail delivery. However, the content of the prenotice letters will remain unchanged.
In developing the questionnaires for the test of the two different layout designs, staff conducted cognitive pretesting and reviewed completed ACS forms to identify issues with navigation that might impact item-level response. The cognitive test targeted respondent groups shown in previous testing to have more difficulty with form navigation. Both the pretesting and the review of forms facilitated the design of the questionnaires to improve both unit and item level response.
All tests will include a toll-free number on the questionnaires, which respondents may call to obtain help in completing the survey or to address more general questions regarding their participation in the ACS. Similarly, materials for all tests will include a URL where respondents can go to obtain additional information about the ACS. If funded, the test of new content will include the CATI and CAPI operations that allow for follow-up with those households that don’t respond by mail.
Test Procedures
As noted above, the 2007 Methods Panel will include more than one test. The first test investigates whether the cost-saving strategy of mailing the prenotice letter using standard postage, holding all other aspects of the mailing operation constant, negatively impacts response to the survey. The primary evaluation measure will be the mail response rate as well as the pattern or timing of response relative to the initial mail out of the questionnaire and the beginning of the next phase of data collection, CATI.
This test will not include CATI or CAPI data collection phases as the ACS production does. However, by using the same mailing strategy as ACS production, we know the point in the mail data collection period when the start of the CATI operation impacts mail response (e.g., the CATI call serves as a reminder, prompting people to return the mail questionnaire). Thus, we can look at patterns and timing of response between mailing the initial questionnaire and the start of the CATI to assess how using standard postage might impact the amount and timing of response during that block of time relative to the ACS production response. As noted above, the control group or comparison group for this study will come from the February panel of ACS production. Computation of the mail response rate will include all returned, non-blank questionnaires received and checked-in on the control system on or before the date the February panel begins CATI operations. If there is not a statistically significant drop in the mail response rate for the sample receiving the prenotice letters mailed using standard postage, the ACS will consider using standard postage to mail the prenotice letter in production.
The second test comparing the grid layout of the basic demographic section (similar to the current ACS layout) to the sequential person design (similar to the anticipated 2010 Census layout) also includes only a mail data collection phase – no CATI or CAPI follow up. However, since the objective of this test focuses on the impact form design has on item-level response, response distributions, patterns of response through the form including form completeness, and to a small extent within household coverage, analysis will include all forms returned, checked-in and captured within six weeks of mail out of the initial questionnaire rather than at the point of identifying the CATI workload. Other than the layout of the questionnaires themselves, the sample sent the grid questionnaire and the sample sent the sequential person design questionnaire would each receive the same mail pieces with the exact same content.
The test assessing the quality of the field of degree question includes mailing half the sample one version of the field of degree question and the other half a second version of the question. One version will use a series of yes/no questions asking if the person received a degree in particular group of related fields. The second version will use a single open-ended question, which asks the person to report the actual degree he/she received. Sampled housing units in each group will receive the same version of the question (i.e., series of yes/no questions or open ended) regardless of whether they participate by mail, CATI or CAPI. Similarly, the CFU Reinterview will ask the same version of the question as asked in the original mail, CATI or CAPI interview.
The text responses in the open-ended version will be coded into the same group of related fields as offered in the version of the questionnaire using a series of yes/no questions. Comparisons of item missing data rates, distribution by type of field and simple response variance between the two versions will indicate which version results in higher quality. In order for OMB and the Census Bureau to consider including either version of the new question on the ACS, the question must obtain similar estimates for each group of related fields as observed from other surveys or other comparable sources.
Contacts for Statistical Aspects and Data Collection
The Census Bureau will collect and process these data as needed for each test. Within the Census Bureau, please consult the following individuals for further information.
Statistical Aspects
Wendy Davis Hicks Chief, ACS Methods Branch
Decennial Statistical Studies Division
Phone: (301) 763-2431
Overall Data Collection
Lisa Blumerman Acting Chief, American Community Survey Office
Phone (301) 763-8050
Attachments
A ACS-12(L)S, Prenotice Letter
B ACS-1, ACS Questionnaire Package
C ACS-20S, Reminder Card
D ACS-1(X)Seq, ACS Questionnaire/Sequential Format
E ACS-1(X)Pro, ACS Questionnaire/Grid Format
F ACS Content Policy
| File Type | application/msword |
| File Title | SUMMARY STATEMENT |
| Author | hicks308 |
| Last Modified By | boyer001 |
| File Modified | 2006-09-28 |
| File Created | 2006-09-28 |