Cqs OMB Part B 4-26-10

SUPPORTING STATEMENT

U.S. Department of Commerce

U.S. Census Bureau

2010 CENSUS PROGRAM FOR EVALUATIONS AND EXPERIMENTS

2010 census quality survey

OMB Control Number 0607-XXXX

Part B. Collection of Information Employing Statistical Methods

1. Universe and Respondent Selection

For the 2010 Census Quality Survey (CQS), the initial sample will be selected from housing units in mailout/mailback areas of the 50 states and the District of Columbia. The sampling frame will exclude housing units in mailout/mailback areas that were mailed a Spanish/English bilingual questionnaire. In addition, group quarters and housing units in any other types of enumeration areas will not be included in the sampling frame. Further, to reduce burden on respondents, any housing units selected for the 2010 Alternative Questionnaire Experiment (AQE), the Content Reinterview Survey (CRS), the Census Coverage Measurement (CCM) operation, or the 2010 ACS sample (entire year) will also be excluded from the sampling frame.

Although the sampling frame consists of those units that are mailed a 2010 Census mail questionnaire, only those (whole housing units) who respond to the census using the mailout/mailback form are invited to complete a reinterview. The goal for the reinterviews is to obtain the same respondent that completed the household’s 2010 Census form.

Response rates for the self-administered reinterviews are difficult to predict since the 2010 CQS is unique in many ways (e.g., it is a reinterview, as opposed to an initial survey request and it is conducted within a census environment). The expected response rate for the Internet ‘push’ reinterview is a rough estimate since the literature on reinterview response rates for web-based surveys is essentially nonexistent. Given the increase in Internet accessibility and usage in the last half of the decade and considering various factors that could negatively affect compliance, we conservatively estimate a 10 percent response rate for the Internet reinterview.

The expected response rate for the mail ‘push’ reinterview is primarily based on the mail response rate for the Census 2000 reinterview (2001 CQS) since the survey design and questionnaire were similar. About 54 percent responded by mail to the 2001 CQS. Thus, we conservatively estimate a 50 percent response rate for the 2010 CQS mail reinterview.

The expected response rates by mode are lowered slightly for the Internet/mail ‘choice’ panel. We assume roughly 43 percent will respond via mail and seven percent will respond via the Internet for a total of 50 percent.

The evaluation will use a stratified random sample to be selected prior to the start of the 2010 Census data collection. Ideally we would prefer to select the sample from the population of 2010 Census mail responses. However, due to time constraints and other operational impediments, this will likely not be possible. Therefore the sample will be pre-selected based on an expected 2010 Census mail response rate of 62 percent.

Approximately 161,290 census housing units will be selected for the Internet ‘push’ reinterview initial sample, approximately 230,415 for the Internet/mail ‘choice’ reinterview initial sample and approximately 32,258 for the mail ‘push’ reinterview intial sample. Of those, we expect roughly 62 percent to return the 2010 Census mail questionnaire. Thus, an estimated 100,000 Internet ‘push’ reinterview invitations, 142,857 Internet/mail ‘choice’ invitations and 20,000 mail ‘push’ reinterview invitations will be mailed out to 2010 Census mail questionnaire respondents (See Figure 1).

Figure 1: Overview of the 2010 Census Quality Survey

The sampling stratification will be based on the audience segmentation clusters that were identified for the 2010 Integrated Communications Program. For the Communications Program, each census tract was assigned to a segmentation cluster based on a factor analysis of various response characteristics. For the 2010 CQS, stratification was based on data correlated with key survey measures (e.g., percentage Internet access at home, Internet home usage frequency). The 2010 CQS will sample tracts in the ‘High Internet Use Stratum,’ which are composed of two of the clusters (‘Advantaged homeowners’ and ‘Single/unattached/mobiles’)¹, both of which are estimated to contain a higher proportion of households that have Internet access and regularly use the Internet. Sample will also be selected in the ‘Medium Internet Use Stratum’ and the ‘Low Internet Use Stratum.’ The mapping of segmentation clusters to the CQS strata are shown in Figure 2.

Figure 2. 2010 Census Quality Survey Strata Description

CQS Stratum Segmentation Clusters Included in CQS Stratum

High Internet Use Advantaged homeowners

Single/unattached/mobiles

Medium Internet Use Average homeowners

Average renters

Low Internet Use Disadvantaged homeowners

Disadvantaged renters

Ethnic homeowners

Ethnic renters

Note that all housing units in each tract are placed in their corresponding stratum.

This evaluation will obtain estimates for the high, medium, and low Internet use areas, but will over-sample housing units from within the two segmentation clusters with expected high Internet usage (i.e., High Internet Use Stratum). For the High Internet Use Stratum, the sampling rate is 2.25 times that for the Low Stratum. The Medium Internet Use Stratum has a sampling rate that is 1.5 times that for the Low Stratum.

2. Procedures for Collecting Information

The reinterviews will be conducted with a sample of 2010 Census mail respondents in order to provide estimates of measurement error associated with the design and content of a self-administered census Internet questionnaire. Since the measurement error structure may differ depending on whether a respondent has only one response mode option (i.e. mail or Internet) versus having a choice between the two modes, we are testing both ‘push’ and ‘choice’ strategies.

The Internet and mail reinterviews will be conducted in late summer of 2010, after the census enumeration activities have been completed, in order to minimize the risk to 2010 Census data collection. However, the reinterviews will be conducted as close to the census enumeration as feasible in order to effectively compare reinterview results to the 2010 Census self-administered mail questionnaire. Presumably, the results collected within the census environment reflect a more generalizable measurement error structure than results from a mid-decade census test instrument. In addition, we hope to capitalize on respondents’ awareness of the 2010 Census to obtain a higher response to the reinterviews than would be possible in the absence of the 2010 Census environment. However, for the Internet reinterview, compliance may suffer to some extent from messages informing potential respondents that there is no Internet response option in the 2010 Census.

The Internet ‘push’ panel reinterview contact strategy will have the same number of contacts as the mail contact strategy that is used for the 2010 Census. This will include an advance letter, initial Internet survey request, reminder postcard, and a final survey request sent only to nonrespondents to the initial survey request. The initial survey request will include an instruction card (in place of a questionnaire) containing information on how to access the survey online. The letter included in the initial survey request asks the respondent to respond to the survey online, but tells them that we will send a paper questionnaire if they are unable to respond online. The final survey request will include information about how to respond online as well as a paper questionnaire to accommodate respondents who are unable or unwilling to respond online.

For the mail ‘push’ panel reinterview, the eligible mail reinterview sample will receive a paper questionnaire in the mail to complete and mail back. This mail reinterview questionnaire will be nearly identical in content to the 2010 Census mail questionnaire but will have a different title and color. In addition, a question about whether the reinterview respondent is the same household member who completed the census form will replace the 2010 Census phone number question, and the form will have a ‘thank you’ that is customized for reinterview responders. The full implementation contact strategy that is used for the 2010 Census will be implemented for the mail reinterview, which includes an advance letter, initial questionnaire, reminder postcard, and targeted replacement questionnaire.

The Internet/mail ‘choice’ panel reinterview contact strategy will be similar to the mail ‘push’ panel contact strategy. It includes an advance letter, initial questionnaire, reminder postcard, and targeted replacement questionnaire. The contact materials will offer the choice of Internet or mail response modes.

The reinterview contact materials will benefit from expert consultation. The letters and reminders will be developed in collaboration with internal and external survey methodologists.

The general methodology for this study involves comparing the Internet reinterview responses with 2010 Census mail questionnaire data to obtain estimates of the gross difference rates for each data item. Similarly, we will compare the mail reinterview responses with 2010 Census mail questionnaire data to get estimates of the gross difference rates for the same data items. Then, we will compare the gross difference rate estimates for the two reinterviews to assess reliability (response variance) of the census Internet questionnaire versus the census mail questionnaire administration. In general, with alpha level of 0.10 and power of 0.80, we expect the minimum detectable statistically significant difference in the national-level gross difference rates to be less than one percentage point.

The bulk of the analysis for this study will focus on households with the same respondent for both the 2010 Census paper questionnaire and the corresponding reinterview. We will match the 2010 Census data to the Internet reinterview data with the same respondents. For cases where the respondents are the same, the paradata will be used to supplement the interpretation of results of the measurement error portion of the analysis.

Similarly, Census data will be matched to the mail reinterview questionnaires with the same respondents.

Additionally, the analysis will include comparisons of item non-response for the Internet and paper modes as one of the indicators of data quality.

Within-household coverage will also be examined for the Internet and paper modes. Although the rostering approach and coverage probes strategy for the Internet questionnaire will be different from the paper questionnaire by design (in order to take advantage of the Internet instrument coverage probes), we will examine the composite effects of these strategy differences since they will also be inherent in future census multi-mode data collections.

Note that differential non-response may be a limiting factor for this study. Reinterview response rates will likely be low, relative to the initial survey request (i.e., the 2010 Census). We will attempt to get an indication of the magnitude of nonresponse bias by comparing demographic characteristics of reinterview respondents and nonrespondents based on data from their 2010 Census paper mode responses. Large differences in the demographic characteristics from the nonresponse analysis may indicate substantial bias in the key survey estimates and reduce the ability to generalize to the population of inference.

3. Methods to Maximize Response

In an effort to maximize response, a reminder postcard will be mailed to all sampled households. In addition, a fourth contact will be mailed to sampled households who do not respond to the initial request. In the mail ‘push’ panel, this fourth contact comes in the form of a replacement questionnaire. In the Internet ‘push’ panel, the fourth contact comes in the form of an additional notification letter containing instructions on how to access the Internet instrument, as well as a paper questionnaire. In the Internet/mail ‘choice’ panel, the fourth contact includes a replacement questionnaire as well as a letter containing instructions on how to access the Internet instrument.

Question-specific help is available in the Internet instrument. A toll-free automated telephone help line will be available for mail respondents.

4. Tests of Procedures or Measures

After the initial design phase, the Internet instrument will undergo comprehensive laboratory usability testing by internal experts. Usability testing will include observing the participants’ behaviors, noting difficulties and comments, and conducting post-testing interviews to gain qualitative feedback about potential confusion. In addition, quantitative measures will also be gathered, including time to complete the survey, eye-tracking variables, key-stroke analysis, and mouse-tracing analysis (we are also currently considering tracing mouse movement for a sample of survey respondents, which would include presentation of an informed consent statement). Eye-tracking analysis from laboratory results is planned for both the invitation letter and the Internet instrument. A second round of testing will incorporate accessibility testing to check for compliance with Section 508 of the U.S. Rehabilitation Act.

All associated contact materials, such as the Internet reinterview invitation and reminders, will go through expert review and/or cognitive testing. In selecting members of the expert review panel, we sought a diverse group of methodological and subject-matter experts. The review panel includes internal and external experts with much collective experience in different areas of expertise, including survey methodology, questionnaire design, and research psychology.

The mail reinterview questionnaire will be nearly identical to the mailout/mailback 2010 Census questionnaire. Thus, the mail questionnaire content has gone through comprehensive cognitive testing in preparation for the 2010 Census. The other contact materials, such as the mail reinterview invitation and reminders, will go through the same expert review and cognitive testing described above for the Internet reinterview materials.

5. Contact(s) for Statistical Aspects and Data Collection

For questions on statistical methods or the data collection described above, please contact Randall Neugebauer of the Decennial Management Division at the Census Bureau (Phone: 301-763-6883 or email [email protected].

Attachments

Attachment A Draft 2010 Census Quality Survey Mail Questionnaire (D-1 R1)

Attachment B Draft Screen Shots of the 2010 CQS Internet Application

Attachment C Draft 2010 Census Quality Survey Letters

Attachment D Draft Internet Instruction Card for Internet Push Panel

Attachment E Draft FAQ and Help Text for 2010 CQS Internet Application

1 As defined in Bates, N., and Mulry, M., (2008), “Segmenting the Population for the Census 2010 Integrated Communications Program,” C2PO 2010 Census Integrated Communications Research Memoranda Series, No. 1, U.S. Census Bureau.

6