OMB CPEX SUPPORTING STATEMENT PART B -05-11-09 - Final Draft For DMD Review_v2

SUPPORTING STATEMENT – PART B

U.S. Department of Commerce

U.S. Census Bureau

2010 CENSUS PROGRAM FOR EVALUATIONS AND EXPERIMENTS

OMB Control Number 0607-XXXX

2010 AQE and 2010 AQE Reinterview Evaluation

1. Universe and Respondent Selection

In terms of the AQE protocol, the experimental forms will be mailed to a national sample of housing units – 30,000 housing units per panel for the race/Hispanic panels and Coverage panels, and a maximum of 20,000 for the Census 2000 comparison panel. Note that sample sizes have not been finalized.

A corresponding, independent AQE reinterview will be conducted with a sample of AQE mail respondents to assess the reliability of both the control and alternative question treatments. Sample sizes for the reinterview will likely not exceed 4000 per panel; sample design for reinterview has not been finalized.

2. Procedures for Collecting Information

Experimental forms will be mailed to a national sample of housing units. Respondents will complete their forms and mail them back to the Census Bureau.

A corresponding, independent AQE reinterview will be conducted over the telephone in June/July (eight week period).

We will attempt to call from respondent-provided phone numbers captured from the paper questionnaires but will implement a ‘look-up’ procedure for sample cases in which the respondent did not provide a phone number. This is the first time that the AQE will incorporate a corresponding reinterview.

Probing questions will also be included in the reinterview as a means of getting a more accurate picture of how the respondents self-identify their race and Hispanic origin. The current plan is to ask to speak to the household member who completed the paper questionnaire; we will be collecting data for the reinterview respondent and a randomly selected member of the household. The reinterview will take roughly seven minutes to conduct.

3. Methods to Maximize Response

The experimental methods will include the mailing of replacement questionnaires (which mimic the experimental treatments) to those who do not respond to the initial questionnaire. Note that Nonresponse followup enumeration will be conducted as in the production 2010 Census for those in the AQE sample who do not return a form by mail (that is, no separate forms will be used for sample households during NRFU).

4. Tests of Procedures or Measures

New questions developed for the race and Hispanic origin AQE treatments will undergo both an expert review and cognitive testing prior to implementation. We will conduct about 60 cognitive interviews among a diverse group of respondents and supplement preliminary findings with recommendations from the expert review panel. New questions developed for the coverage AQE treatment will undergo cognitive testing prior to implementation.

In selecting members of the race and Hispanic origin expert review panel, we sought a diverse group of methodological and subject-matter experts. The review panel includes internal and external experts with different areas of expertise, including survey methodology, questionnaire design, and much collective experience with various race and ethnic groups.

We expect the variances to vary widely depending on the evaluation measure (e.g., mail response rates, item nonresponse rates, race or Hispanic origin distributions, percent multiple Hispanic, percent nonspecific race or ethnic responses, etc.) and population being analyzed. In general, for national-level estimates, with alpha level of 0.10 and power of 0.80 we expect the minimum detectable statistically significant differences to be 2.4 percentage points or less. For stratum-level estimates, we expect the minimum detectable statistically significant differences to be 1.8 percentage points or less.

5. Contact(s) for Statistical Aspects and Data Collection

For questions on statistical methods or the data collection described above, please contact Jason Machowski of the Decennial Management Division at the Census Bureau (Phone: 301-763-4173 or email [email protected], or Erin Love of the Decennial Management Division at the Census Bureau (Phone: 301-763-2034 or email [email protected]).

Deadline Messaging/Compressed Schedule Experiment

1. Universe and Respondent Selection

In this experiment, the universe is households in mailout/mailback areas of the country, that is, the 50 states plus Washington, D.C.

The sample design will include a stratification into three areas based on their anticipated levels of responses. This will be based on the areas identified for the three-tiered replacement questionnaire mailing strategy for the 2010 census. The first is high response areas, the second medium response areas, and the third is low response areas.

The mailout sample size not to exceed 20,000 housing units per panel will be randomly selected by strata, with an over-sampling in the low response areas. Note that the sample size has not been finalized.

The experiment will use a full-factorial design and includes a total of eight panels. These eight panels will include a control group which will receive the standard 2010 mail materials with no specific deadlines or due dates and will use the regular mailing schedule. One panel will receive the mailout materials on a compressed mail schedule, three panels will receive some form of deadline messaging, and three panels combine the compressed schedule with the deadline messages. Note that none of these panels contain any changes to the content or appearance of the census questionnaires, just the other mail pieces and the mailing schedule.

In the compressed mailing schedule treatment, the advance letter, initial questionnaire package, and the reminder postcard will all be mailed out one week later than the regular 2010 production schedule.

The motivation behind this is that it reduces the sometimes lengthy time between when people receive their census mail and the reference date of April 1, which can be two weeks or more later than when they receive the form. For some respondents there may be a disconnect, and by delaying the mailings one week, this difference in time is reduced. Note that the replacement mailings will be conducted on the normal census production schedule for these panels.

The compressed schedule will be conducted in a total of four different panels, by itself and in combination with the three deadline message treatments.

The table below provides an overview of the proposed in-home delivery mailing dates for the normal 2010 census schedule and the experimental compressed schedule. The compressed schedule delivers the materials to homes closer to the April 1 Census date.

Normal 2010 Schedule Compressed 2010 Schedule

Advance letter March 8 - March 10 March 15 - March 17

Initial questionnaires March 15 - March 17 March 22 - March 24

Reminder postcard March 22 - March 24 March 29 - March 31

Census Day April 1 April 1

Replacement questionnaires April 1 - April 10 April 1 - April 10

The wording of the standard 2010 materials that will be used in the control panel is below. The control panel wording does not mention specific due dates or deadlines; it includes generic wording such as promptly, today, and as soon as possible.

Control Panel (2010 Census wording)

Advance letter - “When you receive your form, please fill it out and mail it in promptly.”

Outgoing envelope for IQ - nothing in box underneath “Your Response is Required by Law”

Cover letter for IQ - “Please complete and mail back the enclosed census form today.”

Reminder postcard - “If you have not responded, please provide your information as soon as possible.”

The first deadline message experimental treatment uses a ‘mild’ set of messages. For this treatment, a ‘mail by’ date of April 5 is used in place of the generic control panel wording.

Next, what we are calling the ‘progressive urgency’ messages uses the word “deadline” in the initial questionnaire cover letter, outgoing envelope, and on the reminder postcard. Also, on the reminder postcard, the text again reminds respondents that response is required by law.

Finally, the ‘NRFU motivation’ set of messages uses a different approach. On the cover letter for the initial questionnaire and on the reminder postcard, it warns “Please complete and mail back the enclosed census form by April 5 so that you can avoid a personal visit from an interviewer.” These messages are currently undergoing cognitive pretesting so it is possible that the wording may change slightly.

For all of the experimental panels, no messages will be used on the replacement questionnaires, as households will just receive the normal set of census replacement materials. And, as mentioned earlier, there are no changes to the content or appearance of the census questionnaires.

For the analysis of this experiment, we will be looking at several different measures in addition to response rates to evaluate the overall effectiveness of the compressed mailing schedule and the deadline messages. We will also look at the speed of returns to see if the deadline dates provide a sense of urgency and make some people respond sooner. We will also look at item nonresponse rates; however, we hypothesize that this experiment will have no significant effect on item responses.

Finally, we will attempt to monitor public reactions to the deadline messages, although the specific methodology for this component has yet to be determined.

2. Procedures for Collecting Information

Experimental forms will be mailed to a national sample of housing units. Respondents will complete their forms and mail them back to the Census Bureau.

3. Methods to Maximize Response

The experimental methods will include the mailing of replacement questionnaires (which mimic the experimental treatments) to those who do not respond to the initial questionnaire. Note that Nonresponse followup enumeration will be conducted as in the production 2010 Census for those in the DM/CS sample who do not return a form by mail (that is, no separate forms will be used for sample households during NRFU).

4. Tests of Procedures or Measures

Each of the messages being tested in this experiment have undergone cognitive testing.

With alpha level of 0.10 and power of 0.80 we expect the minimum detectable statistically significant differences to be 1.5 percentage points or less.

5. Contact(s) for Statistical Aspects and Data Collection

For questions on statistical methods or the data collection described above, please contact Jason Machowski of the Decennial Management Division at the Census Bureau (Phone: 301-763-4173 or email [email protected], or Erin Love of the Decennial Management Division at the Census Bureau (Phone: 301-763-2034 or email [email protected]).

Confidentiality/Privacy Notification Experiment

1. Universe and Respondent Selection

The universe for the Confidentiality/Privacy Notification Experiment is housing units in the mailout/mailback areas of the country, that is, the 50 states plus Washington, D.C.

The sample design includes a stratification into three areas based on their anticipated levels of response, that is high, medium, and low response areas. The mailout sample size will not exceed 20,000 housing units for each of the panels and will be randomly selected by strata, with oversampling in the low response areas.

For the analysis of this experiment, we are considering various measures to evaluate the effect of the cover letter treatments. We will examine differences in response rates between the control and treatment panels. We will also examine data quality measures such as the completeness of the questionnaire. In addition, we plan to attempt to monitor public reaction to the treatment messages, however the methodology for this component has not been defined.

The experiment will consist of two panels. The 2010 Census production letters will use the statement “Your answers will only be used for statistical purposes, and no other purpose.” The Panel 1 cover letter will include this production ‘statistical purposes’ statement on the front and will include the data linkage paragraph on the back. The following data linkage statement appears on the back of the cover letter for Panel 1, while it does not appear at all in the 2010 Census production cover letter.

“To improve census results, other government agencies may give us additional information about your household. The additional information we receive is legally protected under Title 13, just like your census answers.”

Note that there is no definitive legal requirement that the Census Bureau provide respondents with a data linkage notification; however, policy considerations come into play and stakeholders and the Census Bureau’s Data Stewardship Executive Policy Committee have recommended the inclusion of such a statement.

In response, the 2006 Working Group developed this revised data linkage statement. The revised statement performed well in qualitative testing, but must be field-tested before it can be used in future censuses. Hence, the Census Bureau approved the addition of this data linkage statement within the Privacy Notification Experiment.

The Panel 2 cover letter will include an alternative ‘statistical purposes’ statement on the front, that is, “Your answers will only be used to produce statistics.” and will again include the data linkage paragraph on the back.

The 2006 Working Group had recommended a revision of this statement because research has consistently indicated that the phrase “for statistical purposes” is not well understood by respondents. The 2006 qualitative research evaluated an alternative statement ("...to produce statistics"). This alternative statement was correctly and consistently interpreted by most respondents, while the standard ‘statistical purposes’ statement has generated few correct and consistent interpretations by cognitive respondents.

2. Procedures for Collecting Information

Experimental forms will be mailed to a national sample of housing units. Respondents will complete their forms and mail them back to the Census Bureau.

3. Methods to Maximize Response

The experimental methods will include the mailing of replacement questionnaires (which mimic the experimental treatments) to those who do not respond to the initial questionnaire. Note that Nonresponse followup enumeration will be conducted as in the production 2010 Census for those in the C/PN sample who do not return a form by mail (that is, no separate forms will be used for sample households during NRFU).

4. Tests of Procedures or Measures

As outlined in section 1, Universe and Respondent Selection, the messages being tested in this experiment were developed from cognitive testing results. With alpha level of 0.10 and power of 0.80 we expect the minimum detectable statistically significant differences to be 1.5 percentage points or less.

5. Contact(s) for Statistical Aspects and Data Collection

For questions on statistical methods or the data collection described above, please contact Jason Machowski of the Decennial Management Division at the Census Bureau (Phone: 301-763-4173 or email [email protected], or Erin Love of the Decennial Management Division at the Census Bureau (Phone: 301-763-2034 or email [email protected]).

Nonresponse Followup (NRFU) Contact Strategy Experiment

1. Universe and Respondent Selection

The precise design and implementation strategy (including the universe and respondent selection) for this experiment is still being developed. The Census Bureau is contending with various logistical challenges associated with implementing this study in a decennial census environment.

The NRFU Contact Strategy Experiment compares the results from two experimental enumerator forms with a control (production) enumerator form.

In selected sites of the country, the enumerator form will have a maximum of five contact attempts, whereas other sampled areas will have a maximum of four contact attempts. These two experimental treatments will be compared to the control form, which has space for up to six contact attempts. Separate forms will be used for the experimental treatments, that is, one type of experimental form will have space for a maximum of five contact attempts and a separate form will have space for a maximum of four attempts.

In order to determine the effectiveness of fewer enumerator contact attempts, we will examine a variety of statistical measures. We will look at the cumulative total of conducted interviews at each contact number and the distribution of outcomes at each contact. We will also look at ‘respondent type’ at last contact to determine differences in proxy rates. In addition, we will look at data quality indicators such as item nonresponse rates and form completeness at last contact. This analysis may also include a cost-benefit component to judge the tradeoffs between the likelihood of more proxy responses and lower quality data with the decrease in costs - methodology of the cost-benefit component are still in the planning stage.

2. Procedures for Collecting Information

Enumerators will collect information for this experiment on the standard Census 2010 enumerator form used in the Non-response Follow-up Operation.

3. Methods to Maximize Response

Because the goal of this experiment is to determine whether data quality remains the same with fewer enumerator contacts, there are no special methods being employed to maximize response.

4. Tests of Procedures or Measures

This experiment represents the first time the Census Bureau will test different numbers of enumerator contacts for the Non-response Follow-up Operation in a decennial census environment. With alpha level of 0.10 and power of 0.80 we expect the minimum detectable statistically significant differences to be 1.0 percentage points or less.

5. Contact(s) for Statistical Aspects and Data Collection

For questions on statistical methods or the data collection described above, please contact Jason Machowski of the Decennial Management Division at the Census Bureau (Phone: 301-763-4173 or email [email protected], or Erin Love of the Decennial Management Division at the Census Bureau (Phone: 301-763-2034 or email [email protected]).

2010 Content Reinterview Evaluation

1. Universe and Respondent Selection

Operationally, the 2010 Content Reinterview and the AQE Reinterview (previously mentioned) are similar. For the Content Reinterview Survey (CRS), we will select a sample of households from the 2010 Census. We will target a sample of respondents who completed the Census 2010 questionnaire either by self-responding to the paper mail-out form or by providing data via enumerator during NRFU.

In the 1990 Census, 12,800 households were interviewed out of an initial sample of 15,500. In Census 2000, 19,897 households were interviewed out of an initial sample of 30,000 households. The 2010 CRS sample size for call attempts under consideration is approximately 11,000 households, however the Census Bureau may expand the sample to 30,000 to equal what was done in Census 2000.

2. Procedures for Collecting Information

We plan to conduct all interviews by telephone rather than personal visit to reduce cost. These interviews will be conducted in June/July of 2010 (eight week period) and will take approximately seven minutes of the respondents’ time.

In the 2000 CRS, person-specific questions were only asked for one randomly selected person in a sampled household. This differed from previous censuses. In 2010, we plan to ask questions about the respondent (the person who completed the paper questionnaire) and about one other randomly-selected person in the household. The details are yet to be determined. We will ask the questions for the 100-percent data items, as well as additional probing questions that focus on obtaining an accurate, respondent-identified race and Hispanic origin.

3. Methods to Maximize Response

The design of this evaluation is still being developed, which includes any methods to maximize response.

4. Tests of Procedures or Measures

The questionnaires used in this evaluation will undergo cognitive testing.

5. Contact(s) for Statistical Aspects and Data Collection

For questions on statistical methods or the data collection described above, please contact Jason Machowski of the Decennial Management Division at the Census Bureau (Phone: 301-763-4173 or email [email protected], or Erin Love of the Decennial Management Division at the Census Bureau (Phone: 301-763-2034 or email [email protected]).

2010 Alternative Group Quarters (GQ) Questionnaire Evaluation

1. Universe and Respondent Selection

The sample for this experiment will be at the Local Census Office (LCO) level and consist of all of the people in GQs that are enumerated with the Individual Census Report (ICR) form for the selected LCO. The size of the experimental universe will consist of approximately 60,000 people. All of the GQs in the selected LCO will receive the experimental ICR form due to the Field difficulties of ensuring that the correct ICR goes to the correct GQ. To gain the maximum amount of data, LCOs with a plethora of college dorms or juvenile institutions are preferred because of the high percentage of respondent completed ICRs found for those GQ types in 2000. In 2000, college dorms had a respondent completed ICR rate of 57.5% and juvenile institutions had a rate of 23.8% (2000 GQE Evaluation). The LCOs selected will have at least a 25% of all their GQs consisting of college dorms or juvenile institutions. Additionally the selected LCOs will have a higher percentage of the GQ population living in college dorms or juvenile institutions compared to the other GQ types.

To select the sample of LCOs that will receive the experimental ICR, Census 2000 data will be used to find Census 2010 LCOs that meet the selection criteria. To correctly implement the experiment in the field, the selected LCOs must be known before the printing of training kits and ICR forms. By selecting 2010 LCOs based on Census 2000 data, there is a risk of growth of non-desirable GQs or closing of college dorms or juvenile institutions, but this risk is judged acceptable.

2. Procedures for Collecting Information

The sampled respondents will provide their responses on an experimental version of ICR. The experimental ICRs will be collected with all other production ICRs during Group Quarters Enumeration.

3. Methods to Maximize Response

No special methods identified at this time to maximize response.

4. Tests of Procedures or Measures

The experimental ICR form will undergo cognitive testing.

5. Contact(s) for Statistical Aspects and Data Collection

For questions on statistical methods or the data collection described above, please contact Jason Machowski of the Decennial Management Division at the Census Bureau (Phone: 301-763-4173 or email [email protected], or Erin Love of the Decennial Management Division at the Census Bureau (Phone: 301-763-2034 or email [email protected]).

2010 Interactive Voice Response (IVR) Customer Satisfaction Survey Evaluation

1. Universe and Respondent Selection

The European Computer Manufacturers Association (ECMA) Script random number generator Math.random() will be used to decide whether a call is a survey candidate. This function returns a random number between 0 and 1.

It will be used with the test: Math.random() < iSurveyPercent/99.999

If the test is true then the call is a survey candidate. (The default is iSurveyPercent=1.) The factor 99.999 is used to ensure that iSurveyPercent=100 yields that every call is a survey candidate, even in case Math.random() can return 1.0; the test also ensures that iSurveyPercent=0 yields that no call is a survey candidate.

The sample size for this customer satisfaction survey is estimated at 665,000 callers. There is an estimated response rate of 7.6%, bringing the estimated number of respondents to 5,016.

2. Procedures for Collecting Information

Customer satisfaction survey responses are being collected via an automated interactive voice response system.

3. Methods to Maximize Response

No special methods identified at this time to maximize response.

4. Tests of Procedures or Measures

This customer satisfaction survey design is based upon similar surveys conducted at the Census Bureau.

5. Contact(s) for Statistical Aspects and Data Collection

For questions on statistical methods or the data collection described above, please contact Jason Machowski of the Decennial Management Division at the Census Bureau (Phone: 301-763-4173 or email [email protected], or Erin Love of the Decennial Management Division at the Census Bureau (Phone: 301-763-2034 or email [email protected]).

21