2012 Methods Panel OMB Supporting Statement Pta_FINAL

SUPPORTING STATEMENT

U.S. Department of Commerce

U.S. Census Bureau

American Community Survey Method Panel Tests

OMB Control Number 0607-0936

Part A. Justification

1. Necessity of the Information Collection

The U.S. Census Bureau requests authorization from the Office of Management and Budget (OMB) to conduct the American Community Survey (ACS) Methods Panel tests.

The ACS collects detailed socioeconomic data from about 3.5 million households in the United States and 36,000 in Puerto Rico each year. Resulting tabulations from that data collection are provided on a yearly basis. The ACS allows the Census Bureau to provide timely and relevant housing and socio-economic statistics for even small levels of geography.

An ongoing data collection effort with an annual sample of this magnitude requires continuous research, testing and evaluations aimed at improving questionnaire content and data collection operations. The ACS Methods Panel is a research program that is designed to address and respond to emerging issues and survey needs. During the 2013-2015 period, the Methods Panel may include testing methods for increasing survey efficiencies, reducing survey cost, lessening respondent burden, and improving response rates. Testing may also include methods that might increase data quality. At this time, plans are in place to propose several tests: a 2013 Questionnaire Design Test, a 2015 ACS Content Test, and a series of Internet tests. Because we cannot anticipate issues that may arise in the production survey or from the proposed studies, we may conduct additional possible testing as needed. Possible additional testing would focus on methods for reducing data collection costs, improving data quality or testing new questions that have an urgent need to be included on the ACS.

Planned Methods Panel Testing:

In September through December 2010, the Census Bureau conducted the 2010 ACS Content Test that included testing revisions to current ACS questions as well as two new questions (Computer ownership and Internet usage, and Parental Place of Birth). During the Content Test, the Census Bureau determined that the ACS paper questionnaire did not contain enough space to accommodate certain configurations of proposed content changes. While selected content from the test does fit on the current form, we need to be proactive to accommodate future content requests on the ACS mail questionnaire. In the 2013 ACS Questionnaire Design Test, we will study the impact of a longer (36-page) questionnaire against our current 28-page form. The experimental treatments are noted below, but we have not finalized the forms for this test at the time. We will also study whether changing the size of the form to a standard size (8.5 x 11) booklet has an impact on response, compared to both the 28- and 36-page forms. The results of this testing will help the Census Bureau to decide which questionnaire format change has the least negative impact on response and data quality.

This test will also include several changes to make the questionnaire more compatible with optical character recognition software, including altering the response box formats for numeric write-in fields to allow them to be captured automatically rather than keyed. This part of the test will allow us to examine whether the form changes impact response behavior as well as to estimate anticipated cost savings from the automatic capture. Lastly, this test will include a test of variations in the relationship question and the marital status series per the OMB initiative to ensure these questions are inclusive of all relationship types and partnerships.

Thus, this test contains four experimental treatment panels, as well as a control panel with a 28-page questionnaire similar to ACS production. The four treatments include a 36-page form (same paper dimension as the control), and 44-page 8.5” x 11” form, a form with the optical character recognition changes, and the form with the relationship and marital status questions. Based on the results from comparing form size, a secondary, follow-up test may be needed to refine the questionnaire identified as the best alternative from this test.

Second, in response to Federal agencies’ requests for new and revised ACS questions, the Census Bureau plans to conduct the 2015 ACS Content Test. We will determine the changes to the current ACS content and the addition of new content through the OMB Interagency Committee for the ACS in 2013. We will then vet all proposed changes with the Interagency Council on Statistical Policy (ICSP) subcommittee for the ACS. OMB must then approve requests for content changes prior to testing. The objective of the 2015 ACS Content Test, for both new and existing questions, is to determine the impact of changing question wording, response categories, and redefinition of underlying constructs on the quality of the data collected. The Census Bureau proposes to evaluate changes to the questions by comparing the revised questions to the current ACS questions, or for new questions, to compare the performance of question versions to each other as well as to other well-known sources of such information. We plan to design the test similar to past content tests, using two experimental panels to compare current versus revised content. We will also use a reinterview to help generate measures of response error.

Third, we want to implement several iterative ACS Internet tests based on issues that arose from two ACS Internet tests conducted in 2011. Both of these tests studied the impact of different notifications of an Internet option in the survey invitations. Production ACS will begin collecting data using the Internet in January 2013. One problem detected in the 2011 tests was the impact to item nonresponse for questions in the later parts of the survey due to Internet break-offs. The Internet tests in 2013-2015 will look at potential ways to restructure messaging and change the Internet design to help reduce break-offs and encourage response in a timely manner. Testing will also include a reexamination of the potential for using the Internet to collect data in Puerto Rico, since results from the 2011 test did not show any distinct advantage. Testing plans are largely undefined at this point, but we will submit more detailed information once plans are solidified.

Possible Additional Methods Panel Testing:

Other considerations for testing include a second Content Reinterview Survey to build upon the results from the first Content Reinterview Survey that is currently in the field. We may need to reevaluate response error properties given the introduction of the web mode, 2013 content changes, and if we see any major issues arising from the current testing,

We are also considering testing designed to improve data collection operations in Group Quarters, such as the introduction of a web option and developing separate questionnaires for different Group Quarters types. There are no specific test plans for these projects at this point.

As previously mentioned, there may also be a need for a follow-up test to the 2013 Questionnaire Design Test. We could use this test to revisit the questionnaire layout of particular content items on the form that is deemed most successful.

Other testing may be considered, but the specific details of these tests are not known at this time. However, these tests would cover similar testing topics of content and methods to address emergent issues or needs. The tests may be conducted on both residential households or group quarters.

The Census Bureau collects data for this survey under Title 13, United States Code, Sections 141, 193, and 221. All data are afforded confidential treatment under Section 9 of that Title.

The Census Bureau is still in the early stages for planning and implementing the proposed tests. Subsequently the materials to be used in the tests have not been developed. This package contains the regular ACS data collection materials that will be modified for use in the proposed tests. For changes to the tests described in this justification, the Census Bureau will submit a non-substantive change request documenting the change.

2. Needs and Uses

The ACS must collect data on a continual basis and aggregate one, three, or five years worth of data to release data for all states, Congressional districts, counties, cities, and small towns down to the census tract and block group level. Essentially the ACS collects data every day of the year, either by mail, Internet (beginning in January 2013), telephone interviews or personal-visit interviews. There are many federal programs that distribute funds based on population and income data from the Census Bureau, including data from the ACS. Federal agencies use ACS data to determine appropriate funding for state and local governments through block grants. State and local governments use ACS data for program planning, administration and evaluation. Thus, the reliability and the quality of the data must remain high in order for the users to rely on the data for funding decisions.

So that the Census Bureau can provide critical information to governments and the private sector, the ACS collects comprehensive demographic, social, economic, and housing statistics covering every community in the nation. The ACS provides a continuous stream of updated information for states and local areas on an annual basis, and has revolutionized the ways the country uses data to understand communities and plan for the future.

ACS Methods Panel testing, such as the Questionnaire Design Test, Internet Tests, and the 2015 Content Test, provide a mechanism to investigate ways to reduce or at least maintain data collection costs and improve the quality of the data.

Information quality is an integral part of the pre-dissemination review of the information disseminated by the Census Bureau (fully described in the Census Bureau’s Information Quality Guidelines). Information quality is also integral to the data collection conducted by the Census Bureau and is incorporated into the clearance process required by the Paperwork Reduction Act.

The ACS Methods Panel research studies and tests are conducted for methodological research purposes and to inform management decisions for the ACS.  The results from these studies are not intended to be used to make representative national or local estimates from the ACS. 

3. Use of Information Technology

All tests will include data collection by Internet (available in English and Spanish), since Internet data collection is proposed to be a part of the production ACS data collection protocol starting in January 2013. All sampled addresses will be mailed a pre-notice letter, a request to respond online, and a reminder postcard. Households that do not respond online in a timely manner will also be mailed a paper questionnaire and supplemental postcard urging response.

Several of the mail pieces include a toll-free number to reach staff at the Telephone Questionnaire Assistance (TQA) center. In the TQA operation, interviewers can complete the ACS interview with a respondent using an automated data collection instrument. Additionally, several mailing pieces include a URL for the ACS where respondents can go to obtain more information about the ACS.

For the 2015 Content Test only, households that do not return their mailed questionnaire will be contacted for a Computer Assisted Telephone Interview (or sent another postcard if there is no phone number for the address) or Computer Assisted Personal Interview (CAPI). During CATI and CAPI interviews in the 2015 Content Test, we will use Computer Audio Recorded Interviewing (CARI) technology to record portions of the interview related to the questions being tested for use in behavior coding. Also, the Content Test will also include a CATI Content Follow-up (CFU), which uses another automated instrument.

4. Efforts to Identify Duplication

The ACS Methods Panel is the only testing vehicle for the ACS. There is no other program designed to improve the ACS.

5. Minimizing Burden

The proposed data collections consist of questions asked of a sample of households or group quarter living facilities, not other businesses or small entities.

6. Consequences of Less Frequent Collection

The 2015 Content Test, Questionnaire Design test, Internet Tests, and potential Group Quarter tests represent one-time, special tests with a defined period for data collection.

7. Special Circumstances

The Census Bureau will collect these data in a manner consistent with the OMB guidelines.

8. Consultations Outside of the Agency

As in the 2011 ACS Internet tests, we will continue to collaborate with researchers outside of the agency who are working on web surveys and usability interfaces to design the Internet tests. We also regularly consult with researchers from Westat, NORC, RTI and other survey and research firms as well as the University of Michigan’s Institute for Survey Research and other academic institutions. In addition, staff regularly review survey methodology literature and attend conferences that present state-of-the-art methods and procedures.

For the 2015 Content Test, we expect to consult many federal agencies, including the OMB and agencies represented on the OMB Interagency Committee for the ACS.

We published a notice in the Federal Register on April 25, 2012 (vol. 77, pg. 24684), inviting the public and other federal agencies to comment on our plans to submit this request. We received one response and the Census Bureau felt it was not relevant to these proposed data collections.

9. Paying Respondents

We do not pay respondents or provide respondents with gifts in the ACS.

10. Assurance of Confidentiality

Methods Panel testing is designed to mimic ACS production operations. As such, we use mostly the same mailing materials for the tests as ACS production operations. Thus, the information that is provided to ACS respondents during production is also provided to respondents in the Methods Panel Tests.

The Census Bureau collects data for this survey under Title 13, United States Code, Sections 141, 193, 221 and 223. All data are afforded confidential treatment under Section 9 of that Title.

In accordance with Title 13, each household, GQ administrator, and each person within a GQ participating in the ACS will be assured of the confidentiality of their answers. A brochure is sent to sample households with the initial mail package and contains this assurance. Households responding using the Internet questionnaire also are presented with additional assurances of their confidentiality and security of their online responses. The brochure mailed to sample GQs with the GQ introductory letter contains assurances of confidentiality. It is also provided to sample GQ residents at the time of interview.

Household members, GQ administrators or GQ residents may ask for additional information at the time of interview. A Question and Answer Guide, and a Confidentiality Notice are provided to respondents, as appropriate. These materials explain Census Bureau confidentiality regulations and standards.

At the beginning of follow-up interviews (CATI and CAPI), the interviewer will explain the confidentiality of data collected and that participation is required by law. For all CAPI interviews, the interviewer will also give the household respondent, GQ administrator, or GQ resident a copy of a letter from the Census Bureau Director explaining the confidentiality of all information provided.

These data collection materials are approved under OMB clearance 0607-0810.

11. Justification for Sensitive Questions

Some of the data we collect, such as race, relationship, marital status, and sources of income may be considered sensitive. The Census Bureau believes that the collection of these types of data is necessary for the analysis of important policy and program issues and has structured the questions to lessen their sensitivity.

12. Estimate of Hour Burden

2013 Questionnaire Design Test: In July 2013, we plan to contact 50,000 residential addresses for the field test portion of the Questionnaire Design Test. We estimate the time to complete the Questionnaire Design Test will be the same as ACS production (40 minutes).

2015 Content Test: From late July through mid-November, 2015, we plan to contact 70,000 residential addresses for the field test portion of the Content Test. We estimate the time to complete the ACS Content Test will be the same as ACS production (40 minutes). Based on the ACS unweighted interview rate of about 65 percent (from previous tests) and an expected followup eligibility rate of 85 percent, we plan to contact approximately 38,675 households that responded during the field test during the Content Follow-up (CFU) portion of the Content Test. The CFU reinterview should average about 15 minutes per household.

Internet Tests: For an average household, the estimated time to complete the ACS is 40 minutes, including the time for reviewing the instructions and answers. Total sample size is 200,000 residential addresses in the United States and 4,000 in Puerto Rico (across six tests in the 3-year clearance period from 2013-2015).

Potential Secondary Questionnaire Design Test: We expect to contact a maximum of 20,000 U.S. addresses in the field test. We estimate the time to complete the survey will be the same as ACS production (40 minutes).

Potential Repeat of Content Reinterview Survey: For an average household, the expected completion time is 20 minutes per reinterview and the yearly sample is 90,000.

Potential Group Quarters (GQ) Methods Test(s): We are considering two separate tests to study the operationalization of 1) an Internet option for GQ residents, and 2) a customized questionnaire for institutional GQs. For each test, we expect to sample a maximum of 75 Group Quarter facilities, and 15 residents within each of those facilities. For each facility, the expected completion time is 15 minutes, and 25 minutes for each resident within the facilities.

Table 1. Response Burden

Test and Test Components	Sampled units	Responses	Estimated Average Burden Hours
Questionnaire Design Test	50,000	50,000	33,334
Internet Tests	204,000	204,000	136,000
2015 Content Test	70,000
Field Test	---------	70,000	46,667
CFU	---------	38,675	9,669
Potential Secondary Questionnaire Design Test	20,000	20,000	13,333
Potential Repeat of Content Reinterview Study	90,000	90,000	30,000
Potential Group Quarters Internet Test	75 (facilities)	1,125 (residents)	488
Potential Group Quarters Test of institutional questionnaire	75 (facilities)	1,125 (residents)	488
Total	434,150*	474,925*	269,978*

*Annual estimates of the number of respondents, responses and reporting burden are calculated by dividing totals above by three years.

13. Estimate of Cost Burden

There are no costs to the respondent other than his/her time to respond to the survey.

14. Cost to Federal Government

As requested in the FY 2013 President’s Budget, the estimated cost of the ACS Methods Panel program is approximately $11.5 million per year. The Census Bureau will pay all costs of the Methods Panel tests.

15. Reason for Change in Burden

This collection is being submitted as revised as it is an ongoing activity. The burden hours are increasing because the Methods Panel is implementing different tests to improve the ACS. The experimental designs and objectives for these tests require additional sample to measure the impact of the changes.

16. Project Schedules

Table 2a. 2013 Questionnaire Design Test

Activities	Time Frame
Develop form designs	November 2011 – April 2012
Cognitive testing	August through December 2012
Update forms designs (as needed)	January 2013
Field test	July - August 2013
Process and Analyze data	September – December 2013
Final results	March 2014

Table 2a. 2015 Content Test

Activities	Time Frame
Identify suggested content for testing	Spring 2013
Cognitive Testing	Fall 2013 through Summer 2014
Field test data collection	End of July – end of-October 2015
CFU data collection	Mid-August- mid-November 2015
Statistically analyze results	January 2016 - August 2016
Final results and recommendations released	September - October 2016

Schedules for all other testing are to be determined.

17. Request Not to Display Expiration Date

We will display the expiration date on the form(s).

18. Exceptions to the Certification

There are no exceptions to the Certification for Paperwork Reduction Act submission.

9