SUPPORTING STATEMENT A
U.S. Department of Commerce
U.S. Census Bureau
OMB Control No. 0607-1020
The Ask U.S. Panel (“the Panel”) will be a probability-based nationally-representative survey panel for tracking public opinion on a variety of topics of interest to numerous federal agencies and their partners, and for conducting experimentation on alternative question wording and methodological approaches.
The Panel will be developed through a multi-year effort. The first year of data collection will focus on conducting a large-scale field Pilot Test. In the Pilot Test, a sample consisting of approximately 1,700 people that will be recruited and surveyed as a proof-of-concept to refine methods that can be used to recruit a final panel. The current request is for the Pilot Test.
The goal of the Panel is to ensure availability of frequent data collection for nationwide estimates on a variety of topics and a variety of subgroups of the population, meeting standards for transparent quality reporting of the Federal Statistical Agencies and the Office of Management and Budget (OMB).
The Panel is an interagency effort. Representatives from Census, the Economic Research Service, the Bureau of Labor Statistics, the National Center for Health Statistics, the National Center for Science and Engineering Statistics, the National Center for Education Statistics, the Department of Defense, Department of Transportation, Department of Labor and the Social Security Administration are guiding the design, content and methodological rigor of the Panel.
Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.
The programs that provide essential statistical information for use by governments, businesses, researchers, and the public are carried out by independent agencies and spread across several departments within an agency. These agencies are referred to as Federal Statistical Agencies and they serve the public by providing independent, non-partisan data about the country. This clearance request aims to pilot test methods for an online research panel that could be available for robust public opinion, methodological research, and rapid-response data collection for the common good.
This long-term objective supports the recommendations of the Commission on Evidence Based Policymaking (https://www2.census.gov/adrm/fesac/2017-12-15/Abraham-CEP-final-report.pdf) in several ways:
Promotes a multi-year learning agenda that supports the generation and use of data as evidence;
Enables coordination of the public sector’s evidence-building activities; and
Streamlines the approval processes for new data collections and uses existing flexibilities in procurement policy.
The statistical community at large has been struggling with near-real-time measurement of key areas, including:
Privacy and confidentiality opinions and preferences;
Public attitudes towards data collection and use of administrative records;
Methodological choices regarding online instrument design decisions;
Survey design choices regarding wording and contact timing;
Messaging strategies to increase response rates; and
Novel data collection needs due to emerging national events, like the COVID-19 pandemic.
OMB Standard 1.4 from the Office of Management and Budget, Standards and Guidelines for Statistical Surveys, requires Federal Agencies to appropriately pretest data collection instruments prior to fielding them. The Panel will eventually offer a platform on which to conduct pretests with a nationally representative audience.
Existing commercial online panel alternatives typically fail to meet OMB’s standards for transparency, which require sufficient detail on data collection and estimation methods to allow reproducibility, and sufficient detail on data quality and representativeness to enable OMB to evaluate the fitness for purpose.1
Access to a pool of pre-recruited panel members, including historically undercounted populations, will help researchers at the Census Bureau (Census) and other Federal Agencies better understand public opinion related to federal data collection, including administrative data matching, privacy, and confidentiality, and will facilitate methodological testing. The Panel will consist of an entirely new, representative probability sample of U.S. adults who are not members of an existing survey panel. The addition of targeted subgroups and general replenishment to supplement the existing panel will likely be desirable in the future.
The Pilot will answer critical methodological questions about ability to recruit and retain historically undercounted population groups in the panel. The Pilot will provide proof of concept for the use of tablets by non-internet households, the use of alternate (in-person, phone, text) nonresponse follow-up, and the effects of those methodologies on retention. Results from the Pilot will be used to refine the methodology a future full panel.
The baseline survey will pilot collecting demographic data that would later be used to assess representativeness of the panel and to understand potential nonresponse bias in future surveys. Items on the baseline survey would also be available for weighting adjustment and subsampling in future surveys.
The first topical survey that will be used as a proof-of-concept will be developed from the Census Barriers, Attitudes and Motivators Survey (CBAMS). The CBAMS questionnaire will be adapted from that used prior to the 2020 Census and will be used to measure intercensal mindsets towards the Census Bureau and Census Bureau data collections. The Pilot topical survey will be a field test of the survey instrument to be used to observe barriers, attitudes, and motivators towards the Census Bureau. This questionnaire will be submitted through a future 30-day notice.
Information quality is an integral part of the pre-dissemination review of the information disseminated by the Census Bureau (fully described in the Census Bureau's Information Quality Guidelines). Information quality is also integral to the information collections conducted by the Census Bureau and is incorporated into the clearance process required by the Paperwork Reduction Act.
All data will be collected in an automated form, either online or via computer-assisted telephone interviewing (CATI) or computer-assisted personal interviewing (CAPI) in the nonresponse follow-up.
This research does not duplicate any other data collection or research being done by the Census Bureau or other Federal agencies. The purpose of this clearance is to stimulate additional research, which would not be done under other circumstances due to time constraints. This research will involve collaboration with staff from other agencies. All efforts would be collaborative in nature, and no duplication in this area is anticipated.
To the maximum extent possible, we will make use of previously collected data by agencies, external data sources, and results from previous collections of survey.
While this research may encounter small business owners as members of the population, the target population is neither small businesses nor other small entities.
The Pilot Test is necessary to demonstrate whether this new data collection methodology would meet OMB’s standards for transparency, which require sufficient detail on data collection and estimation methods to allow reproducibility, and sufficient detail on data quality and representativeness to enable OMB to evaluate the fitness for purpose.2
requiring respondents to report information to the agency more often than quarterly;
Respondents are not required to participate in the Pilot. Participation is voluntary. Respondents will be incentivized for participation in the Pilot.
requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;
Respondents are not required to participate in the Pilot. Participation is voluntary. Respondents will not be asked to prepare written responses outside of the Pilot collection.
requiring respondents to submit more than an original and two copies of any document;
Respondents are not required to participate in the Pilot. Participation is voluntary. Respondents will not be asked to submit multiple copies of any documents.
requiring respondents to retain records, other than health, medical, government contract, grant-in- aid, or tax records for more than three years;
Respondents are not required to participate in the Pilot. Participation is voluntary. Respondents will not be required to retain records for more than 3 years.
in connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study;
The Pilot will be designed to produce valid and reliable results that can be generalized to the intended universe or it will be clearly noted otherwise.
requiring the use of a statistical data classification that has not been reviewed and approved by OMB;
The Pilot will not require the use of a statistical data classification that has not been reviewed and approved by OMB.
that includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; or
The Pilot (baseline and topical surveys) will impose confidentiality requirements that are authorized by statute or regulation, and will follow any policies consistent with such statutes or regulations. These surveys will allow for sharing of data with other agencies as authorized by law, for compatible confidential use.
requiring respondents to submit proprietary trade secret, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information's confidentiality to the extent permitted by law.
Respondents are not required to participate in the Pilot. Participation is voluntary. No respondents will be asked to submit proprietary trade secret information. All data will have all applicable, legally-required confidentiality protections applied.
Consultation with representatives of those from whom information is to be obtained or those who must compile records should occur at least once every 3 years - even if the collection of information activity is the same as in prior periods. There may be circumstances that may preclude consultation in a specific situation. These circumstances should be explained.
The Census Bureau published a notice in the Federal Register on December 07, 2021 soliciting public comments on our plans to submit this request (86 FR 69220). We received a total of seven comments. Two comments were not substantively related to this data collection. One commenter focused on the distinction in representation between urban and rural participants in the panel. We will address this concern by analyzing pilot test data and, based on the outcome of this analysis, potentially incorporating the findings into future plans (to be presented in future 30-day notices). Another commenter recommended that the Census Bureau use this new platform to investigate the undercount of young children. Consideration will be given to this topic for inclusion in a Pilot Test topical survey that will be presented in a future 30-day notice.
Several commenters suggested that the current effort is duplicative of existing products in the marketplace. According to OMB, existing commercial online panel alternatives typically fail to meet OMB’s standards for transparency, which require sufficient detail on data collection and estimation methods to allow reproducibility, and sufficient detail on data quality and representativeness to enable OMB to evaluate the fitness for purpose.3 This panel is not duplicative of existing commercial online panel alternatives, since it is being designed in a manner that will meet OMB’s standards for transparency by providing sufficient detail on data collection and estimation methods to allow reproducibility as well as sufficient detail on data quality and representativeness to enable OMB to evaluate fitness for purpose. Results from the pilot test will be presented in any subsequent requests for data collection to OMB via additional 30-day public notices.
One commenter noted a concern with opening the panel to “government and other non-profit researchers and policy makers." The instant request is only for the pilot study to develop a proof of concept and refine methods. Approval for the build out and use of the full panel will be the subject of subsequent 30-day notices.
Several commenters noted that “the methodology being implemented by the Bureau is evolving, and more work will be needed to ensure the validity of the methodology and the utility of the proposed data collection.” The Census Bureau agrees, thus the plans to complete the Pilot study. Very detailed methodology for the pilot is specified in this package. Results from the pilot will determine future methodology and plans for the panel.
A final critique is noted that “the Bureau underestimates the respondent burden and cost estimates of the Ask U.S. Panel.” The pilot will produce specific data on recruitment rates and associated cost which will allow us to refine cost estimates and make design decisions for the panel. These results will also be made public in future 30-day notices for additional data collection.
In the spirit of transparency, the Census Bureau will publish 30-day public notices at each point in time that a new design characteristic is brought into this panel.
The Census Bureau published a notice in the Federal Register on March 01, 2022 again soliciting public comments. We received a total of three comments. The commenters disagreed with OMB’s statement that existing panels do not meet OMB’s standards for transparency. Results from the pilot study described herein will be presented to the OMB and any future plans for pursuance of a full panel will be discussed with OMB at that point. Two commenters ask questions and pose additional concerns about planned procedures for the future of the panel described in this package. The current PRA request is for the pilot only. Any future uses of this panel or program will be submitted to the OMB and the public in future 30-day notices.
The Census Bureau published a notice in the Federal Register on April 4, 2022 soliciting public comments specifically on the topical survey content. We received a total of three comments. The first comment suggested adding a well-being question to the survey content. This will be considered for future iterations of both this and other related Census Bureau data collections. The other two comments addressed concerns about duplication, cost, quality and efficiency. The PRA is concerned with identifying duplication in burden to respondents not in methods for collecting data, thus the Census Bureau responses in these statements focus on identifying possible duplication among government data collections. Building a panel that can accommodate government needs would reduce burden on the public if the alternative is a series of independent, government-specified, large-sample surveys.
The commenters point out the various panels that exist in the marketplace. It is worth noting that there are significant differences across the private solutions in terms of methodology. Different companies have designed panels using different methodologies to optimize in different ways. This shows that there is no single best way to build a panel for all purposes and that specific methods should be tailored to specific use cases. The intention of the research questions for the Ask U.S. Pilot are to determine the optimal methodology for a government sponsored panel that could be used in the future for "influential scientific, financial, or statistical information."[1] In addition, a panel created specifically with government requirements in mind will allow for permission-based linkages to other government records, which, to our knowledge, is not available with many existing panel providers.
Commenters also cite the situations in which OMB approved the use of panels by government agencies. We suspect that in many of these instances, like the one referenced for the Census Bureau, the government used private sector panels for collections that were deemed not “influential scientific, financial or statistical information.” In the referenced Census Bureau case, the research was for branding and advertising. The OMB guidance distinguishes between influential and non-influential information (MEMORANDUM FOR THE HEADS OF EXECUTIVE DEPARTMENTS AND AGENCIES (whitehouse.gov)). Limited disclosure of information about the existing panels and concerns about data quality have prevented OMB from granting approval for panel use for influential statistics in the past.
The commenters raise concerns about cost and quality of a new panel created to meet government guidelines. We argue that the purpose of a pilot test is to gain empirical data on cost and quality of a government-sponsored panel. Literature has shown that sponsorship matters and government data collections consistently receive better response rates than that of non-governmental sponsors. These are reasons to believe that a government-sponsored panel may have different outcomes with regard to recruitment and retention than a private-sector-sponsored panel. This pilot seeks to provide empirical evidence to that question. Additionally, it is possible that some of the research and literature published pre-COVID about panel recruitment and retention is out-of-date given changes in the population during the pandemic regarding internet uptake and use, views towards government surveys and incentive effectiveness.
Particularly speaking to cost, the Census Bureau does have considerable expertise conducting mixed mode surveys, including longitudinal ones. The Census Bureau draws on decades of experience from which to create cost estimates and manage budgets for a longitudinal survey program.
During development, the Census Bureau has consulted with the following individuals and agencies:
Doug Williams and Jennifer Edgar (BLS),
Mark Denbaly and Jeff Gonzalez (USDA ERS),
Stephen Blumberg and Paul Scanlon (CDC - NCHS),
Andrew Zuckerberg (NCES),
Laith Alattar (SSA),
Jennifer Sinibaldi (NCSES),
Julie Parker, Cha-Chi Fan and Clara Reschovsky (DOT/BTS),
William Marton (HHS),
Gordon Willis and Richard Moser (NIH),
David Beall, Chip Berry, Carolyn Hronis and Gerson Morales (EIA),
Cynthia Gilham (CSPC),
Kathleen Holland and Mary-Helen Risler (IRS),
CJ Krizan and Christina Yancy (DOL),
Chris Moore (EPA),
Cleo Redline (ED),
Karyen Chu (FDIC),
Kathryn Aikin (FDA),
Jeff Larrimore, Alicia Lloro, and Ellen Merry (FRB), and
Cathy Flynn, Malikah Dorvil, Carol Newell, Jeremy Hall and Paul Rosenfeld (DOD).
Because retention is critical to a longitudinal, panel design, incentives will be offered to respondents. As incentives remain one of the most effective ways to encourage survey participation, we informed our panel incentive structure by reviewing existing longitudinal surveys and panels (Table 4.1) and making adjustments commensurate with burden.
Table 4.1 Incentive Structure in Longitudinal Federal Surveys and Nonprofit Panels
Survey |
Enrollment Incentive/Wave 1 |
Panel |
|
Incentive |
Maintenance |
||
SIPP (historical) |
$40* |
$40 |
none |
PSID |
$75-150 |
$1/minute |
Unknown |
ECLS-B |
$50 |
$30/survey + children’s book |
none |
NLSY |
$40 |
$70 $100 early bird |
none |
NSCAW |
$50 caregiver $10-20 children |
$10-20 children $50 young adults |
none |
ANES 2008-2009 |
$2 prepaid + $10 promised $5 NRFU |
$10/month + free internet access if needed |
Unknown |
HRS |
$100 |
$80 |
Unknown |
American Trends (Pew) |
$10 |
$10/survey |
none |
National Survey Project Cohort of the American Life Panel (RAND) |
$200 |
$25/month |
none |
FDA Tobacco Panel |
$35 |
$15/survey |
none |
GALLUP |
none |
none |
none |
Understanding America (USC) |
$5 prepaid $15 promised $15 welcome package |
$20/survey (30 min) $10 bonus for sleepers |
none |
* This incentive is historical.
The Pilot recruitment incentive design includes the following:
Initial Invitation: $5 prepaid incentive sent with the initial invitation to complete a household roster.
Early Bird: A $5 early bird incentive for completing the household roster within 1 week of the survey invitation mailing. Early bird incentives encourage more timely responses, reducing the need for and costs associated with nonresponse follow up (NRFU).
NRFU Household Roster: $5 for household roster completion.
Baseline Questionnaire: $20 baseline incentive.
Topical Surveys: $10 for each topical survey (~15-minute average).
Incentives will be paid based on respondents’ preference for cash, an electronic gift card, a physical gift card, or a mailed check. Using a mailed incentive option ensures that respondents with limited internet use will be able to use their earned incentive.
Once recruited, panelists will respond to topical survey invitations via web mode. Using a unique login, panel members can access the website by computer, tablet, or smartphone to complete a topical survey, update contact information, view published results about the panel, or access technical support.
Each topical surveys data collection will take place over a 4-week period. Pilot panelists will receive their first topical survey up to 4 weeks after recruitment. Each topical survey will be approximately 15 minutes long and panelists will receive $10 upon completion.
All respondents who participate in research under this clearance will be informed that the information they provide will not be made available in any way that would personally identify them and that their participation is voluntary. This disclosure will be made prior to any data collection.
The Pilot is being developed under a cooperative agreement awarded by the Census Bureau pursuant to the Consolidated Appropriations Act of 2021, Public Law 116-260, Section 110. The collection of data for the baseline recruitment in this Pilot is authorized by 13 U.S.C. §§ 8(b), 131, 141, 161, 181, 182, and 193; 49 U.S.C. § 329; 20 U.S.C. §9543; Section 1110(a) of the Social Security Act as amended (42 U.S.C. 1310(a)); 7 U.S.C. § 3318; 7 U.S.C. 2204(a); Section 306 of the Public Health Service Act (42 U.S.C. 242k); and 10 U.S.C., Sections 1782.
Data collection in this Pilot for Census Bureau sponsored topical surveys is authorized by Title 13, Sections, 141, 182, and 193. Data collection in this Pilot for Department of Defense topical surveys is authorized by 10 U.S.C., Section 1782.
The confidentiality of information collected on topical surveys in this panel is assured by one or more of the following authorities, as applicable: CIPSEA, Title 13 of the United States Code, or other titles of the United States Code which assure the confidentiality of collected information.
The baseline Pilot survey will include the following disclosures:
The Department of Defense topical panel survey will include the following disclosures:
The Department of Defense is conducting this voluntary study under the authority of 10 U.S.C. Section 1782. The purpose of collecting this information is to improve and inform future surveys. Your privacy is protected by the Privacy Act of 1974 (5 U.S.C. Section 552a).
The Census Bureau topical survey will include the following disclosure:
The U.S. Census Bureau is conducting this voluntary study under the authority of 13 U.S.C. Sections 141182, and 193. The purpose of collecting this information is to improve and inform future surveys.
Your privacy is protected by the Privacy Act of 1974 (5 U.S.C. Section 552a). The Privacy Act permits the information provided to be shared by Census Bureau staff for work-related purposes as identified in the Privacy Act System of Records Notices (SORNs) COMMERCE/CENSUS-5, Decennial Census Programs.
Most of the questions that are included on the Pilot questionnaires are not of a sensitive nature and should not pose a problem to the respondents. However, it is possible that some potentially sensitive questions may be included in questionnaires that are tested under this clearance. One of the purposes of the testing is to identify such questions, determine sources of sensitivity, and alleviate them insofar as possible before a larger production survey (independent of the Panel) is administered.
For information collections involving questions of race/ethnicity, the agency will ensure that the OMB Standards for the Classification of Federal Data on Race and Ethnicity are followed, unless we are specifically testing these questions. In that situation, OMB will be made aware of the proposed changes and the related research agenda.
Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. Unless directed to do so, agencies should not conduct special surveys to obtain information on which to base hour burden estimates. Consultation with a sample (fewer than 10) of potential respondents is desirable. If the hour burden on respondents is expected to vary widely because of differences in activity, size, or complexity, show the range of estimated hour burden, and explain the reasons for the variance. Generally, estimates should not include burden hours for customary and usual business practices.
If this request for approval covers more than one form, provide separate hour burden estimates for each form and aggregate the hour burdens.
Provide estimates of annualized cost to respondents for the hour burdens for collections of information, identifying and using appropriate wage rate categories. The cost of contracting out or paying outside parties for information collection activities should not be included here. Instead, this cost should be included under ‘Annual Cost to Federal Government’ (Item #14).
Among the 4,800 households sampled for the Pilot, approximately 1,664 household respondents will complete the 5-minute household roster. This totals approximately 139 hours (maximum, if every sampled participant was successfully contacted and completed the screener). We anticipate that approximately 2,779 adults will be selected based on the residential household roster information. An additional 2,456 adults (1,756 active-duty military service personnel and 700 non-military spouses) will be randomly selected directly from military administrative data.
The baseline questionnaire will take approximately 20 minutes for each panel member. Each panel member will complete 1 topical survey at 15 minutes each during the Pilot period. This would be a total burden of 35 minutes for each of the 1,700 recruited panel members (1,500 residential, 100 active duty, and 100 non-military spouse sample members) and a total of approximately 992 hours.
Thus, the total burden hours for this pilot is 1,131.
The estimated total annual respondent cost burden based on these hours is $30,921. For individuals, the wage rate is $25.72 per hour based on hourly earnings for employees as reported by the Bureau of Labor Statistics.
Type of Respondent |
Expected Number of Respondents |
Average Burden per Response (in hours) |
Total Burden Hours |
Hourly Wage Rate |
Total Respondent Cost |
Household Rosters1 |
1,664
|
0.08 |
139
|
$2.06 |
$286 |
Panel Members |
1,700 |
0.58 |
992 |
$21.43 |
$21,251 |
Total |
|
|
1,131 |
|
$21,537 |
1 Residential sample only
2 All samples (i.e., residential, active duty, and non-military spouse)
There is no cost to respondents for participating in the research we are conducting under this clearance, except for their time to complete the questionnaire.
The Cooperative Agreement under which the Pilot data is being collected will cost the government a total of $3.5 million from inception (September 2020) through completion of the Pilot (September 2022). Census Bureau annual staff time is estimated at approximately $200,000 (or one FTE).
This is a new collection and a new program. The 60-day notice announced plans for the full data collection. This 30-day notice seeks approval for the Pilot only. A future 30-day notice will seek approval for additional work on this program.
The Census Bureau plans to release a summary of the Pilot test data.
The agency plans to display the expiration date for OMB approval of the information collection on all instruments.
The agency certifies compliance with 5 CFR 1320.9 and the related provisions of 5 CFR 1320.8(b)(3)
1 See OMB guidance: https://www.whitehouse.gov/wp-content/uploads/2021/04/standards_stat_surveys.pdf and Q&A 34 and 72 here: questions and answers when designing survey (archives.gov)
2 See OMB guidance: https://www.whitehouse.gov/wp-content/uploads/2021/04/standards_stat_surveys.pdf and Q&A 34 and 72 here: questions and answers when designing survey (archives.gov)
3 See OMB guidance: https://www.whitehouse.gov/wp-content/uploads/2021/04/standards_stat_surveys.pdf and Q&A 34 and 72 here: questions and answers when designing survey (archives.gov)
Page
|
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | DOC PRA TOOLS 2020 |
| Subject | 2020 |
| Author | Dumas, Sheleen (Federal) |
| File Modified | 0000-00-00 |
| File Created | 2022-05-27 |