The Office of Management and Budget
Paperwork Reduction Act Submission
|
Prepared by:
THE CORPORATION FOR NATIONAL AND COMMUNITY SERVICE
CONTENTS
Part B
STATISTICAL METHODS OF DATA COLLECTION
B1 Potential Respondent Universe
B2 Sampling Method and Respondent Universe
B2.1 Sampling Plan
B2.2 Procedures to Deal With Non-Response
B3 Pre-Testing of Procedures
B4 Persons Responsible for Statistical Aspects of the Design
Part B
STATISTICAL METHODS OF DATA COLLECTION
B1 Potential Respondent Universe
The Census Bureau conducts the November Civic Engagement Supplement in conjunction with the Current Population Survey (CPS), for which the universe is 118 million households. From this universe, the Census Bureau selects a sample of approximately 72,000 households each month, of which approximately 59,000 households are eligible for interview. We actually interview about 54,000 households each month. The items in the Civic Engagement supplement are asked, as appropriate, for all members of these households. All civilian household members age 18 and up (citizens and non-citizens) are eligible for the civic engagement questions.
B2 Sampling Method and Respondent Universe
B2.1 Sampling Plan
Attachment D gives an overview of the CPS sample design and weighting methodology and response rates. The statistical properties of these supplemental items will fall within those associated with the CPS itself.
B2.2 Procedures to Deal With Non-Response
The Census Bureau maintains response rates and data accuracy for the CPS at high levels through interviewer instruction, self-study training, and follow-up of refusal interviews with more experienced senior interviewers. Additionally, they closely monitor data output and response rates, and conduct extensive debriefs of CPS interviewers and call center staff to identify potential problems with the survey. (Refer to Item 5 of Attachment D for a discussion of the CPS nonresponse.)
In even-numbered years, when the Civic Engagement Supplement is conducted along with the Voting Supplement, nonresponse rates have tended to be somewhat higher than those observed on a typical CPS supplement. To control for potential nonresponse bias, the Census Bureau has constructed special weights that incorporate information about volunteering (unpaid work done through or for an organization) that is gathered from the CPS Volunteering Supplement.
Previous Civic Engagement Supplements have used a nonresponse bias adjustment that utilized the fact that their sample universes had a significant overlap with the universe for the CPS Volunteer Supplement, conducted just two months prior. Among Volunteer Supplement respondents, volunteerism rates were compared between Civic Engagement respondents and between nonrespondents. A nonresponse bias adjustment factor was generated on that basis, which was then used to adjust for nonresponse bias in the Civic Engagement Supplement.
B3 Pre-Testing of Procedures
Abt Associates conducted two rounds of cognitive testing of the November 2011 Civic Engagement supplement questions. The first round of testing was conducted February 23-March 1, 2011, and the second round was conducted April 28-May 2, 2011. The goal of each round was to test respondents' comprehension of the new questions, to test the flow of the supplement, to detect and repair major recall difficulties, to ascertain the sensitivity or inappropriateness of any questions, and to gauge the operational feasibility of the supplement. The cognitive interviewing protocols included probing questions embedded within the supplement and debriefing questions at the end of the questionnaire. The final form of the questionnaire appears in Attachment A1.
The first round consisted of in-person cognitive interviews with 15 respondents at the Abt Associates Cognitive Testing Laboratory in Bethesda, MD. Respondents were selectively recruited for diversity with respect to age, sex, and socioeconomic status. Interviews took approximately one hour and were video recorded for subsequent analysis. The cognitive interviewing protocol consisted of both preexisting items and additional items being considered for inclusion. The new items were integrated into a series of items from the November 2009 Civic Engagement supplement (the most recent nonelection year version) in a sequence that was designed to provide an appropriate substantive “flow.”
Overall, the survey was well received. Respondents felt most questions were easy to answer and generally reported being interested in question topics. There was little consensus on which questions might be sensitive. The item most consistently mentioned as problematic measures confidence in various institutions. A number of respondents felt that it was difficult to provide an answer for institutions about which they were ambivalent (trusting some members of the institution but not others). Testing also revealed variation in how respondents interpreted several of the institutions on the list. In order to improve the question, the list of institutions was ultimately reduced from ten to three, another response option was added to accommodate respondents with no confidence, and interviewer instructions were developed to address potential respondent uncertainty about what the question was asking.
A detailed summary report of Round 1 testing is provided in Attachment A2, and a summary of the debriefing responses is provided in Attachment A3. Revisions to the questionnaire resulting from this test led to the need for another round of testing.
The second round of cognitive interviews was also conducted in-person with 15 respondents at the Abt Associates Cognitive Testing Laboratory. A sumary of the second round of testing is provided in Attachment A4. Round 2 was used to evaluate the questionnaire modifications made based on the results of Round 1. Probes embedded in the questionnaire and debriefing questions were both used to gain insight on respondents’ interpretation of the items, strategies for responding, and any cognitive difficulties experienced.
Results from the two rounds of testing revealed, in general, respondents found the supplement questions interesting and easy to answer. One notable exception was an item in the “confidence in institutions” battery on religious institutions. This item elicited negative feedback in both rounds of cognitive interviewing and has been dropped from the supplement. Another item asking about trust in one’s neighbors was revised substantially from Round 1 to Round 2. Even with the revised Round 2 wording, respondents’ interpretations of trust varied. In response, new interviewer instructions have been added to the supplement, including definitions for both “trust” and “people in your neighborhood.” Another notable change made to the supplement prior to Round 2 was on a question about frequency of voting in local elections. The revised wording provides examples of local elections, and the Round 2 testing indicated that this improved comprehension and helped most respondents to provide an appropriate response.
Other results from testing included elimination of problematic questions, rephrasing of certain question fragments, omission of words, addition of transitional language, and revision of answer categories to make the questions more understandable. While some questions led respondents to pause and think through their answer, overall, the respondents received the questions in a positive manner.
B4 Persons Responsible for Statistical Aspects of the Design
The Census Bureau will collect and process the data. Within the Census Bureau, the following individuals may be contacted for further information on data collection, operations, and analysis:
Statistical Design
Patrick Flanagan Chief, Current Population Surveys Branch
Demographic Statistical Methods Division
(301) 763-4290
Data Collection/Survey Design
Christopher J. Laskey Assistant Division Chief, Continuing Surveys
Demographic Surveys Division
(301) 763‑5312
Annex
A1. Final Voting and Civic Engagement supplement questionnaire
A2. Cognitive Testing (Round 1) Summary Report, Abt Associates, Draft March 11, 2011
A3. Cognitive Testing (Round 1) Debriefing Report, Abt Associates, Draft March 14, 2011
A4. Cognitive Testing (Round 1) Summary and Debriefing Report, Abt Associates, Draft May 10, 2011
B. CPS-263 (MIS-1) Advance Letter
C. BC-1428 Confidentiality Brochure
D. Overview of CPS Sample Design and Methodology
| File Type | application/msword |
| File Title | Part A |
| Author | MAbravan |
| Last Modified By | Dietz, Nathan |
| File Modified | 2011-08-25 |
| File Created | 2011-08-25 |