Attachment M - Secondary Metrics for the Analysis of the 2016 Content Test

Attachment M: Secondary Metrics for the Analysis of the 2016 Content Test

The Census Bureau proposes to evaluate changes to the questions by comparing the revised questions in the Test Treatment to the current ACS questions in the Control Treatment (for new questions we will compare the performance of question versions to each other). The primary metrics of interest in this comparison include the following: response distributions, item missing data rates, comparisons against benchmark data (when available), and response reliability (as measured by gross difference rates and the index of inconsistency). In addition to these metrics, which may also be examined by mode if there is sufficient data, some of the questions being tested have additional secondary metrics that are important to consider in determining if the Test Treatment wording should be implemented in the future. Examples of these secondary metrics for each topic being tested are as follows:

Race and Hispanic Origin

• Does the item missing data rate for the Ancestry question change for Control and Test versions?

• How do the estimates of Race and Hispanic Origin differ by socioeconomic and demographic characteristics (age, sex, poverty level, income, educational attainment, fertility, tenure, and household size)?

Telephone

• How does the proportion of housing units with no telephone service in each treatment compare with the proportion reported in the most current National Health Interview Survey Wireless Substitution Early Release Program?

• For each version of the question, how often do households report having a smartphone in the Computer question, but answer “no” to the Telephone Service question? Does either version have a higher rate of inconsistency or are they about the same?

Computers

• How do the proportions for each category of computers in each treatment compare with proportions found in the Current Population Survey (CPS) and from surveys done by the Pew Research Center?  

• Do the changes to the Computers question decrease the proportion in the ‘Some other’ category?

Internet Access

• How do the proportions in each treatment compare with proportions found in the Current Population Survey?  

• Among households that reported having a handheld device (smartphone plus tablet categories in Test) on the Computers question, is the proportion with a paid Internet subscription higher in the Test version than in then Control versions?

Internet Subscription

• How do the proportions of mobile broadband subscribers compare to Pew Research findings as well as the most recent Current Population Survey results?  

• Among households that reported having a smartphone or tablet computer in the Computers question, is the proportion reporting “Yes” to “Mobile broadband” higher in Test than in Control?

Journey to Work – Mode of Commute

• How do the proportion of respondents marking one of the three rail categories combined compare between Test and Control versions?

• How do the Test and Control response distributions compare, in metro areas with high levels of light rail usage?

• How do the Test and Control response distributions compare, in metro areas with high levels of overall rail usage?

Journey to Work – Time Leaving Home for work

• Is the proportion of respondents who leave home at a time that ends in 5 or 0 (an indication of rounding) comparable between Test and Control versions?

Number of Weeks Worked

• How does the proportion of full-time, year-round workers (age 16+) for each treatment compare with Current Population Survey estimates?

• How does the distribution of the number of weeks worked for each treatment compare with the comparable distribution from Current Population Survey?

• How does the proportion of year-round workers (age 16+) for each treatment compare with Survey of Income and Program Participation estimates at the national-level?

• How does the distribution of weeks worked for each treatment compare with the comparable distribution from Survey of Income and Program Participation?

• Is the proportion of year-round full-time workers (worked 50+ weeks) greater in the Test version than in the Control version?

Results from both Control and Test versions will be compared with administrative data, contingent upon the receipt and matching of the Longitudinal Employer-Household Dynamics program administrative data. It is possible that any analysis with these data will not be feasible in the time frame necessary to make a decision. The Longitudinal Employer-Household Dynamics data (and the Quarterly Census of Employment and Wages, in particular) can indicate whether someone participated in the labor force in a given quarter for all states, assuming their job was covered by unemployment insurance. With some assumptions, these data can give upper and lower bound estimates for the numbers of people who worked year-round and part-year. If the data is available, the following research questions will be answered:

• Is there a difference in the mean number of quarters worked in the Test sample and the Control sample?

• Is there a difference in the income amounts in the Quarterly Census of Employment and Wages in the Test sample and the Control sample?

• Do the Test and Control samples compare to the administrative data in a similar fashion?

• How does the median income compare between Test and Control responses?

Class of Worker

• How do the Control and Test estimates of unpaid family workers compare with the 2015 Current Population Survey estimate?

• How do the Control and Test estimates of the three government employee categories (Federal, State, and Local) compare with estimates from the 2015 Annual Survey of Public Employment and Payroll?

• How do the Control and Test estimates of self-employed (incorporated and non-incorporated) workers compare with estimates from the 2015 Survey of Business Owners?

• In the paper mode, are there more multiple responses to the Class of Worker question in Test than in Control?

• How often is the coding output value of the Class of Worker question changed by coding clerks from the coding input value?

• In the telephone interviewing (CATI) mode, are the Control and Test rates of “unspecified” responses the same?

• For Test treatment, how consistent are Class of Worker responses with write-in responses about industry (employer name and kind of industry) and the final industry code? In particular, is the reporting of “Active Duty” in the Class of Worker Test version consistent with the Industry write-in responses and Industry code?

• Is the reporting of self-employment income and wages income consistent with the class of worker response?

• Do those persons reported as unpaid family workers work at least 15 hours per week? How often is this not the case?

Industry and Occupation

• For Industry, are the Control and Test codeable rates the same?

• For Occupation, are the Control and Test codeable rates the same?

• For Industry, are the Control and Test interim referral rates the same?

• For Occupation, are the Control and Test interim referral rates the same?

• Are the Control and Test distributions of eligible persons among the Standard Occupational Classification major groups (as indicated by the occupation code) the same?

• Are the Control and Test distributions of eligible persons among the North American Industry Classification System industry sectors (as indicated by the industry code) the same?

• Do the changes to the Industry and Occupation questions result in more specificity in the four write-in responses?

• For each of the four Industry and Occupation write-in fields, are the mean character counts for Test and Control the same?

• For each of the four Industry and Occupation write-in fields, are the mean word counts for Test and Control the same?

• For both Industry and Occupation, are the coding time means for Test and Control the same?

Retirement, Survivor, and Disability income

• How do the proportions in each treatment compare with published Current Population Survey Annual Social and Economic Supplement data?

• Are the estimated mean and median of aggregate survivor, disability, and retirement income for eligible persons higher for the Test treatment than for the Control treatment?

• How do the distributions of persons among the income ranges compare between the Control and Test treatments? Are they significantly different? Are proportion estimates for any single range significantly different between the Control and Test treatments?

• Is the absolute value of the net difference rate for combined Retirement, Survivor, and Disability Income Recipiency smaller for the Test treatment than for the Control treatment?

• For each of the income ranges, is the absolute value of the net difference rate for combined Retirement, Survivor, and Disability income smaller for the Test treatment than for the Control treatment?

Health Insurance

• How do the proportions in each treatment compare with proportions found in other surveys?  

• Does the Test version of the Health Insurance question have a lower proportion of persons with multiple types of Health Insurance compared to the Control version?

• Does the Test version of the Health Insurance question result in a smaller proportion of persons who are reported to have both employer-provided insurance and insurance purchased directly compared to the Control version?

• Does the Test version of the Health Insurance question decrease the proportion of persons who write-in an “other” type of health insurance coverage compared to the Control version?

• Which version needs to be more heavily recoded using the Premium/Subsidy questions?

• Which version produces estimates of Marketplace coverage (subsidized and unsubsidized) that more closely match benchmarks?

• Is there a difference in coverage types, especially means-tested and direct-purchase, between Test and Control responses, by state groups (expansion vs. non-expansion; federal marketplaces vs. state-based)?

• If the sample size permits, is there a difference in coverage types when divided by eligibility for Medicaid and subsidies (both dependent on type of state) using poverty ratios, state, and age?

• How often were the edit checks triggered in the Test version? Is the proportion of persons with more than one type of coverage smaller in Test responses than in Control responses? This research question will only use responses from the CATI, CAPI and Internet modes.

• In CATI/CAPI interviews, how often was the option for respondents to volunteer the same type of coverage as Person 1 (for Persons 2+) used in the Test version? How often did the interviewer “back up” to the Health Insurance Intro screen in order to follow this new path?

Premiums/Subsidies

• Are estimates of the proportions of persons who pay a premium comparable to estimates from Current Population Survey Annual Social and Economic Supplement and National Health Interview Survey?

• Additionally, if timely data are available, we will compare results to benchmarks from administrative data. This could include Medicaid participation as well as participation in the Health Insurance Marketplace (only in states with Federal-based marketplaces).

• Which version has a greater proportion of individuals who are in the universe for the Premium question? (Coming from Health Insurance question.)

• Which version has a greater proportions of individuals who are in the universe for the Subsidy question? (Coming from Premium question.)

• Using the Health Insurance question, Premium question and Subsidy question in combination, which version (Control/Test) produces a higher rate of marketplace coverage?

• Using the Health Insurance question, Premium question and Subsidy question in combination, which version (Control/Test) produces a higher rate of subsidized marketplace coverage?

• Using the Health Insurance question, Premium question and Subsidy question in combination, which version (Control/Test) produces a higher rate of unsubsidized marketplace coverage?