NATES 2013 Pilot Test Survey Change Memo

Date: January 31, 2021

To: Shelly Martinez, OMB

From: Sharon Boivin, NCES

Through: Kashka Kubzdela, NCES

Re: Changes in the National Adult Training and Education Survey (NATES) 2013 Pilot Test Resubmission (OMB# 1850-0803 v.74)

This memo accompanies the second set of clearance documents for the National Adult Training and Education Survey (NATES) 2013 Pilot Test (OMB# 1850-0803 v.74) and outlines the changes that were made from the previously approved set of clearance documents for this study (OMB# 1850-0803 v.72 approved on September 24, 2012).

The changes consist of improvements to the sample design for the NHES screener test being conducted as part of the pilot and improvements to the NATES topical questionnaire. These changes have an associated increase in respondent burden (700 additional respondents asked to complete the 3-minute screener), but no change to the cost to the federal government.

NATES/NHES Screener Test

After discussions with Andy Zukerberg of NCES and the Census Bureau, we are requesting approval to increase the screener test sample size from 500 to 1,200 cases and to split the sample between 720 respondents who will receive the new NATES screener that enumerates adults in the household and 480 respondents who will receive the original NHES screener (with no adult enumeration). A concurrent administration of the two screener options will allow NCES to evaluate the direct effect of adding adult enumeration to the screener without the confounds of time and context that would result if we were to compare the response rates on the NATES screener to the original response rate achieved by the NHES screener in its full scale administration in 2012. Increasing the test sample size from 500 to 1,200 respondents will provide enough power to compare response rates in this more robust test of the NATES screener. This increase in sample size changes the burden estimates as follows (cells affected by the change are highlighted in yellow):

Original Burden Estimate:

Table 1.  Estimated response burden for the NATES Pilot

Instrument	Unit1 sample size	Expected response rate (%)	Expected number of completed units2	Avg. unit completion time (mins)	Total burden hours
Mail Screener	500	60	270	3	14
Topical Survey3	10,000	65	5,850 (8,775 individual responses)	22.5	2,194
Nonresponse Survey	1,670	60	1,000	5	83
Total	NA	NA	7,120 (10,045 individual responses)	NA	2,291
1 Unit = Household 2 Approximately 10 percent of the address sample is expected to be undeliverable. 3 Based on NCES testing, it takes an average of 15 minutes for a person to complete the questionnaire. Each sampled unit will receive either 3 individual questionnaires or a booklet that includes the full set of questionnaire items for up to three eligible household members. The average unit completion time is estimated to be 22.5 minutes based on the expectation that on average responding units will complete one and a half questionnaires.

Revised Burden Estimate:

Table 1 (Revised). Estimated response burden for the NATES Pilot

Instrument	Unit1 sample size	Expected response rate (%)	Expected number of completed units2	Avg. unit completion time (mins)	Total burden hours
Mail Screener	1,200	60	720	3	36
Topical Survey3	10,000	65	5,850 (8,775 individual responses)	22.5	2,194
Nonresponse Survey	1,670	60	1,000	5	83
Total	NA	NA	7,570 (10,495 individual responses)	NA	2,313
1 Unit = Household 2 Approximately 10 percent of the address sample is expected to be undeliverable. 3 Based on NCES testing, it takes an average of 15 minutes for a person to complete the questionnaire. Each sampled unit will receive either 3 individual questionnaires or a booklet that includes the full set of questionnaire items for up to three eligible household members. The average unit completion time is estimated to be 22.5 minutes based on the expectation that on average responding units will complete one and a half questionnaires.

NATES Topical Questionnaire

While working with Census to finalize the questionnaire, we identified a few issues that we felt needed to be addressed prior to printing the questionnaire. The following table summarizes the changes, other than typos or formatting, that are reflected in the revised questionnaire:

Page number	Question number	Change
1	Above A	We are adding an “Introduction” header to improve instrument appearance and to make it clearer that this is a section the survey.
1	A	We are adding the instruction “No one in your household needs to complete any other questionnaires.” in order to be consistent with question B.
2	Q5	We knew at the end of cognitive lab work that at least one respondent found these response options confusing. At the time, our solution was to emphasize some of the text. Upon further reflection, we feel that the following changes will better clarify the response options: Previous response options: __ ONE OR MORE for work-related reasons __ ALL for personal interest New response options: __ I got ONE OR MORE certifications or licenses for work-related reasons __ I did NOT GET ANY certifications or licenses for work-related reasons
7	Q28	We had changed the text of the stem during cognitive lab work, but had forgotten to change the response options to match the new stem. Thus we are changing both response options from “For a job…” to “Related to a job…”
9	Instruction between Q37 and Q38	We now realize this instruction will be missed by respondents who are told to skip to Q39; it is also not clear to which questions the instruction applies. The instruction is thus being moved to come after Q35 and expanded to read “The rest of this section asks about these college classes. If you are on a school break, please respond for the classes you were taking before you went on break.” We are not putting this instruction before Q35 because Q35 was carefully crafted to capture all postsecondary education and the correct kinds of school breaks; we want all respondents to read that question first.
11	Q47	We corrected an error in our definition of adult basic education, changing the first response option in this section from “Basic reading, writing or arithmetic (instruction for adults at the elementary or high school level)” to “Basic reading, writing or arithmetic (instruction for adults below the high school level)”
12	Q50	To match the stem (and the rest of this section), we are changing the text in the last response option from “Not relevant—I was self-employed or not employed when I took the course or instruction” to “Not relevant—I was self-employed or not employed when I took the instruction or training.”
13	Q58	We are deleting this skip because it is unnecessary, is easily missed, and adds complexity to an already complex survey section.
14	Q63-Q64	We are concerned about respondents missing the skip on Q63, so we are exploring the option of reversing the order of Q63 and Q64 (which would then move the skip up to Q62). In consultation with Census, we will decide on the best ordering. (If we reverse the order, we will also modify the skip on Q60 accordingly.)
15	Q79	In order to generate “foreign born” counts that will parallel those in the ACS, we are separating the 2nd response option into: ___In a U.S. territory (Puerto Rico, Guam, American Samoa, U.S. Virgin Islands, or Northern Marianas)  Go to question 82 ___Outside the U.S. (in a foreign country)