Memorandum revised

Volume I EES-NHES Cog Labs 2nd Round -revised.docx

NCES Cognitive, Pilot, and Field Test Studies System

Memorandum revised

OMB: 1850-0803

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 1850-0803 can be found here:

Document [docx]

Download: docx | pdf

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: March 25, 2015

TO: Shelly Martinez, OMB

FROM: Chris Chapman, NCES

THROUGH: Kashka Kubzdela, NCES

SUBJECT: Employment and Earnings Survey – Second Round of Cognitive Testing (OMB# 1850-0803 v.130)

Submittal-Related Information

The following material is being resubmitted under the National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803) which allows NCES to improve the methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, pilot tests, focus groups, and cognitive interviews. This resubmission reflects information that was missing from an earlier submission, and that addresses questions that were raised in communications with OMB. This request is to conduct a second round of cognitive interviews to aid in development of employment and earnings items for the National Household Education Survey (NHES) program, and items that NCES was required to develop in response to regulatory language cleared by OMB in Otocber 2014 (34 CFR 668.406¹). The first round of cognitive interviews for this development effort and recruitment for the second round were approved separately in January 2015 (#1850-0803 v.123 and v. 125, respectively). The data collection for this study is being carried out for NCES by RTI International (contract # ED-IES-13-C-0070).

In preparation for the submission, NCES reviewed existing person-level surveys conducted regularly by the United States Census Bureau (Census) and the Bureau of Labor Statistics (BLS) for existing data collection approaches and item sets, and organized meetings that included these agencies. Representatives from the Census Bureau included Charles Nelson (Assistant Division Chief of the Economic Characteristics Division at Census) and Alfred Gottschalk (Chief of the Labor Force Statistics Branch at Census). Anne Polivka, Chief of the Statistics Employment Research Division, represented BLS. In addition, Katherine Abraham, former director of BLS and a recent member of the President’s Council of Economic Advisors, was instrumental in revisions to items that are included in this submission. Appendix 1 below provides item source annotations referncing parallel items from the March Current Population Survey (CPS), which is developed and fielded jointly by Census and BLS and used for official poverty statistics, and from the National Longitudinal Study of Youth (NLSY) fielded by BLS.

Results from the cognitive interviews will be used to inform development of an instrument to be pilot tested in the 2016 NHES as part of the Adult Training and Education Survey (ATES). The ATES survey is designed to collect nationally-representative data on the attainment of non-degree credentials and the labor market characteristics of individuals with such credentials. As part of its submission for OMB clearance for the full scale 2016 NHES, NCES will propose a small pilot test to compare collecting point estimates versus categories for respondent earnings.

The current income categorization approach is not well suited to study potential returns to education or to study poverty. NCES understands that it is not the agency primarily responsible for measuring income, but the results will be used to help address its research needs and limitations of existing approaches noted by the Census Bureau and BLS.

If it is feasible to collect point estimates, NCES can derive variables reflecting poverty status to include on public use files, providing licensed analysts will have more sensitivity for modeling the relationship between credential attainment and labor market outcomes. The cognitive interviews described in this submittal will test the proposed survey item language. For ease of reference, the items proposed here for testing are referred to as the Employment and Earnings Survey.

Apart from the need for the NHES research, NCES is required by regulation to develop an earnings survey to support gainful employment program evaluations (see 34 CFR 668.406 as specified in final regulations published in the Federal Register in October 2014). Per regulatory requirements, the survey is to be published by July 1, 2015. NCES is responsible for developing the survey and the technical standards to which programs must adhere in its administration. The proposed cognitive interviews will thus also allow NCES to complete the second of four steps in the development ad testing of the gainful employment survey, which will be used by institutions to appeal program-level debt-to-earnings ratios calculated by the U. S. Department of Education (ED) under the gainful employment regulations (34 CFR Parts 600 and 668). The earnings component of the debt-to-earnings ratio (D/E ratio) is provided by the Social Security Administration, but institutions may calculate an alternative earnings measure by administering a survey to program graduates. Institutions that choose to submit alternate earnings appeal information will survey all students from programs who graduated during the same period that ED used to calculate the D/E ratios, or a comparable period as defined in 668.406(b)(3) of the regulations. The survey will provide an additional source of earnings data for ED to consider before deciding on final D/E ratios for programs subject to the gainful employment regulations. Programs with final D/E ratios that fail to meet the minimum threshold may face sanctions, including the possible loss of Title IV (federal financial aid) program funds.

Three rounds of cognitive testing and a 2-stage pilot test will refine the Employment and Earnings Survey questions in order to maximize the quality of data collected and provide information on issues with important implications for the survey design. The initial survey and plans for its testing were developed with input from a Technical Review Panel (TRP) that met on December 2, 2014 (membership of the panel is provided in Attachment I). The results of the first round of testing, which was conducted in January and February 2015, were shared with a second panel of earnings experts convened by the National Institute of Statistical Sciences (NISS) on February 27, 2015 (membership of the NISS panel is included in Attachment II, and a copy of the report from the first round of testing is included as Attachment VI). The most significant recommendation from the first round of testing was to combine all income questions into a single multi-part question in order to help respondents correctly distinguish sources of income without double counting. Neither the time burden imposed by the survey nor the sensitivity of the items was reported as a concern by participants. The most significant recommendation from the NISS panel was to revamp the questions to help respondents understand the need to report income from ALL sources and to help them recall their earnings information from two years prior to data collection. To improve recall, the revised survey asks respondents to focus first on the job they held the longest (per approaches tested previously for CPS), and then on all other jobs. This approach is designed to help respondents think about their actual employment situation in 2013, with the goal to elicit total earnings and to not count earnings for the longest job separately.

This request is to conduct the second round of cognitive testing of the Employment and Earnings Survey. In addition to the aforementioned Attachemnts I, II, and IV, this submission includes the informed consent form for cognitive interview participants (Attachment III), the questionnaire to be tested in the cognitive interviews (Attachment V), and the protocol for conducting the cognitive interviews (Attachment VI).

Design and Context

Like the first round, the second round of cognitive testing of the Employment and Earnings Survey will be conducted for NCES by its contractor, RTI International (RTI), in three of RTI’s U.S. office locations – Research Triangle Park, NC; Washington, DC; and Chicago, IL. Participants will be recent graduates (up to 4 years following completion of the credential) from institutions in the vicinity of RTI’s offices, including online program graduates residing in the same geographic areas. RTI will recruit a sufficient numbers of graduates to ensure 15 completed interviews. The recruitment material and the screener for participants were approved on January 30, 2015 (#1850-0803 v.125).

The desired sample will include a mix of respondents who are (1) recipients of a degree or certificate from any level of private for-profit institutions and (2) recipients of a certificate, associates degree, or other vocational credential at public and private not-for-profit, 2-year and less-than-2-year institutions. To help inform gainful employment regulations, the sample will overrepresent graduates from private for-profit less-than-2-year and 2-year institutions. Such schools had higher than average metric failure rates in the analysis conducted to support the development of the gainful employment regulations. Table 1 provides a distribution of the desired minimum number of interview participants by control and level of institution.

Table 1: Desired minimum number of respondents, by institution control and level

Institution control	Institution level	Credential type(s)	Minimum number of respondents
Private for-profit	Less-than-2-year	Vocational certificates	4
	2-year	Vocational certificates and associates	4
	4-year	All	2
Private not-for-profit	Less-than-2-year and 2-year	Vocational certificates and associates	2
Public	Less-than-2-year and 2-year	Vocational certificates and associates	3

As in the first round, cognitive testing will involve one-on-one interviews in which the respondent is asked to “think aloud” as he or she answers survey questions and to answer additional questions about the items they just answered. The survey is being developed and tested as a paper-and-pencil instrument. Each testing session will last about 40 minutes. In order to encourage participation in this study, potential respondents will be offered a $40 incentive for their participation. Arrangements will be made to allow NCES staff to observe the testing and a recording of each interview will be available to NCES for review.

Assurance of Confidentiality

Cognitive interview participants will be informed that their participation is voluntary and that their responses may be used only to help inform the survey design and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002, 20 U.S.C. § 9573). Participants will be assigned a unique student identifier (ID) that will be created solely for file management and used to keep all materials together. The participant ID will not be linked to the student’s name in any way. Participants will sign an informed consent form (Attachment III) which will be kept separate from the interview files and notes in a locked cabinet in a secure room for the duration of the study and destroyed after the final report is submitted.

Schedule for Development of the Employment and Earnings Survey

This second round of interviews is scheduled to take place in mid March 2015. The final draft of the survey will be used in a pilot test with approximately 3,400 program graduates beginning in April 2015. All survey item development activities need to be completed by the end of June 2015 for inclusion in the clearance request for the 2016 collection of NHES and to inform the gainful employment regulations that will be posted in the Federal Register by July 1, 2015. The request for the pilot test will be submitted to OMB as soon as all required materials become available.

Estimate of Respondent Burden

To yield 15 completed interviews, we anticipate screening up to 60 individuals for eligibility to achieve the desired distribution of program graduates. The recruitment and screening process, which has been approved and underway, was estimated to take on average 5 minutes per person.

Table 2: Estimate respondent burden for recruitment for second round of cognitive interviews

Activity	Number of respondents	Number of responses	Minutes per respondent	Maximum total burden hours
Screening*	60	60	5	5
Cognitive interview	15	15	40	10
Study Total	15	15		10

*Burden for screening was approved in January 2015, and thus is not counted in the total burden requested in this submission.

Cost to Federal Government

The cost to the federal government for developing and cognitive and pilot testing the survey will be $555,570, which includes contractor staff time, incentives, and project materials.

Appendix 1

Item Source Annotation for Extant Census and BLS Data Collections

Question 1. Was not derived from existing federal person-level income surveys, but is needed to verify that the respondent matches existing frame information.

Question 2. This leading “No/Yes” item summarizes several pages of employment questions that precede earnings questions in the March CPS. This is the same approach as used in the National Youth Longitudinal Survey income questions (please see https://www.nlsinfo.org/content/cohorts/nlsy97/other-documentation/questionnaires, under round 16 income – item YINC – 1400).

Question 2a asks about earnings from the job held the longest during a calendar year and was derived from March CPS items Q47a (instruction to focus on the job held the longest), and Q48aa and Q48aad (request for all earnings from the longest job)(please see http://www2.census.gov/programs-surveys/cps/techdocs/cpsmar14.pdf, appendix D).

Question 2b asks about earnings from all other jobs and was derived from March CPS items Q49b1d and Q49B1A.

Questions 3 and 3a ask specifically about self-employment earnings. The items were derived from parallel items in NSLY, particularly items YINC-2000 and YINC-2100. The approach to asking about earnings is also similar to item Q48b in the March CPS. Note that in concordance with recommendations from expert reviewers, the concept of ‘net earnings’ was clarified for item 3a in the current survey.

Questions 4 and 4a ask about earnings from other jobs outside of regular jobs. It is based on items Q73A1 T and Q731 in the March CPS. Wording modifications were needed to fit the population of interest for the current earnings survey.

Questions 5 and 5a ask about any other earnings and are based on item Q731 in the March CPS. An adjustment was made because the March CPS item focuses on income and the items in the earnings survey focuses on earnings as opposed to overall income.

Attachment I

Employment and Earnings Survey (EES) Technical Review Panel Members

Geri Anderson

Special Asssistant to the President for External Affairs

Aims Community College

5401 West 20th Street

Greeley, CO 80632

Phone: (303)503-4074


Signature of Person Obtaining Consent		Date

Printed Name of Person Obtaining Consent

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Wine, Jennifer S.
File Modified	0000-00-00
File Created	2021-01-27

Memorandum revised

Volume I EES-NHES Cog Labs 2nd Round -revised.docx

NCES Cognitive, Pilot, and Field Test Studies System