Justification

Volume I EES-NHES Cog Labs 2nd Round.docx

NCES Cognitive, Pilot, and Field Test Studies System

Justification

OMB: 1850-0803

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 1850-0803 can be found here:

Document [docx]

Download: docx | pdf

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: March 3, 2015

TO: Shelly Martinez, OMB

FROM: Sharon Boivin, NCES

THROUGH: Kashka Kubzdela, NCES

SUBJECT: Employment and Earnings Survey – Second Round of Cognitive Testing (OMB# 1850-0803 v.130)

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803) which allows NCES to improve the methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, pilot tests, focus groups, and cognitive interviews. This request is to conduct a second round of cognitive interviews to aid in development of employment and earnings items for the National Household Education Survey (NHES) program. These items will also inform the U.S. Department of Education’s development of a new gainful employment survey that will be used by postsecondary programs to survey graduates about their earnings. The first round of cognitive interviews for this development effort and recruitment for the second round were approved separately in January 2015 (#1850-0803 v.123 and v.125, respectively). The data collection for this study is being carried out for NCES by RTI International (contract # ED-IES-13-C-0070).

As background, the NHES collects earnings information as part of the Adult Training and Education Survey. This survey is designed to collect nationally-representative data on the attainment of non-degree credentials and the labor market characteristics of individuals with such credentials. As part of its submission for OMB clearance for the full scale 2016 NHES, NCES will propose a small pilot test to compare collecting point estimates versus categories for respondent earnings. If it is feasible to collect point estimates, NCES can derive variables reflecting poverty status to include on public use files, providing licensed analysts will have more sensitivity for modeling the relationship between credential attainment and labor market outcomes. The cognitive interviews described in this submittal will test the proposed survey item language. For ease of reference, the items proposed here for testing are referred to as the Employment and Earnings Survey.

The proposed cognitive interviews will also inform the gainful employment survey, which will be used by institutions to appeal program-level debt-to-earnings ratios calculated by the U.S. Department of Education (ED) under the gainful employment regulations (34 CFR Parts 600 and 668). The earnings component of the debt-to-earnings ratio (D/E ratio) is provided by the Social Security Administration, but institutions may calculate an alternative earnings measure by administering a survey to program graduates. Under the regulations, NCES is responsible for developing the survey and the technical standards to which programs must adhere in its administration. Institutions that choose to submit alternate earnings appeal information will survey all students from programs who graduated during the same period that ED used to calculate the D/E ratios, or a comparable period as defined in 668.406(b)(3) of the regulations. The survey will provide an additional source of earnings data for ED to consider before deciding on final D/E ratios for programs subject to the gainful employment regulations. Programs with final D/E ratios that fail to meet the minimum threshold may face sanctions, including the possible loss of Title IV (federal financial aid) program funds.

Two rounds of cognitive interviews will refine the Employment and Earnings Survey questions in order to maximize the quality of data collected and provide information on issues with important implications for the survey design. The initial survey and plans for its testing were developed with input from a Technical Review Panel (TRP) that met on December 2, 2014 (membership of the panel is provided in Attachment I). The results of the first round of testing, which was conducted in January and February 2015, were shared with a panel convened by the National Institute of Statistical Sciences (NISS) on February 27, 2015 (membership of the NISS panel is included in Attachment II, and a copy of the report from the first round of testing is included as Attachment VI). The most significant recommendation from the first round of testing was to combine all income questions into a single multi-part question in order to help respondents correctly distinguish sources of income without double counting. Neither the time burden imposed by the survey nor the sensitivity of the items was reported as a concern by participants. The most significant recommendation from the NISS panel was to revamp the questions to help respondents understand the need to report income from ALL sources and to help them recall their earnings information from two years prior to data collection. To improve recall, the revised survey asks respondents to focus first on the job they held the longest, and then on all other jobs. This approach, recommended by panel member Katharine Abraham and agreed to by the other panel members, is designed to help respondents think about their actual employment situation in 2013. The goal is to elicit total earnings, not to count earnings for the longest job separately.

This request is to conduct the second round of cognitive testing of the Employment and Earnings Survey. In addition to the aforementioned Attachemnts I, II, and IV, this submission includes the informed consent form for cognitive interview participants (Attachment III), the questionnaire to be tested in the cognitive interviews (Attachment V), and the protocol for conducting the cognitive interviews (Attachment VI).

Design and Context

Like the first round, the second round of cognitive testing of the Employment and Earnings Survey will be conducted for NCES by its contractor, RTI International (RTI), in three of RTI’s U.S. office locations – Research Triangle Park, NC; Washington, DC; and Chicago, IL. Participants will be recent graduates (up to 4 years following completion of the credential) from institutions in the vicinity of RTI’s offices, including online program graduates residing in the same geographic areas. RTI will recruit sufficient numbers of graduates to ensure 15 completed interviews. The recruitment material and the screener for participants were already approved on January 30, 2015 (#1850-0803 v.125).

The desired sample will include a mix of respondents who are (1) recipients of a degree or certificate from any level of private for-profit institutions and (2) recipients of a certificate, associates degree, or other vocational credential at public and private not-for-profit 2-year and less-than-2-year institutions. To help inform gainful employment regulations, the sample will overrepresent graduates from private for-profit less-than-2-year and 2-year institutions. Such schools had higher than average metric failure rates. Table 1 provides a distribution of the desired minimum number of interview participants by control and level of institution.

Table 1: Desired minimum number of respondents, by institution control and level

Institution control	Institution level	Credential type(s)	Minimum number of respondents
Private for-profit	Less-than-2-year	Vocational certificates	4
	2-year	Vocational certificates and associates	4
	4-year	All	2
Private not-for-profit	Less-than-2-year and 2-year	Vocational certificates and associates	2
Public	Less-than-2-year and 2-year	Vocational certificates and associates	3

As in the first round, cognitive testing will involve one-on-one interviews in which the respondent is asked to “think aloud” as he or she answers survey questions and to answer additional questions about the items they just answered. The survey is being developed and tested as a paper-and-pencil instrument. Each testing session will last about 40 minutes. In order to encourage participation in this study, potential respondents will be offered a $40 incentive for their participation. Arrangements will be made to allow NCES staff to observe the testing and a recording of each interview will be available to NCES for review.

Assurance of Confidentiality

Cognitive interview participants will be informed that their participation is voluntary and that their responses may be used only to help inform the survey design and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002, 20 U.S.C. § 9573). Participants will be assigned a unique student identifier (ID) that will be created solely for file management and used to keep all materials together. The participant ID will not be linked to the student’s name in any way. Participants will sign an informed consent form (Attachment III) which will be kept separate from the interview files and notes in a locked cabinet in a secure room for the duration of the study and destroyed after the final report is submitted.

Schedule for Development of the Employment and Earnings Survey

This second round of interviews is scheduled to take place in mid March 2015. The final draft of the survey will be used in a pilot test with approximately 3,400 program graduates beginning in April 2015. All survey item development activities need to be completed by the end of June 2015 for inclusion in the clearance request for the 2016 collection of NHES and to inform the gainful employment regulations that will be posted in the Federal Register by July 1, 2015. The request for the pilot test will be submitted to OMB as soon as all required materials become available.

Estimate of Respondent Burden

To yield 15 completed interviews, we anticipate screening up to 60 individuals for eligibility to achieve the desired distribution of program graduates. The recruitment and screening process, which has been approved and underway, was estimated to take on average 5 minutes per person.

Table 2: Estimate respondent burden for recruitment for second round of cognitive interviews

Activity	Number of respondents	Number of responses	Minutes per respondent	Maximum total burden hours
Screening*	60	60	5	5
Cognitive interview	15	15	40	10
Study Total	15	15		10

*Burden for screening was approved in January 2015, and thus is not counted in the total burden requested in this submission.

Cost to Federal Government

The cost to the federal government for developing and cognitive and pilot testing the survey will be $555,570, which includes contractor staff time, incentives, and project materials.

Attachment I

Employment and Earnings Survey (EES) Technical Review Panel Members

Geri Anderson

Special Asssistant to the President for External Affairs

Aims Community College

5401 West 20th Street

Greeley, CO 80632

Phone: (303)503-4074


Signature of Person Obtaining Consent		Date

Printed Name of Person Obtaining Consent

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Wine, Jennifer S.
File Modified	0000-00-00
File Created	2021-01-27

Justification

Volume I EES-NHES Cog Labs 2nd Round.docx

NCES Cognitive, Pilot, and Field Test Studies System

Justification

OMB: 1850-0803

Submittal-Related Information

Design and Context

Assurance of Confidentiality

Schedule for Development of the Employment and Earnings Survey

Estimate of Respondent Burden

*Burden for screening was approved in January 2015, and thus is not counted in the total burden requested in this submission.

Cost to Federal Government

Introduction

Purpose

The interview is being conducted on behalf of the National Center for Education Statistics, within the U.S Department of Education, by RTI International, a research organization with headquarters in Research Triangle Park, North Carolina.

Procedures

Study Duration

We do not anticipate that any of the discussion topics will make you uncomfortable or upset. However you may refuse to answer any question or take a break at any time.

| Your Benefits |

There are no direct benefits to you from participating in this study.

Confidentiality

Future Contacts

Your Rights

Your Questions