Justification

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: January 28, 2015

TO: Shelly Martinez, OMB

FROM: Sharon Boivin, NCES

THROUGH: Kashka Kubzdela, NCES

SUBJECT: Employment and Earnings Survey Recruitment for 2^nd Round of Cognitive Testing (OMB# 1850-0803 v.125)

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803) which allows NCES to improve the methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, pilot tests, focus groups, and cognitive interviews. This request is to recruit participants for a second round of cognitive interviews to aid in development of employment and earnings items for the National Household Education Survey (NHES) program. These items will also inform the U.S. Department of Education’s development of a new gainful employment survey that will be used by postsecondary programs to survey graduates about their earnings. The first round of cognitive interviews for this development effort were approved in January 2015 (#1850-0803 v.123). The data collection for this study is being carried out for NCES by RTI International.

The NHES collects earnings information as part of the Adult Training and Education Survey. This survey is designed to collect nationally-representative data on the attainment of non-degree credentials and the labor market characteristics of individuals with such credentials. As part of its submission for OMB clearance for the full scale 2016 NHES, NCES will propose a small pilot test to compare collecting point estimates versus categories for respondent earnings. If it is feasible to collect point estimates, NCES can derive variables reflecting poverty status to include on public use files and licensed analysts will have more sensitivity for modeling the relationship between credential attainment and labor market outcomes. The 2^nd round of cognitive interviews described in this submittal will test the proposed survey item language. For ease of reference, the items proposed for testing are referred to as the Employment and Earnings Survey.

The planned cognitive interviews will also inform the gainful employment survey, which will be used by institutions to appeal program-level debt-to-earnings ratios calculated by the U.S. Department of Education under the gainful employment regulations (34 CFR Parts 600 and 668). The earnings component of the debt-to-earnings ratio (D/E ratio) is provided by the Social Security Administration, but institutions may calculate an alternative earnings measure by administering a survey to their program graduates.

NCES will use the cognitive interviews to refine the the Employment and Earnings Survey questions to maximize the quality of data collected and provide information on issues with important implications for the survey design, such as the following:

• The manner and degree to which respondents comprehend the questions and instructions;

• Memory search strategies commonly used and cues found to facilitate recall;

• Memory demands of the questions and the answering strategies used;

• Respondents’ ability to make calculations and judgments;

• Sources of burden and respondent stress; and

• Response domain coverage by the provided response options.

This request is to recruit and screen potential respondents for the 2^nd round of cognitive testing of the Employment and Earnings Survey. This submission includes recruitment and screening materials only. The materials for the cognitive interview themselves, including the informed consent document, the survey items revised from the 1^st round, and the interview protocol, will be submitted as part of a separate clearance memo after a planned meeting of a National Institute for Statistical Sciences (NISS) panel reviews the survey items on February 27, 2015. Because of the tight timeline for this project resulting from the goal of informing the gainful employment regulations, NCES requests approval for 2^nd round of recruitment prior to submitting the 2^nd round cognitive interview documents.

Design and Context

Cognitive testing of the Employment and Earnings Survey will be conducted for NCES by its contractor, RTI International (RTI), in three of RTI’s U.S. office locations – Research Triangle Park, NC; Washington, DC; and Chicago. Participants will be recent graduates (up to 4 years following completion of the credential) from institutions in the vicinity of RTI’s offices, including online program graduates residing in the same geographic areas. RTI will recruit sufficient numbers of graduates to ensure 15 completed interviews. To recruit graduates, RTI will advertise on a variety of websites including Facebook and Craigslist, as well as use direct outreach such as postings at area businesses. In addition, institutional representatives on the TRP have volunteered to assist with recruiting their graduates. Attachment I presents the materials that will be used for recruitment of cognitive interview participants. Once a candidate expresses interest in participating in the cognitive interview, one of the designated recruiters will use a set of scripted screening questions (Attachment II) to determine his/her eligibility for the cognitive interview. The script contains questions on the participant’s current enrollment status, completion of a program within the reference period, type of program, and institution attended. The recruitment material and the screener are the same as those approved for the 1^st round.

The desired sample will include a mix of respondents who are (1) recipients of a degree or certificate from any level of private for-profit institutions and (2) recipients of a certificate, associates degree, or other vocational credential at public and private not-for-profit, 2-year and less-than-2-year institutions. To help inform gainful employment regulations, the sample will overrepresent graduates from private for-profit less-than-2-year and 2-year institutions. Such schools had higher than average metric failure rates. Table 1 provides a distribution of the desired minimum number of interview participants by control and level of institution.

Table 1: Desired minimum number of respondents, by institution control and level

Institution control	Institution level	Credential type(s)	Minimum number of respondents
Private for-profit	Less-than-2-year	Vocational certificates	4
	2-year	Vocational certificates and associates	4
	4-year	All	2
Private not-for-profit	Less-than-2-year and 2-year	Vocational certificates and associates	2
Public	Less-than-2-year and 2-year	Vocational certificates and associates	3

As in the 1^st round, cognitive testing will involve one-on-one interviews in which the respondent is asked to “think aloud” as he or she answers survey questions and to answer additional questions about the items they just answered. The survey is being developed and tested as a paper-and-pencil instrument. Each testing session will last about 40 minutes. In order to encourage participation in this study, potential respondents will be offered a $40 incentive for their participation. Arrangements will be made to allow NCES staff to observe the testing and a recording of each interview will be available to NCES for review.

Assurance of Confidentiality

Cognitive interview participants will be informed that their participation is voluntary and that their responses may be used only to help inform the survey design and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002, 20 U.S.C. § 9573). Participants will be assigned a unique student identifier (ID) that will be created solely for file management and used to keep all materials together. The participant ID will not be linked to the student’s name in any way. Participants will sign an informed consent form (to be submitted as part of the 2^nd round of cognitive interviews data colelction clearance package) which will be kept separate from the interview files and notes in a locked cabinet in a secure room for the duration of the study and destroyed after the final report is submitted.

Schedule for Development of the Employment and Earnings Survey

RTI will begin recruiting for the second round of testing upon receiving OMB clearance and will submit a second clearance package for the conduct of the cognitive interviews after revising the survey items based on results from the 1^st round of cognitive interviews and recommendations from the NISS panel at the end of February 2015. This 2^nd round of interviews will take place in mid March 2015. The final draft of the survey will be used in a pilot test with approximately 3,400 program graduates beginning in April 2015. All survey item development activities need to be completed by the end of June 2015 for inclusion in the clearance request for the 2016 collection of NHES and to inform the gainful employment regulations that will be posted in the Federal Register by July 1, 2015. The requests for the second round of cognitive interviews and for the pilot test will be submitted to OMB as soon as all required materials become available.

Estimate of Respondent Burden

To yield 15 completed interviews, we anticipate screening up to 60 individuals for eligibility to achieve the desired distribution of program graduates. The screening process, on average, is estimated to take about 5 minutes per person.

Table 2: Estimate respondent burden for recruitment for 2^nd round of cognitive inetrviews

Activity	Number of respondents	Number of responses	Minutes per respondent	Maximum total burden hours
Screening	60	60	5	5
Cognitive interview	15	15	40	10
Study Total	60	60		5

Cost to Federal Government

The cost to the federal government for developing and cognitive and pilot testing the survey will be $555,570, which includes contractor staff time, incentives, and project materials.

3