Memorandum United States Department of Education
Institute of Education Sciences
National Center for Education Statistics
DATE: January 28, 2015
TO: Shelly Martinez, OMB
FROM: Sharon Boivin, NCES
THROUGH: Kashka Kubzdela, NCES
SUBJECT: Employment and Earnings Survey Recruitment for 2nd Round of Cognitive Testing (OMB# 1850-0803 v.125)
The following material is being submitted under the National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803) which allows NCES to improve the methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, pilot tests, focus groups, and cognitive interviews. This request is to recruit participants for a second round of cognitive interviews to aid in development of employment and earnings items for the National Household Education Survey (NHES) program. These items will also inform the U.S. Department of Education’s development of a new gainful employment survey that will be used by postsecondary programs to survey graduates about their earnings. The first round of cognitive interviews for this development effort were approved in January 2015 (#1850-0803 v.123). The data collection for this study is being carried out for NCES by RTI International.
The NHES collects earnings information as part of the Adult Training and Education Survey. This survey is designed to collect nationally-representative data on the attainment of non-degree credentials and the labor market characteristics of individuals with such credentials. As part of its submission for OMB clearance for the full scale 2016 NHES, NCES will propose a small pilot test to compare collecting point estimates versus categories for respondent earnings. If it is feasible to collect point estimates, NCES can derive variables reflecting poverty status to include on public use files and licensed analysts will have more sensitivity for modeling the relationship between credential attainment and labor market outcomes. The 2nd round of cognitive interviews described in this submittal will test the proposed survey item language. For ease of reference, the items proposed for testing are referred to as the Employment and Earnings Survey.
The planned cognitive interviews will also inform the gainful employment survey, which will be used by institutions to appeal program-level debt-to-earnings ratios calculated by the U.S. Department of Education under the gainful employment regulations (34 CFR Parts 600 and 668). The earnings component of the debt-to-earnings ratio (D/E ratio) is provided by the Social Security Administration, but institutions may calculate an alternative earnings measure by administering a survey to their program graduates.
NCES will use the cognitive interviews to refine the the Employment and Earnings Survey questions to maximize the quality of data collected and provide information on issues with important implications for the survey design, such as the following:
The manner and degree to which respondents comprehend the questions and instructions;
Memory search strategies commonly used and cues found to facilitate recall;
Memory demands of the questions and the answering strategies used;
Respondents’ ability to make calculations and judgments;
Sources of burden and respondent stress; and
Response domain coverage by the provided response options.
This request is to recruit and screen potential respondents for the 2nd round of cognitive testing of the Employment and Earnings Survey. This submission includes recruitment and screening materials only. The materials for the cognitive interview themselves, including the informed consent document, the survey items revised from the 1st round, and the interview protocol, will be submitted as part of a separate clearance memo after a planned meeting of a National Institute for Statistical Sciences (NISS) panel reviews the survey items on February 27, 2015. Because of the tight timeline for this project resulting from the goal of informing the gainful employment regulations, NCES requests approval for 2nd round of recruitment prior to submitting the 2nd round cognitive interview documents.
Cognitive testing of the Employment and Earnings Survey will be conducted for NCES by its contractor, RTI International (RTI), in three of RTI’s U.S. office locations – Research Triangle Park, NC; Washington, DC; and Chicago. Participants will be recent graduates (up to 4 years following completion of the credential) from institutions in the vicinity of RTI’s offices, including online program graduates residing in the same geographic areas. RTI will recruit sufficient numbers of graduates to ensure 15 completed interviews. To recruit graduates, RTI will advertise on a variety of websites including Facebook and Craigslist, as well as use direct outreach such as postings at area businesses. In addition, institutional representatives on the TRP have volunteered to assist with recruiting their graduates. Attachment I presents the materials that will be used for recruitment of cognitive interview participants. Once a candidate expresses interest in participating in the cognitive interview, one of the designated recruiters will use a set of scripted screening questions (Attachment II) to determine his/her eligibility for the cognitive interview. The script contains questions on the participant’s current enrollment status, completion of a program within the reference period, type of program, and institution attended. The recruitment material and the screener are the same as those approved for the 1st round.
The desired sample will include a mix of respondents who are (1) recipients of a degree or certificate from any level of private for-profit institutions and (2) recipients of a certificate, associates degree, or other vocational credential at public and private not-for-profit, 2-year and less-than-2-year institutions. To help inform gainful employment regulations, the sample will overrepresent graduates from private for-profit less-than-2-year and 2-year institutions. Such schools had higher than average metric failure rates. Table 1 provides a distribution of the desired minimum number of interview participants by control and level of institution.
Table 1: Desired minimum number of respondents, by institution control and level
Institution control |
Institution level |
Credential type(s) |
Minimum number of respondents |
Private for-profit |
Less-than-2-year |
Vocational certificates |
4 |
|
2-year |
Vocational certificates and associates |
4 |
|
4-year |
All |
2 |
Private not-for-profit |
Less-than-2-year and 2-year |
Vocational certificates and associates |
2 |
Public |
Less-than-2-year and 2-year |
Vocational certificates and associates |
3 |
As in the 1st round, cognitive testing will involve one-on-one interviews in which the respondent is asked to “think aloud” as he or she answers survey questions and to answer additional questions about the items they just answered. The survey is being developed and tested as a paper-and-pencil instrument. Each testing session will last about 40 minutes. In order to encourage participation in this study, potential respondents will be offered a $40 incentive for their participation. Arrangements will be made to allow NCES staff to observe the testing and a recording of each interview will be available to NCES for review.
Cognitive interview participants will be informed that their participation is voluntary and that their responses may be used only to help inform the survey design and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002, 20 U.S.C. § 9573). Participants will be assigned a unique student identifier (ID) that will be created solely for file management and used to keep all materials together. The participant ID will not be linked to the student’s name in any way. Participants will sign an informed consent form (to be submitted as part of the 2nd round of cognitive interviews data colelction clearance package) which will be kept separate from the interview files and notes in a locked cabinet in a secure room for the duration of the study and destroyed after the final report is submitted.
RTI will begin recruiting for the second round of testing upon receiving OMB clearance and will submit a second clearance package for the conduct of the cognitive interviews after revising the survey items based on results from the 1st round of cognitive interviews and recommendations from the NISS panel at the end of February 2015. This 2nd round of interviews will take place in mid March 2015. The final draft of the survey will be used in a pilot test with approximately 3,400 program graduates beginning in April 2015. All survey item development activities need to be completed by the end of June 2015 for inclusion in the clearance request for the 2016 collection of NHES and to inform the gainful employment regulations that will be posted in the Federal Register by July 1, 2015. The requests for the second round of cognitive interviews and for the pilot test will be submitted to OMB as soon as all required materials become available.
To yield 15 completed interviews, we anticipate screening up to 60 individuals for eligibility to achieve the desired distribution of program graduates. The screening process, on average, is estimated to take about 5 minutes per person.
Table 2: Estimate respondent burden for recruitment for 2nd round of cognitive inetrviews
Activity |
Number of respondents |
Number of responses |
Minutes per respondent |
Maximum total burden hours |
Screening |
60 |
60 |
5 |
5 |
Cognitive interview |
15 |
15 |
40 |
10 |
Study Total |
60 |
60 |
|
5 |
The cost to the federal government for developing and cognitive and pilot testing the survey will be $555,570, which includes contractor staff time, incentives, and project materials.
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
Author | Wine, Jennifer S. |
File Modified | 0000-00-00 |
File Created | 2021-01-27 |