NPSAS Justification

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: September 3, 2010

TO: Shelly Martinez, OMB

FROM: Tracy Hunt-White, NCES

THROUGH: Kashka Kubzdela, NCES

SUBJECT: NPSAS:12 Cognitive Testing 2010 (OMB No. 1850-0803 v.32)

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803) that provides for NCES to improve methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, focus groups, and cognitive interviews.

The request for approval described in this memorandum is to conduct cognitive testing of the student questionnaire items to be used in the 2011-12 National Postsecondary Student Aid Study (NPSAS:12) field test. OMB approved on 7/15/2010 (OMB# 1850-0803 v. 32) a request to begin recruiting and screening activities for the cognitive testing. This package provides information about plans and procedures for conducting the cognitive testing, presents the questionnaire items to be tested, and includes final item wording and the probes to be administered. NCES will also begin the OMB clearance process on another package in September 2010 (OMB # 1850-0666) for approval to conduct student interviews and collect student records from postsecondary institutions for the NPSAS:12 field test, scheduled to begin in March 2011.

Background

Housed in NCES’s Postsecondary, Adult, and Career Education Division, NPSAS:12 is a comprehensive study of how students and their families pay for postsecondary education. In addition to providing cross-sectional information about college costs and financing, NPSAS:12 will serve as the base year data collection for the next Beginning Postsecondary Students Longitudinal Study (BPS:12/14, BPS:12/17), provided the BPS options will be exercised by NCES. As a longitudinal study, BPS is able to investigate persistence and enrollment in less-than-2-year, 2-year, and 4-year institutions, transfer and graduation rates, employment, and student loan debt. BPS, which samples both traditionally- and nontraditionally-aged postsecondary students, is the only nationally-representative, longitudinal survey of first-time beginners in postsecondary education.

Working together with a distinguished team of researchers and its contractor, RTI International, NCES has reconceptualized BPS to better elaborate the decision-making processes of postsecondary students. The BPS student interview, a subsection of the NPSAS:12 base year interview, is also being redesigned to reflect this new conceptualization. Drawing upon human capital theory, the redesign team is developing a model of student decision-making in which choices are based upon probabilistic expectations of the rewards and costs of alternative choices, and grounded in imperfect – and evolving – information.

In January 2010, NCES received approval to conduct focus groups – the first stage in a multistage qualitative evaluation comprised of focus groups and cognitive interviews. This qualitative evaluation will inform refinement of items used in previous surveys as well as the development of items which will help to elaborate the postsecondary choices of the first-time beginning (FTB) population. Focus groups were conducted to help the redesign team move from conceptualization to instrument development. Additionally, the focus groups were used to improve a select set of existing questions in the NPSAS and BPS interviews, particularly items involving financial aid terminology that is possibly unfamiliar to students (e.g. private loans) and items used to determine eligibility for the BPS cohort. The first stage has been completed and the project team is now ready for the second stage – cognitive testing (see the included schedule).

Instrument development was informed by the results of the focus groups, and will be tested with postsecondary students in RTI’s cognitive lab. The cognitive testing process will enable the instrumentation team to:

• examine the thought processes affecting the quality of answers provided to survey questions,

• understand the extent to which terms in questions are comprehended,

• evaluate the memory demands of the questions,

• evaluate the ability of respondents to make calculations and judgments,

• determine appropriate presentations of response categories,

• assess the time it takes to complete the interview,

• assess the navigational problems users face, and

• identify sources of burden and respondent stress.

Once the cognitive testing is complete, RTI will submit the items and test results from both evaluations (focus groups and cognitive testing) to the Q Bank¹ -- a federal repository of questions that have been tested along with the results.

Design and Context

Cognitive Testing. Survey Methodologists from RTI’s Program for Research in Survey Methodology (PRISM) will conduct cognitive interviews with recruited participants. RTI-PRISM staff have extensive experience in all types of cognitive interviewing methodologies, and NPSAS cognitive interview protocols will include both “think aloud” data capture, and scripted probing. A "think aloud" interview is one in which the respondent is instructed to tell the interviewer everything that he/she is thinking about in answering a survey question. Probes for selected items will be concurrent (asked at the same time the subject answers the questions) and retrospective (asked during a debriefing session) and will be prepared ahead of time. In addition, methodologists will employ spontaneous probes as needed.

RTI is drawing cognitive interview participants from the Washington, D.C. area² as well as from the greater Research Triangle Park, NC area where RTI’s North Carolina-based Laboratory for Survey Methods and Measurement is located. Both locations are home to a large number of postsecondary institutions, making them particularly well-suited for recruiting the target population. In the event that we are unable to recruit adequate numbers of students to come to the RTP or DC labs, we will make arrangements with institutions in DC and/or RTP to conduct the cognitive interviews on campus. We will ensure that the space is private so the interviews cannot be overheard (such as a study room in a library or classroom).

A total of forty-eight cognitive interviews are planned. Interviews will be conducted in person, in order to evaluate the instrument. Students will be selected from all levels of postsecondary institutions (4-year, 2-year, and less than 2-year) and will be selected to provide broad representation of the FTB population based on age, sex, and other key characteristics. An abbreviated questionnaire will collect standard demographic information about each of the participants during the screening process. Students will be compensated for their participation in the interview. Attachment I provides additional detail about recruitment procedures. Attachment II presents the screening questions that will be used to determine eligibility for cognitive interview participation. Attachment III contains the Assurance of Confidentiality. A facsimile of item wording and generic probes is presented in Attachment IV.

The cognitive interviews will be held in facilities that are centrally located, easily accessible by car and public transportation, and allow for audio recording. Interviews will be conducted at times convenient for student schedules. The survey is estimated to take about 45 minutes, and additional time is needed for the “think aloud” and probe responses. In total, the cognitive interview process is expected to last about two hours for each participant. Interviews will be led by an RTI-PRISM staff member with considerable expertise in cognitive interviewing and professional training in cognitive and social psychology and survey methodology. The recordings will be made available to NCES for review.

Immediately following the conclusion of each interview, methodologists will review the cognitive interview recordings and notes, highlighting potential themes that may have arisen. Following each interview, the digital audio recording will be archived for qualitative analysis. RTI-PRISM will organize their observations and summarize the common themes, insights, and ideas emerging from each of the interviews into a report that will be submitted to RTI’s instrumentation partners and to NCES.

Assurance of Confidentiality

Cognitive interview participants will be informed that their participation is voluntary and their responses are protected by federal statute (Education Sciences Reform Act of 2002, Public Law 107-279, Section 183) and may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Attachment III). Students will be assigned a unique student identifier (ID), which will be created solely for file management and used to keep all student materials together. The student ID will not be linked to the student name in any way or form. The signed student consent forms will be kept separately from student interview files and notes, in a locked cabinet in a secure room for the duration of the study, and will be destroyed after the final report is released.

Schedule for NPSAS:12 OMB requests and related activities

Clearance for Focus Groups	Start	End
Develop focus group protocols	10/29/2009	12/14/2009
Package for Focus Groups to OMB (1850-0803 v.20)	12/18/2009	Approved 1/4/2010
Prepare for focus groups	1/4/2010	1/30/2010
Conduct Focus Groups	2/1/2010	4/30/2010
TRP meeting 1 (discuss results of Focus groups and plans for cognitive testing for item development)	7/13/2010	7/14/2010
Clearance for Cognitive Testing
Package for Cog Testing Recruiting to OMB (1850-0803 v.32)	7/8/2010	Approved 7/15/2010
Begin Recruiting	8/1/2010	12/10/2010
Develop/refine items for cognitive testing	5/1/2010	8/30/2010
Prepare package for Cognitive testing	8/1/2010	9/3/2010
OMB review of Cognitive package	9/3/2010
Conduct cognitive testing	9/15/2010	12/15/2010

Note: Shaded items have been completed.

Estimate of Hour Burden

In total, 48 individual cognitive interviews are planned. Each interview session is expected to last approximately 2 hours, to allow cognitive interview participants to review the entire survey, while providing think aloud responses and responding to probes.

Estimated respondent burden

Institution Level	Number of Recruits	Hours per recruit	Hours for recruiting	Number selected	Hours per respondent	Hours for Cognitive Testing
4‑year	25	.066 hr (4 minutes)	1.67	16	2.0	32.0
2-Year	25	.066 hr (4 minutes)	1.67	16	2.0	32.0
Less that 2-year	25	.066 hr (4 minutes)	1.67	16	2.0	32.0
Totals	75		5	Up to 48*		96

* A maximum of 48 interview participants will be included.

Estimate of Costs for Recruiting and Paying Respondents

Given the importance of recruiting a representative range of respondents and the difficulty typically experienced recruiting portions of the target population, especially students who are both working and enrolled in an institution, we will offer prospective respondents $50 for completing the 120-minutes cognitive interview. This amount was approved on 7/15/2010 by OMB in the memorandum submitted to begin recruiting and screening.

Estimate of Cost Burden

There are no direct costs to participants.

Cost to Federal Government

The cost of conducting the cognitive interviews will be $62,862. The following table provides the overall project cost estimates:

Cognitive Interviews, 2010
1 Staff Costs	$28,587
2 Other Project Materials	$5,131
Newspaper Advertisements for Recruiting	$2,678
Incentives for Student Participation	$2,400
Photocopy/Shipping	$53
Indirect Costs	$24,966
Total Cost of Task Without Fee	$58,684
Fixed Fee	$1,950
Award Fee	$2,228
Total Cost of Task	$62,862

1 http://wwwn.cdc.gov/QBANK/Report/Miller_2005.pdf

2 The recruitment approval (1850-0803 v. 32) indicated NC. We since determined that it will be advantageous to recruit respondents both in NC and DC areas. The attached approved recruitment materials reflect the addition of the DC location.