Justification Memo

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: July 8, 2010

TO: Shelly Martinez, OMB

FROM: Tracy Hunt-White, NCES

THROUGH: Kashka Kubzdela, NCES

SUBJECT: NPSAS:12 Cognitive Testing 2010 Recruitment (OMB No. 1850-0803 v.32)

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803) provides for NCES to improve methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, focus groups, and cognitive interviews.

The request for approval described in this memorandum is to begin recruiting and screening activities in August 2010, in preparation for September 2010 cognitive testing of the student questionnaire items to be used in the 2011-12 National Postsecondary Student Aid Study (NPSAS:12). This initial package provides information about plans and procedures for conducting the cognitive testing, and presents a flow chart of the data elements currently being developed, which will be the subject of the September 2010 cognitive interviews. On September 2^nd, 2010, the second package, containing the questionnaire items to be cognitively tested, will be submitted to OMB for approval. The September package will include final item wording and the probes to be administered. NCES will also submit another package in the fall of 2010 to request approval to conduct student interviews and collect student records from postsecondary institutions for the NPSAS:12 field test, scheduled to begin in March 2011.

Background

Housed in NCES’s Postsecondary, Adult, and Career Education Division, the NPSAS:12 is a comprehensive study of how students and their families pay for postsecondary education. In addition to providing cross-sectional information about college costs and financing, NPSAS:12 will serve as the base year data collection for the next Beginning Postsecondary Students Longitudinal Study (BPS:12/14, BPS:12/17), provided the BPS option will be exercised by NCES. As a longitudinal study, BPS is able to investigate persistence and enrollment in less-than-2-year, 2-year, and 4-year institutions, transfer and graduation rates, employment, and student loan debt. BPS, which samples both traditionally- and nontraditionally-aged postsecondary students, is the only nationally-representative, longitudinal survey of first-time beginners in postsecondary education.

Working together with a distinguished team of researchers and its contractor, RTI International, NCES is reconceptualizing the BPS study to better elaborate the decision-making processes of postsecondary students. The BPS student interview, including the NPSAS:12 base year interview, is also being redesigned to reflect this new conceptualization. Drawing upon human capital theory, the redesign team is developing a model of student decision-making in which choices are based upon probabilistic expectations of the rewards and costs of alternative choices, and grounded in imperfect – and evolving – information.

In January 2010, NCES received approval to conduct focus groups – the first stage in a multistage qualitative evaluation comprised of focus groups and cognitive interviews. This qualitative evaluation will inform refinement of items used in previous surveys as well as the development of items which will help to elaborate the postsecondary choices of the first-time beginning (FTB) population. Focus groups were conducted to help the redesign team move from conceptualization to instrument development. Additionally, the focus groups were used to improve a select set of existing questions in the NPSAS and BPS interviews, particularly items involving financial aid terminology that is possibly unfamiliar to students (e.g. private loans) and items used to determine eligibility for the BPS cohort. The first stage has been completed and the project team is preparing for the second stage – the cognitive testing (see the included schedule).

Instrument development is currently underway, informed by the results of the focus groups, and will be tested with postsecondary students in RTI’s cognitive lab. The cognitive testing process will enable the instrumentation team to:

• examine the thought processes affecting the quality of answers provided to survey questions,

• understand the extent to which terms in questions are comprehended,

• evaluate the memory demands of the questions,

• evaluate the ability of respondents to make calculations and judgments,

• determine appropriate presentations of response categories,

• assess the time it takes to complete the interview,

• assess the navigational problems users face, and

• identify sources of burden and respondent stress.

Once the cognitive testing is complete, RTI will submit the items and test results from both evaluations (focus groups and cognitive testing) to the Q Bank¹ -- a federal repository of questions that have been tested along with the results.

Design and Context

Cognitive Testing. Survey Methodologists from RTI’s Program for Research in Survey Methodology (PRISM) will conduct cognitive interviews with recruited participants. RTI-PRISM staff have extensive experience in all types of cognitive interviewing methodologies, and NPSAS cognitive interview protocols will include both “think aloud” data capture, and scripted probing. A "think aloud" interview is one in which the respondent is instructed to tell the interviewer everything that he/she is thinking about in answering a survey question. Probes will be concurrent (asked at the same time the subject answers the questions) and retrospective (asked during a debriefing session) and will be prepared ahead of time. In addition, methodologists will employ spontaneous probes as needed.

RTI will draw cognitive interview participants from the greater Research Triangle Park, NC area where RTI’s North Carolina-based Laboratory for Survey Methods and Measurement is located and where a large number of postsecondary institutions are located, making it particularly well-suited for recruiting the target population. A total of forty-eight cognitive interviews are planned. Interviews will be conducted in person and by telephone in order to evaluate both a self-administered web and interviewer-administered telephone version of the insturment. Students will be selected from key institution levels (4-year, 2-year, and less than 2-year) and will be selected to provide representation of the FTB population based on age, sex, and other key characteristics. An abbreviated questionnaire will collect standard demographic information about each of the participants during the screening process. Students will be compensated for their participation in the interview. Attachment I provides additional detail about recruitment procedures. Attachment II presents the screening questions that will be used to determine eligibility for cognitive interview participation. Attachment III contains the Assurance of Confidentiality. A flow chart of the data elements that is serving as the basis for the current instrument development work is presented in Attachment IV.

The cognitive interviews will be held in a facility that is centrally located, easily accessible by car and public transportation, and allows for professional audio recording. Interviews will be conducted at times convenient for student schedules. Each interview, of approximately 2 hours in duration, will be led by an RTI-PRISM staff member with considerable expertise in cognitive interviewing and professional training in cognitive and social psychology and survey methodology. The recordings will be made available to NCES for review.

Immediately following the conclusion of each interview, methodologists will review the cognitive interview recordings and notes, highlighting potential themes that may have arisen. Following each interview, the digital audio recording will be archived for qualitative analysis. RTI-PRISM will organize their observations and summarize the common themes, insights, and ideas emerging from each of the interviews into a report that will be submitted to RTI’s instrumentation partners and to NCES.

Assurance of Confidentiality

Cognitive interview participants will be informed that their participation is voluntary and their responses are protected by federal statute (P.L. 107-279, Title I, Part E, Sec. 183) and may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Attachment III). Students will be assigned a unique student identifier (ID), which will be created solely for file management and used to keep all student materials together. The student ID will not be linked to the student name in any way or form. The signed student consent forms will be kept separately from student interview files and notes, in a locked cabinet in a secure room for the duration of the study, and will be destroyed after the final report is released.

Schedule for NPSAS:12 OMB requests and related activities

Clearance for Focus Groups	Start	End
Develop focus group protocols	10/29/2009	12/14/2009
Package for Focus Groups to OMB	12/18/2009
Prepare for focus groups	1/4/2010	1/30/2010
Conduct Focus Groups	2/1/2010	4/30/2010
TRP meeting 1 (discuss results of Focus groups and plans for cognitive testing for item development)	7/13/2010	7/14/2010
Clearance for Cognitive Testing
Package for Cog Testing Recruiting to OMB	7/8/2010
Begin Recruiting	8/1/2010)	12/10/2010
Develop/refine items for cognitive testing	5/1/2010	8/30/2010
Prepare package for Cognitive testing	8/1/2010	9/1/2010
OMB review of Cognitive package	9/2/2010
Conduct cognitive testing	9/15/2010)	12/15/2010

Estimate of Hour Burden

Forty-eight individual cognitive interviews, are planned. Each interview session is expected to last approximately, but no longer than 2 hours, to allow cognitive interview participants to review the entire survey, while providing think aloud responses and responding to probes. This package, however, only seeks approval for the burden associated with recruitment, approximately 5 hours.

Estimated respondent burden

Institution Level	Number of Recruits	Hours per recruit	Hours for recruiting	Number selected	Hours per respondent	Hours for Cognitive Testing	Total Burden Hours
4‑year	25	.066 hr (4 minutes)	1.67	16 (8 Web/8 CATI)	2.0	32.0	33.67
2-Year	25	.066 hr (4 minutes)	1.67	16 (8 Web/8 CATI)	2.0	32.0	33.67
Less that 2-year	25	.066 hr (4 minutes)	1.67	16 (8 Web/8 CATI)	2.0	32.0	33.67
Totals	75		5	Up to 48* (24 Web/24 CATI)		94	102

* A maximum of 48 interview participants will be included.

Estimate of Costs for Recruiting and Paying Respondents

Given the importance of recruiting a representative range of respondents and the difficulty typically experienced recruiting portions of the target population, especially students who are both working and enrolled in an institution, we will offer prospective respondents $50 for completing the 120-minutes cognitive interview.

Estimate of Cost Burden

There are no direct costs to participants.

Cost to Federal Government

The cost of conducting the cognitive interviews will be $62,862. The following table provides the overall project cost estimates:

Cognitive Interviews, 2010
1 Staff Costs	$28,587
2 Other Project Materials	$5,131
Newspaper Advertisements for Recruiting	$2,678
Incentives for Student Participation	$2,400
Photocopy/Shipping	$53
Indirect Costs	$24,966
Total Cost of Task Without Fee	$58,684
Fixed Fee	$1,950
Award Fee	$2,228
Total Cost of Task	$62,862

1 http://wwwn.cdc.gov/QBANK/Report/Miller_2005.pdf

7