NPSAS 2012 Cog Testing 2011 Memo REVISED

NPSAS 2012 Cog Testing 2011 Memo.docx

NCES Cognitive, Pilot, and Field Test Studies System

NPSAS 2012 Cog Testing 2011 Memo REVISED

OMB: 1850-0803

Document [docx]
Download: docx | pdf

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics



DATE: July 14, 2011

TO: Shelly Martinez, OMB

FROM: Tracy Hunt-White, NCES

THROUGH: Kashka Kubzdela, NCES

SUBJECT: NPSAS:12 Full Scale Cognitive Testing 2011 (OMB No. 1850-0803 v.53)



Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803) that provides for NCES to improve methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, focus groups, and cognitive interviews.


The request for approval described in this memorandum includes recruiting and screening activities and cognitive testing of questionnaire items for the 2011-12 full scale National Postsecondary Student Aid Study (NPSAS:12). This package provides information about plans and procedures for conducting the cognitive testing in summer 2011.

Background

Housed in NCES’s Postsecondary, Adult, and Career Education Division, NPSAS:12 is a comprehensive study of how students and their families pay for postsecondary education. In addition to providing cross-sectional information about college costs and financing, NPSAS:12 will serve as the base year data collection for the next Beginning Postsecondary Students Longitudinal Study (BPS:12/14, BPS:12/17), provided the BPS options will be exercised by NCES. As a longitudinal study, BPS is able to investigate persistence and enrollment of first-time beginning (FTB) students in less-than-2-year, 2-year, and 4-year institutions, transfer and graduation rates, employment, and student loan debt. BPS, which samples both traditionally- and nontraditionally-aged postsecondary students, is the only nationally-representative, longitudinal survey of first-time beginners in postsecondary education.

Field test instrument development was informed by the results of focus groups held in the spring of 2010 (OMB# 1850-0803 v.20) and by cognitive testing conducted in the fall of 2010 (OMB# 1850-0803 v.36). The cognitive testing described in this memorandum will be conducted to inform final, full scale instrument development. The cognitive testing process will enable the instrumentation team to:

  • examine the thought processes affecting the quality of answers provided to survey questions,

  • understand the extent to which terms in questions are comprehended,

  • evaluate the memory demands of the questions,

  • evaluate the ability of respondents to make calculations and judgments,

  • determine appropriate presentations of response categories,

  • assess the time it takes to complete the interview,

  • assess the navigational problems users face, and

  • identify sources of burden and respondent stress.

Once the cognitive testing is complete, RTI will submit the items and test results to the Q Bank1 -- a federal repository of questions that have been tested along with the results.

Design and Context

Survey Methodologists from RTI’s Program for Research in Survey Methodology (PRISM) will conduct cognitive interviews with recruited participants. RTI-PRISM staff have extensive experience in all types of cognitive interviewing methodologies, and NPSAS cognitive interview protocols will include both “think aloud” data capture, and scripted probing. A "think aloud" interview is one in which the respondent is instructed to tell the interviewer everything that he/she is thinking about in answering a survey question. Probes for selected items will be concurrent (asked at the same time the subject answers the questions) and retrospective (asked during a debriefing session). In addition, cognitive interviewers will employ spontaneous probes as needed. Examples of probes are provided in Attachment V.


RTI is drawing cognitive interview participants from the Research Triangle Park, NC area where RTI’s North Carolina-based Laboratory for Survey Methods and Measurement is located. Recruitment will also be conducted in the Washington, D.C. area where RTI also has offices. Both locations are home to a large number of postsecondary institutions, making them particularly well-suited for recruiting the target population. In the event that we are unable to recruit adequate numbers of students to come to the RTP or DC offices, we will make arrangements with institutions in DC and/or RTP to conduct the cognitive interviews on campus. We will ensure that the space is private so the interviews cannot be overheard (such as a study room in a library or classroom). Attachment I provides additional detail about recruitment procedures.

A total of 30 cognitive interviews are planned. Interviews will be conducted in person and, if necessary, by telephone. Students will be selected from all levels of postsecondary institutions (4-year, 2-year, and less than 2-year) and will be selected to provide broad representation of the FTB population based on age, sex, and other key characteristics. An abbreviated questionnaire will collect standard demographic information about each of the participants during the screening process. Attachment II presents the screening questions that will be used to determine eligibility for cognitive interview participation. Attachment III contains the Assurance of Confidentiality.

The cognitive interviews will be held in facilities that are centrally located, easily accessible by car and public transportation, and allow for audio recording. Interviews will be conducted at times convenient for student schedules. The cognitive interview process is expected to average 90 minutes for each participant but could take as long as 2 hours. Interviews will be led by an RTI-PRISM staff member with considerable expertise in cognitive interviewing and professional training in cognitive and social psychology and survey methodology. The recordings will be made available to NCES for review.


The NPSAS:12 field test was completed on June 30, 2011. Based upon review of field test data and monitored telephone interviews, Attachment IV identifies the interview items selected to be the focus of full scale cognitive interviews. Probing on these items and feedback from cognitive interviews will help to determine more efficient and effective ways to ask questions of respondents.  Cognitive interview participants will complete the entire questionnaire to provide respondents with overall context for the interview items of focus identified in Attachment IV. While cognitive interview techniques, including think-alound questions and probes, will be focused primarily on these items, feedback from respondents on all interview items will be accepted and considered for the improvement of the full scale instrument. Example probes are identified in Attachment V.


Immediately following the conclusion of each interview, methodologists will review the cognitive interview recordings and notes, highlighting potential themes that may have arisen. Following each interview, the digital audio recording will be archived for qualitative analysis. RTI-PRISM will organize their observations and summarize the common themes, insights, and ideas emerging from each of the interviews into a report that will be submitted to RTI’s instrumentation partners and to NCES.

Assurance of Confidentiality

Cognitive interview participants will be informed that their participation is voluntary and their responses are protected by federal statute (Education Sciences Reform Act of 2002, 20 U.S.C. § 9573) and may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Attachment III). Students will be assigned a unique student identifier (ID), which will be created solely for file management and used to keep all student materials together. The student ID will not be linked to the student name in any way or form. The signed student consent forms will be kept separately from student interview files and notes, in a locked cabinet in a secure room for the duration of the study, and will be destroyed after the final report is released.

Schedule


The NPSAS:12 technical review panel (TRP) will meet on August 16-17, 2011, and the cognitive interviews will take place August through September, 2011.


The performance of individual survey items in the NPSAS:12 field test is being analyzed and will be reported at the August TRP. Should the TRP meeting lead to changes to the proposed here content of the cognitive interviews, we will come back to OMB to seek approval of the revised items, hoping for a quick review of the revisions to allow the study to stay on schedule. We are submitting the OMB cognitive testing package now, so that we may start the recruiting and screening of cog lab participants as soon as possible.  We plan to conduct the actual cognitive interviews soon after the TRP, in late-August through early September so that we will be able to begin the clearance process on the full-scale student data collection package in October using the final instrument.



Estimate of Hour Burden

In total, 30 individual cognitive interviews are planned. Interview sessions are expected to average 90 minutes but could last up to 2 hours to allow cognitive interview participants to review the entire survey and items of focus.

Estimated respondent burden

Institution Level

Number of Respondents

Number of Responses

Hours per Respondent

Total Burden Hours

4‑year Postsecondary Institution

Student Recruitment & Screening

16

16

.066 hr

(4 minutes)

1

2-Year Postsecondary Institution

Student Recruitment & Screening

16

16

.066 hr

(4 minutes)

1

Less that 2-year Postsecondary Institution

Student Recruitment & Screening

16

16

.066 hr

(4 minutes)

1

4‑year Postsecondary Institution

Student Cognitive Interview

10

(5 Web/5 CATI)

10

2.0

20

2-Year Postsecondary Institution

Student Cognitive Interview

10

(5 Web/5 CATI)

10

2.0

20

Less that 2-year Postsecondary Institution

Student Cognitive Interview

10

(5 Web/5 CATI)

10

2.0

20

Totals

48*

78

-

63

* A maximum of 30 interview participants will be conducted.


Estimate of Costs for Recruiting and Paying Respondents

We will offer prospective respondents $40 to thank them for their time and effort in completing the 90-120-minutes cognitive interview.

Estimate of Cost Burden

There are no direct costs to participants.

Cost to Federal Government

The estimated cost to the federal government for conducting cognitive interviews for NPSAS:12 is $50,149. Included in the contract estimates are contractor staff time, incentives, and project materials such as advertisements for recruiting.

File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
File TitleMemorandum
Authormcominole
File Modified0000-00-00
File Created2021-02-01

© 2024 OMB.report | Privacy Policy