NPSAS 2012 Field Test Student Data Part B C D

2011-12 National Postsecondary Student Aid Study (NPSAS:12)

Student Interview and Student Records

Supporting Statement Part B, C, and D

Request for OMB Review

(OMB # 1850-0666 v.8)

Submitted by

National Center for Education Statistics

U.S. Department of Education

September 27, 2010

B. Collection of Information Employing Statistical Methods

The sampling design for the full-scale NPSAS:12 study is presented in Section B.2. The purpose of the NPSAS:12 field test is to fully test all procedures, methods, and systems of the study in a realistic operational environment prior to implementing them in the full-scale study. Specific plans for such field test activities are provided below.

1. Respondent Universe

The students eligible for inclusion in the sample are those who were enrolled in a NPSAS-eligible institution in any term or course of instruction at any time between July 1, 2010 and April 30, 2011 for the field test and between July 1, 2011 and April 30, 2012 for the full-scale study, and who were:

enrolled in either (a) an academic program; (b) at least one course for credit that could be applied toward fulfilling the requirements for an academic degree; (c) exclusively non-credit remedial coursework but who the institution has determined are eligible for Title IV aid; or (d) an occupational or vocational program that required at least 3 months or 300 clock hours of instruction to receive a degree, certificate, or other formal award; and

not currently enrolled in high school; and

not enrolled solely in a GED or other high school completion program.

Field test institutions with continuous enrollment will be asked to provide lists with students enrolled between July 1, 2010 and March 31, 2011, instead of April 30, 2011, so that all lists can be received by mid-April, which allows time for field test data collection to be completed on schedule by June 30, 2011. This will not be an issue for the full-scale study.

NPSAS-eligible institutions are required during the relevant academic year (2010-11 for the field test and 2011-12 for the full-scale study) to:

offer an educational program designed for persons who have completed secondary education; and

offer at least one academic, occupational, or vocational program of study lasting at least 3 months or 300 clock hours; and

offer courses that are open to more than the employees or members of the company or group (e.g., union) that administers the institution; and

have a signed Title IV participation agreement with the U.S. Department of Education;

be located in the 50 states or the District of Columbia; and

be other than a U.S. Service Academy.

Institutions providing only a vocational, recreational, or remedial courses or only in-house courses for their own employees are excluded. U.S. Service Academies are excluded because of their unique funding/tuition base. The eligibility requirements are consistent with those used in all previous NPSAS rounds, with three exceptions: (1) the last requirement was new for NPSAS:2000; (2) offering more than just correspondence courses was no longer a requirement beginning with NPSAS:04; and (3) Puerto Rico has been excluded from the sample.

2. Statistical Methodology

In this section, we describe the field test student sample design, including our plans for sampling students from enrollment lists.¹ Plans for conducting these activities in the full-scale study (which will be revised after the field test in consultation with NCES) will be described in a later clearance package.

Based on past experience, we expect to obtain, minimally, overall 95 percent student eligibility rates and overall 70 and 75 percent student interview response rates in the field test and full scale, respectively. The expected field test student sample sizes and sample yield are presented in table 7. As indicated in the table, the field test will be designed to sample about 4,500 students, and the full-scale will be designed to sample about 117,000 students.

We plan to oversample students enrolled in for-profit institutions. While only about 11 percent of the student population is enrolled in for-profit institutions, these students receive about 25 percent of disbursed federal aid. Over-sampling will ensure adequate representation of this analytically important sub-group.

As mentioned in Section A, we will identify potential first-time beginning students (FTBs), and the remaining undergraduate students will be classified as other undergraduates. The NPSAS sampling rates for students identified as potential FTBs and other undergraduate students will be adjusted to yield the appropriate sample sizes after accounting for the expected false positive and false negative rates by sector. Tables 7 and 8 do not include the adjusted sample sizes, but a large percentage of the sample will be comprised of potential FTBs in order to obtain a BPS field-test sample yield of at least 1,000.

Potential FTBs, other undergraduates, masters students, doctoral-research/scholarship/other students, doctoral-professional practice, and other graduate students will be sampled at different rates in order to control the sample allocation, as has been done since NPSAS:2000. Using different rates will allow us to obtain the target sample sizes and is a technique necessary in the full-scale study to meet analytic objectives for defined domain estimates.

Table 7. NPSAS:12 field test expected student sample sizes and yield

Institutional sector	Sample students				Eligible students				Interview Respondents				Responding students per responding institution
	Total	FTBs	Other undergraduate students	Graduate students	Total	FTBs	Other undergraduate students	Graduate students	Total	FTBs	Other undergraduate students	Graduate students
Total	4,530	2,529	1,801	200	4,304	2,397	1,716	191	3,000	1,627	1,239	134	20
Public
Less-than 2-year	58	41	16	0	42	30	12	0	34	24	10	0	13
2-year	1,627	855	772	0	1,587	834	753	0	1,092	566	526	0	31
4-year non-doctorate-granting	280	115	148	17	268	110	141	16	206	84	111	12	17
4-year doctorate-granting	630	230	336	63	583	213	312	58	466	169	254	43	22
Private not-for-profit
Less-than-4-year	75	54	21	0	73	53	20	0	54	39	15	0	15
4-year non-doctorate-granting	302	158	111	33	291	152	106	32	213	111	80	22	9
4-year doctorate-granting	218	105	48	65	213	103	47	63	160	78	37	45	8
Private for-profit
Less-than-2-year	183	135	48	0	119	88	32	0	74	54	20	0	18
2 year	294	217	78	0	287	211	76	0	179	130	48	0	19
4 year	863	618	222	22	842	603	217	22	522	372	138	13	26

NOTE: FTB = first time beginner.

Table 8. NPSAS:12 full-scale preliminary student sample sizes and yield

Institutional sector	Sample students				Eligible students				Interview Respondents				Responding students per responding institution
	Total	FTBs	Other undergraduate Students	Graduate students	Total	FTBs	Other undergraduate students	Graduate students	Total	FTBs	Other undergraduate students	Graduate students
Total	117,256	39,404	64,786	13,067	111,420	37,191	61,489	12,740	83,553	28,867	45,492	9,194	60
Public
Less-than 2-year	1,250	652	598	0	1,006	525	482	0	643	353	289	0	51
2-year	41,626	13,482	28,145	0	38,875	12,590	26,284	0	28,035	9,690	18,345	0	84
4-year non-doctorate-granting	8,190	1,777	5,274	1,140	7,994	1,734	5,147	1,112	6,458	1,503	4,307	648	55
4-year doctorate-granting	19,560	3,514	11,883	4,162	19,180	3,446	11,652	4,081	15,842	3,053	9,708	3,081	79
Private not-for-profit
Less-than-4-year	1,606	852	754	0	1,443	766	677	0	838	471	368	0	49
4-year non-doctorate-granting	8,460	2,410	3,888	2,162	8,238	2,347	3,786	2,105	6,847	2,080	3,269	1,499	31
4-year doctorate-granting	7,275	1,546	1,629	4,100	7,112	1,512	1,593	4,008	5,756	1,325	1,416	3,014	32
Private for-profit
Less-than-2-year	3,888	2,123	1,765	0	3,343	1,825	1,518	0	1,816	1,051	765	0	48
2 year	6,245	3,410	2,836	0	5,973	3,261	2,712	0	4,288	2,447	1,841	0	50
4 year	19,156	9,638	8,015	1,503	18,255	9,185	7,638	1,432	13,029	6,893	5,184	952	67

NOTE: FTB = first time beginner.

We will employ a variable-based (rather than source-based) definition of a study member, similar to that used in NPSAS:08 and NPSAS:04, with revisions deemed necessary by NCES. Specifically, a study member will be defined as any sample member who is determined to be eligible for the study and, minimally, has valid data from any source for the following:

student type (undergraduate or graduate);

date of birth or age;

gender; and

at least 8 of the following 15 variables:

• dependency status;

• marital status;

• any dependents;

• income;

• expected family contribution (EFC);

• degree program;

• class level;

• first-time beginner (FTB) status;

• months enrolled;

• tuition;

• received federal aid;

• received non-federal aid;

• student budget;

• race; and

• parent education.

We expect the rate of study membership to be about 90 percent.²

The six student sampling strata will be:

1. first-time beginning undergraduate students

2. other undergraduate students

3. masters students

4. doctoral-research/scholarship/other students

5. doctoral-professional practice students³

6. other graduate students.⁴

As was done in past rounds of NPSAS, certain student types (potential FTBs, other undergraduates, masters students, doctoral-research/scholarship/other students, doctoral-professional practice students, and other graduate students) will be sampled at different rates to control the sample allocation. Differential sampling rates facilitate obtaining the target sample sizes necessary to meet analytic objectives for defined domain estimates in the full-scale study.

Creating student sampling frames. Sample institutions will be asked to provide an electronic student enrollment list. The following data items will be requested for NPSAS-eligible students enrolled at each sample institution. Most of these items are similar to what was collected in past NPSAS studies.

Name

Social Security number (SSN)

Student ID number (if different from SSN)

Student level (undergraduate, masters, doctoral-research/scholarship/other, doctoral-professional practice, other graduate)

First-time Beginner (FTB) indicator

Class level of undergraduates (first year, second year, etc.)

Date of birth (DOB)

High school graduation date (month and year)

CIP code or major

Indicator of whether the institution received an ISIR (electronic record summarizing the result of the student’s FAFSA processing) from CPS

Contact information (local and permanent street address and phone number and school and home e-mail address)

These data items are described in greater detail in section C.

As with NPSAS:04 and NPSAS:08, we will request locating data from institutions concurrent with the collection of student lists used for sample selection. This will allow web-based student record collection and interviewing to begin almost immediately after sample selection and thus help us meet the very tight schedule for data collection, data processing, and file development, especially in the full-scale study. For institutions unwilling to provide location data for all students on enrollment lists, we will request locating data only for sample students immediately after the sample is selected.

The FTB indicator, class level, date of birth, and high school graduation year will be used to identify and oversample potential FTBs, as described below. High school graduation year has not been requested on lists in the past for NPSAS, so we will test the feasibility of this request in the field test.

CIP code and major have been collected in the past for NPSAS in order to help identify baccalaureate recipients who are business majors, so that they could be undersampled in NPSAS years that spin off the B&B Longitudinal Study. In NPSAS:08, CIP code and major were also used to oversample STEM majors who were not SMART grant recipients. Currently, there is no need to oversample or undersample any students based on CIP and major, but we plan to ask for CIP and major on the list in the field test in case a need arises. Our experience shows that institutions can easily provide this information on the list without added burden. For the full-scale study, we will re-visit this issue to see if it is necessary to ask for these items on the list.

The following section describes our planned procedures to securely obtain, store, and discard sensitive information collected for sampling purposes.

Obtaining student enrollment lists. To ensure the secure transmission of sensitive information on the enrollment lists, we will provide the following options to institutions: (1) upload encrypted student enrollment list files to the project’s secure website using a login ID and “strong” password provided by RTI, (2) provide an appropriately encrypted list file via e-mail (RTI will provide guidelines on encryption and creating “strong” passwords). 

We expect that few institutions will ask to provide a paper list. However, in the event that an institution is unable to transmit data via the secure electronic methods of transmission outlined above, we will accept faxes sent to a secure electronic fax machine. To ensure the fax transmission is sent to the appropriate destination, we will require a test run with nonsensitive data prior to submitting the transcripts to eliminate errors in transmission from misdialing. RTI will provide schools with a fax cover page that includes a confidentiality statement to use when transmitting individually identifiable information.

List files received via e-fax are stored as electronic files on the e-fax server, which is housed in a secured data center at RTI. These files will be copied to a project folder that is only accessible to project staff members. Access to this project folder will be set so that only those who have authorized access will be able to see the included files. The folder will not even be visible to those without access. After being copied, the files will be deleted from the e-fax server. The files will be stored on the network that is backed up regularly to avoid the need to recontact the institution to provide the data again should a loss occur. RTI’s information technology service (ITS) will use standard procedures for backing up data, so the backup files will exist for 3 months.

Identifying FTBs during the base year. Accurately qualifying sample members as FTBs is a continuing challenge. This is important because unacceptably high rates of misclassification (e.g., false positives) can and have resulted in (1) excessive cohort loss with too few eligible sample members to sustain the longitudinal study, (2) excessive cost to “replenish” the sample with little value added, and (3) inefficient sample design (excessive oversampling of “potential” FTBs) to compensate for anticipated misclassification error.

We will take steps early in the NPSAS:12 listing and sampling processes to improve the rate at which FTBs are correctly classified for sampling. First, in addition to an FTB indicator, we will request that enrollment lists provided by institutions (or institution systems) include class level, student level, date of birth, and high school graduation date. Students identified by the school as FTBs but also identified as in their third year or higher and/or not an undergraduate student will not be classified as FTBs for sampling. Additionally, students appearing to be dually enrolled at the postsecondary school and in high school based on the high school graduation date will also not be eligible for sampling. If the FTB indicator is not provided for a student on the list but the student is 18 years old or younger and does not appear to be dually enrolled, the student will be classified as an FTB for sampling. Otherwise, if the FTB indicator is not provided for a student on the list and the student is over the age of 18, then the student will be sampled as an “other undergraduate” but will be part of the BPS cohort if identified during the interview as an FTB.

Second, prior to sampling we will match students over the age of 18 listed as potential FTBs to National Student Loan Data System (NSLDS) records to determine if any have a federal financial aid history pre-dating the NPSAS year (earlier than July 1, 2010 for the field test and July 1, 2011 for the full-scale). Since NSLDS maintains current records of all Title IV grant and loan funding, any students with data showing disbursements from the prior year can be reliably excluded from the sampling frame of FTBs. Given that about 60 percent of FTBs receive some form of Title IV aid in their first year, this matching process will not be able to exclude all listed FTBs with prior enrollment, but will significantly improve the accuracy of the listing prior to sampling, yielding fewer false positives. Only students over 18 years of age will be sent to NSLDS because most students 18 and younger are FTBs. Matching to NSLDS would be expected to identify about 22 percent of the cases matched as false positives (based on NPSAS:04 data). If it is not feasible to send all potential FTBs to NSLDS for matching, we will consider sending only potential FTBs in sectors with historically high false positive rates.

An interim step under consideration is matching the remaining sample (or a subset thereof,) to the National Student Clearinghouse (NSC) for further narrowing of potential FTBs based on the presence of evidence of earlier enrollment. Due to the high cost of the NSC service, we are weighing the costs and benefits of sending all potential FTBs resulting from NSLDS matching who are 19 or older versus sending a subsample of these potential FTBs to NSC for matching. Matching to both NSC and NSLDS would be expected to identify about 35 percent of the cases matched as false positives (based on NPSAS:04 data).

Third, we will set our FTB selection rates taking into account the error rates observed in NPSAS:04 and BPS:04/06 within each sector. As shown in table 9, some institution sectors were better able to accurately identify their students as FTBs. While the sample selection rates will take into account these false positive error rates, we do anticipate achieving an improvement in accuracy from the NSLDS and NSC record matches and will adjust the selection rates accordingly. To the extent possible, we will examine the sector-level FTB error rates in the field test to determine the rates necessary for full-scale student sampling.

False starters are students who enrolled in a course at the postsecondary level but did not earn credit, and these students have been included as FTBs in the past. Because they are difficult to accurately identify, our experience has shown that many false positives are false starters. We have agreed with NCES to limit the FTB sample to pure FTBs.

Finally, we will revise the screening questions used to identify FTBs in institutional student records and the student interview. Question wording to determine FTB eligibility will be pre-tested in focus groups and in RTI’s cognitive laboratory; both evaluations will be conducted in advance of the field test.

Table 9. Weighted false positive rate observed in FTB identification, by sector: NPSAS:04

Sector in NPSAS:04	False positive rate (weighted)
Public
Less-than 2-year	64.4
2-year	72.5
4-year non-doctorate-granting	26.8
4-year doctorate-granting	27.0
Private not-for-profit
Less-than-4-year	63.1
4-year non-doctorate-granting	43.4
4-year doctorate-granting	15.2
Private for-profit
Less-than-2-year	63.1
2 years or more	70.0

FTB = first time beginner.

Quality control checks for lists. Several checks on quality and completeness of student lists will be implemented before the sample students are selected. The lists will fail quality control checks if student level and/or the FTB indicator are not included on the list. Additionally, the unduplicated total of students at the undergraduate and graduate levels on each institution’s student list will be checked against the latest IPEDS unduplicated enrollment data from the 12-Month Enrollment Survey. Lists will be unduplicated by student ID number. Based on past experience, only counts within 50 percent of non-imputed IPEDS counts will pass edit checks.

Because IPEDS does not have unduplicated annual FTB counts, the unduplicated FTB counts on each institution’s student list will be checked separately against the fall enrollment counts from the IPEDS Fall Enrollment Survey. The check of FTBs will fail if the count for any unduplicated list is at least 50 percent less than the IPEDS FTB count. The list FTB counts are expected to almost always be more than the IPEDS counts because the IPEDS counts are not annual counts. This check will identify institutional enrollment lists that under-report FTBs. We will re-evaluate these checks after the field test for use in the full-scale study.

Additionally, contact information will be checked carefully for each enrollment list as well as for each student sample. Past experience shows that some institutions may provide enrollment lists with contact information matched improperly with student name. Comparing e-mail addresses with student names will help ensure that each student’s contact information matches.

Institutions failing the edit checks will be re-contacted to resolve the discrepancy and verify that the institution coordinator who prepared the student list(s) clearly understood our request and provided a list of the appropriate students. When we determine that the initial list provided by the institution is not satisfactory, we will request a replacement list. We will proceed with selecting sample students when we have either confirmed that the list received is correct or have received a corrected list.

Selection of sample students. Students will be sampled on a flow basis as student lists are received using a stratified systematic sampling procedure. The student sampling strata will be:

FTB undergraduate,

other undergraduate

masters,

doctoral-research/scholarship/other,

doctoral-professional practice,

other graduate.

Lists will be unduplicated by student ID number prior to sample selection. In addition, all samples will be unduplicated by Social Security number (SSN) between institutions. In prior NPSAS studies, we found several instances in which this step avoided multiple selections of the same student. However, we also learned that the ID numbers assigned to non-citizens may not be unique across institutions; thus when duplicate IDs are detected, but the IDs are not standard SSNs (do not satisfy the appropriate range check), we will check the student names to verify that they are indeed duplicates before deleting the students.

Although no paper lists are expected to be received because there were none in NPSAS:08, we will still be prepared to sample students from paper lists. Any paper lists received will be keyed into an electronic format and then sampled in the same manner as electronic lists. QC will be done for both the keying and sampling of the students.

Sample yield for the field test and full-scale NPSAS data collections will be monitored by institutional and student sampling strata, and the sampling rates will be adjusted early, if necessary, to achieve the desired sample yield.

Although simpler procedures would suffice for the NPSAS:12 field test, the student sampling procedures planned for the field test will be comparable to those planned for the full-scale study to provide a thorough evaluation before being implemented for the full-scale sampling.

Re-interview. A subsample of about 300 interview respondents will be randomly selected to be re-interviewed⁵ to enable analysis of the reliability of selected items in the field test instrument. Because most of the new field test interview items are for FTBs only, the selection of these new items comprise the majority of the re-interview. Including only FTBs in the sample for the re-interview maximizes the number of respondents administered the re-interview items. The Case Management System (CMS) will be programmed to randomly select this subsample.

Quality control checks for sampling. Quality control (QC) is very important for sampling and all statistical activities. Statistical procedures used in NPSAS:12 will undergo thorough quality control checks. RTI has developed technical operating procedures (TOPs) that describe how to properly implement statistical procedures and QC checks. We will employ a checklist for all statisticians to use to ensure that appropriate QC checks are performed.

Some specific sampling QC checks will include, but will not be limited to, checking that the:

institutions and students on the sampling frames all have a known, non-zero probability of selection;

distribution of implicit stratification for institutions is reasonable;

weighted institution size measures sum to the frame size;

number of institutions and students selected match the target sample sizes; and

sample weight for each institution and student is the inverse of the probability of selection.

Tracing prior to the start of data collection. Once the sample is selected, RTI will conduct several batch database searches to prepare the sample for the start of student interviews. The first steps in the batch tracing process will be to match to the U.S. Department of Education's Central Processing System (CPS) and the National Change of Address (NCOA) database to obtain updated contact information. Any new information collected from CPS or NCOA matches will be added to the NPSAS locator database and will be used to attempt to match to Telematch to capture any updated telephone numbers needed for the start of CATI data collection. Batch locating is the final step before the start of data collection.

3. Methods for Maximizing Response Rates

Response rates in the NPSAS:12 field test and full-scale study are a function of success in two basic activities: identifying and locating the sample members involved, then contacting them and gaining their cooperation. Two classes of respondents are involved: institutions, and students who were enrolled in those institutions. Institutions will be asked to provide data from institutional records for sampled students. In this section, we describe our plans for maximizing response to the request for data from institutional records. We also present our plans for maximizing response to the student survey.

1. Collection of Data from Institutional Records

Our plans for contacting and communicating with institutions, beginning with the process of list acquisition, are designed to ensure the cooperation of as many institutions as possible and to establish rapport with institutional staff. This process will include sending the chief administrator of each institution a package of descriptive materials about the study, follow-up telephone calls to obtain the chief administrator’s consent and cooperation, and asking the chief administrator to designate an Institutional Coordinator (IC) who will serve as our primary point of contact.

All institution coordinators receive information that informs them about the purposes of NPSAS, describes their tasks, and assures them of our commitment to maintaining the confidentiality of data. Written materials will be provided to coordinators explaining each phase of the study, as well as their role in each. Training of institution coordinators is geared toward the method of data collection selected by the institution (see below). The system used for collecting institutional record data is a World Wide Web application; and the website, accessible only with an ID and password, provides institution coordinators with instructions for all phases of study participation. Copies of all written materials, as well as answers to frequently asked questions, are available on the website.

Experienced RTI staff from RTI’s Call Center Services (CCS) carry out these contacts and are assigned to specific institutions, which remain their responsibility throughout the data collection process. This allows RTI staff members to establish rapport with the institution’s staff and provides those individuals with a consistent point of contact at RTI. Staff members are thoroughly trained in basic financial aid concepts and in the purposes and requirements of the study, which helps them establish credibility with the institution staff.

As an additional means of maximizing institutional participation, we have secured endorsements from 26 professional associations for NPSAS:12 (see Appendix G). RTI also sponsors an exhibit booth at NASFAA’s national conference, which allows project management to meet staff from participating institutions and to answer any questions regarding the study, collection of institutional record data or institution burden.

RTI will offer several options for providing the Student Records for sampled students (as in prior NPSAS studies), and invite the coordinator to select the methodology that is least burdensome and most convenient for the institution. The optional methods for providing student record data are:

Student Records obtained via a web-based data entry interface. The web-based data entry interface is flexible and allows the coordinator to enter data in one of two data entry modes. One data entry mode resembles a spreadsheet (referred to as “grid mode”) and as such, the coordinator can view and edit multiple student records at a time. The other data entry mode displays one student at a time, and the coordinator may enter data in a top to bottom fashion before moving onto the next student. 

Student Records obtained by completing an Excel workbook. An Excel workbook will be created for each institution and will be preloaded with the sampled students’ ID, name, and SSN (if available). To facilitate simultaneous data entry by different offices within the institution, the workbook contains a separate worksheet for each of the following topic areas: Financial Aid, Enrollment, Locating and Contact info, Demographics. The user will download the Excel worksheet from the secure NPSAS institution website, enter the data, and then upload the data to the website. Validation checks occur both within Excel as data are entered and when the data are uploaded via the website.

Student Records obtained by uploading CSV (comma separated values) files. Institutions with the means to export data from their internal database systems to a flat file may opt for this method of supplying Student Records. Over the past two NPSAS studies, the number of institutions providing data files has increased. Institutions that select this method will be provided with detailed import specifications, and all data uploading will occur through the project’s secure website.

Institution coordinators who elect to use the web-based data entry interface will receive a manual that provides detailed instructions for accessing and using the site. This manual, which is currently under revision to reflect recent improvements in the system, will be available early in 2011. In the interim, the manual used in NPSAS:08 is provided as Appendix I, for informational purposes.

Prior to data collection, student records are matched to the U.S. Department of Education Central Processing System (CPS)—which contains data on federal financial aid applications—for locating purposes and to reduce the burden on the institutions for the student record abstractions. The vast majority of the federal aid applicants (about 95 percent) will match successfully to the CPS prior to Student Records data collection. During data collection, institutions will be asked to provide the student’s last name and Social Security number for the small number of federal aid applicants who did not match to the CPS on the first attempt. After Student Records data collection ends, we will submit the new names and Social Security numbers to CPS for file matching. Any new data obtained for the additional students will be delivered on the Electronic Code Book (ECB) with the data obtained prior to Student Records data collection.

2. Student Survey: Self-Administered Web and CATI

Methods for maximizing response to the study survey include: (1) tracing of sample members; (2) thorough training for all staff involved in data collection; (3) use of a sophisticated case management system; (4) a carefully designed survey instrument; and (5) detailed plans for averting and converting refusals.

1. Tracing of Sample Members

To achieve the desired response rate, we propose an integrated tracing approach that consists of up to 12 steps designed to yield the maximum number of locates with the least expense. During the field test, we will evaluate the effectiveness of these procedures for the full-scale study effort. The steps of our tracing plan include the following elements.

• Matching student list information with NCOA, Telematch, CPS, and other databases, which will yield locating information for the students sampled for NPSAS:12.

• Providing a system for moving locator information obtained during collection of student record data quickly into CATI so that this new information can be put to immediate use

• Lead letter and other mailings as necessary to sample members. A personalized letter (signed by an NCES official) and study brochure will be mailed to all sample members to initiate data collection. This letter will include a toll-free 800 number, study website address, and study ID and password, and will request that sample members complete the self-administered interview over the Internet. A few days after the lead letter mailing, an email message mirroring the letter will also be sent to sample members.

• Conducting batch tracing before data collection and before and after the start of CATI as needed

• Advance tracing prior to the start of CATI efforts. Not all schools will be able to give complete or up-to-date locating information on each student, and some cases will require more advanced tracing, before mailings can be sent or the cases can be worked in CATI. RTI plans to conduct batch tracing on all cases to obtain updated address information prior to mailing the lead letters. This step will minimize the number of returned letters and maximize the number of early completes. To handle cases for which mailing address, phone number, or other contact information is invalid or unavailable, RTI plans to conduct advance tracing of the cases prior to lead letter mailout and data collection. This advance tracing will involve searching for address and telephone information. As lead information is found, additional searches will be conducted through interactive databases to expand on leads found. This will be an important step in the tracing components because of the nature of this sample. After locating information is found, more advanced database searches, such as Experian, will be used, to provide more comprehensive information for the individual.

• CATI tracing

• Pre-intensive tracing including FastData and Accurint. We plan to send cases to both FastData and Accurint to identify a new phone number, to minimize the number of cases requiring more expensive intensive interactive tracing. Through FastData we can tap into 260 million consumer records and over 33 million public records. We are also able to access a national directory assistance database—updated daily—of over 156 million phone numbers. FastData has also recently added a more comprehensive cell phone search (SuperPhones & Phone+Premium) built into existing searches; obtaining reliable cell phone numbers is becoming an increasingly critical component of locating and interviewing this population. Accurint is a flexible search vendor capable of providing a variety of contact information for a very low cost per case. This vendor provides an indicator that the phone number returned has been verified as accurate and belonging to a subject in the past 24 hours. Accurint uses SSN to search, making it a viable tool for NPSAS:12 and the follow-up studies due to the high percentage of SSNs we expect to obtain on the student enrollment lists and through tracing sources and student interviews.

Conducting intensive in-house tracing, including proprietary database searches. RTI’s tracing specialists conduct intensive interactive searches to locate contact information for sample members. In NPSAS:08, about 60 percent of sample members requiring intensive tracing were located, and about 59 percent of those located responded to the interview. Intensive interactive tracing differs from batch tracing in that a tracer can assess each case on an individual basis to determine which resources are most appropriate and the order in which they should be used. Intensive interactive tracing is also much more detailed due to the personal review of information. During interactive tracing, tracers utilize all previously obtained contact information to make tracing decisions about each case. These intensive interactive searches are completed using a special program that works with RTI’s CMS to provide organization and efficiency in the intensive tracing process. Sources that may be used, as appropriate, include credit database and insurance database searches through Experian, ChoicePoint, various public websites, and other integrated database services. (Although LexisNexis and TransUnion have been used in the past on RTI studies, they are no longer used for batch or interactive searches because the costs have increased while the results have become substandard. FastData has replaced TransUnion as a source for obtaining SSNs because FastData does this more cost-effectively.

• Conducting NPSAS List Completer (NLC) searches. NLC is an RTI software application that compiles all information available for the school and sample members to Tracing Services for additional address, phone, and e-mail searches to be made. The application will send Tracing Services the school name, school web address, and total number of students to be worked. If student name, address, and phone number are available, this information will also be sent to the NLC. Tracing Services will then use the school web page directly to conduct searches and update records in the NLC for any new school or student information found. This application was used on the most recent round of IES’s National Study of Postsecondary Faculty and more than 4,000 new email addresses were located. Many major universities have student directory information available, and RTI feels this application could allow for additional success on NPSAS:12.

• Conducting searches for hard-to-locate sample members on social networking sites such as MySpace and Facebook. We propose to set up a project Facebook page that can be used to send a message to sample members located on Facebook. Facebook is currently the fastest-growing social networking site with about 125 million users at the time of this writing.

• University, college, or personal web pages.

• An online video, using stop-motion filmmaking and Lego figures, to communicate information about NPSAS:12 to sample members. RTI will seek to collaborate with the Partnership for Public Service to ensure that the content and presentation of the video resonate with the NPSAS sample. An experiment will be conducted to determine whether use of the video increases participation rates. Additionally, voluntary debriefing questions will be included to help better understand reactions to the video.

2. Training for Data Collection Staff

Telephone data collection will be conducted by staff in RTI’s Call Center Services unit, including Quality Control Supervisors (QCSs), Help Desk Agents (HDAs), Telephone Interviewers (TIs), and Refusal Conversion Specialists. Training programs for these staff members are critical to maximizing response rates and collecting accurate and reliable data.

Quality control supervisors, who are responsible for all supervisory tasks, will attend project-specific training for QCSs, in addition to the content of the HDS and TI training. They will receive an overview of the study, background and objectives, and the data collection instrument through a question-by-question review. Supervisors will also receive training in the following areas: providing direct supervision during data collection; handling refusals; monitoring interviews and maintaining records of monitoring results; problem resolution; case review; specific project procedures and protocols; reviewing CATI reports; and monitoring data collection progress.

Training for HDAs, who assist sample members who call the project-specific toll-free line, and Telephone Interviewers is designed to help staff become familiar with and practice using the Help Desk application and survey instrument, as well as to learn project procedures and requirements. Particular attention will be paid to quality control initiatives, including refusal avoidance and methods to ensure that quality data are collected. Both HDAs and TIs will receive project-specific training on telephone interviewing, and HDAs will receive additional training specifically geared toward solving technical problems and answering questions from web participants regarding the study or related to specific items within the interview. They will also be able to unlock cases, reissue passwords, and respond to sample member e-mail messages, using prepared text approved by the NCES Contracting Officer’s Representative. At the conclusion of training, all HDAs and TIs must meet certification requirements by successfully completing a certification interview. This evaluation consists of a full-length interview with project staff observing and evaluating interviewers, as well as an oral evaluation of interviewers’ knowledge of the study’s Frequently Asked Questions.

3. Case Management System

Student interviews will be conducted using a single web-based survey instrument for both self-administered and CATI data collection. The data collection activities will be accomplished through the Case Management System (CMS), which is equipped with the following capabilities:

• on-line access to locating information and histories of locating efforts for each case;

• state-of-the-art questionnaire administration module with full “front-end cleaning” capabilities (i.e., editing as information is obtained from respondents);

• sample management module for tracking case progress and status; and

• automated scheduling module which delivers cases to interviewers and incorporates the following features:

• Automatic delivery of appointment and call-back cases at specified times. This reduces the need for tracking appointments and helps ensure the interviewer is punctual. The scheduler automatically calculates the delivery time of the case in reference to the appropriate time zone.

• Sorting of non-appointment cases according to parameters and priorities set by project staff. For instance, priorities may be set to give first preference to cases within certain sub-samples or geographic areas; cases may be sorted to establish priorities between cases of differing status. Furthermore, the historic pattern of calling outcomes may be used to set priorities (e.g., cases with more than a certain number of unsuccessful attempts during a given time of day may be passed over until the next time period). These parameters ensure that cases are delivered to interviewers in a consistent manner according to specified project priorities.

• Restriction on allowable interviewers. Groups of cases (or individual cases) may be designated for delivery to specific interviewers or groups of interviewers. This feature is most commonly used in filtering refusal cases, locating problems, or foreign language cases to specific interviewers with specialized skills.

• Complete records of calls and tracking of all previous outcomes. The scheduler tracks all outcomes for each case, labeling each with type, date, and time. These are easily accessed by the interviewer upon entering the individual case, along with interviewer notes, thereby eliminating the need for a paper record of calls of any kind.

• Flagging of problem cases for supervisor action or supervisor review. For example, refusal cases may be routed to supervisors for decisions about whether and when a refusal letter should be mailed, or whether another interviewer should be assigned.

• Complete reporting capabilities. These include default reports on the aggregate status of cases and custom report generation capabilities.

The integration of these capabilities reduces the number of discrete stages required in data collection and data preparation activities and increases capabilities for immediate error reconciliation, which results in better data quality and reduced cost. Overall, the scheduler provides a highly efficient case assignment and delivery function by reducing supervisory and clerical time, improving execution on the part of interviewers and supervisors by automatically monitoring appointments and call-backs, and reducing variation in implementing survey priorities and objectives.

4. Survey Instrument Design

In January 2010, NCES received approval to conduct focus groups – the first stage in a multistage qualitative evaluation comprising focus groups and cognitive interviews. This qualitative evaluation will inform refinement of items used in previous surveys as well as the development of items which will help to elaborate the postsecondary choices of the first-time beginning (FTB) population. Focus groups were conducted to help the redesign team move from conceptualization to instrument development. Additionally, the focus groups were used to improve a select set of existing questions in the NPSAS and BPS interviews, particularly items involving financial aid terminology that is possibly unfamiliar to students (e.g. private loans) and items used to determine eligibility for the BPS cohort. The first stage has been completed and the project team is preparing for the second stage –cognitive testing, which is scheduled to take place in September 2010. Results of the focus groups and cognitive testing will guide the development of final survey wording⁶

To specify and program the survey, NPSAS:12 will employ a cutting-edge web-based instrument and deployment system, created by RTI, known as Hatteras. NPSAS:08 was the first NCES study to use Hatteras, which has proven to be a flexible, collaborative system allowing for instrument designers, programmers, and NCES management to work together on instrument development. The instrument specifications stored in database tables are used to produce web pages dynamically. Hatteras provides multimode functionality, whereby the survey instrument is created one time and can be used for self-administration, CATI, CAPI, or data entry. Hatteras provides multilanguage support and is compatible with RTI’s Instrument Development and Documentation System (IDADS) to provide NCES-specific data documentation directly from the instrument specifications.

Below are some of the basic questionnaire administration features of the web-based instrument:

• Based on responses to previous questions, the respondent or interviewer is automatically routed to the next appropriate question, according to predesignated skip patterns.

• The web-based interview automatically inserts “text substitutions” or “text fills” where alternate wording is appropriate depending on the characteristics of the respondent or his/her responses to previous questions.

• The web-based interview can incorporate or preload data about the individual respondent from outside sources (e.g., previous interviews, sample frame files, etc.). Such data are often used to drive skip patterns or define text substitutions. In some cases, the information is presented to the respondent for verification or to reconcile inconsistencies.

• With the web/CATI instrument, numerous question-specific probes may be incorporated to explore unusual responses for reconciliation with the respondent, to probe “don’t know” responses as a way of reducing item non-response, or to clarify inconsistencies across questions.

• Coding of multi-level variables. An innovative improvement to previous NPSAS data collections, the web-based instrument uses an assisted coding mechanism to code text strings provided by respondents. Drawing from a database of potential codes, the assisted coder derives a list of options from which the interviewer or respondent can choose an appropriate code (or codes if it is a multi-level variable with general, specific, and/or detail components) corresponding to the text string.

• Iterations. When identical sets of questions will be repeated for an unidentified number of entities, such as children, jobs, schools, and so on, the system allows respondents to cycle through these questions as often as is needed.

In addition to the functional capabilities of the CMS and web instrument described above, our efforts to achieve the desired response rate will include using established procedures proven effective in other large-scale studies we have completed. These include:

• Providing multiple response modes, including self-administered and interviewer-administered options.

• Offering incentives to encourage response (see incentive structure described below).

• Prompting calls initiated prior to the start of data collection to remind sample members about the study and the importance of their participation.

• Assigning experienced CATI data collectors who have proven their ability to contact and obtain cooperation from a high proportion of sample members.

• Training the interviewers thoroughly on study objectives, study population characteristics, and approaches that will help gain cooperation from sample members.

• Providing the interviewing staff with a comprehensive set of questions and answers that will provide encouraging responses to questions that sample members may ask.

• Maintaining a high level of monitoring and direct supervision so that interviewers who are experiencing low cooperation rates are identified quickly and corrective action is taken.

• Making every reasonable effort to obtain an interview at the initial contact, but allowing respondent flexibility in scheduling appointments to be interviewed.

• Providing hesitant respondents with a toll-free number to use to telephone RTI and discuss the study with the project director or other senior project staff.

• Thoroughly reviewing all refusal cases and making special conversion efforts whenever feasible (see next section).

5. Refusal Aversion and Conversion

Recognizing and avoiding refusals is important to maximize the response rate. We will emphasize this and other topics related to obtaining cooperation during data collector training. Supervisors will monitor interviewers intensely during the early days of data collection and provide retraining as necessary. In addition, the supervisors will review daily interviewer production reports produced by the CATI system to identify and retrain any data collectors who are producing unacceptable numbers of refusals or other problems.

After encountering a refusal, the data collector enters comments into the CMS record. These comments include all pertinent data regarding the refusal situation, including any unusual circumstances and any reasons given by the sample member for refusing. Supervisors will review these comments to determine what action to take with each refusal. No refusal or partial interview will be coded as final without supervisory review and approval. In completing the review, the supervisor will consider all available information about the case and will initiate appropriate action.

If a follow-up is clearly inappropriate (e.g., there are extenuating circumstances, such as illness or the sample member firmly requested that no further contact be made), the case will be coded as final and will not be recontacted. If the case appears to be a “soft” refusal, follow-up will be assigned to an interviewer other than the one who received the initial refusal. The case will be assigned to a member of a special refusal conversion team made up of interviewers who have proven especially adept at converting refusals.

Refusal conversion efforts will be delayed for at least one week to give the respondent some time after the initial refusal. Attempts at refusal conversion will not be made with individuals who become verbally aggressive or who threaten to take legal or other action. Refusal conversion efforts will not be conducted to a degree that would constitute harassment. We will respect a sample member’s right to decide not to participate and will not impinge this right by carrying conversion efforts beyond the bounds of propriety.

Incentives. As described in the justification section (section A), we have proposed to offer incentive payments differentially to members of the sample population depending on their modeled likelihood of participation. Our approach to maximizing the response of all sample members—and thereby limiting potential bias—involves an incentive payment to reimburse the respondent for time and expenses. The NPSAS:12 field test will be used to conduct an experiment to examine the effectiveness of this methodology, which is hypothesized to increase weighted response rates and reduce nonresponse bias. Additional detail about the planned field test experiment is provided in section B.4.

4. Tests of Procedures and Methods

The NPSAS:12 field test will include two planned data collection experiments: the first involves the use of an informational video during the initial contact with sample members to increase participation by sample members and to establish “branding” for the new longitudinal BPS cohort. An experiment will be conducted to determine whether the sample members who view the video have higher rates of participation than those who do not view the video.

The second experiment involves testing a new approach aimed specifically at using incentives to minimize bias rather than to increase response rates. The two experiments, which are described below, are independent of each other and both procedures are planned for use in the NPSAS:12 full scale study, subject to demonstration of their effectiveness in the field test.

1. Experiment # 1: Increasing Survey Participation Using Informational Video

Initial contacts with sample members at the start of a data collection have traditionally been made using letters, e-mails, and sometimes telephone calls, with each form of communication including similar information about the study. These communications were designed to inform the sample member about the survey, how to complete the survey, and what incentives or benefits would be associated with participation.

RTI proposes to perform an experiment using an online video to communicate survey information⁷, and to include characters and images from the video in communications to sample members. This communication technique was used late in data collection on a sample of B&B:09 survey non-participants. A link to the video, posted on YouTube, was e-mailed to the non-participants and during the week following the e-mail response rates doubled compared to the previous week. Later, a postcard featuring characters from the video was mailed to remaining non-participants and resulted in 2.4 times as many survey completions during the week after the mailing as the previous week. However, because the video and postcard were mailed to all non-participant sample members, the impact of the video (versus traditional non-video) reminders on response rates could not be assessed.

The hypothesis is that video and visual images are more effective methods of communicating information to, and engaging, sample members, than are the traditional methods of contact. To test this hypothesis, half of the field test sample, selected at random, will receive the treatment (a link to a video will be included in initial contact materials) and their participation rates⁸ will be compared to a control group, the other half of the field test sample, who will receive the materials in the traditional format (postal mail or e-mail without a link to the video). Support for this hypothesis would be manifested in an increased participation rate for the treatment group⁹.

A second hypothesis to be tested is that the use of recognizable characters and themes from the video will improve recognition of, and response to, follow-up surveys. Such “branding” would be particularly beneficial to longitudinal studies, such as BPS and B&B, which require one or more follow-up interviews following the NPSAS base year data collection. Branding is the application of names, designs, symbols, and recognizable characters to products to increase recognition and identification in users (Gladden, Milne, & Sutton, 1998). Through branding, related products can be conceptualized as a single entity in a consumer’s memory (Aaker, 1991).

Presented without a unifying “brand,” sample members initially contacted to participate in NPSAS may not easily recognize the connection between it and the differently named follow-ups, BPS or B&B. The hypothesis is that “branding” using the NPSAS video will improve survey participation in subsequent rounds of the longitudinal study. To test this hypothesis, RTI proposes to use characters from the NPSAS:12 field test video in subsequent communications with those members of the BPS:12 field test cohort who were initially exposed to the video during NPSAS:12. Treatment group participation rates for panel maintenance and interviewing will be compared to the control group to evaluate the effectiveness of the branding.

The experimental design to test the second hypothesis will be informed by the outcome of the first experiment. If the treatment group’s participation rate is greater than the control, the treatment may be given to the entire sample in the follow-up study to increase overall participation rates. Participation would be measured between the NPSAS treatment and control groups to assess the impact of branding.

Video design. Video has long been recognized as an effective way to communicate information and the proliferation of high-speed internet connections, digital video, and user-friendly software has made the task of creating and sharing video much easier. Traditional-aged postsecondary students, members of “Generation Y” or the “Net Generation,” the most technology savvy generation in U.S. history, are accustomed to viewing videos for entertainment and educational purposes, and many have experience creating and sharing their own videos (Shaw & Fairhurst, 2008). The video proposed for this experiment will use stop-motion filmmaking and Lego figures, blocks, and accessories. Along with audio dialogue, the Lego materials will be used to visually represent key information such as the methods of completing the survey and major themes the survey will address. In addition to its cost effectiveness this method is selected because it is an entertaining and perhaps unexpected method of communication, which increases the likelihood that sample members will watch the video.

Use of characters from the NPSAS:12 video in communications to sample members may help draw their attention to these materials. For example, a postcard containing a close-up photo of a Lego figure may be more eye-catching than more traditional stock photos and graphics. Furthermore, sample members may begin to associate recognizable characters from the videos and mailings with the Department of Education survey.

Implementation. For the first hypothesis, that an informational video will increase survey participation rates, a randomly-assigned treatment group will receive an initial letter (as described in the data collection overview) that includes an invitation to view an informational video about the survey. The video will be posted to YouTube and a URL will be included in the letter. YouTube will be used rather than an RTI website because of its likely familiarity to sample members. E-mails sent to sample members subsequent to the initial letter will include hyperlinks to the video on YouTube, and may include graphics from the video as well.

Testing of the second hypothesis, that characters from the video will increase recognition of follow-up NCES studies and thereby improve participation rates, cannot be tested until future field tests are designed for these studies (e.g. in the BPS:12/14 field test). However, this hypothesis could be tested by placing the BPS-qualifying NPSAS treatment group sample members in a second, BPS-specific treatment group. Communications with this BPS treatment group would again make use of the Lego characters and a BPS-specific video would be offered.

For each hypothesis, participation rates for the treatment and control groups will be compared, with significance testing performed on the results.

In addition to the experimental analysis, survey respondents will also be asked voluntary debriefing questions to help better understand their reactions to the video.

2. Experiment # 2: Response Propensity Approach

Nonresponse bias in sample surveys can lead to inaccurate estimates and compromise data quality. In the NPSAS:12 field test, we plan to test a new methodology, developed by RTI, that will minimize nonresponse bias by targeting cases that have a low likelihood of responding and a high likelihood of contributing to nonresponse bias. We describe this methodology, and our plans for conducting an experimental evaluation of its effectiveness, in this section.

Survey organizations commonly address nonresponse bias by attempting to increase the survey response rate. This step is usually accomplished by pursuing the nonresponse cases most likely to be interviewed. However, this approach may not be successful in reducing nonresponse bias even if higher response rates are achieved–in fact, nonresponse bias could even be increased by adding more cases that are similar to those that have already responded (Merkle and Edelman 2009). If low propensity (i.e., difficult to complete) cases are brought into the response pool, we anticipate that this will not only increase the weighted response rate and result in less biased survey estimates. This is the hypothesis we intend to test with this experiment.

RTI is currently undertaking an initiative, modeled on the Responsive Design methodologies developed by Groves (Groves and Heeringa, 2006), to develop new approaches to improve survey outcomes that incorporate different responsive and adaptive features. Although still in the development phase, RTI has implemented several of these procedures on recent studies and have published preliminary results (Rosen, et al., 2011; Peytchev, et al., 2010). RTI’s experimental approach aims to reduce nonresponse bias by using multiple sources of data to produce models that estimate a sample member’s response propensity prior to the commencement of data collection. After we empirically identify sample members as low and high response propensity, we will apply incentive amounts differentially by experimental group, and use other data collection interventions (such as prompting, multiple email contacts, whatever may be appropriate for the sample) in an attempt to maximize the average response propensity across all sample members.

The ultimate goal of the approach is to minimize bias by targeting the cases that, based on the available data, are expected to have a low response propensity and a high likelihood of contributing to nonresponse bias. Because the propensity-modeling plan considers respondent information more inclusively and broadly, it is expected that it will also be able to determine which cases would potentially contribute most to minimization of bias in estimates, and ensure that these cases receive priority, via an effective treatment.

The Methodology. Ideally, the first objective of the response propensity approach is to use information that is known prior to data collection (e.g., frame variables) and incorporate information obtained during data collection (e.g., data that describes the survey process, for example) to develop a predictive model of a given sample member’s propensity to respond. The methodology proceeds as follows:

Step 1: Identify variables which predict propensity to respond and estimate a case’s propensity to respond to an interview.

Step 2: Target high and low propensity cases appropriately with special interventions to encourage participation.

Step 3: Evaluate the predictive ability of the response propensity model and determine if bias is reduced in key survey variables.

In theory, a good response propensity model will predict high and low propensity cases (i.e., those that are predicted to be less likely to respond.). Targeting low likelihood cases for special treatment should increase the presence of low propensity cases in the response pool, which could in turn, reduce bias in the survey estimates. We seek to test our assumptions in the NPSAS:12 field test.

Step 1. Mining past rounds of NPSAS and external data sources

NPSAS:12 provides an opportunity to examine the utility of the response propensity approach for a cross-sectional study when sample member data are more limited than in other survey designs (such as longitudinal studies). As the base-year of a longitudinal study (BPS:12/14/17), NPSAS:12 can also further inform the design and use of propensity models in longitudinal studies. To our knowledge, NPSAS:12 will be the first cross-sectional study in which this approach is tested. While prior round survey response data for the NPSAS:12 sample members will not be available, data for similar populations are available from previous rounds of NPSAS (since NPSAS is a periodically repeated cross-sectional study). Thus, our investigation begins prior to the start of the field test. Data from prior waves of NPSAS, specifically, NPSAS:04, have been examined to detect predictors of a sample member’s eventual response outcome. In preliminary modeling, we found the following NPSAS:04 variables to be predictive of a sample member’s response propensity:

• Student attends a 4-year institution;

• Student attends a less –than-2- year institution;

• Student attends a public school;

• Student attends a private-for-profit school;

• Student is a first-time-beginner;

• Student is in a doctoral program;

• Student is an undergraduate;

• Student is in school full-time;

• Mother is a college graduate;

• Father is a college graduate;

• Student has missing  CPS data. 

Step 2. Setting Up Experimental and Control Groups

Estimated response propensities will be used to divide the sample into two, roughly equal groups: a low and a high response propensity group. We will use the median calculated response propensity from the NPSAS:04 modeling work as our high/low cutoff in NPSAS:12.   The median response propensity from NPSAS04 is 0.6102 which is reflective of the overall response rate. The low propensity group will be those sample cases we predict will be difficult to complete and likely to introduce nonresponse bias if they remain nonrespondents. Within the low propensity group, cases will be assigned, at random, to either a “low propensity experimental” group, which will be offered a $45 incentive or a “low propensity control” group, which will be offered a $30 incentive. For the “high propensity control” group, we will offer the same level, $30, and for the “high propensity experimental” group, offer an incentive of $15. Based on our prior experience with NPSAS, we believe that a higher respondent incentive will encourage participation from low propensity cases, and we know from experience that $30 is effective with this population. Figure 2 outlines the data collection design and the associated incentive offers.

Figure 2. NPSAS:12 field test propensity experiment design

Step 3. Evaluating the Results

We intend to evaluate the experimental results by examining how well our model predicts response outcomes and by investigating whether our treatments affected response rates and, minimized bias in key survey estimates. First, we will examine response rates for each group defined by the estimated response propensity to determine how well our assigned response propensities actually predicted the survey outcome. Response rates will also be examined to ensure that the overall response rates for the experimental groups is equal to or better than to the control groups and the control groups with each other’s response rate. While the goal of this approach is to minimize bias (and not necessarily to increase response rates), we want to be sure that there is no negative impact to response rates for either the high or low propensity group, and that bias is minimized in the low propensity group as a result of the response propensity approach.

We will then address whether the variance of the response propensity, , was lowered, and whether the association between the response propensity and selected key survey variables, , was reduced, thus minimizing nonresponse bias in survey estimates of means and proportions. Nonresponse bias analyses will be conducted to estimate the bias prior to any weight adjustments to compare the magnitude of bias between the treatment and control groups. This analysis will compare respondents and nonrespondents to the interview and will inform data collection procedures for the full-scale study. While there are no plans to compute an interview weight or do nonresponse bias analysis between interview respondents and nonrespondents in the full-scale study, it will be important to know whether item-level bias can be reduced by increasing interview response among low-propensity cases.

The full-scale analysis file will contain study members, where a study member is based on having sufficient data regardless of data source. That is, there will be study members who do not respond to the interview, but for whom key data from other sources are available. The full-scale response rates, weights, and nonresponse bias analysis will be based on study members. Therefore, we will estimate bias at the item level (rather than at the interview response level) and compare the magnitude of the bias between the treatment and control groups. While we will compute nonresponse weight adjustments (as described in section 1) to make the field test sample representative of the population, the nonresponse bias analysis will use the unadjusted base weights in order to properly measure the bias reduction associated with the experimental treatment.

In addition to reducing nonresponse bias in the interview data, the response propensity approach also has benefits for imputation procedures. Missing data are imputed from responses obtained by “donors” who are subject to bias. Another result of the response propensity approach is a cleaner donor pool, which should produce more precise estimates for key survey items and require less imputation overall for interview nonrespondents. A small subset of items from the field test interview will be imputed to allow a comparison of the post-imputation distributions across the experimental and control groups.

Dr. Jeffrey Rosen of RTI will lead design and analysis of the propensity experiment for NPSAS:12. Dr. Rosen is also working on a similar experiment for the third follow-up of ELS:2002 and the first follow-up of HSLS:09. Recently, Dr. Rosen has published analyses (Rosen, et al,2011; Peytchev et al, 2010) of survey response propensity modeling and is thus well qualified to analyze the experimental results. Dr. Rosen will work closely with other members of the NPSAS project team to ensure close coordination of the propensity modeling activities being conducted on both NCES projects.

c. Experimental Design

The two experiments will test the hypotheses outlined below. The experimental design includes estimation of the minimum difference between the control and treatment groups necessary to detect statistically significant differences.

Null Hypotheses

1. There will be no difference in participation rates during the early response period for those who receive the survey materials with a link to the informational video and those who receive the survey materials without a link to the informational video.¹⁰

2. Response rates will not be lower among the low-propensity treatment group than in the low-propensity control group.¹¹

3. Response rates will not be lower among the high-propensity treatment group than in the high-propensity control group.

4. There will be no difference in response rate between the low-propensity treatment group and the high-propensity control group.

5. There will be no difference in unit nonresponse bias between low-propensity treatment and control groups.

Detectable Differences

The differences between the control and treatment groups necessary to detect statistically significant differences are shown in Table 10. Several assumptions were made in computing detectable differences:

1. Detectable differences with 95 percent confidence were calculated as follows:

   1. Hypotheses 1, 4, and 5 assume a two-tailed test.

   2. Hypotheses 2 and 3 assume an one-tailed test.

2. The sample will be equally distributed across experimental cells.

3. Approximately half of the sample will be assigned to the high propensity group, and approximately half of the sample will be assigned to the low propensity group. 

4. Analysis of the informational video experiment will include sample members who responded to the invitation to complete the survey but were determined to be ineligible for the study. For the response propensity experiment, all ineligible cases will be excluded from the analysis, even if they are not determined ineligible until the interview.

5. The participation rate for the control group for experiment 1 (the video) will be 35 percent.¹²

6. The response rate for the control group for experiment 2 (response propensity) will be 70 percent.¹³

7. Unit nonresponse bias for the control group for experiment 2 will be ten percent.¹⁴

8. The statistical tests will have 80 percent power with an alpha of 0.05.

9. The study design effect will be about 2.0.

10. The intraclass correlation will be about 0.2.

Table 10. Detectable differences for field test experiment hypotheses

Hypothesis		Control group			Treatment group			Detectable difference with 95 percent confidence
		Definition	Sample size		Definition	Sample size
1		No video	2,265		Video	2,265		5.1
2		$30 for low propensity cases	1,076		$45 for low propensity cases	1,076		6.3
3		$30 for high propensity cases	1,076		$15 for high propensity cases	1,076		6.3
4		$30 for high propensity cases	1,076		$45 for low propensity cases	1,076		7.1
5		$30 for low propensity cases	1,076		$45 for low propensity cases	1,076		4.0

5. Reviewing Statisticians and Individuals Responsible for Designing and Conducting the Study

Names of individuals consulted on statistical aspects of study design along with their affiliation and telephone numbers are provided below.

Name	Affiliation	Telephone
Dr. Lutz Berkner	MPR	(510) 849-4942
Dr. Susan Choy	MPR	(510) 849-4942
Ms. Christina Wei	MPR	(510) 849-4942
Dr. John Riccobono	RTI	(919) 541-7006
Dr. James Chromy	RTI	(919) 541-7019
Mr. Peter Siegel	RTI	(919) 541-6348
Dr. Jennifer Wine	RTI	(919) 541-6870

In addition to these statisticians and survey design experts, the following statisticians at NCES have also reviewed and approved the statistical aspects of the study: Dr. Tom Weko, Dr. Tracy Hunt-White, and Ms. Linda Zimbler.

6. Other Contractors’ Staff Responsible for Conducting the Study

The study is being conducted by the Postsecondary, Adult, and Career Education (PACE) division of the National Center for Education Statistics (NCES), U.S. Department of Education. NCES’s prime contractor is the RTI International (RTI). RTI is being assisted through subcontracted activities by MPR Associates, Branch Associates, Kforce Government Solutions, Inc. (KGS), Research Support Services, Millennium Services 2000+, Inc., and consultants. Principal professional staff of the contractors, not listed above, who are assigned to the study are identified below:

Name	Affiliation
Dr. Alvia Branch	Branch Associates
Dr. Cynthia Decker	Consultant
Ms. Andrea Sykes	Consultant
Mr. Dan Heffron	KGS
Ms. Carmen Rivera	Millennium Services
Ms. Vicky Dingler	MPR
Ms. Laura Horn	MPR
Ms. Alexandria Radford	MPR
Dr. Jennie Woo	MPR
Dr. Alisú Shoua-Glusberg	RSS
Mr. Jeff Franklin	RTI
Mr. Tim Gabel	RTI
Ms. Christine Rasmussen	RTI
Ms. Melissa Cominole	RTI
Ms. Kristin Dudley	RTI
Mr. Brian Kuhr	RTI

3. Overview of Analysis Topics and Survey Items

The two NPSAS:12 data collection instruments (including a facsimile of the student interview and a table showing data elements to be collected from student records) are presented in Appendix H. Many of the data elements to be used in NPSAS:12 appeared in the previously approved NPSAS:04 and NPSAS: 96 studies (the last NPSAS studies to include a BPS cohort). Additional items will also be included in NPSAS:12. These items have been tested with focus groups, and cognitive testing will be conducted prior to field test data collection.

NPSAS is a particularly complex survey because it uses a large variety of sources and several sources may be available for the same data element. These sources include:

Student records: Student-level data from institutional records collected through a secure web application. These include records from the registrar, bursar, and financial aid office.

Student interviews: Data from student interviews using either the web-based self-administered or telephone interview.

ACT: Data from American College Testing service files on ACT college entrance examinations and student questionnaires.

FAFSA: Data from the Central Processing System (CPS) for Free Application for Federal Student Aid (FAFSA), which includes student and parent demographic, income and asset information, and expected family contribution used in need analysis. The records are called Institutional Student Information Reports (ISIR).

IPEDS: Data from the Integrated Postsecondary Education Data System (IPEDS) which includes institutional characteristics and enrollment.

NSLDS: Data from the U.S. Department of Education's National Student Loan Data system (NSLDS), which has a record of all individual student loans ever borrowed and all Pell Grant payments since 1994.

NSC: Data from the National Student Clearinghouse’s Student Tracker file, which includes student-level data on institutions attended, enrollment dates, and degree completion.

SAT: Data from the College Board files of SAT college entrance examinations.

4. References

Aaker, D. A. (1991). Managing Brand Equity. New York: Free Press

Abreu, D., and Winters, F. (1999). Using Monetary Incentives to Reduce Attrition in the Survey of Income and Program Participation. Paper presented at the annual conference of the American Association of Public Opinion Research.

Biber, M., Link, M., Riccobono, J. and Siegel, P. (2000) National Postsecondary Student Aid Study: 2000 Field Test Methodology Report (NCES 2000-17) U.S. Department of Education. Washington, DC: National Center for Education Statistics. Retrieved 4/7/06 from http://nces.ed.gov/pubs2000/200017.pdf.

Cominole, M., Siegel, P., Dudley, K., Roe, D., and Gilligan, T. 2004 National Postsecondary Student Aid Study (NPSAS:04) Full Scale Methodology Report (NCES 2006–180). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

Couper, M., Mathiowetz, N., Singer, E. (1995). Related Households, Mail Handling and Returns to the 1990 Census. International Journal of Public Opinion Research. 7(2):172-177.

Curtin, R., Presser, S., and Singer, E. (2005). Changes in Telephone Survey Nonresponse over the Past Quarter Century. Public Opinion Quarterly, 69(1): 87-98.

Dillman, D.A. (2000). Mail and Internet Surveys: The Tailored Design Method. New York: John Wiley & Sons, Inc.

Fahimi, M. (2005, August). Analysis of Incentive Experiment Data for NPSAS. Paper presented at The Sixth International Conference on Social Science Methodology, Amsterdam, Netherlands.

Gladden, J. M., Milne, G. R., & Sutton, W. A. (1998). A conceptual framework for assessing brand equity in Division I college athletics. Journal of Sports Management, 12(1), 1-19.

Goldstein, K.M., and Jennings, M.K. (2002). The Effect of Advance Letters on Cooperation in a List Sample Telephone Survey. Public Opinion Quarterly, 66(4): 608-617.

Groves, Fowler, Couper, Lepkowski, Singer, Tourangeau. 2004. Survey Methodology. Hoboken, New Jersey: John Wiley co.

Groves, R. M., & Heeringa, S. (2006). Responsive design for household surveys: tools for actively controlling survey errors and costs. Journal of the Royal Statistical Society Series A: Statistics in Society, 169(Part 3), 439-457.

Heberlein, T.A., and Baumgartner, R. (1978). Factors Affecting Response Rates to Mailed Questionnaires: A Quantitative Analysis of the Published Literature. American Sociological Review, 43(4): 447-462.

Kovac, M. and Markesich, J. (2002, May). Tiered Incentive Payments: Getting the Most Bang for Your Buck. Paper presented at the annual conference of the American Association of Public Opinion Research, St. Pete Beach, Florida.

Leeuw, E., Hox, J., Korendijk, E., and Lensvelt-Mulders, G. (2006). The Influence of Advance Letters on Response in Telephone Surveys: A Meta-Analysis. Paper presented at the Second International Conference on Telephone Survey Methodology, Miami, Florida.

Merkle, D. M., & Edelman, M. (2009). An Experiment on Improving Response Rates and Its Unintended Impact on Survey Error. Survey Practice, (March)

Moore, D. (2001, August). The Effect of Repetitive Token Incentives and Priority Mail on Response to Physician Surveys. Paper presented at the annual meeting of the American Statistical Association.

Peytchev, A., S. Riley, J.A. Rosen, J.J. Murphy, and M. Lindblad. (2010). Reduction of Nonresponse Bias in Surveys through Case Prioritization. Survey Research Methods, 4(1), 21-29

Riccobono, J., Cominole, M., Siegel, P., Gabel, T., Link, M., and Berkner, L. (2005). National Postsecondary Student Aid Study, 1999-2000 (NPSAS:2000) Methodology Report (NCES 2002–152). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

Riccobono, J., Siegel, P., Cominole, M., Dudley, K., Charleston, S., and Link, M. (2005). 2004 National Postsecondary Student Aid Study (NPSAS:04) Field Test Methodology Report (NCES 2005–02). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

Rosen, J., Murphy, J. J, A. Peytchev, Riley, S., & Lindblad, M;. The Effects of Differential Interviewer Incentives on a Field Data Collection Effort. Forthcoming in Field Methods.

Schouten, B., Cobben, F., & Bethlehem, J. (2009). Indicators for the representativeness of survey response. Survey Methodology, 35(1), 101-114.

Shaw, S., & Fairhurst, D. (2008). Engaging a new generation of graduates. Education + Training, 50(5), 366-378.

Singer, E., Van Hoewyk, J., Gebler, T., Raghunathan, T., and McGonagle, K. (1999). The Effect of Incentives on Response Rates in Interviewer-Mediated Surveys. Journal of Official Statistics, (15)2: 217-230.

U.S. Department of Education, Office of the Under Secretary, Policy and Program Studies Service, The Impacts of Regular Upward Bound: Results from the Third Follow-Up Data Collection, Washington, D.C., 2004.

Wine, J., Cominole, M., Wheeless, S., Bryant, A., Gilligan, T., Dudley, K., and Franklin, J. (2006). 2004/06 Beginning Postsecondary Students Longitudinal Study (BPS:04/06) Field Test Methodology Report (Forthcoming). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

Yu, J., and Cooper, H. (1983). A Quantitative Review of Research Design Effects on Response Rates to Questionnaires. Journal of Marketing Research 20(1) 36-44.

1 Plans for sampling students from enrollment lists were also described in the OMB package approved in July 2010, since that activity will likely begin before approval of the current request. They are repeated in this package for reference.

2 NPSAS has many administrative data sources, along with the student interview. Key variables have been identified across the various data sources to determine the minimum requirements to support the analytic needs of the study. Sample members who meet these minimum requirements will be classified as study members. These study members will have enough information from these multiple sources to be included in the NPSAS analysis files.

3 Past rounds of NPSAS have included samples of first-professional students. However, IPEDS is in the process of replacing the term first-professional with doctoral-professional practice. We will work with the sample institutions when requesting enrollment lists to ensure that they understand how to identify doctoral-research and doctoral-professional practice students.

4 “Other graduate” students are those who are not enrolled in a degree program, such as students in post-baccalaureate certificate programs or students just taking graduate courses.

5 Re-interviews will be conducted approximately 3 to 4 weeks after the initial interview and will contain a subset of items (either new items or those that have been difficult to administer in the past). Re-interviews will be conducted in the same administration mode as the initial interview.

6The survey item wording included in Part C will be tested in the cognitive lab. Based on cognitive testing findings, the item wording will be revised, and final item wording will be provided January 11, 2011.

7 RTI will collaborate with The Partnership for Public Service to help ensure that the video content and presentation resonate with sample members.

8 The survey participation rate includes sample members who responded to the invitation to complete the survey but were determined to be ineligible for the study.

9 The video will be posted on YouTube, which allows tracking of the number of times the video has been viewed but does not include data on the identity of individual viewers. 

10 This hypothesis examines participation rate, rather than response rate, because the outcome of interest is the sample member’s attempt to complete the survey, even if he or she is determined to be ineligible.

11 The purpose of this hypothesis is to ensure that the propensity treatment has no detrimental effect on overall response rate.

12 35 percent is used here as a baseline because it is consistent with the participation rate obtained during the early response period in NPSAS:08.

13 70 percent is used here as a baseline based on past experience and is consistent with the sample sizes discussed above.

14 10 percent is generally considered the maximum acceptable value for unit nonresponse bias.