B&B Incentive Allocation Change Memo

MEMORANDUM OMB # 1850-0729 v.10

DATE: January 2, 2013

TO: Rochelle W. Martinez

Office of Information and Regulatory Affairs, Office of Management and Budget

FROM: Ted Socha

B&B Project Officer, National Center for Education Statistics

THROUGH: Kashka Kubzdela

National Center for Education Statistics

SUBJECT: Baccalaureate and Beyond Longitudinal Study of 2008/2012 (B&B:08/12) Responsive Design Incentive Allocation Change Request

This request is to make a change to a portion of the responsive design model procedures described and approved in OMB # 1850-0729 v.8 (NOA 06/05/2012). In the approved submission, we proposed to conduct extensive case review (ECR) as the treatment intervention in the third phase of the Baccalaureate and Beyond Longitudinal Study of 2008/2012 (B&B:08/12) responsive design model. However, we have become concerned that the planned case review, by itself, will not be a sufficiently effective treatment to impact the response of high-distance treatment cases due to operational and feasibility issues.

By its nature, the number of man-hours needed to complete ECR per case is very high, thus making the rollout of this treatment to approximately 600 cases time exorbitant. That said, some cases would receive the treatment up to two weeks later than others, calling into question the validity of any result—positive, neutral, or negative—from the Phase 3 treatment. This time rollout issue also has the possibility to negatively impact our data collection effort for those who aren’t included in the treatment group. In the original proposal, our plans were to make ECR unavailable to non-treatment cases during Phase 3, but in retrospect we realize that some of these cases could directly benefit from ECR during this period and we don’t want to miss the opportunity to do so. The below outlined proposal offers us a cleaner and more efficient treatment intervention.

We request to modify the planned treatment for Phase 3 to be a prepaid cash incentive sent via a FedEx letter with tracking and delivery confirmation (up to this point, we’ve sent the physical mailings via USPS first-class mail). This method of incentivization is supported by the results that the Education Longitudinal Study (ELS) 2002 Third Follow-up 2012 team has seen in encouraging response among their high-distance cases. Also, prepaid cash incentives were experimented with in the B&B:09 field test and found to be effective at improving overall response.

We are not requesting additional incentive amounts, rather, we propose to reallocate the total amount so that a $5 bill be prepaid and the balance of the (already offered) incentive amount to be paid upon interview completion. Prior to sending the FedEx package with the prepaid incentive, we will conduct another round of tracing to ensure that we have the most up-to-date locating data.

For reference, the section describing the responsive design model from the approved OMB package (OMB # 1850-0729 v.8) is included below (Attachment 1). Additionally, the letter informing the selected sample members of the incentive change is attached for your approval (Attachment 2).

Attachment 1 – Excerpt from approved submission (OMB # 1850-0729 v.8)

Supporting Statement Part B.3.d

Experimental Design

Our proposed experiment is predicated on the assumption that, to reduce nonresponse bias, response rates among high-distance cases must be increased. To that end, prior to the start of data collection all sample cases will be randomly assigned to control and treatment groups (see table 17 for a description of the experimental groups). Then, treatment group cases with a Mahalanobis value above a to-be-determined cut point will be targeted during data collection at three points in time. At each:

• Mahalanobis values will be evaluated for all remaining nonrespondents;¹

• cases will be assigned to low- and high-distance groups on the basis of the cut point; and

• treatment cases within the high-distance group will be eligible for interventions as defined below.

Table 17. Experimental design

Data collection step	Description	Time frame	Mahalanobis distance	High-distance (based on Mahalanobis distance)		Low Distance (based on Mahalanobis distance)
				Control	Treatment	Control	Treatment
1	Initial Invitation and CATI-Light	July 2012 –March 2013	Calculate Mahalanobis for all cases.  Assign cases to high/low distance based on a cut point	$20, $35, or $55 (based on predicted response propensity)
2	Full CATI	Begins in October 2012	Evaluate Mahalanobis for nonrespondents. Assign cases to high/low distance based on a cut point		Additional $151
3	Extensive Case Review	Begins in November 2012	Evaluate Mahalanobis for nonrespondents. Assign cases to high/low distance based on a cut point	Late review	Additional $15 +Early review	Late review	Late review
4	Abbreviated interviews	Begins in January 2013	Evaluate Mahalanobis for nonrespondents. Assign cases to high/low distance based on a cut point	Late abbreviated	Additional $15 +early abbreviated	Late abbreviated	Late abbreviated

¹ Once a case becomes eligible for the additional $15, they remain eligible for the additional $15 even if they move into the low-distance group.

Treatments

Treatment 1 (Month 3) – additional incentive. The first three months of data collection will include web data collection and “CATI-light”, which involves a minimal number of phone calls, mainly to prompt web response. After the first three months, Mahalanobis values will be evaluated for the remaining nonrespondents, and cases above the cut point will be offered a $15 incentive in addition to their original offer ($20, $35, or $55 based on their response propensity score). Once a case becomes eligible for the additional $15, they remain eligible for the additional $15 even if they move into the low-distance group later.

Treatment 2 (Month 4) – extensive case review. After an additional month of data collection, Mahalanobis values will be evaluated again for remaining nonrespondents, and those above the new cut point (determined based on the remaining nonrespondents at month 4) will receive early extensive case review. Project staff will review the CMS-CATI events log, along with any paradata available for a particular case (e.g., availability of e-mail address, parent address, etc.), to identify any specific actions that may be considered to encourage the sample member’s participation. Cases eligible for extensive case review will also be prioritized in the CMS. The high-distance nonrespondents in the control group and all low-distance nonrespondents will receive extensive case review, but on the regular schedule (i.e., 6 weeks later).

Treatment 3 (Month 6) – abbreviated interview. After an additional two months of data collection, Mahalanobis values will be evaluated again for remaining nonrespondents, and those above the cut point (determined based on the remaining nonrespondents at month 6) will be offered an abbreviated interview. The high-distance nonrespondents in the control group and all low-distance nonrespondents will receive an abbreviated interview, but on the regular schedule (i.e., 6 weeks later).

Attachment 2 – Responsive Design Phase 3 Treatment Letter

January 18, 2013

«fname» «mname» «lname», «suffix» Study ID: «caseid»

«addr1»

«addr2»

«city», «state» «zip»-«zip4»

Dear «fname»:

First of all, let me wish you a very happy and prosperous new year. As I'm sure you're aware by now, we would like you to complete a survey for the U.S. Department of Education's National Center for Education Statistics (NCES) called the Baccalaureate and Beyond Longitudinal Study (B&B).

B&B collects information on how earning a bachelor’s degree affects the lives of college graduates as it pertains to their transitions to the workforce. In addition to employment questions, the interview covers topics such as any experiences in additional education, earnings and expenses, family formation, and personal and professional goals. To ensure that B&B is representative of all 2007-08 graduates, we need your participation.

You will receive a <<$IncAmt>> check as a token of our appreciation<< in addition to the $5 bill included in this letter>>. The interview takes approximately 35 minutes to complete.

To complete the interview by telephone with a professional interviewer, call the B&B Help Desk toll-free at 1-877-262-4440. If you wish to complete the interview over the Web, simply log onto our secure website:

https://surveys.nces.ed.gov/bb/

Study ID = «caseid»

Password = «password»

Please note that the survey is not designed for mobile handheld browsers and that the password is case sensitive and must be entered exactly as it appears here.

Your responses will be used only for statistical purposes and may not be disclosed or used in identifiable form for any other purpose, except as required by law (20 U.S.C., § 9573). If you have any questions or concerns about the study itself, please contact the B&B Project Director, Melissa Cominole, toll-free at 1-877-225-8470 (email: [email protected]) or me at 202-502-7383 (e-mail: [email protected]v).

Thank you in advance for your participation in this important study. Your cooperation is greatly appreciated and needed to make this study a success.

Sincerely,

Ted Socha

NCES Project Officer

National Center for Education Statistics

U.S. Department of Education

Enclosures

«panelinfo»

The National Center for Education Statistics (NCES) of the U.S. Department of Education is authorized by federal law (Education Sciences Reform Act of 2002 - ESRA 2002, 20 U.S.C., § 9573) to conduct the Beginning Postsecondary Students Longitudinal Study. NCES will authorize only a limited number of researchers to have access to information that could be used to identify individuals. They may use the data for statistical purposes only and are subject to fines and imprisonment for misuse.

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number of this information collection is 1850-0729. The time required to complete this voluntary information collection is estimated to average 25 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving the interview, or comments or concerns regarding the status of your individual interview, write directly to: Ted Socha, National Center for Education Statistics, 1990 K Street, NW, Washington, DC 20006.

1 While the Mahalanobis values will not change during data collection, the cases who are nonrespondents will change. Therefore, the cut point could potentially change and an individual respondent’s classification as high-or low-distance may change as well.