B&B OMB Memo Panel Maintenance

Memorandum

United States Department of Education Institute of Education Sciences National Center for Education Statistics

DATE: Aug 16, 2010

TO: Shelly Martinez & Brian Harris-Kojetin, OMB

FROM: Ted Socha, NCES

THROUGH: Kashka Kubzdela, NCES

SUBJECT: Summary of revised approach for proposed panel maintenance activity for B&B:08/12 field test

OMB approval (No. 1850-0729 v.2) for the field test and full-scale data collections of the 2008/09 Baccalaureate and Beyond Longitudinal Study (B&B:08/09) was received in February 2008. The B&B longitudinal study (both B&B:08/09 and B&B:08/12) and its base year data collection, and the National Postsecondary Student Aid Study (NPSAS:08), are authorized under the Education Sciences Reform Act of 2002 (P.L. 107-279, Title 1 Part C) and in accordance with Section 183 of that law. This memorandum serves to summarize a change to panel maintenance activities for the B&B:08/12 field test cohort. Panel maintenance is to be performed in the Fall 2010, in advance of the B&B:08/12 field test data collection, which will commence in the summer of 2011.

Overview

The primary purpose of the B&B series of studies is to focus on the value of obtaining a bachelor’s degree, and to track the paths of recent graduates into employment and additional education at three time points: the final year of undergraduate education, one year following award, and five years following award. B&B:08/09 was the first follow-up study of baccalaureate recipients identified as part of the 2007-08 National Postsecondary Student Aid Study (NPSAS:08). The B&B:08/09 sample consisted of those students who were eligible to participate in NPSAS:08 and completed requirements for a bachelor’s degree during the 2007–08 academic year at postsecondary institutions in the United States, the District of Columbia, and Puerto Rico. A field test sample was created to test procedures developed for each data collection in the year prior to full-scale data collection – 2007, 2008, and 2011.

Maintaining the integrity of the sample for a longitudinal study is key to its success. For a longitudinal study of a young cohort where one or more years elapse between waves, effective tracing involving multiple strategies is critical for locating success. Young adults, particularly recent college graduates, are traditionally difficult to locate because they move frequently; they have likely changed physical addresses several times between the base year and final interview. In addition to the challenge of locating sample members who have moved, others may have married, changing their names in the process.

This submission to OMB requests approval for a change to a proposed panel maintenance (locating) activity. In our prior clearance, we sought permission to send an address-update mailing to all sample members. In this change request, we are seeking approval to contrast that approach against an address-update mailing that includes instructions on how to access a short informational video designed to pique a sample member’s interest in the B&B study and encourage him or her to confirm or update locating information. Video is an effective and popular form of communication, and viewing short online videos through sites such as YouTube is commonplace among traditionally-aged recent college graduates. The video is intended to be entertaining while explaining why it is important for sample members to update their contact information and how to do so.

To evaluate the effectiveness of the YouTube video, we propose the use of an experimental design. A randomly assigned treatment group will receive an email and a folded-postcard mailing containing address-update instructions and a link to the video while a control group will receive the same communications but without references or links to the video. This design will be simple to implement and will enable comparison of the two panel maintenance approaches.

A similar communication using a stop action Lego video was used late in the B&B:08/09 full-scale data collection with survey nonrespondents. A link to the video, posted on YouTube, was emailed to the nonrespondents and followed by a second email with the sample member’s survey log in credentials (sent separately to ensure that log in information was not forwarded should the link to the video be forwarded). During the week following the email, response rates doubled compared to the previous week. Later, a postcard featuring characters from the video was mailed to remaining nonrespondents and resulted in 2.4 times as many survey completions. Because the video and postcard were mailed to all nonrespondents, the effect of the video versus traditional (non-video) reminders could not be evaluated.

Summary of Planned Changes

This section summarizes the changes planned for the panel maintenance activity for B&B:08/12 field test sample members. Topics addressed include the development and content of the video, dissemination to sample members, the experimental design for its implementation, and the estimated burden.

The video will be created using stop-motion filmmaking techniques and Lego mini-figures and blocks. This technique was used in a similar application near the end of B&B:08/09 full-scale data collection in an effort to encourage remaining sample members to complete the survey. For this panel maintenance video the character of “Ed” from the Department of Education will be portrayed as looking for someone, who turns out to be a B&B sample member. Ed’s message is that it is common to move in the years following college, but that without accurate locating information we cannot include previous sample members in the next study wave. Sample members will be encouraged to visit the study website to update or confirm their contact information. A transcript of the draft dialogue for the video is included in Attachment A; a copy of the address update form is provided in Attachment B; and a copy of the address update instructions included on the postcard is provided in Attachment C. Although there will be no way to know definitively that a sample member viewed the video, address update rates will be compared between groups to determine if there was a higher rate of update participation in the video group.

The half of the field test sample that will be randomly assigned to the treatment group will receive a folded-postcard mailing and an email containing instructions for updating addresses (Attachment C) and a link to the YouTube Lego video. YouTube was chosen because of its name recognition, providing confidence that the link is legitimate and not directing them to an inappropriate or unsafe website. The other half of the sample will be assigned to a control group, which will also receive an email and folded-postcard mailing but without a link to or mention of the video.

The email and postcard will briefly remind sample members of their participation in B&B:08/09 and inform them that we are seeking updated contact information so we can invite them to participate in the next survey in 2011. The email to treatment group members will encourage them to use the embedded link to watch a short video “starring Ed from the US Department of Education” to get more information. Emails and postcards to both treatment and control groups will include a URL for a website where address updates can be entered and a toll-free number for sample members without internet access.

Based on their user name (Study ID), the website for treatment group members will include the question, “Did you watch the B&B informational video featuring Lego characters?” This will help provide a context from which to evaluate the influence of the Lego video on the address update activity.

Both groups will update their contact information at the same website. All updated locating information for sample members will be stored for use during data collection, alongside any current or historical locating information generated through tracing activities. The number of responses received from members of the treatment and control groups will be compared. Random assignment should evenly distribute sample members who have not moved since the B&B:08/09 field test data collection and therefore will not need to respond to this panel maintenance.

We will employ a target power level set at 0.80 with an alpha=0.05 to determine any significant positive effect of the Lego video and report our findings to OMB in our 2011 field test package. That said, we expect to have a minimal detectable difference of about 3.8%.

The panel maintenance address update is expected to take approximately 3-4 minutes to complete. The video sent to the treatment group is approximately 2 minutes long. From the anticipated total sample of 1760, it is expected that between 20% and 25% of sample members will respond with updated addresses (between 352 and 440 respondents). Based on this response rate, the total time spent on address updates by the sample members will be approximately 18-29 hours. If approximately half of the respondents also watch the video this will add 6-7 hours. Therefore the total estimated burden is 24-36 hours.