ED Responses to OMB Passback

Date: March 22, 2011

To: Shelly Martinez, OMB

Through: Kashka Kubzdela, NCES

From: Tracy Hunt-White, NPSAS:12 Project Officer, NCES

Subject: Responses to OMB’s Passback of March 7^th and 17^th, 2011, for NPSAS:12 Field Test Student Data Collection (OMB# 1850-0666 v.8)

During a telephone call, on March 7th, 2011, OMB requested:.

1. More information on how predictive NPSAS:12 response propensity model was using NPSAS:04 data.

2. For NCES to consider doing an experiment on the high propensity group by offering a lower incentive (i.e., $20 vs. $30).

3. For NCES to discuss in more detail the increased use of paradata (when available), frame data, and adaptive response design in propensity modeling, and to discuss in this context equity, which is a public policy concern, and other competing concerns, such as bias, cell sizes, or overall cost effectiveness of the study.

Regarding the conversation on changes to the response propensity model for NPSAS:12, attached are the following documents:

• A memo outlining NCES’s reasons for changing the Response Propensity Design and what was learned from the preliminary modeling

• A revised Part A with track changes

• A revised Part B with track changes

Based on OMB’s feedback, NCES is going to do an experiment on the high response propensity group: $15 and $30. For the low propensity group we will test $30 and $45. We recognize that the goal is to try to stay below $50 dollars for incentives and we felt that $45 was different enough from $30 to be able to detect a difference in response. NCES has done experiments in the past where $10 increments between incentive amounts did not really allow to detect any differences. We know that $30 has been effective for achieving good response rates in the past and this will give us a chance to see if its effectiveness varies by response propensity. Attached is also a document summarizing how the questionnaire has changed by section.

Response to written MOB passback of March 17th, 2011:

1. Where is the video (we at least need a script and screen shots)?

On pages F-29 and F-30 of Appendix F of the OMB package, we provide the script for the Lego video. Unfortunately, we had the wrong label.  It has now been fixed in Appendix F (see track changes) along with an updated script.  Also, the script is provided in Attachment 1 to this memorandum.  Selected scenes from the video are also provided as an enclosure (see Lego.pdf). The wav file for the Lego video is too large to send, but we can provide a CD upon request.

2. Did RTI either get permission from Lego to use its figures in this manner or did it get a legal determination that this permission was unnecessary?

In January 2010, when we first began developing and using LEGO bricks to develop videos for data collection purposes, we consulted with intellectual property staff in RTI’s Office of Corporate Counsel (OCC). Their opinion was that the LEGO bricks used for the videos are considered to be generic and in the public domain. Moreover, none of the characters created were copyrighted images (as, for example, a Darth Vader LEGO character would be). Before the video is used for NPSAS:12, however, we are confirming directly with the LEGO Corporation that use of their products for this purpose is acceptable.

3. Response propensity models

1. There isn’t a clear relationship between response propensity and nonresponse bias in this discussion.  Getting more low propensity cases won’t reduce bias if propensity to respond isn’t related to the substantive variables on the survey.  Other than a brief mention of bias, it isn’t clear what analysis in support of bias reduction has been done.  Specifically, the variables with explanatory power would need to be correlated with both response propensity and substantive variables of interest to be useful.  Please respond to this concern.

Bias reduction among key survey items is something we are proposing to test for the first time with the NPSAS:12 field test. Because of the way bias was calculated in NPSAS:04 and NPSAS:08, the design makes it difficult (or nearly impossible) to tie substantive variables to bias. In our approach for NPSAS:12, we are proposing to test a theory of how survey studies might reduce bias in a simple, cost effective manner.  Our assumption, grounded in theory, is that if we are able to identify (a priori) and bring more “low propensity to respond” cases into the response pool, bias in key survey variables should be reduced by reducing the variability in response propensities and including more of the type of people who tend to be omitted in surveys. This working theory is currently being tested in a number of studies where we will soon be able to show empirically whether the approach is successful in reducing bias in key survey estimates.

2. Related, use of paradata is a common strategy in responsive design, and quite a bit of work has been done on paradata since 2004.  We understand that your analysis showed limited explanatory power from the 2004 data.  However,

   1. Did that analysis actually look at correlation to substantive variables of interest rather than just response?

We did not look at correlations between paradata and variables of substantive interest because we felt that the universe of paradata available in NPSAS:04 was of limited use.  We instead proposed to change the design.  We are not proposing a responsive design for the NPSAS:12 field test since we will not be adjusting our survey protocol based on paradata we collect during the field test.  However, we do understand the value of responsive or adaptive design and plan to collect paradata in the field test that might inform the full scale collection.  Furthermore, we will descriptively analyze our field test results across categories of paradata that we will be collecting, with the goal of informing the full scale collection and potentially implementing a well conceived responsive design. 

2. Also, what paradata might NCES collect in the field test that could improve the model for the full scale survey? 

The paradata items we will collect and potentially make use of include:

• Positive match from Telematch (or equivalent)

• One or more email addresses available

• Advance letter comes back undeliverable

• Postcard comes back undeliverable

• Positive email message from respondent (wanting information, etc.)

• Negative email message from respondent (pending refusal, etc..)

• Positive call made by respondent (wanting information, etc…)

• Negative call made by respondent (pending refusal, etc.)

• Ever logged in but did not get past first items 

• Ever logged in and got through enough to consider partial complete

3. What does this sentence in the memo mean: “Unlike the initial design, propensity scores will be assigned across institutions, rather than within institutions.”?

This statement simply means that the cut point between low and high propensity cases will be determined across the sample as a whole rather than within institutions.

3. Besides a higher incentive, which other of the possible interventions discussed in SS B4 b are being used for the low propensity cases in the field test? 

   1. If none, why was a higher incentive the only intervention proposed?  What is the evidence that a higher incentive is more effective than the other items enumerated?

SS B4 b (page 43) was not precise in explaining our approach. We have corrected the text to indicate that all other interventions, which RTI uses throughout the normal course of data collection, will be applied equally across the two propensity groups. To do otherwise would confound the effects of the incentives and the response propensity experiment. Consistent with the survey research literature, our past experience with NPSAS and similar student populations has shown that incentives are the most cost effective intervention for encouraging response.

4. Please update SS B4 with the current status of the modeling work.  What is there is still future tense and doesn’t even conclude yet that the approach is feasible to test.

We have updated SS B4 to include the results of the modeling performed using NPSAS:04 data. 

5. Please list the specific variables that are being used in the modeling.

The model includes the following NPSAS:04 variables:

• Student attends a 4-year institution;

• Student attends a less –than-2- year institution;

• Student attends a public school;

• Student attends a private-for-profit school;

• Student is a first-time-beginner;

• Student is in a doctoral program;

• Student is an undergraduate;

• Student is in school full-time;

• Mother is a college graduate;

• Father is a college graduate;

• Student  has missing  CPS data. 

6. Please justify why exactly half of the sample are identified as “low” and half as “high” propensity.  Do the data really have such a neat cut point, is this a sample size issue, or some other reason?  It seems like to dilute experimental results to have individuals with similar propensities in the two different groups.

With respect to the sample size, since this is the first time this approach has been implemented in a cross-sectional design, we are adopting the design with the greatest simplicity. 

Nonetheless, OMB’s point is well noted and our proposed solution is to use the median calculated response propensity from the NPSAS:04 modeling work as our high/low cutoff in NPSAS:12.   The median response propensity from NPSAS04 is 0.6102 which is reflective of the overall response rate.  Assuming a roughly similar overall response rate in NPSAS:12, that cutoff is sensible to use.  The placement of the cutoff is essentially arbitrary, and we now think the best place to put it is at the median response propensity in NPSAS:04. This is especially true since we are relying on NPSAS:04 to assign propensities in NPSAS:12.   When we do the analysis, we will investigate cases around the median response propensity to be sure the placements of those cases are not influencing results too heavily.

7. As NCES knows, OMB is concerned about equity issues in using incentives.  However, we also are very supportive of efforts to address bias and cost effectiveness.  One practical consideration is to avoid incentivizing two individuals at different levels who are likely to know of each others’ participation in the survey.  For example, two individuals who attend the same university.  Is attending the same institution inherent in the model to avoid this problem?  If not, we recommend constraining the model in this regard.

We have not added constraints in this manner and we understand that this cannot be controlled completely.  It is useful to note that institutional type is a major factor in assigning response propensities.  In other words, it is likely that students from the same institution who are likely to be in contact with each other (doctoral students, undergraduates, first-time beginners) are also likely to have roughly the same response propensity.  We are, of course, randomly assigning cases to groups, so it may be the case that similar students are randomly assigned to different experimental and treatment categories, which is actually desirable from an experimental standpoint.  We can, however, add some safeguards.  If a student calls and asks for the same incentive as her friend, we propose to provide the higher incentive.  We will be sure to remove any student in this situation from the analysis.  

4. Reinterviews – are these cases being equally drawn from high and low propensity cases?  A conditional 80% response rate seems a little high.  Is NCES sure it has enough cases in sample in case that estimate is off?

We do plan to randomly select reinterview cases from both the high and low propensity cases. We will, first, select potential reinterview cases from within a portion of the completed interviews separately for the high and low propensity cases (1 potential reinterview case from every 5 completed high propensity interviews and 1 potential reinterview case from every 5 completed low propensity interviews). We are not certain that the interview respondents will be equally split between the high and low propensity cases, so we will monitor counts of reinterview cases and adjust the reinterview sampling rates, as necessary, to ensure that half of the cases are selected from among high propensity completes and half from among low propensity completes. Because of the constraints of budget and schedule, reinterviews have typically been conducted with starting samples of about 300 students with the expectation that at least 80 percent will participate. Even if we do not meet the 80% response rate, we have in past studies been able to conduct analyses with a 67% response rate (NPSAS:04) and a 47% response rate (NPSAS:08).

5. By what process will NCES shave 16 minutes off of the questionnaire for full scale?  What are the likely candidates?

In the original supporting statement, the burden table indicated that the field test interview would average 38.1 minutes. This value was calculated using the expected average time in the interview for first time beginning students (45 minutes) and all other students (30 minutes), weighted for the number of interviews expected with each category of student. The full-scale interview estimate of 25.1 minutes was calculated in the same way based on a 35 minute interview for first time beginners and a 20 minute interview for all other students.

The difference of 13 minutes is anticipated because of reductions and refinements we expect to make between the field test and full-scale data collections. We will monitor field test interviews, evaluate field test data in collaboration with NCES, and use feedback from the TRP to inform the contents of the questionnaire items for the full-scale interview. Candidates for items to be removed from the questionnaire will be those that consistently present respondents with difficulty during interviews, or for which the data appear inconsistent. Priorities on necessary items will be set using assistance from the TRP members. In addition, the entire Opinions section of the field test interview will not be present in the full-scale study, cutting several items from the interview.

6. Please provide a list of any new content on the survey that was not a part of last fall’s cognitive testing.

With the exception of three questions in the Opinion section (focused on students’ experience with the interview itself) and one question in the locating section (focused on gathering information for subsequent contacting), no entirely new content has been developed since the conclusion of last fall’s cognitive testing. Attachment 2 summarizes items proposed in the original cognitive testing submission that had some revision (e.g., format changes or wording improvements for the sake of clarity or validity) during the iterative testing process as well as the four new items developed after testing.

Attachment 1. NPSAS:12 Field Test Lego Video Script

Text in [brackets] describes scenes:

[Sample member (SM) goes to the mailbox and gets out a letter]

SM: “The National Postsecondary Student Aid Study? What do they need me for? This is going in the trash…”

[Helicopter with Ed on the side]

“Ed: Sounds like I’m needed!”

[Ed lowers into view, starts conversation with SM]

Ed: “Before you throw that away, I’ve got a few things I’d like to tell you.”

SM: “Who are you?”

Ed: “I’m Ed, from the U.S. Department of Education. We want you to complete our survey so we can learn more about how students and their families finance their education.

SM: “What kind of things will you ask me?”

Ed: “We’re interested in your education after high school [student throwing mortarboard].

The kind of degree or certificate you are studying for. [classroom scene with teacher]

Whether or not you received scholarships, loans, or help from family to pay for school. [show a Lego figure hand and arm holding a Financial Aid Form]

Whether you are working while you’re in school? [Lego figure working as a cashier]

The number of people in your household? [scene of one Lego figure, then several others appear]

We can offer you $30 to thank you for your time. [Lego character with money appearing in his hand]

[Scene back in front of house]

SM: “Well, I’m pretty busy…”

Ed: “We make it easy for you. You can complete the survey by phone [Lego character using phone], or on the web using your own computer. [Lego character at computer] It won’t take long.

[Scene back in front of house]

SM: “OK, but one more question. How do I know my answers will stay confidential?

Ed: “Federal law requires that we protect all data collected. Protecting your data is our number one concern.” [Lego policeman closes a vault/safe]

[Scene back in front of house]

SM: “That sounds pretty good. I’ll do it!

[Ed turns and talks to the camera]

Ed: “You, too, can help us make the National Postsecondary Student Aid Study a success!

[Lego people appearing, holding hands, and swaying back and forth]

Ed: You’re in a special position to provide us vital information on your experiences attending school and paying for your education. With your participation this study will help Congress evaluate federal aid policies and programs to help students like you find success in college and beyond.

[Back in front of house]

SM: “Thanks, ED!!”

Ed: “Thank you for taking our surveyyyyyyyyyyyy!”

Display contact screen:

ED (speaking over the final “credit” slide with the info below typed on it):

“Call us toll free at 1-XXX-XXX-XXXX to complete your interview today! “

“And don’t forget to check your e-mail for more information about your participation in this study.”

Attachment 2. Revised and New Content Since the Original Cognitive Testing Submission

SECTION ONE: Revised Content Contained in Original Cognitive Submission

Note: All items below have been tested in cognitive labs. However, they represent items that underwent some change by virtue of that testing since the submission of the original cognitive testing package.

Enrollment:

N12UGSTATVER:

Are you primarily an undergraduate working toward a bachelor's degree in a joint program?

-Yes/No

Education Experiences:

N12AP1:

While in high school, did you take any Advanced Placement (AP) courses or Advanced Placement exams?

(Advanced placement (AP) exams are those offered in specialized study areas such as biology or chemistry, and credit earned by these exams may be accepted by colleges. We will ask you about any credits earned in the International Baccalaureate (IB) program next.)

-Yes/No

N12AP2:

While in high school, how many...

(If you did not take any AP courses or exams while in high school, enter a "0" in each of the boxes below.)

-AP courses did you take

-AP exams did you score a 3 or greater on

N12AP3A

For the AP exam on which you scored a 3 or greater, did [Y_NPSCHL]...

-Use it only to place you in a higher level course, but not give academic credit for it? Yes/No

-Award you academic credit? Yes/No

N12AP3B

For how many of the [N12APNUMEXM] AP exams on which you scored a 3 or greater did [Y_NPSCHL]...

- Place you in a higher level course, but not award you any academic credit

- Award you academic credit

-Neither place you in a higher level course nor award you any academic credit

N12IB1

While in high school, did you participate in the International Baccalaureate (IB) Diploma Program?

-Yes/No

N12IB2

Were you awarded the IB diploma?

-Yes/No

N12ALTCRS

[If T_CURENR=1]:Are all, some, or none of the classes you are currently taking at [Y_NPSCHL] taught... (If you are not currently taking any classes, select “None” for each of the below.) [Else]: During the last term you attended [Y_NPSCHL] in the 2010-2011 school year, were all, some, or none of the classes you were taking taught… (If you were not taking any classes during the last term you attended [Y_NPSCHL] in the 2010-2011 school year, select “None” for each of the below.)

-Only on-line: All/Some/None

-At night (starting after 6:00 p.m. Monday-Thursday): All/Some/None

-On the weekend (starting after 6:00 p.m. Friday or any time Saturday or Sunday): All/Some/None

N12ONLINEATT

[If T_CURENR=1]: If my on-line courses were not available, I still would have attended school this term. [Else]: If my on-line courses were not available, I still would have attended school in the 2010-2011 school year.

-Scale: 1-5

N12NIGHTATT

[If T_CURENR=1]: If my night courses were not available, I still would have attended school this term. [Else]: If my night courses were not available, I still would have attended school in the 2010-2011 school year.

-Scale: 1-5

N12WKNDATT

[If T_CURENR=1]: If my weekend courses were not available, I still would have attended school this term. [else]: If my weekend courses were not available, I still would have attended school in the 2010-2011 school year.

-Scale: 1-5

N12SRVMATRX:

How often did you use each of the following services, and how important was each of these services in your decision to stay at [Y_NPSCHL]? How often (once/every term/monthly/weekly/daily/never); How important (Not at all important/somewhat important/important/very important)

-Financial aid services

-Academic support services (for example, tutoring or writing centers)

-Academic advising

-Career planning or job placement services

-Personal counseling services

-Student health services

-Childcare services

-Services for veterans

N12OCCWAGEL

[If N12EXJBTL ne missing] Once you begin working as a(n) [N12EXJBTL], what is the least amount of money you would reasonably expect to make each year? (Provide your best guess if you are unsure of the amount.) [ELSE] What is the least amount of money you would reasonably expect to make each year when you are finished with your education? (Provide your best guess if you are unsure of the amount.)

-amount per year

N12OCCWAGEH

[If N12EXJBTL ne missing] Once you begin working as a(n) [N12EXJBTL], what is the most amount of money you would reasonably expect to make each year? (Provide your best guess if you are unsure of the amount.) [ELSE] What is the most amount of money you would reasonably expect to make each year when you are finished with your education? (Provide your best guess if you are unsure of the amount.)

-amount per year

Financial Aid

N12DCLN2K

[if N12FEDLN=1]: If you had been offered an additional $2,000 college loan, would you have accepted the loan? [else]: If you had been offered a $2,000 college loan, would you have accepted the loan?

-Yes/No

N12LN2KNO

What is the main reason why you wouldn't accept the additional loan?

-Your college costs are already covered with money from other sources

-You would be concerned that you might not be able to repay the loan amount

-You simply would not want more debt

N12LNED1K

[if N12FEDLN=1]: If you had been offered an additional $1,000 college loan to be used only for paying educational expenses like tuition and books, would you have accepted the loan? [else]: If you had been offered a $1,000 college loan to be used only for paying educational expenses like tuition and books, would you have accepted the loan?

-Yes/no

N12LNLV1K

[if N12FEDLN=1]: If you had been offered an additional $1,000 college loan to be used only for paying your living expenses like food and rent, would you have accepted the loan? [else]: If you had been offered a $1,000 college loan to be used only for paying your living expenses like food and rent, would you have accepted the loan?

-Yes/no

Current Employment

N12LIFECHNG

Tell [if web: us/if TIO: me] how much you agree with this statement by providing a number from 1 to 5, with 1 being "strongly disagree" and 5 being "strongly agree." If you would like to, you can use a decimal value such as a 2.5 or a 4.5 to answer this question. [If T_STAT=2] My life would be better overall if I -finished my education. [Else] My life would be better overall if I finished my [T_DEGREN].

-Scale: 1-5

N12CHNGWAYS

[If N12LIFECHNG in (3.5, 4, 4.5, 5)] In which of the following ways do you think your life would be better if you finished your education? (Else) In which of the following ways do you think your life would be worse if you finished your education?

-Choosing how and when you will complete important work tasks

-Flexibility to make time for friends or family

-Flexibility to make time for leisure pursuits

-Being seen as an expert in your field

-Helping others

-Yearly income

-Other way not listed

N12WRKLIMITS

[If T_CURENR=1] Does working while you are attending [Y_NPSCHL]... [Else] Did working when you last attended [Y_NPSCHL] during the 2010-2011 school year...

-If T_CURENR=1]: Limit the number of classes you can take? [Else]: Limit the number of classes you could take? Yes/no

-Make it difficult to schedule your classes? Yes/no

-Limit your access to campus facilities or services? Yes/no

Income and Expenses

N12FAMHLPORD

[If N12MARR=2] Of the money received from family or friends other than your spouse, about how much do you use each term or semester to pay for: [Else] Of the money received from family or friends, about how much do you use each term or semester to pay for: ($0, $1-$500, $501-$1,000, $1,501-$2,000, More than $2,000)

- Basic living expenses such as rent and food (or room and board) for you and your spouse and t_fill1 [Else if N12MARR=2] Basic living expenses such as rent and food (or room and board) for you and your spouse [Else if N12DEP2 gt 0] Basic living expenses such as rent and food (or room and board) for you and your t_fill1 [Else] Basic living expenses such as rent and food (or room and board)

-All other expenses you may have including expenses for you and your spouse and t_fill1 [Else in N12MARR=2] All other expenses you may have including expenses for you and your spouse [Else if N12DEP2 gt 0] All other expenses you may have including expenses for you and your t_fill1 [Else] All other expenses you may have

- Tuition, fees, and other educational expenses

N12CCUSEORD

About how much have you charged on your credit cards each term or semester to pay for: ($0, $1-$500, $501-$1,000, $1,501-$2,000, More than $2,000)

-Basic living expenses such as rent and food for you and your spouse and t_fill1 [Else if N12MARR=2] Basic living expenses such as rent and food for you and your spouse [Else if N12DEP2 gt 0] Basic living expenses such as rent and food for you and your t_fill1 [Else] Basic living expenses such as rent and food

-All other expenses you may have including expenses for you and your spouse and t_fill1 [Else in N12MARR=2] All other expenses you may have including expenses for you and your spouse [Else if N12DEP2 gt 0] All other expenses you may have including expenses for you and your t_fill1 [Else] All other expenses you may have

- Tuition, fees, and other educational expenses

N12DISCOUNTA

Imagine you have a choice between receiving $750 in one month, or $1500 one year from today. This gift is guaranteed to be paid whether you would choose to take the $750 in one month or wait to receive $1500 one year from today. Would you prefer...

-$750 one month from today

-$1500 one year from today

N12DISCOUNTB

You just said you would wait a year to receive $1500 rather than take $750 in one month. Starting with $750, what’s the least amount of money you’d be willing to wait a year for?

-Scale: $750-$1500

N12DISCOUNTC

Imagine you have a choice between receiving $750 in one month, or $1125 six months from today. This gift is guaranteed to be paid whether you would choose to take the $750 in one month or wait to receive $1125 six months from today. Would you prefer...

-$750 one month from today

-$1125 six months from today

N12DISCOUNTD

You just said you would wait six months to receive $1125 rather than take $750 one month from today. Starting with $750, what’s the least amount of money you’d be willing to wait six months for? <p><br /> You just said you would wait six months to receive $1125 rather than take $750 one month from today. </p><br />Starting with $750, what&rsquo;s the least amount of money you&rsquo;d be willing to wait six months for?

-Scale: $750-$1125

<p><br /> Imagine you have a choice between receiving $750 in one month, or $1125 six months from today. This gift is guaranteed to be paid whether you would choose to take the $750 in one month or wait to receive $1125 six months from today. </p><br />Would you prefer...<p><br /> Imagine you have a choice between receiving $750 in one month, or $1500 one year from today. This gift is guaranteed to be paid whether you would choose to take the $750 in one month or wait to receive $1500 one year from today. </p><br /> Would you prefer...

Background

N12IMGRAGE

How old were you when you came to the United States?

-Younger than 6

-Between 6 and 12

-Between 13 and 17

-Older than 17

SECTION TWO: New Content Developed After Cognitive Testing

Locating

N12CELLPRO

Please provide the name of your cell phone service provider.

Opinions

OPEDVID

Please help us to improve our future surveys by answering these few questions about your experience with the NPSAS survey. Did you receive information about a YouTube video featuring “ED?”

-Yes, and you viewed the video

-Yes, but you didn’t visit the site to view the video

-No

OPEDVID2

Did the video help you decide to participate in the NPSAS interview?

-Yes/no

OPMOBDEV

Did you access the NPSAS:12 study website on a mobile device such as a cell phone, iPad, iPhone, smartphone, or other mobile device?

-Yes/no