Field Test Results

RTI International

1

2008-12 Baccalaureate and Beyond
Longitudinal Study

www.rti.org

Selected Results of the B&B:08/12
Field Test

RTI International is a trade name of Research Triangle Institute.

Propensity Modeling Design

Bivariate analyses were used to narrow candidate list

C&RT analysis was done to check for interaction
effects in the initial list of candidate variables

Regression analyses were conducted to confirm
multivariate relationships

A list of candidate variables was developed based on
the propensity modeling literature and previous
experience with this population

Model development used variables from NPSAS:08 to
predict response to B&B:08/09

RTI International

2

RTI International

•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•

Age
Interview response status (responded/did not
respond) *
Responded during early completion period indicator
Responded before prompting started indicator *
Case received a prompting letter indicator
Ever refused indicator
Call count *
Located for NPSAS:08 indicator
NCOA match indicator
ACCURINT match indicator
NSLDS match indicator *
Federal aid amount received
CPS match indicator
TELEMATCH match indicator *
Institution control
Parents’ education *

Propensity Modeling Design (continued)

* Significant at p < .05

Data from the base year
study (NPSAS:08)

3

RTI International

Student address on file indicator
Parent address on file indicator
“Other” address on file indicator *
Email address on file indicator
Student phone number on file indicator
Parent phone number on file indicator
“Other” phone number on file indicator

Propensity Modeling Design (continued)

* Significant at p < .05

Contact data available at •
the start of the first follow- •
•
up (B&B:08/09)
•
•
•
•

4

1

Propensity Modeling Design (continued)

2

3

4

5
6
Propensity decile

7

8

9

10

Response rate by propensity decile for incentive experiment control
group

Can We Predict Response?

Propensity scores ranged from .36 to .96 with a mean
of .77

Final distribution was 65% low propensity, 35% high
propensity

Predicted propensity scores were reviewed and a cut
point determined

B&B:08/12 sample was scored using B&B:08/09
variables and parameter estimates from the
development model

RTI International

5

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

RTI International

6

Response rate

41.49%

58.51%

Low propensity

Nonrespondents
Respondents

Did The Model Predict Participation?
9.54%

90.46%

High propensity

4.17

Mean relative bias

Overall
All

6.84
7.29
3.94

Low propensity only
Incentive amount same as offered in B&B:08/09 (control)
Incentive amount $15 more than offered in B&B:08/09
(treatment)
High propensity with….
Low propensity control
Low propensity treatment

4.22
7.05

7.01

4.08

All with low propensity cases treated as nonrespondents
High propensity
Low propensity

Group

Unit Level Bias Analysis – B&B:08/12 FT

Note: Control cases only. χ2 = 88.34; p < .001

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

RTI International

8

RTI International

7

Response rate

Unit Level Bias Analysis – B&B:08/09 FS

All with low and medium propensity cases treated as
nonrespondents

All with low propensity cases treated as nonrespondents

All

3.72

17.89

9.40

3.90

Mean relative bias

All with NPSAS:08 respondents who were B&B:08/09
nonrespondents excluded

12.39

Group

All with NPSAS:08 respondents who were B&B:08/09
nonrespondents treated as respondents and double
nonrespondents treated as nonrespondents

Overall

RTI International

9

RTI International

98.8

$18,839

12.4

26.1

High
(Top 1/3)
22.7

$35,613
40.9
28.8
69.6
$877
8.6
15.5
23.2
40.6
97.0
.5

97.3

$21,060

12.9

32.8

Low
(Bottom 2/3)
18.4

$35,612
41.0
28.8
69.6
$877
8.6
15.5
23.2
40.6
97.0
.5

97.3

$21,060

12.9

32.8

Low
Control
18.4

$39,602
41.5
29.66
66.9
$926
7.7
13.0
16.7
43.6
97.5
.5

97.2

$33,620

15.8

29.0

Low
Treatment
18.5

Significant at p < .05

$32,271
37.6
29.8
67.5
$867
6.3
14.6
17.7
42.3
99.6
.3

Key Variables Analysis
Earned graduate degree
Received industry certification or occupational
license
Received vocational or technical certificate
Amount of private student loans since bachelor's
degree
Worked for pay since earning bachelor's degree

Significant at p < .10

Current employment: Salary
Current employment: Hours per week
Looking for a job
Has retirement account
Monthly rent or mortgage payment amount
Financial stress: Phone
Financial stress: Mortgage/rent/utility bill
Financial stress: Food
Married
Citizen
Number of dependent children
10

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2

Propensity quintile

3

4

5

Control

Treatment (+15)

Response rate by propensity quintile for low-propensity
treatment and control groups

1

1

2

Propensity quintile

3

4

5

Control

Treatment (-$15)

Response rate by propensity quintile for high-propensity treatment and
control groups

Can We Lower Monetary Incentives for the High-propensity
Cases Without Affecting Response Rates?

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Can We Successfully Target Low-propensity Sample
Members With Increased Monetary Incentives?

RTI International

11

12

RTI International

Response rate
Response rate

Response rate

100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
1

2

3

4

5

6

High Propensity (1/3 of sample)

Treatment = $15 less

7

8

9

10

Control

Lower incentive

Higher incentive

B&B:08/12 Field Test Response Rates, by Incentive Amount
and Propensity Level

Treatment = $15 more

Low Propensity (2/3 of sample)
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%

Treatment

B&B:08/12 Field Test Response Rates, by Propensity Level and
Experimental Condition

RTI International

13

14

RTI International

Response rate

–

–

–

Higher monetary incentives are one way of targeting cases at the
high end of the of the low-propensity group, but this may not reduce
nonresponse error

Low-propensity cases contributed a small amount to overall unit
level bias but did not change parameter estimates significantly

We can predict propensity to respond well

RTI International

–

Higher monetary incentives are not very effective at increasing
response among cases at the lowest end of the propensity
continuum

Propensity Experiment Conclusions

–

Overall response rates in the high-propensity group are not affected
by a decrease in monetary incentives. However, the average call
count (a measure of “level of effort” to obtain a complete interview)
was significantly higher among the treatment group

15

Low

Propensity Level

Middle 40%

Lowest 30%

Percent of Full-scale Sample

$20

$35

$55

Full-Scale Recommendations – Incentives

Medium

Highest 30%

Incentive Offer

High

RTI International

16

–

–

–

–

–

$10 incentive for address update prior to data collection

Revisit the utility of social network contacting/locating (has not
been very effective in the past, but revisiting Facebook/LinkedIn,
etc.)

Consider increased use of the more costly interactive tracing
searches, such as Choice Point

New tracing sources (Spokeo, Fast Data’s Premium Address
Service, etc.)

Pre-Intensive tracing

Locating

Full-Scale Recommendations – Survey Methods

RTI International

17

Survey Methods (continued)

–

–

–

More tailored messages

Contact parents

More frequent contacts

Communication

RTI International

18

Survey Methods (continued)

Select pool of highly skilled interviewers

CATI Strategy

Consider targeted field effort

Closer monitoring of low-propensity cases

–

–

Contacts
Melissa Cominole
919-990-8456
[email protected]

Bryan Shepherd
919-316-3482
[email protected]

Abbreviated interview, after unsuccessful attempts to obtain a
complete interview

–

Offering alternate data collection methods

RTI International

19

RTI International

20