Phase Boundary Experiment

NSFG OMB Attachment D3 OMB No. 0920-0314

Summary of Feasibility Study for a Phase 2 Boundary Shift

Background and rationale:

The standard, approved fieldwork protocol for phase 1 (weeks 1-10) of each 12-week NSFG fieldwork period is to send a pre-notification letter as a standard USPS letter, provide no incentive for completion of the household screening interview, and provide a $40 incentive for the main interview with the selected respondent, at the time she or he is beginning the interview. The Phase 2 protocol, used for weeks 11-12 of the quarter, begins by selecting a subsample of about 1/3 of the remaining active, nonresponding cases at that point, for the interviewers to continue working on. Phase 2 adds a pre-paid $5 incentive for the household screening interview and a pre-paid $40 for the main interview, in addition to the $40 provided in person to the respondent at the time of the interview. The Phase 2 advance materials are delivered at additional cost in a UPS mailer, as they contain the advance incentives of $5 for the household screener or $40 for the main interview.

We have observed that NSFG response rates have been declining since the start of NSFG fieldwork under the current contract, but this has been especially true for the phase 1 response rates. Figure 1 shows the phase 1 response rate (Phase 1 RR), the final response rate (Final RR), and the difference between the two (essentially, the phase 2 rate, (Final RR minus Phase 1 RR)) for each quarter from 1 through 22, conducted from Sept 2011 through March 2017 (i.e, under the current NSFG contract which spans the current and prior NSFG ERB protocols). For each rate, there is also a line based on the estimated slope from the regression of the rate on the quarter. The slope coefficient for phase 1 is negative and significant, indicating a downward trend in response rates for phase 1. The slope coefficient for the final response rate is also negative and significant, indicating a negative impact of phase 1’s decline on the final response rate. However, the slope coefficient for the difference (the phase 2 response rate) is positive but not significant, indicating that it did not decrease over time. Thus, while the effectiveness of the phase 1 protocol has been declining, the phase 2 protocol has remained fairly consistent.

The initial design, implemented at the start of continuous interviewing in June 2006, called for a 10-week phase 1 and a 2-week phase 2. This phase 1 duration within the 12 –week quarter represented approximately the amount of time during which interviewer hours remained efficient and cost-effective and key indicators from the NSFG achieved stability. This threshold was based on extensive analysis of calls, yield, response rates and key indicators from data collection for the 2002 NSFG (Groves et al, 2005). Moving the phase boundary earlier by one week is now proposed as a relatively conservative but meaningful change to this design, increasing the amount of time spent in phase 2, thereby potentially addressing the declining overall response rates due to phase 1 response rate declines, without large risk to the benefits of phase 1 for the stability of indicators. A larger portion of the sample will be exposed to the phase 2 protocol and less time will be spent attempting to complete the survey under a phase 1 protocol that has become less effective over time. While increased time spent in phase 2 may bring some additional costs of the additional incentives, we hypothesize that the benefits for the overall response rate and efficiency of fieldwork will outweigh those costs.

Regardless of the results of this feasibility pilot test, we will not propose any further shifts to the phase 2 boundary. The only possible modification we will make is to propose a full experiment of this phase boundary (at the experimental 10-week mark or current protocol’s 11-week mark).

Proposed pilot test:

We propose a conservative, feasibility pilot test of moving the phase boundary to 1 week earlier in the quarter. This test would be done on a small scale (one quarter) in order to determine if the data collection produces lower response rates or higher costs, which would rule out subsequent testing or implementation. If neither of these outcomes occurs, then we would propose in a separate ERB amendment to conduct an experiment in additional quarters, accumulating results after each quarter. A preliminary description of such an experiment is described in the “Comparisons” section below.

The proposed pilot test has some direct costs associated with it. In order to change the boundary of when the second phase begins, we would need to conduct the sampling and subsequent implementation of the second phase design differentially across the sample. That is, we would need to select two samples for the second phase (1 starting at week 10 per current protocol, and 1 starting at week 9), organize the phase 2 advance mailings to be sent at 2 different times in the quarter, and update interviewers’ SurveyTrak data (SurveyTrak is the electronic sample management system used to keep track of the progress and status of sample lines) at two different times. Note that the content of the advance mailings for phase 2 will be identical for the “early” and “normal” timing groups, and unchanged from what is currently approved for use in phase 2 – only the timing would change.

Table 1 includes each of the steps in the process, showing the timing of these steps under our current fieldwork protocol with the phase boundary at week 10 and the proposed timing of these steps when the phase boundary is moved 1 week earlier. The numbers in parentheses are the weeks/days of the 12 week (84-day) quarter. Under the current procedure, the Phase 2 sample is selected on Tuesday (or early Wednesday) of week 10. The mailings are assembled and sent on Wednesday via UPS. Updates to the interviewers’ SurveyTrak data occur on Sunday morning (7am EST). The second phase of the quarter begins on Sunday morning. Under the experimental design, each of these dates is shifted earlier by one week.

Our proposal is to select five (out of 35) PSUs to implement the phase 2 boundary shift. These PSUs will be selected randomly using a stratified design. We will form the strata using some broad characteristics, including patterns in the primary cost/efficiency indicator: hours per interview (HPI). The trend in HPI in weeks 1 through 8 will be monitored to determine whether it is going up during week 8 relative to previous weeks or whether it is holding steady or going down during week 8. These strata will be further refined using the size and urbanicity of the PSUs to create 5 strata and then draw 1 PSU from each stratum for the experimental treatment. This approach will allow us to assess the phase boundary shift in a variety of circumstances related to HPI patterns in weeks 1 through 8. However, we will not have the power to make firm statistical conclusions in this small-scale test lasting one quarter and based on 5 PSUs.

This pilot test will be run concurrently with, but independently from, the recently approved mailed screener experiment (amendment 3 approved by ERB 3/21/17). We will randomly select the five PSUs for the pilot test and ensure no overlap with the areas used for the mailed screener experiment. Excluding only five PSUs should not interfere with the sample size needed for the mailed screener experiment.

Comparisons:

Once the feasibility pilot test has been implemented for one quarter, we will compare the five PSUs to their own past performance as well as to that of other PSUs in the current quarter. The metrics for the comparison will be HPI and response rate.

If data collection operations do not produce substantially lower response rates or higher costs in the test quarter, we will then submit a separate amendment to propose a full-scale experiment. The full-scale experiment would be powered based on the results of the feasibility pilot test and may run for several quarters with large sample sizes in the experimental condition. A decision would then be made on whether or not to request approval to implement this phase boundary change into the main NSFG fieldwork protocol, based on the efficiency and response rate outcomes.

Direct Costs

The assumptions underlying estimated direct costs are as follows: We assume that the sample sizes for phase 2 would increase by 70%, given that the phase 2 field period was increased from two weeks to three weeks, and assuming a somewhat higher per-week yield in phase 2 compared to current phase 2 yield. For five PSUs, this represents an increase of 28 screener lines and 23 main lines that will now get the phase 2 incentives. The pre-paid incentives cost $1,060 (28*$5 +23*$40) for these additional cases. There will also be $371 in shipping/postage (53 * $7) for these additional lines exposed to the Phase 2 protocol. We also estimated the additional costs of drawing an additional (week 9/Phase 2) sample, preparing the additional UPS mailings, and releasing the sample to the field to be $376.

In sum, the total direct costs to implement the pilot test for the phase boundary shift are, under these assumptions, estimated to be about

$1,807: ($1,060+$371+$376)

These assumptions are based on predicting slightly higher efficiency (lower HPI and costs) with an expanded phase 2, the desired outcome. Lowered efficiency (higher HPI) during phase 2 is also a possible outcome, since interviewers would have a relatively small sample to work for a longer period of time.

Reference

Groves RM, Benson G, Mosher WD, Rosenbaum J, Granda P, Axinn W, Lepkowski J, Chandra A. 2005. Plan and operation of the 2002 National Survey of Family Growth. Vital and Health Statistics Series 1, No. 42. Hyattsville, MD: National Center for Health Statistics.

Figure 1.

Table 1.

Sample selection and related processes for a 12-week quarter (84 days) of data collection: Timing under current protocol and early Phase 2 protocol

Activity	Timing under current protocol	Proposed timing with earlier Phase 2
Sampling Dry Run	Friday (week 9; day 62)	Friday (week 8; day 55)
Request Imprest Cash	Friday (week 9; day 62)	Friday (week 8; day 55)
Purchase/print materials for phase 2	Friday (week 9; day 62)	Friday (week 8; day 55)
Sampling- Phase 2	Tuesday (week 10- morning; day 66)	Tuesday (week 9 morning; day 59)
Address list generated (Data Ops)	Tuesday (week 10- morning; day 66)	Tuesday (week 9 morning; day 59)
Check WebTrak for updated results	Wednesday (8am week 10; day 67)	Wednesday (8am week 9; day 60)
Print letters and assemble mailing	Wednesday (10am week 10; day 67)	Wednesday (8am week 9; day 60)
Mailing	Wednesday (week 10; day 67)	Wednesday (week 9; day 60)
Survey Trak Update; Phase 2 begins	Sunday (week 11; day 71)	Sunday (week 10; day 64)

     

4