Attachment F - ATUS Cash Incentive Analysis Report

Evaluation of an Experiment Testing the Use of Prepaid Monetary Incentives in the American Time Use Survey

November 2022

Douglas Williams

Robin Kaplan

Office of Survey Methods Research

Executive Summary

Since 2003, the American Time Use Survey (ATUS) has produced a valuable time series on how Americans spend their time, being the only federal survey to provide information on unpaid, nonmarket work. Since its inception, ATUS has used a promised incentive, in the form of debit cards, to encourage households for which a telephone number is not available (regular incentive cases) to call in and participate in the ATUS. Due to the substantial costs associated with managing, monitoring, and administering physical debit cards, the ATUS program investigated the use of alternative, prepaid cash incentives.

The prepaid incentive experiment tested three conditions, including $5 and $10 versus no incentive. This experiment was in place of the use of $40 debit cards. Given the potential cost savings with the prepaid incentives, the experiment was expanded to include cases with the largest decreases in response, which included sampled persons (designated persons or DPs) age 15 to 24 (hard to reach cases). The experiment was conducted from December 2019 through September 2021. The data collection period had been extended from the original end date of September 2020 due to disruptions from the COVID-19 pandemic.

Analyses were conducted to evaluate the effectiveness of each prepaid incentive amount on response, respondent compositions, and key measures. Comparisons are also made to response with the use of a $40 debit card for previous years to assess whether prepaid incentives result in equivalent levels of response.

Response: Both the $5 and $10 incentive conditions had positive effects on response for both regular incentive and hard to reach cases. Only the $10 condition yielded response that was significantly greater than the no incentive condition. The percentage point increase in response was greater for regular incentive cases than hard to reach cases as response was initially lower for regular incentive cases. Increases in response for hard to reach cases was driven by increases for DPs age 15 to 19. While the $10 prepaid incentive yielded significantly higher response, when compared to response from prior years using debit cards and adjusting for expected declines in response, the $10 prepaid incentive is expected to be about 6 to 7 percentage points lower than what would be expected with a promised $40 debit card.

Respondent Compositions: The significantly higher response for the $10 incentive condition yielded differences in respondent compositions, suggesting the incentive brings in different types of respondents from those that responded without an incentive. The regular incentive group saw increases in younger DPs, those that are Black or African American, and those with lower socioeconomic status. For the hard to reach group, there were fewer changes in respondent compositions, with small increases in those with some college or income between $25 thousand and $50 thousand.

Key Measures: Questions collecting employment status and school enrollment were used as key measures to see how these are affected by prepaid incentives. For regular incentive cases, the $10 incentive yielded more respondents who are not in the labor force and more reporting currently enrolled in high school or college. For the hard to reach cases, differences for employment status were opposite that for regular incentive cases with increases in those reporting being currently employed. The opposite pattern reflects differences likely inherent between the regular incentive and hard to reach groups, since the hard to reach group includes only those age 15 to 24.

Salience and Level of Effort: Regular incentive cases were required to call in, since there was no telephone number available, thus assessing any increases in salience or changes in level of effort were restricted to hard to reach cases. Overall, salience was high across all conditions, exceeding 80%, as measured by whether the respondent recalled receiving the advance materials. Recall of the mail materials was highest for those in the $10 incentive condition. Level of effort was measured as the number of contacts required across all eligible cases per completed interview. The number of contacts was lowest for the $10 incentive condition, potentially yielding a savings that could nearly cover the cost of the prepaid incentive.

Conclusion: The $10 prepaid incentive was effective at significantly increasing response for both regular incentive and hard to reach cases. However, it is not projected to yield rates of response equivalent to that previously observed for $40 promised debit cards for regular incentive cases. The prepaid incentive does address cost issues associated with the labor for processing and managing debit cards, and funding losses due to crossing fiscal years for cards that are not utilized. Further research is recommended to identify approaches to increasing the saliency of a prepaid incentive or amounts to increase response.

Background

The American Time Use Survey (ATUS) measures how people in the United States allocate their time across various activities on a given day. Additionally, information is collected about where and with whom each activity occurred. This information makes ATUS the only federal survey to provide information on unpaid, nonmarket work.

Data collection for the ATUS has been conducted continuously since 2003 producing a valuable time-series useful for identifying changes in how people spend their time. Interviews are conducted by telephone through Computer Assisted Telephone Interviewing (CATI). Persons selected for the ATUS are drawn from households completing the closing wave (8^th wave) of the Current Population Survey (CPS). These participants are asked to report on activities that occurred on the previous day of the interview.

While the ATUS sample is drawn from households participating in the CPS, for many cases a phone number may not be available. This can be due to the household not having telephone access, or refusal to provide a telephone number. The CPS overcomes this with in-person interviewing, however the ATUS is only conducted via CATI. To ensure the ATUS is representative of all persons in the U.S., additional efforts are required to reach these sample members.

Since data collection started in 2003, ATUS has mailed $40 debit cards as a promised incentive to sample members for which a phone number is unavailable. These mailings include an appeal to call in to complete the survey, a brochure about ATUS, and an inactive $40 debit card. Upon completing the survey, respondents are provided a PIN to activate the card. Starting in 2008, the promised incentive was expanded to include cases for which a phone number was available, but was determined to be in a non-working status during ATUS data collection. These cases were mailed the same materials as no telephone cases, but not until the nonworking status was determined. These case groups accounted for 8.3 percent of the ATUS sample during 2014-2016.

Since ATUS sample members are from households previously completing the CPS, descriptive information is available on cases without a telephone, or with a non-working number. A review of compositions showed that, for the 2014-16 sample, these cases tended to include more minorities, younger persons, or have lower socioeconomic status. Efforts to include these groups are important to minimize any potential bias in ATUS data.

While the use of debit cards has been successful for eliciting participation from sample members who ATUS would otherwise be unable to contact, their use includes substantial labor costs for managing, monitoring, and administering the debit cards. These labor costs exceed the total value of the debit cards. Further, due to the fiscal year funding of ATUS, unused debit cards that cross into another fiscal year cannot be used to fund debit cards outside the funded fiscal year. Overall, approximately one-third of debit cards are cashed (reflecting respondents completing the ATUS and given the activation PIN). This results in a substantial monetary loss to the ATUS program. For a detailed review of the use of debit card incentives, compositions, and cost see the ATUS Incentive Study Proposal.

Pre-paid Cash Incentive Experiment

As an alternative to debit cards, the ATUS program considered prepaid cash incentives. Switching to prepaid cash incentives would not eliminate labor costs, but would greatly reduce these costs since the tracking and reporting of debit cards would no longer be needed. Losses due to crossing fiscal years would also be avoided. The ATUS program considered using either a $5 prepaid incentive or a $10 prepaid incentive. With each amount the cost to the ATUS program would be roughly one-half to two-thirds the cost of debit cards. The ATUS program proposed using this savings to expand the use of incentives to other groups with the lowest response to the ATUS.

The projected cost savings to the ATUS program will be beneficial, but only if the alternative use of prepaid cash incentives motivates response similar to that observed for debit cards. Prepaid incentives are known to be more effective than promised incentives (see Mercer et al. 2015; Singer and Ye 2013), however, it is important to empirically demonstrate what effect this will have on the ATUS program. To measure the effect of prepaid cash incentives, the ATUS program worked with the Office of Survey Methods Research (OSMR) to design a study testing different prepaid amounts. This followed the amounts specified above and a control condition that did not receive any incentive. This study focused on the same group that has historically been sent a $40 debit card (no telephone cases). To determine if the use of incentives could be expanded to other underrepresented groups, the study also targeted 15 to 24 year old sample members. This group has shown the largest decrease in response over time to the ATUS, with a 24 percentage point decrease between 2003 and 2017. The objectives of the ATUS incentive study are summarized below:

1. To test the effectiveness of different prepaid monetary incentive amounts on response to the ATUS.

   1. Compared to no incentive, is $5 or $10 more effective in motivating response?

   2. How do any increases in response to a prepaid incentive compare to the historical use of $40 promised debit cards?

2. Are prepaid incentives effective for motivating response in hard to reach, underrepresented groups, specifically 15 to 24-year-old sample members.

   1. Compared to no incentive, is $5 or $10 more effective in motivating response?

Methods

As described previously, there were two groups eligible for the incentive study. The first group included sampled cases without a telephone number, referred to as regular incentive cases. Cases that are found to have some form of non-working phone number were not eligible for the study. These are referred to as expanded incentive cases. All expanded incentive cases received a mailing with $5 and an appeal to call in to complete the survey. The second group included sampled cases where the selected household member (designated person [DP]) was aged 15 to 24 and a phone number was provided for the case, referred to for this study as hard to reach (HTR) cases.

The first group, regular incentive cases, included 1,800 sampled DPs that were randomly assigned to one of the three incentive conditions: $0 (no incentive), $5, or $10. Assignment to an incentive condition, occurred once a designated person was selected during the ATUS sample process. The second group, hard to reach cases, included 4,400 sampled DPs¹ that were separately randomly assigned to one of the three incentive conditions: $0 (no incentive), $5, or $10. Assignment again occurred after a designated person was selected.

All cases sampled for ATUS are mailed advance materials. For cases in the incentive study, these materials were modified to reference the prepaid incentive (if assigned to an incentive condition) and an appeal to call in and make an appointment to complete the ATUS. The advance mailing includes a letter indicating who was selected and a brochure about the survey. During the ATUS interview, respondents were asked whether they received the advance mailing. While response to this is not recorded as a data file output, this information was collected as paradata by Census.

Data collection for the incentive study ran from December 2019 through September 2021. Originally, data collection was to end in September 2020, however the COVID-19 pandemic disrupted data collection activities. All data collection activities for ATUS were placed on hold on March 19, 2020 and did not resume until May 2020. Due to the time required to resume mailing operations, ATUS data collection and the incentive study did not resume until the June 2020 sample. This resulted in the loss of sample for the months of January 2020 through May 2020.

This report provides findings from an analysis of the data collected from the incentive study. Results examine the effect of different prepaid incentive amounts on response, respondent compositions, and key measures. Comparisons are made to response with the use of $40 debit cards for previous years to assess whether prepaid incentives result in equivalent levels of response. These comparisons to historical response should be viewed with caution. Response to the ATUS, like many other surveys, has observed declines in response. Examining response since 2003, ATUS response has declined approximately 1 percentage point per year. Specifically, in 2017, response to the ATUS was 45.6 percent, compared to 2021 with a response rate of 39.4 percent. This period also covers disruption due to the COVID-19 pandemic, which may have accelerated these declines.

Findings

Response Rates

The primary research question for the ATUS incentive study is whether prepaid cash monetary incentives motivate participation, and if so, is there a differential effect by amount? Table 1 presents response rates for regular incentive (no telephone number available) and hard to reach groups (DPs age 15 to 24). Both the $5 and $10 conditions yielded positive increases in response compared to no incentive. However, only the $10 condition yielded statistically significant increases. For the regular incentive group, response was 6 percentage points higher than the no incentive condition (F = 7.5, p < 0.01) and 4.5 percentage points higher than the $5 condition. For the hard to reach group, response was 3.6 percentage points higher than the no incentive condition (F = 4.38, p < 0.05) and 0.6 percentage points higher than the $5 condition.

Table 1: Response rates for regular incentive and hard to reach age groups by incentive condition¹

	$0	$5	$10
Regular incentive (n)	600	600	600
All cases	14.6%	16.1%	20.6%
Hard to reach (n)	1,500	1,500	1,400
All cases	27.4%	30.4%	31.0%
Age 15 to 19	26.4	30.5	31.0
Age 20 to 24	28.6	30.2	30.9%

1. The U.S. Census Bureau reviewed this data product for unauthorized disclosure of confidential information and approved the disclosure avoidance practices applied to this release. CBDRB-FY22-POP001-0160.

Looking closer at the hard to reach group, the significant increase in response for the $10 condition was driven by the youngest DPs. For DPs age 15 to 19, the increase in response was significantly higher at 4.6 percentage points (F = 3.95, p < 0.05), compared to DPs age 20-24 that yielded a nominal increase of 2.3 percentage points.

Another important research question is how prepaid incentives compare to historical use of a promised incentive in the form of a debit card with a greater value of $40. Table 2 shows response for regular incentive cases between 2017 and 2019.

Table 2: Yearly response rates for regular incentive cases with promised debit card incentives

	2017	2018	2019
Response	38.5%	32.5%	33.1%

Source: ATUS Case History file 2017 through 2019 (public use files accessed from: https://www.bls.gov/tus/#data)

Comparing the observed significantly higher rate of response for the $10 condition of 20.6% to even the lowest observed rate of 32.5% shows a much lower level of survey response with the prepaid incentive. As noted earlier, ATUS continues to experience declines in response over time. Table 2 shows a decrease of 6 percentage points between 2017 and 2018 (with a small increase between 2018 and 2019) for regular incentive cases. Due to the prepaid incentive experiment, direct comparisons cannot be made to corresponding time periods. Additionally, there may have been further effects on response due to the COVID-19 pandemic that affected the prepaid incentive study. Excluding 2020, when ATUS data collection was shut down for three months, response to the ATUS (overall response) decreased from 42.0% in 2019 to 39.4% in 2021. Assuming a continuing rate of decline similar to that observed for regular incentive cases between 2017 and 2018, a projected rate for 2021 (using debit cards) would likely still be at least 6 to 7 percentage points higher than the rate of 20.6% observed for the $10 prepaid condition.

Summary Findings: Response to the ATUS.

Considering the effect of prepaid incentives, $10 was observed to significantly increase response for both the regular incentive cases and hard to reach cases.

While a prepaid incentive of $10 significantly increased response for regular incentive cases, compared to no incentive, the level of response achieved is likely to be at least six to seven percentage points lower than what could be expected from the use of a promised $40 debit card.

Respondent Compositions

Respondent compositions are compared for each incentive condition to the compositions for the sample. This was done as there is concern that monetary incentives elicit participation from sample members with similar demographic compositions. That is, bringing in more respondents who are similar to those that participate without an incentive. Due to the generally low rate of response across all conditions, no group closely matches the distribution for the sample. Generally, weighting adjusts for these differences, however those with low representation would require larger weights and thus increase variance. If monetary incentives can elicit participation from underrepresented groups, this can improve the utility of the collected data.

Table 3: Distribution of sample compositions and compositions for respondents in each incentive condition for regular incentive group

	Sample	$0	$5	$10
Regular Incentive Group (n)1	1,800	100	100	150
Age
Not provided	8.6%	4.5%	3.1%	2.4%
15 to 24	10.1	3.4	10.3	11.2
25 to 34	15.8	18.0	10.3	12.0
35 to 44	14.3	19.1	15.5	9.6
45 to 54	13.5	12.4	9.3	16.8
55 to 64	14.0	10.1	19.6	16.8
65 to 74	12.1	20.2	13.4	16.0
75 and over	11.6	12.4	18.6	15.2
Sex
Male	47.3	44.9	44.3	52.8
Female	52.7	55.1	55.7	47.2
Race
White only	69.5	76.4	72.2	76.0
Black or African American	23.9	18.0	18.6	18.4
Other	6.7	5.6	9.3	5.6
Ethnicity
Hispanic or Latino	20.2	20.2	18.6	16.0
Non-Hispanic	79.8	79.8	81.4	84.0
Educational Attainment
High school diploma or less	48.4	28.1	39.2	36.0
Some college or associate’s degree	24.5	27.0	27.8	25.6
Bachelor’s degree or higher	27.1	44.9	33.0	38.4
Annual Income
Less than $25,000	29.1	11.2	23.7	17.6
$25,000 to $49,999	29.2	33.7	23.7	35.2
$50,000 to 74,999	15.1	14.6	15.5	16.0
$75,000 or more	26.6	40.5	37.1	31.2

1. The U.S. Census Bureau reviewed this data product for unauthorized disclosure of confidential information and approved the disclosure avoidance practices applied to this release. CBDRB-FY22-POP001-0160.

For regular incentive cases, Table 3 shows that without any incentive, groups that are underrepresented tend to be young, male, Black or African American, and have lower socioeconomic status (lower educational attainment or lower annual income). By comparison, the $10 incentive condition increased representation of some of these groups, specifically those who are younger and of lower socioeconomic status. Conversely, males and those who are non-Hispanic were overrepresented. Interestingly, distributions for race were similar, continuing to overrepresent those who identify as White only.

For hard to reach cases, Table 4 shows fewer differences between the no incentive and $10 incentive condition. Both show lower proportions of respondents in the lowest socioeconomic groups (educational attainment and annual income). As with regular incentive cases, there are larger proportions observed for males and non-Hispanics.

Table 4: Distribution of sample compositions and compositions for respondents in each incentive condition for hard to reach group

	Sample	$0	$5	$10
Hard to Reach Group (n)1	4,400	400	450	450
Sex
Male	49.9%	47.0%	56.2%	52.5%
Female	50.2	53.0	43.8	47.5
Race
White only	74.0	76.8	79.6	74.9
Black or African American	15.9	11.0	11.0	11.4
Other	10.0	12.3	9.4	13.7
Ethnicity
Hispanic or Latino	24.5	22.5	24.5	19.7
Non-Hispanic	75.5	77.5	75.5	80.3
Educational Attainment
High school diploma or less	66.1	61.5	61.8	60.5
Some college or associate’s degree	25.9	26.5	25.4	28.3
Bachelor’s degree or higher	8.0	12.0	12.8	11.2
Annual Income
Less than $25,000	19.6	17.0	19.1	15.7
$25,000 to $49,999	23.9	19.5	19.1	25.1
$50,000 to 74,999	18.1	18.8	18.0	14.6
$75,000 or more	38.4	44.8	43.8	44.6

1. The U.S. Census Bureau reviewed this data product for unauthorized disclosure of confidential information and approved the disclosure avoidance practices applied to this release. CBDRB-FY22-POP001-0160.

Recalling Table 1, the effectiveness of the $10 incentive was observed for those age 15 to 19 relative to those age 20 to 24. This would have been expected to result in a higher proportion of respondents with a high school diploma or less. Table 5 shows response rates for each category of educational attainment and annual income. The difference in distributions is due to the differential effect of the incentive across educational attainment and annual income categories. Those with some college or an associate’s degree and those with an annual income between $25,000 and $49,999 had the largest increase (6.1 and 8.1 percentage points respectively).

Table 5: Response by educational attainment and annual income for hard to reach cases

	$0	$5	$10	$10 - $0
Hard to Reach Group (n)1	1,500	1,500	1,400	--
Educational Attainment
High school diploma or less	25.4%	28.8%	28.1%	2.7%
Some college or associate’s degree	28.1	29.3	34.2	6.1
Bachelor’s degree or higher	42.9	46.6	44.3	1.4
Annual Income
Less than $25,000	23.9	29.5	24.7	0.8
$25,000 to $49,999	22.5	25.7	30.6	8.1
$50,000 to 74,999	27.7	28.6	27.3	-0.4
$75,000 or more	32.1	34.5	35.9	3.8

1. The U.S. Census Bureau reviewed this data product for unauthorized disclosure of confidential information and approved the disclosure avoidance practices applied to this release. CBDRB-FY22-POP001-0160.

Summary Findings: Changes in Respondent Compositions.

For regular incentive cases, the prepaid $10 incentive provided a better representation of respondent compositions compared to the no incentive condition, especially for those with the lowest educational attainment and those with lowest annual income.

For hard to reach cases, the prepaid $10 incentive did not show changes (with sex being an exception) in the distribution of respondent compositions compared to no incentive. The incentive was most effective at increasing participation from those with some college or an associates degree, and those in the second lowest annual income category.

Key Measures

Key measures are reviewed to determine whether the prepaid incentive affects distributions for these measures. This can be an indication that the incentive is bringing in different respondents. The measures examined include labor force status and school enrollment (including high school and college).

Table 6 shows distributions of labor force status and school enrollment for regular incentive cases. Cases receiving a prepaid incentive show changes to both these distributions. Focusing on the $10 condition compared to the no incentive condition, we see that for labor force status a lower percentage of employed DPs with a commiserate increase from those not in the labor force. The percentage of those enrolled in school is also much higher – more than double of those not receiving an incentive. Looking back to Table 3, these changes make sense since the $10 incentive increased the percentage of respondents age 15 to 24. This age cohort has a greater likelihood of school enrollment and not being in the labor force.

Table 6: Distribution of labor force status and school enrollment by incentive condition for regular incentive cases

	$0	$5	$10	20192
Regular Incentive (n)1	100	100	150	418
Labor Force Status
Employed	62.9%	47.4%	51.2%	59.1%
Unemployed	3.4	7.2	2.4	3.4
Not in labor force	33.7	45.4	46.4	35.2
School Enrollment				2503
Enrolled (in HS or College)	11.9	18.4	27.8	13.6

1. The U.S. Census Bureau reviewed this data product for unauthorized disclosure of confidential information and approved the disclosure avoidance practices applied to this release. CBDRB-FY22-POP001-0160.

2. Source: ATUS Respondent, Case, and Roster files for 2019 (public use files accessed from: https://www.bls.gov/tus/datafiles-2019.htm)

3. School enrollment is subset to respondents age 15 to 49 and thus these cases are a subset within regular incentive cases.

Table 6 also includes the unweighted 2019 estimate for these measures. Caution should be taken with this comparison, as the COVID-19 pandemic resulted in dramatic effects on labor market participation. While the $10 incentive condition is much different for both measures, this could reflect true change, that the different incentive types ($10 prepaid cash vs $40 promised debit card) motivate different groups to participate, or differences due to differential response. While differences are observed, any cause is only speculative. The comparison is included for illustration.

Table 7 provides the same comparisons for the hard to reach group. Comparing the $10 incentive condition to the no incentive condition, there is an increase in the percent employed with a commiserate decrease from those not in the labor force. For the proportion enrolled in school the proportions are essentially the same. The increase in employment is in line with Table 5 that shows a large increase for those with an annual income between $25,000 and $49,999. The equivalence for school enrollment is unexpected given that Table 5 shows a large increase from those with some college or an associate’s degree. This shows that the incentive, while bringing in more respondents, it is bringing in respondents similar to those that participated without an incentive.

As with Table 6, unweighted distributions for these measures are provided for 2019 in Table 7. What is striking is how similar these are to the $10 incentive condition. Again, caution should be taken when drawing any conclusions from this. The hard to reach group in 2019 was not provided any incentive, so expectations are that this should most reflect the no incentive condition. As noted above, there are many alternative explanations for this observation. The COVID-19 pandemic had dramatic effects on the labor market where the differences could reflect true change, or this could reflect changes in compositions due to differential response. As with Table 6, comparisons to 2019 are for illustration only.

Table 7: Distribution of labor force status and school enrollment by incentive condition for hard to reach cases

	$0	$5	$10	20192
Hard to Reach Group (n)1	400	450	450	675
Labor Force Status
Employed	47.3%	52.1%	54.0%	54.1%
Unemployed	9.5	8.8	9.6	8.4
Not in labor force	43.3	39.1	36.3	37.5
School Enrollment
Enrolled (in HS or College)	56.0	54.6	56.3	57.5

1. The U.S. Census Bureau reviewed this data product for unauthorized disclosure of confidential information and approved the disclosure avoidance practices applied to this release. CBDRB-FY22-POP001-0160.

2. Source: ATUS Respondent, Case, and Roster files for 2019 (public use files accessed from: https://www.bls.gov/tus/datafiles-2019.htm)

Summary Findings: Key Estimates.

Both the regular incentive and hard to reach cases show changes in labor force status when comparing the $10 incentive condition to no incentive. Changes were modest and in opposite directions for each group. The regular incentive cases saw a decrease in the proportion employed, while hard to reach cases saw an increase. For school enrollment, the regular incentive group saw a dramatic increase in the proportion enrolled, while hard to reach cases saw no change.

Overall, the prepaid $10 incentive appears to bring in respondents that differ for some key measures.

Salience and Level of Effort

Another approach to assessing the effectiveness of incentives is to assess respondent recall of mail materials and any effects on level of effort, or in the case of ATUS, number of contacts required to obtain a completed interview. For these analyses, paradata is reviewed to assess whether recall of the letter increased under incentive conditions, or if the number of contacts to a completed interview were lower for incentive conditions. These analyses include only hard to reach cases, since regular incentives cases would require the respondent to call in².

Whether the advance materials were recalled by participating respondents demonstrates if an incentive was effective at increasing the salience of these. Incentives can be expected to cause recipients to give more attention to the mail materials and therefore give more consideration to the survey request. Additionally, the novelty of receiving cash may cause better recall of the materials when an interviewer makes contact. Similar effects are expected to reduce the number of contacts required to complete an interview, since the respondent should have a better understanding of the purpose of the call and potentially expecting the contact.

Table 8: Distribution for salience (recall of letter) and level of effort measures by incentive condition for hard to reach group

	$0	$5	$10
Hard to reach (n)1	400	450	450
Recalled letter	82.0%	83.5%	84.6%
Contacts/Complete	64.8	58.9	57.4

1. The U.S. Census Bureau reviewed this data product for unauthorized disclosure of confidential information and approved the disclosure avoidance practices applied to this release. CBDRB-FY22-POP001-0160.

Table 8 shows the distributions for these measures for each incentive condition. Recall of the advance materials was generally high across all groups, exceeding 80 percent. For each incentive condition, increases are observed with the $10 condition showing the highest recall at 84.6 percent. For the average number of contacts to complete an interview, they were notably high across all conditions, peaking at nearly 65 contacts per completed interview. This demonstrates the difficulty of the telephone mode in reaching respondents and gaining cooperation. For each incentive condition the average decreases, indicating less effort to gain cooperation, with the lowest average for the $10 incentive condition with 57.4 contacts per completed interview.

While the average number of contacts per complete remains high across all conditions, the reduction of 7.4 contacts per complete for the $10 condition over the no incentive condition is substantial. The prepaid $10 incentive not only yields more completed interviews, but a monetary savings in interviewer time. For example, if each contact attempt cost $1, a reduction of 7.4 contact attempts ($7.40) nearly covers the cost of the $10 incentive.

Summary Findings: Salience and Level of Effort.

For hard to reach cases (where a telephone number is available), the prepaid $10 incentive demonstrated a modest increase in salience as assessed by recall of the mailed materials. There was a substantial reduction in the level of effort, saving on average more than seven contacts per complete.

Conclusions and Recommendations

Due to the high cost of managing debit cards and the financial loss associated with funds that cross fiscal years, the ATUS program conducted an experiment to test the effectiveness of cash prepaid incentives. This focused on regular incentive cases, those without a telephone number, yet because of the potential costs savings associated with prepaid incentives, the experiment was expanded to cover cases showing the greatest decline in response over time, called hard to reach cases (DPs age 15 to 24).

Considering response to the ATUS, only the $10 condition yielded significant increases in response, while increases for the $5 condition were nominal and in a positive direction. The increase in response was much larger for regular incentive cases at 6 percentage points compared to hard to reach cases at 3.6 percentage points. The differential effect is due to regular incentive cases having lower initial response, and therefore more room to grow; and that increases for the hard to reach group were largely driven by younger DPs (age 15 to 19).

The $10 incentive condition also showed differences in respondent compositions, suggesting the incentive is bringing in different types of respondents. This observed effect differed between the regular incentive and hard to reach groups. The regular incentive group saw increases in younger DPs, those that are Black or African American, and those with lower socioeconomic status. For the hard to reach group, there were fewer changes in respondent compositions, with small increases in those with some college or income between $25 thousand and $50 thousand.

For the key measures of employment status and school enrollment, differences were again observed for the $10 incentive condition. For regular incentive cases, there were increases in those not in the labor force and those currently enrolled in high school or college. This is likely a result of increases in response in the extremes for age. There was greater response from those 15 to 24 compared to no incentive, while an overrepresentation of those 65 and older compared to the sample. These are groups that have a greater likelihood to not be in the labor force. For the hard to reach group, differences in employment status were in the opposite direction with an increase of those who are employed. This likely reflects an inherent difference since this group is comprised of DPs age 15 to 24 and the observation that the increase in response was largest for those with some college education.

Measures to assess any increases in salience or decreases in level of effort were restricted to hard to reach cases. Regular incentive cases were required to call in and thus similar measures would not be meaningful. The differences observed for these measures show that a prepaid incentive resulted in small increases in recall of the letter which was already high, exceeding 80% of participating respondents. The decreases in level of effort were greater as measured by the average number of contacts per completed case. This suggests that the cost of the incentive could nearly be covered by the reduction in effort of trying to reach and gain cooperation from respondents.

Overall, the findings from the ATUS prepaid incentive experiment support offering a $10 prepaid incentive that was most effective at increasing response, bringing in respondents with different compositions, and reducing the level of effort required to reach these respondents. However, this neglects another objective of the incentive experiment, which was to compare the prepaid incentive to the historical use of debit cards provided in advance with the promise of $40 for regular incentive cases. While direct comparisons are not possible, comparisons were made to previous years. Even under conditions presuming large decreases in response, especially in the context of the COVID-19 pandemic, we conclude that the $10 prepaid incentive would still fall about 6 to 7 percentage points below the level of response expected with promised debit cards.

The greater effect of the promised debit card compared to a prepaid incentive is unexpected and counter to the survey literature (see for example Mercer et al. 2015 and Singer and Ye 2013). There are several hypotheses for this observation. The first, is the large disparity in incentive amounts. The prepaid amount was $10, while the promised amount was $40. Lower prepaid incentive amounts are generally as or more effective than promised amounts of equal or greater value, however the disparity in these amounts may have been too much. Another hypothesis, is that the debit card may have provided other mechanisms to induce potential respondents to open the advance materials. The debit card is physically heavier and noticeable without opening the envelope. This may have elicited enough curiosity to get recipients to open the advance mailer and thus be exposed to the survey appeal.

While the $10 prepaid incentive is not projected to be as effective (in terms of survey response) as the use of a $40 debit card, it did show positive benefits and addresses issues associated with the management and cost structure of debit cards. Any decision regarding the use of prepaid versus promised incentives should weigh the cost benefits of moving from debit cards with the likely decrease in response. It should be noted these decreases would be much greater without any incentive at all.

Future Research

The prepaid incentive experiment clearly demonstrated that $10 was effective at motivating participation, and that $10 is near the minimum amount required. The fact that the observed effectiveness was not equal to a promised incentive of greater value and that, for regular incentive cases, response only reached 21 percent, suggests room for improvement. Below we offer suggestions for additional research aimed at increasing response or the saliency of the incentive.

1. Test a larger incentive amount, such as $20. The $10 condition was the minimally effective amount, compared to $5 that resulted in only nominal increases. A larger amount may be required to match the much larger amount promised by the debit card.

2. Test making the incentive visible. Previous research has shown that making the incentive visible through a window in the mailing increases the saliency and effectiveness of the incentive (DeBell 2020)³. The ATUS program should consider a similar process, or other methods to inform the mail recipient an incentive is included in the mail material (e.g., messaging on the envelope).

3. Test the use of a dummy debit card. If, as hypothesized, the physical heft of a debit card in an envelope increases saliency, a dummy debit card can be included. Text within the letter would reference the cash in lieu to provide an immediate debit or token of appreciation for their participation.

4. Test a combined prepaid and promised incentive. This continues the use of a debit card, but with a lower value (e.g., $10), combined with a prepaid $5 or $10 incentive. This leverages a larger amount, but the full amount is only provided to ATUS participants (and still less than $40), and the potential salience from the heft of a debit card to open the mail materials. A $5 prepaid amount is revisited as when combined with a promised incentive, may provide significant results. The continued use of a debit card does reintroduce the management and processing burden of debit cards. However, the amount that needs to be covered is one-quarter the current debit card amount, greatly reducing cross fiscal year funding loss.

Of these proposals, the second proposal would introduce a known positive benefit to response (by making the incentive visible), and require minimal changes to mailing procedures, or cost.

References

DeBell, M., Maisel, N., Edwards, B., Amsbary, M., and Meldener V. (2020). Improving Survey Response Rates with Visible Money. Journal of Survey Statistics and Methodology. 8(5), pp. 821-831.

Mercer, A., Caporaso, A., Cantor, D., and Townsend, R. (2015). How Much Gets You How Much? Monetary Incentives and Response Rates in household Surveys. Public Opinion Quarterly. 71(1), pp. 105-129.

Singer, E., and Ye, C. (2013). The Use and Effects of Incentives in Surveys. The ANNALS of the American Academy of Political and Social Science. 645(1), pp. 112-141.

1 The sample sizes reported reflect rounding in accordance with Census disclosure avoidance practices. The U.S. Census Bureau reviewed this data product for unauthorized disclosure of confidential information and approved the disclosure avoidance practices applied to this release. CBDRB-FY22-POP001-0160.

2 For mail material recall, only respondents are included since this data cannot be collected from nonrespondents. Contacts per complete is the sum of all contacts across cases that are known to be eligible and with unknown eligibility divided by the sum number of completed interviews.

3 The Consumer Expenditure Survey (CE) program is currently considering a test making prepaid incentives visible in advance mailings. A similar process by the ATUS program may introduce consistency between survey programs at Census.