Request for OMB Approval of a Generic Clearance for Collection of Information
Retirement Savings Module of the Household Financial Survey
SUPPORTING
STATEMENT
B. COLLECTIONS
OF INFORMATION EMPLOYING STATICAL METHODS
Treasury and the Center for Social Development (CSD) at Washington University in St. Louis will employ a variety of statistical methods in this research effort to ensure the validity and usefulness of findings. The Household Financial Survey (HFS), an already functioning, privately-funded, large-scale survey of low- and moderate-income (LMI) households, will serve as the platform for launching a new Treasury-funded module of survey items (the Retirement Savings Module) aimed at assessing participants’ reaction to the myRA program as well as collecting baseline data on retirement preparedness, attitudes, and expectations among the targeted LMI population. Based on previous years of experience, the team estimates that roughly 8,000 respondents will participate. These participants are drawn from the universe of LMI households who file federal income taxes through the TurboTax Freedom Edition (TTFE). Participation in the survey, and specifically the Retirement Savings Module, is fully voluntary. Statistical techniques will be used to test the representativeness of survey respondents compared to the universe of TTFE tax filers. In the event that the survey sample is not representative of the population universe, statistical techniques, such as sample weighting and propensity score analysis, will be employed to allow for valid inference to be drawn from the sample data and applied to the universe. The research is exploratory and largely descriptive in nature. Findings are not intended to be generalizable to a broader population, but rather specifically focus on LMI online tax filers with refunds. The information collected will inform efforts by the Department of Treasury in designing and administering public education outreach and other aspects of the myRA program.
Describe Potential Respondent Universe
Respondent
universe:
The
respondent universe consists of households that file federal income
taxes using the TurboTax Freedom Edition and that are owed a tax
refund. Households qualify to use the program by meeting at least one
of the following criteria: (1) household adjusted gross income below
$30,000, (2) active military duty with household adjusted gross
income below $60,000, or (3) qualification to receive the earned
income tax credit. In the past several years, the number of
households using TTFE and receiving a refund has been around 800,000.
The universe is restricted to those households that have a federal
income tax refund because the research will investigate the demand
for using part of the tax refund to fund a retirement account.
This research is made possible by an existing collaborative relationship between Intuit, the makers of TurboTax, and a research team consisting of CSD at Washington University in St. Louis and Duke University. This team, known as the Refund to Savings (R2S) initiative, has researched the TTFE population extensively for the past three years, including the collection of HFS data beginning in 2013. The proposed research will consist of a new Treasury-funded Retirement Savings Module to be included in the 2015 version of HFS. The low-income criteria for the TTFE results in a population that aligns closely with the LMI population, one targeted population for the myRA program .
Establishments |
State Government Units |
Local Government Units |
Households |
Persons |
0 |
0 |
0 |
800,000 |
0 |
Total:
Expected
response rate:
The
research team expects roughly 8,000 households to participate in the
overall HFS, including the Retirement Savings Module, which
represents a response rate of around 1 percent of the respondent
universe. This estimate is based on past applications of the HFS
survey in previous years. Response rates in past iterations of the
survey have been highly correlated with the amount of monetary
incentives offered to participants. An offer of a $20 prepaid debit
card resulted in a response rate of approximately 10 percent, while
the response rate was close to 1 percent when no incentives were
offered.
Concerns about selection bias arise both with samples that respond to offers of incentives and those that respond in the absence of incentives. For the 2015 HFS, the research team proposes a hybrid approach to sampling that will limit the number of households that are invited to the survey but increase the response rate among those that are invited. Among those invited to participate, a monetary incentive will be offered. As mentioned above, the resulting respondent sample will be tested for representativeness with respect to the universe of TTFE tax filers.
The sampling procedure will essentially amount to a random selection of 1 in 2 TTFE tax filers for invitation to participate in the survey, which should equate to roughly 400,000 tax filers. Those that are invited will be offered a $5 incentive, which if historical trends continue, should result in about a 2 percent response rate from the invited sample. In total, the research team thus expects around 8,000 high quality responses that will represent a valid sample of the 800,000 tax filer universe of the study.
Collection
conducted previously:
Similar
privately-funded HFS collections have been conducted during the 2013
and 2014 tax filing seasons, although covering different material and
not covered by the PRA clearance requirements. The federally-funded
Retirement Savings Module will be included for the first time in
2015, but the past iterations of the survey provide data that has
informed response rate and burden estimates, as well as evidence
regarding the representativeness of survey respondents.
Describe Procedures for Collection of Information
Statistical
methodology for stratification and sample selection:
All
tax filers using TTFE and receiving a federal income tax refund are
included in the respondent universe. For this reason, stratified
sampling will not be employed. Instead, a random sample will be
invited to participate in the survey. Based on estimates discussed
above, roughly half of the respondent universe will be invited to
participate, and 2 percent of that sample will likely complete the
survey when incentives are offered. The first selection (50% invited)
will be random, while the second selection (2% complete) is
self-directed, and thus not assumed to be random. However, the
representativeness of the sample is an empirical question to be
tested. In past iterations, the self-selected sample has been shown
to be highly representative of the respondent universe based on
observable factors, such as income, refund amount, age, and number of
dependents. If the sample is found to suffer from sampling bias,
sampling weights can be employed to correct for the problem.
Estimation
procedure:
Estimates
are based on historical data from previous versions of the HFS
survey.
Degree
of accuracy needed for the purpose described in the justification:
The expected
sample size will support exploratory analysis and the production of
descriptive statistics to address the research questions provided in
Part A.
Unusual
problems requiring specialized sampling procedures:
There
are no unusual specialized sampling procedures being employed in this
work.
Any
use of periodic (less frequent than annual) data collection cycles to
reduce burden:
Data
for this module will be collected at one point in time, coinciding
with the date of tax-filing.
Describe Methods to Maximize Response Rates
Methods
to maximize response rates:
The main tool used to maximize response rates is the offer of
monetary incentives to participants. As discussed above, this method
has been used in the past and has been shown to produce response
rates at least ten times higher than in instances where incentives
were not offered. Invitations to the survey will be presented in
their tax filing experience using TTFE and, thus, will hopefully be
seen as coming from a trustworthy source. Clear instructions will be
provided to potential participants at the outset of the survey, as
well as an estimate of the time needed to complete the survey. Survey
items will also be designed to be simple to understand and as brief
as possible.
Accuracy
and reliability of information collected:
The
research team expects highly accurate and reliable data to be
collected through this research. Data from past surveys has been
matched to tax data as a validity check and has proven to be
extremely accurate for the variables that can be matched.
Collections
based on sampling:
The
sampling technique to be employed is described above.
Describe Any Test of Procedures or Methods
The methods employed for this research have been used successfully in past years. The research team developed the survey instrument, drawing on significant experience administering surveys of the targeted population and reflecting significant input from Treasury regarding the desired information to be produced. Many survey items are straightforward assessments of past or current retirement-related behaviors and are often framed as dichotomous response items (“yes/no questions”). Other items ask respondents to react to hypothetical products or focus on more abstract concepts, such as risk tolerance or confidence in retirement preparedness.
The guiding framework for instrument development focused on understandability and positive participate experience in addition to producing valid and reliable data. Questions were designed to be clear and unambiguous. Similarly formatted question have been used in previous iterations of the Household Financial Survey and very seldom have respondents reported issues with clarity of the questions or discomfort with the survey design.
Two additional techniques were employed to further ensure quality construction of the survey module: consultation with experts in the field and small scale pretesting with participants who would meet the qualifications of participants in the R2S study. First, the research team solicited feedback on the general framework of the survey during an in-person meeting with experts in research and policy surrounding household finances and asset building. Many of these experts also have extensive practical experience in instrument development, survey administration, and data analysis of households. These experts include:
Clinton Key - Pew Charitable Trust
Rourke O’Brien – Harvard University
Amy Brown - The Ford Foundation
Frank DeGiovanni – The Ford Foundation
Keith Ernst – Federal Deposit Insurance Corporation
Beadsie Woo – Annie E. Casey Foundation
Jennifer Tescher – Center for Financial Services Innovation
Ben Keys – University of Chicago
Ray Boshara – Federal Reserve Bank of St. Louis
After incorporating extensive feedback from these experts and Treasury officials, select individuals (Key, O’Brien, Brown, DeGiovanni, and Ernst) were provided drafts of the survey and asked to make additional suggestions to improve accurate measurement, clarity for participants and quality of the design. Recommendations from the experts were incorporated into the draft survey instrument.
The survey draft was then programmed into Qualtrics survey software for pretesting. A convenience sample of less than 10 participants was recruited to complete the survey and provide extensive feedback to the research team. Participants, who met the income criteria for participation in R2S (i.e. household income less than $30,000) were asked to consider:
Question clarity and language comprehension
Intent of each question (i.e. what the participant perceives the question is asking)
Response options (unnecessary options; missing options)
Order of questions and overall user experience
Whether any of the questions made them uncomfortable
To stimulate the experience as closely as possible, the pretest instrument was emailed to participants and they were asked to complete it online in a location of their choosing. The research team collected information from pre-testers both in writing and verbally by following up with pretest participants in-person and via phone to collect feedback. In addition, the Qualitrics software allowed the research team to examine the length of time spent on each question in order to further evaluate the usability of the instrument. Feedback from pretest participants was used to improve the survey instrument.
Based on the extensive survey development undertaken, the pretesting conducted, and the relatively straightforward nature of the questions, the research team is confident that the instrument is solid and will produce valid and reliable data.
Provide the name and Number of Individuals Consulted on Statistical Aspects of the Design and Name of Agency Unit
Several individuals have been consulted on the statistical aspects of the research design.
Michal Grinstein-Weiss, PhD
Position: Associate Professor, Washington University in St. Louis
Phone: 314-935-8201
Shenyang Guo, PhD
Position: Professor, Washington University in St. Louis
Phone: 314-935-3092
Blair Russell, PhD
Position: Senior Statistical Data Analyst, Washington University in St. Louis
Phone: 314-935-3710
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Blair Russell |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |