SupportingStatement Part B_Survey of Registered Voters Living Overseas_5_5_15

SUPPORTING STATEMENT – PART B

Survey of Registered Voters Living Overseas– 0704-TBD

B.  COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1.  Description of the Activity

The Federal Voting Assistance Program (FVAP) issued Request for Quote (RFQ – 881873) for research and analysis services to examine the demographic profile of registered overseas civilian voters, quantify the population of these voters by country, and calculate a voter registration and participation rate for the population using records from state and local election officials. The component of this research covered under this Information Collection submission pertains to a survey of the overseas U.S. civilian registered population, focused on voting-relevant characteristics such as educational attainment, income, and mobility. This data will allow FVAP to compare with greater certainty the voting behaviors of overseas registered U.S. civilians to those of the registered civilian voting age population (CVAP) and active duty military (ADM) population who share similar characteristics. The initial phase of the survey will be conducted with 5,000 registered voters living outside the U.S. at the time of the 2014 election to confirm contact methodology. A secondary phase will then soon follow with an additional sample of 40,000. The total number of potential participants to be contacted is 45,000. It is anticipated that the survey will yield a response rate of approximately 20%, for a total of 9,000 completed surveys.

2.  Procedures for the Collection of Information

The following activities will be conducted as part of this survey data collection effort:

1. Develop a survey of overseas civilians

2. Administer the survey

3. Manage received survey data

4. Analyze the survey data

FVAP will collect data from the registered overseas U.S. civilian population through a mixed mode, mail and on-line survey of a sample of registered civilians living overseas at the time of the 2014 general election. Conducting such a survey will allow FVAP to collect specific, accurate information on voting relevant demographic variables (such as age, gender, race/ethnicity, education, and mobility) to make the comparisons between the overseas, domestic, and ADM populations that are so important to FVAP’s mission. This collection will also provide important information on awareness and voting behaviors that might help address complex and/or unexpected findings within and across countries. Specifically, this data will address overseas voter awareness of FVAP, use of and satisfaction with FVAP services and other resources that these voters use. The survey instrument is designed to parallel FVAP’s Post-Election Survey of ADM members and the Census’ Current Population Survey (CPS), facilitating FVAP’s ability to compare the registration and voting behavior of the overseas U.S. civilian population, domestic CVAP, and ADM.

The survey will utilize a mixed-mode approach in which respondents will receive an initial mail contact directing them to a web site where they can complete an online survey. Respondents who do not respond to the online survey will then be sent a paper copy. This mixed-mode design¹ has significant benefits over soliciting potential respondents by email. Specifically, it ensures that all registered U.S. civilians living overseas have an equal probability of being contacted and having the potential to participate, rather than just those with a listed email address and Internet access; as a result, this increases the likelihood that the final sample of respondents will be more representative of this population with respect to Internet use/access. In addition, costs can be high for printing and mailing paper surveys, especially to an international population; pushing respondents to complete an online survey will lower costs by limiting the number of paper surveys that are sent out.²

The communication strategy will be a two-pronged approach involving up to four international mailings and four email messages per participant, as well as a follow-up postcard sent to a participant’s domestic address for cases in which the international mailings do not yield a response.

The physical materials sent to each voter’s overseas address will include:

1. A cover letter including the URL for the online survey.

2. A second, similarly worded cover letter including the survey URL.

3. A cover letter including the survey URL and a paper copy of the survey with return postage.

4. A fold-over postcard reminder including the survey URL.

If these mailings do not result in a survey response or other contact, a follow-up postcard will be sent to the individual’s domestic address of record. It is anticipated that this will increase the overall response rate by including responses from some individuals who were residing overseas during the 2014 general election but have subsequently returned to the United States.

The email contacts will correspond to the physical mailings, using similar wording and design choices. They will include:

1. An initial email message including the URL for the online survey.

2. A second, similarly worded email message including the survey URL.

3. A third email message including the survey URL.

4. A short reminder email message including the survey URL.

The physical and electronic mailings will be sent concurrently. The second contact will be sent seven days after the initial invitation, the paper survey will be sent ten days after that, and the reminder postcard will be sent fourteen business days following the paper copy of the survey. Once a respondent has completed the survey, no further communications will be sent.

Because the survey frame will not include email addresses for all overseas registered voters, the post survey analysis will appropriately weight the email and non-email samples to control for potential error introduced by this two-pronged approach. Using all means of contact available will help mitigate the effects of uncertain overseas mail delivery and other obstacles that may arise from trying to contact a population that has been historically difficult to survey.

The first phase of the survey will be administered to a sample of 5,000 respondents, using the actual survey instrument and administration plan, which will test the effectiveness of three different incentive plans (described in detail below) to ensure that the most effective method of generating responses will be employed in the full survey. The second phase of the fielding will proceed using the incentive plan identified as most effective. The incentive (if any) will be included with the initial invitation letter to each respondent. During the second phase of the fielding, an invitation to participate in a web-based survey will be sent to a sample of 40,000 additional overseas voters who were registered during the 2014 election cycle, for a total of 45,000 survey participants.

1. Estimation procedures.

Sampling probabilities will be known at each stage of the sampling process and will be incorporated into the post-stratification weights used for estimation. Sampling design decisions, including the decision of how to allocate the sample across different domains, are key aspects of a study design and have meaningful implications for survey estimation, often involving competing goals and tradeoffs among options. The sampling plan for this study cannot be created yet because the survey frame consisting of absentee voters in the November 2014 general election has not yet been assembled. However, the main decisions and priorities are described below.

In this study, one pair of competing goals involves the desire to make estimates of sufficient precision at the country level, while also aiming to produce accurate estimates overall. Thus, although a proportional allocation of sample across countries could lead to more accurate estimates overall, it will be necessary to use a form of stratified sampling in order to achieve necessary levels of precision in smaller countries. Further, given the interest in examining the impact of State-level voting policies, the sampling plan will take State into account in order to ensure sufficient sample size for comparisons across groups of States according to differences in absentee voting policies.

Additionally, the initial phase of the survey may reveal that individual-level voter demographics and vote history may be associated with different response rates. If this is the case, in order to decrease the overall variance of weighted estimates, the sampling plan for the full survey will be modified to allocate a higher proportion of the sample to individuals whose survey participation will result in more precise final estimates.

Thus, for this study, variables that will be used in sample stratification are geography (e.g., country of mailing address and State in which the voter is registered) and voter characteristics as measured on the voter file (e.g., demographics and vote history).

After the survey data has been collected, full-sample weights will be developed. At a minimum, these weights will account for differential selection and response rates across strata.

Sample weighting will accomplish the following objectives:

• Compensate for differential probabilities of selection among sample members.

• Reduce biases occurring because the characteristics of non-respondents may have been different from those of the respondents.

• Improve the precision of the survey-based estimates.³

After data collection is complete, experienced project leaders will oversee and direct each analysis (including planning and execution) to ensure proper procedures are applied, utilize reasonable analytical assumptions, and apply the correct approach given the characteristics of the data and objectives of the analysis. All analysis will be repeated by a second analyst to ensure error-free results. Some of the approaches used will include:

• Regression analysis (OLS, 2SLS, logit/probit, etc.).

• Machine learning model building (CART).

• Geographic analysis including thematic and LISA maps.

2. Degree of accuracy needed for the Purpose discussed in the justification. The goal of using the survey data is to support an analysis of voting rates similar to that applied to the comparison between the ADM and domestic civilian population using the ADM Post-Election Survey. For a comparison between overseas and domestic civilians to have a similar degree of reliability, the number of survey completes for this survey should be similar to the number of completes for the ADM Post-Election Survey. Consequently, the aim will be to obtain approximately 8,000 completed surveys. Assuming a 20% response rate (with the use of incentives)⁴, this implies that approximately 40,000 total survey invitations should be sent in the second stage of survey fielding.

To calculate the number of surveys necessary for the first wave of the survey, a preliminary power analysis has been conducted to determine the number of observations necessary to identify differences in voting and response propensity between respondents subject to different incentive regimes. Based on this analysis, approximately 1,000 completed surveys should facilitate a reliable analysis of the performance of different incentives in the first phase of the survey. Assuming response rates of approximately 15% for no incentives and 20% for potential respondents subject to incentives, this corresponds to approximately 5,000 total surveys that will be sent in this phase. Added together with the survey invitations necessary for the second phase, this yields a total of 45,000 survey invitations that will be sent out.

3. Unusual problems requiring specialized sampling procedures.

Not applicable

4. Use of periodic or cyclical data collections to reduce respondent burden. N/A

3.  Maximization of Response Rates, Non-response, and Reliability

Like all surveys, there is concern that non-response, both to the survey as a whole as well as to specific questions, will undermine the accuracy of estimated voting propensity of the overseas registered population, and thus the validity of any comparisons made with the CVAP. Low response rates could undermine the accuracy of estimates of participation propensity by increasing sampling variability and thus uncertainty in how well the sample reflects the greater population. This problem would be reflected in larger confidence intervals around the estimates. More problematic, if non-response is systematically related to unobserved demographic or geographic characteristics that are also relevant to voting, is that the estimated model of participation may be invalid even if the final sample is large. In fact, the IRS’s 2011 Survey of Individuals Living Abroad, which is most comparable to our proposed effort, did not use incentives and achieved an overall response rate of 32%.⁵ This response rate is well below the 80% recommended by Office of Management and Budget (OMB).

A common method of increasing response rates in surveys is to provide incentives during the initial contact of the potential respondent and/or on condition of completing the survey. The use of incentives in increasing response rates is well-documented in the literature.⁶ The incentive can take the form of a small cash payment or an in-kind gift. Incentives typically act as a gesture of goodwill and thus induce a socially favorable response and a sense of obligation in the recipient. Unfortunately, there is little research assessing the effectiveness of incentives in raising response rates in international surveys, and what little empirical work that has been done on the topic has reported inconsistent results.⁷

The symbolic nature of incentives and varying perceptions across cultures make it difficult to predict how overseas absentee ballot requesters will respond to specific types of incentives. Existing research has evaluated several types of incentives, including cash⁸, commemorative stamps⁹, and bookmarks¹⁰; these studies took place in several different countries and generally targeted participants who were not U.S. citizens. For this reason, the first phase of the survey will be used to assess the effect of cash and in-kind incentives on response for this particular population.

There is reason to believe that the effectiveness of an incentive scheme in inducing response will vary to a greater degree in a multicountry survey than in a survey of a population in a single country. Specifically, cash incentives in the form of U.S. dollars may be of little value to individuals who live in countries that use another currency. In addition, the potential cultural and political diversity of the overseas U.S. population that results from the population being spread over many different countries means that the effectiveness of in-kind incentives may vary based on host country. Finally, in countries whose mail systems are less reliable, there is a concern that mail may be tampered with and the incentive may not reach the potential respondent, thus raising cost with limited potential benefit.

Due to this uncertainty, the first phase of the survey will be used to determine which incentive to use in the second, larger phase of the survey. Based on the previously cited research, three incentive options will be tested. Each potential respondent contacted in the initial phase of the survey will be assigned to one of the three incentive conditions, with each type of incentive being sent to one-third of the potential respondents. The sampling plan for this phase will be created in such a way as to create a balanced sample across each incentive plan so that differences in the observed response rates for each incentive will be not be attributable to differences in observable demographic characteristics among individuals selected to receive each one. This will allow survey administrators to explore the effects of different incentives on response rates overall, determine whether certain incentives minimize nonresponse error more than others, and observe whether respondents in certain countries or regions are more likely to have their mail tampered with and the incentive removed.

4.  Tests of Procedures

Online survey logistics: The online survey will be thoroughly tested by employees of the contractor who will enter at least 20 test surveys through the web-based portal. Hard copies of the survey responses entered will be compared to responses captured by the web-based technology. Should the data check reveal errors, necessary changes will be made to the web-based technology and the checks will be conducted again until 100% accuracy is achieved.

Incentive plans:

As described above, three incentive plans that will be tested in the initial phase of the survey, in which invitations will be sent to 5,000 potential participants. The incentive conditions include:

• No incentive

• Monetary incentive (approximate value $2)

• FVAP-branded magnet (approximate value: $0.50)

A preferred incentive program will be chosen based on both the observed response rates and the representativeness of the sample. If two or more incentive plans yield equivalent results, the most cost-effective option will be used (including the control condition of no incentive). The full survey administration will utilize only this chosen incentive in its mailings to the remaining 40,000 individuals in the sampling frame. The average value of the gift or incentive sent will therefore be less than or equal to $2 per potential respondent.

Communication strategy: During the initial phase of the survey, the timing of the mailing plan will be evaluated by observing when the earliest participants from each country or region complete the survey. If the specified periods between mailings do not allow enough time to ensure that participants receive consecutive mailings in the intended order, the period will be extended as necessary.

5.  Statistical Consultation and Information Analysis

1. Provide names and telephone number of individual(s) consulted on statistical aspects of the design.

Dr. Brian Griepentrog
Director of Research

Fors Marsh Group
Phone: 571 858 3798

Email: [email protected]

2. Provide name and organization of person(s) who will actually collect and analyze the collected information.

Dr. Brian Griepentrog
Director of Research

Fors Marsh Group
Phone: 571 858 3798

Email: [email protected]

1Lonna Rae Atkeson, Alex N. Adams, and R. Michael Alvarez, “Nonresponse and Mode Effects in Self- and Interviewer-Administered Surveys”, Political Analysis, published online May 28, 2014, doi: 10.1093/pan/mpt049.

2 Approximately 60% of participants in the IRS’ mixed mode survey of international tax filers completed their survey online. Tiffanie N. Reker, David C. Cico, and Saima S. Mehmood, “Taxpayer Experience of Individuals Living Abroad: Service Awareness, Use, Preferences, and Filing Behaviors”, http://www.irs.gov/pub/irs-soi/12rescontaxpayexperience.pdf

3Skinner, C. J., Holt, D. & Smith, T. M. (Eds.). (1989). Analysis of complex surveys. New York: Wiley.

4 Based on prior unpublished work by FMG and its partners with general adult populations

5 It should be noted that past surveys of the overseas citizen population conducted by FVAP and the Census Bureau between 2004 and 2008 do not serve as useful comparisons for the present effort as they did not have well defined frames and used vastly different methodologies.

6 Singer, E., & Ye, C. (2013). The use and effects of incentives in surveys. The ANNALS of the American Academy of Political and Social Science, 645(1), 112-141.

Coughlin, S. S., Aliaga, P., Barth, S., Eber, S., Maillard, J., Mahan, C., ... & Williams, M. (2013). The effectiveness of a monetary incentive on response rates in a survey of recent US veterans. Survey Practice, 4(1).

Church, A. H. (1993). Estimating the effect of incentives on mail survey response rates: A meta-analysis. Public Opinion Quarterly, 57(1), 62-79.

Fox, R. J., Crask, M. R., & Kim, J. (1988). Mail survey response rate a meta-analysis of selected techniques for inducing response. Public Opinion Quarterly, 52(4), 467-491.

7 Keown C.F. (1986) “Foreign Mail Surveys: Response rates using monetary incentives.” Journal of International Business Studies 16: 151-153.

8 Armstrong J.S. and Yokum J.T. (1994) “Effectiveness of Monetary Incentives: Mail Surveys to Members of Multinational Professional Groups.” Industrial Marketing Management 23: 133-136.

9 Dawson S. and Dickinson D. (1988) “Conducting International Mail Surveys: The Effect of Incentives on Response Rates with an Industry Population.” Journal of International Business Studies 19(3): 491-496.

10 Jobber D., Mirza H., and Wee K.H. (1991) “Incentives and Response Rates to Cross-national Business Surveys: A Logit Model Analysis.” 22(4): 711-721.

4