CTAS Survey Methodologies

2019 CTAS Phone and Online Survey Methodologies

(Excerpts from PCG’s 2019 CTAS Work Plan)

Task 4: Collect Attitudinal Data

RAAS has specified their preference to continue the multi-mode data collection methodology, comprised of telephone and online panel random sampling to ensure a representative sample of the general public. PCG has proposed the collection of a total of 2,000 completed surveys, split evenly between the telephone and online data collection methodologies, for a margin of error of +/- 2% at the 95% confidence level. The survey is exactly the same as in 2018 and will be an average of 20 minutes with no open-ended questions.

Research Population

The research population for this study is defined as U.S. adults, aged 18 or over.

Telephone Methodology

The telephone sampling approach will include random digit dialing (RDD) of households with landlines in the continental U.S., augmented by an RDD sample of cell phone numbers in order to capture the opinions of U.S. adults in cell phone only (CPO) households. The phone survey participants are selected via RDD of household telephone numbers from working phone area codes and exchanges and by computer generation of the last 4 digits. Since the RDD sampling system is totally computer-based, it provides an equal probability of selection for each and every telephone household. Thus, the sample represents telephone households with both listed and unlisted phones in their proper proportions.

In order to reflect current usage patterns in the U.S., we will collect 500 interviews via RDD from a traditional landline telephone sampling frame, and 500 interviews via RDD from a sampling frame of cell phone numbers, including cell phone only (CPO) and dual users. The total telephone sample size will be 1,000 completed interviews, with a margin of error equal to +/- 4% at the 95% confidence level.

All telephone interviews will be conducted by PCG’s phone subcontractor who will work under the leadership of PCG’s project manager. All interviewers will be professionally trained in interviewing techniques and in IRS surveys prior to the commencement of fielding, and they will be continuously monitored and supervised throughout the fielding period. The interviewing methodology used will be Computer Assisted Telephone Interviewing (CATI). All data are entered and cleaned through the CATI system during the interviewing process, which eliminates editing and keypunch operations. A screening question ensures that respondents are at least 18 years old. For those agreeing to participate in the survey, interviewers administer the survey questionnaire guided by the computer assisted telephone interviewing (CATI) process. All sample phone numbers selected by the sampling procedure are subject to an initial call, followed by at least four follow-up attempts to complete an interview. Last year, the response rate was 2.1% for landline phones and 3.5% for cellphones. Phone response rates are declining every year, so we expect the response rate to be similar to or a little lower than last year’s response rate.

Online Methodology

In conjunction with telephone data collection, PCG is planning to collect a total online sample size of 1,000 completed online surveys, with a margin of error equal to +/- 4% at the 95% confidence level.

PCG will program a standard online version of the questionnaire. The survey ‘look and feel’ will have a professional design as used in the 2018 survey, and will not identify the IRS as the project sponsor. The online survey will be thoroughly tested internally at PCG to ensure the programming reflects the survey content (questions, response options) as well as any skip logic. These skip patterns are built into the survey program code and are tested along with the survey content. PCG will test each branching of the skip patterns to ensure that the question sequence is programmed correctly. A test survey link will then be shared with the RAAS team for review and comments. PCG will revise the online program as needed based on RAAS feedback. PCG will also send the email invitation text to RAAS for review.

In addition to the standard version of the programmed survey, PCG will also program a second version of the survey that will conform to the technical standards for web-based Intranet and Internet Information and Applications of Section 508 of the Rehabilitation Act of 1998. Having the two versions ensures the best user experience for those reading the survey themselves and for those using JAWS software to read the survey to them. This survey version will also be thoroughly tested internally at PCG, before providing RAAS with a survey link for their own testing process.

In order to achieve survey data that is representative of the U.S. adult online population, PCG has subcontracted with Ipsos to provide the online sample from their probability based online panel, KnowledgePanel^® (Please see Appendix). This panel uses an Address-Based Sampling (ABS) methodology which is random-by-mail to recruit members and each quarter a stratified random sample of addresses is selected to replenish the panel. The sampling frame used is the universe of all U.S. residential addresses secured from the latest Delivery Sequence File of the U.S. Postal Service.

Ipsos will send out survey invitations to the panel respondents via email containing a link to the survey on the PCG server. Ipsos will send invitations and reminders about a an “opinion survey” with no details regarding IRS sponsorship. We expect a completion rate of at least 50% for this survey (the completion rate in 2017 was 64%). The survey has been confirmed as non-FTI (Federal Taxpayer Information), it is completely anonymous, and no individual respondent will be providing personally identifiable information.

Task 5: Complete Basic Analyses

Weighting Strategy

PCG will follow the same weighting strategy as was utilized in the Base Period. The survey data from each data collection mode will be weighted separately to allow for analysis of each sample separately, and comparatively. The two samples will also be combined, and an additional ‘blended’ weight will be used.

The entire telephone sample, consisting of landline and cell phone samples, will be weighted on demographics and telephone usage variables to demographically represent the adult U.S. population according to the most recent national population estimates. This weighting scheme will account for dual cell phone and landline users.

The online sample will be weighted using demographic variables only.

In merging the telephone and online survey datasets, PCG will use a blended weighting to achieve a final dataset that matches the U.S. adult population on CPS-based demographic variables. This weighting will be based on demographic variables as well as behavioral/attitudinal questions about internet and television use as well as attitudes about trying new products.

Analytical Planning

PCG will review an interim survey dataset in the latter stage of fielding. The dataset will be cleaned and basic frequencies for each question will be generated. A review of this initial analysis in addition to our experience analyzing last year’s data will lead to PCG’s analysis and reporting plan for the final dataset(s). PCG will share this plan with RAAS and a joint-team meeting will occur to discuss the proposed plan and whether it sufficiently meets RAAS’s needs.

2