Memo to OMB on Supplemental Fraud Supplement

U.S. Department of Justice

Office of Justice Programs

Bureau of Justice Statistics

Washington, D.C. 20531

MEMORANDUM

To: Jennifer Park

Office of Statistical and Science Policy

Office of Management and Budget

Through: Melody Braswell

Clearance Officer

Justice Management Division

Jeri M. Mulrow

Acting Director

Bureau of Justice Statistics

From: Rachel Morgan and Lynn Langton

Bureau of Justice Statistics

Date: March 2, 2017

Re: BJS Request for OMB Clearance for Cognitive Testing of the Supplemental Fraud Survey (SFS) under the NCVS Generic Clearance Agreement (OMB Number 1121-0325)

The Bureau of Justice Statistics (BJS) requests clearance for cognitive interviewing tasks under the OMB generic clearance agreement (OMB Number 1121-0325) for activities related to the National Crime Victimization Survey Redesign Research (NCVS-RR) program. The requested set of cognitive interviewing efforts will focus on finalizing the screener and incident report questionnaires which comprise the full NCVS Supplemental Fraud Survey (SFS) and measure individual financial fraud victimization. The screener and incident report will ultimately be administered to all NCVS survey respondents 18 years of age and older. The primary purpose of the screener section of the SFS is to measure the prevalence of a range of different types of fraud, whereas the incident report will capture additional details about the consequences of and victims’ reactions to specific fraud victimization experiences.

Once the instrument has been finalized through these cognitive testing approaches, it will be administered as a supplement to the National Crime Victimization Survey. OMB approval for the full administration of the SFS will be sought later in 2017.

Under this clearance, the full SFS instrument will be tested using face-to-face cognitive interviewing techniques. Face-to-face cognitive testing will be conducted iteratively with up to 90 persons over multiple rounds of testing, including interviews with financial fraud victims and non-victims. Two versions of the instrument (versions 3 and 4) will be tested with up to 45 persons each; however, if it becomes clear that one version is working significantly better than the other, it is possible we would start administering that version only, but without exceeding the total of 90 interviews. Several versions of the screener were tested via an online crowdsourcing platform (discussed in detail below) under a prior OMB generic clearance. Because of the apparent strengths and weaknesses, it was determined that these two versions of the instrument should be further tested in person. The purpose of this face-to-face cognitive testing is to ensure that the questions are accurately identifying victims of different types of financial fraud, while screening out those with negative financial experiences not rising to the level of fraud. The testing will also ensure that the incident report questions are understood and are appropriate for victims of different types of financial fraud.

Background on the Project and Instrument Development

Financial fraud is a major problem for individuals and for society, but our understanding of the scope of the problem has been hampered by a lack of valid, national statistics. Key sources of crime statistics in the United States, including the NCVS and the Federal Bureau of Investigation's Uniform Crime Reports, have historically focused on traditional property crimes like burglary and larceny and have not attempted to measure the prevalence of fraud.

One of the impediments to the inclusion of fraud in national data collections on crime has been the lack of a clear definition for the term “fraud.” Because no systematic categorization existed, researchers and practitioners often classified fraud types based on different characteristics, including communication method (e.g., cyber fraud, mail fraud), product marketed (e.g., lottery fraud, securities fraud), strategy employed (e.g., advance fee fraud, overpayment fraud), group targeted (e.g., elder fraud), and/or fraudster characteristics (e.g., employee fraud, occupational fraud). This led to a proliferation of overlapping and often confusing definitions and categorizations that hampered the generation of valid fraud prevalence estimates as well as the understanding of the mechanisms and consequences of fraud.

To address the need for a fraud classification system, the Financial Fraud Research Center (FFRC), a joint project of the Stanford Center on Longevity and the FINRA Investor Education Foundation (FINRA Foundation), collaborated with BJS to develop a standardized fraud classification scheme. The purpose was to group and organize fraud types meaningfully and systematically into a definitional framework that could be translated into survey questions that could be administered as a supplement to the NCVS.

The taxonomy was reviewed by an extended review panel consisting of a wider scope of fraud and measurement researchers and practitioners. Input from the extended review panel helped refine the taxonomy by addressing potential areas of overlap or confusion. As a final validation step, to assess comprehensiveness and applicability, the taxonomy was tested using consumer complaint data from the Federal Trade Commission’s (FTC) Consumer Sentinel Network database. Three-hundred consumer fraud complaint cases were classified using the taxonomy coding scheme. This validation step using FTC data identified gaps in the taxonomy and areas where clearer definitions were needed. The objective was to ensure that the taxonomy captured the full range of common scams perpetrated against consumers and that the definitions reflected consumers’ actual experiences. Based on the consumer complaint data, parts of the taxonomy were reorganized and amended with additional fraud types. The final report and taxonomy are available at: http://fraudresearchcenter.org/2015/07/framework-for-a-taxonomy-of-fraud/.

Using the taxonomy as the basis for instrument development, BJS, working in collaboration with the FFRC, developed an instrument to measure the key categories and attributes of financial fraud. The resulting instrument was designed to measure the annual prevalence of seven general types of financial fraud – consumer investment fraud, consumer products and services fraud, employment fraud, prize and grant fraud, phantom debt collection fraud, charity fraud, and relationship and trust fraud – and to capture more detailed information about the fraud incident experienced most recently, including:

• Information needed for coding detailed fraud types based on the taxonomy

• Mode of initial contact

• Method used for transferring funds

• Monetary losses

• Victim reporting behaviors

The FFRC used the instrument to conduct their own cognitive testing in 2015, and to administer the survey to approximately 2,000 web-based respondents in early 2016. The FFRC study found a much higher prevalence of fraud than anticipated based on prior research. However, the study also included a narrative option in the web-based survey that allowed respondents to provide a description of what had happened to them. The narratives revealed that a large proportion of respondents who responded affirmatively to the screening questions about fraud did not in fact experience something that would rise to the level of criminal fraud.

Results of Cognitive Testing via Crowdsourcing

In September 2016, BJS revised the SFS screener instrument to address the type I errors identified through the FFRC’s web survey and obtained OMB approval to cognitively test the new version using crowdsourcing techniques. From October 2016 through February 2017, three iterative rounds of crowdsourcing were conducted with a total of 300 respondents. Round 1 was tested with 150 respondents, round 2 with 75 respondents, and round 3 with 75 respondents. The results of this crowdsourcing informed the screener versions included in this clearance.

The first version of the screener included questions on negative financial experiences not rising to the level of fraud to allow respondents to report on these experiences separately from the items used for fraud estimation. This version was tested in round 1 with 150 respondents (version 1 screener). The screener used a top down approach for estimation and respondents were asked specific behavioral questions measuring the seven general types of financial fraud (level 2 in the taxonomy) – consumer investment fraud, consumer products and services fraud, employment fraud, prize and grant fraud, phantom debt collection fraud, charity fraud, and relationship and trust fraud. Respondents were also asked about negative financial experiences and identity theft victimization; these questions were included so we could determine if respondents were excluding these experiences when asked the fraud questions. If respondents endorsed any of these behaviors, they were asked to provide a few sentences describing their situation. These narratives were helpful when determining if a particular experience constituted fraud.

Findings from this round indicated that respondents often experienced negative financial experiences and that the distinctions in question wording between the negative financial experience questions and fraud questions were not clear enough. The narratives suggested high levels of false positive responses to the fraud items. In an attempt to reduce the false positive responses, follow-up questions were added after each screener to further refine the measures based on whether the respondent was reimbursed for the losses by the person or company involved in the potential fraud.

Round 2 of crowdsourcing tested this revised screener with 75 respondents (version 2 screener). Respondents were administered the behavioral fraud questions included in round 1 along with follow-up questions asking if they received any of their money back following this experience or were still in contact with the person or entity that took their money. These follow up items are intended to capture the legal definition of fraud, which requires demonstration that there was an explicit intent to deceive for monetary gain. Follow-up questions were only included with the behavioral questions measuring fraud and not with the questions focused on negative financial experiences or identity theft victimization. As with round 1, the round 2 screener utilized a top down approach to estimation and would allow BJS to produce prevalence estimates for the fraud types at level 2 in the taxonomy and summing all level 2 fraud types would allow for a comprehensive estimate of financial fraud.

Overall, the round 2 screener performed well and it appeared that the follow-up questions narrowed the scope of the types of experiences that were considered fraud. However, in some instances the follow-up questions also appeared to screen out cases of fraud that should have been included.

Round 3 of crowdsourcing tested a different screener than round 2. The round 3 screener was tested with 75 respondents (version 3 screener). Questions in this screener ask about more general experiences with fraud rather than specific types of fraud. Ultimately, the round 3 instrument will allow for estimates of certain types of fraud on the taxonomy but an overall measure of personal financial fraud would be limited to the summation of the certain types being measured. This screener would not produce a comprehensive estimate like the round 2 screener.

To ensure that respondents are reporting incidents that rise to the level of fraud, the follow up questions measuring the legal criteria for fraud were again included in the round 3 screener. From a legal perspective, if the company returns the individual’s money or if the individual never tried to get it back in the first place, it is not possible to demonstrate that the offender intended to defraud the victim. Correspondingly, the follow up items ask whether the victim was reimbursed by the person or company and if not, whether he or she tried to get their money back. These follow up items eliminated the need for the questions about negative financial experiences.

Overall, the round 3 screener worked equally well as the round 2 screener but demonstrated some evidence that the follow up items could be reducing type I error but potentially introducing type II error. Additional face-to-face person cognitive testing of this screener version will clarify the extent to which this is or is not occurring.

After these three rounds of crowdsourcing were completed, a new version of the screener was developed as a hybrid of round 2 and round 3 to maintain the focus on specific categories of fraud in the screener. This approach addressed the challenges of moving respondents from the screener to the crime incident report when negative financial experiences were also included in the screener in addition to fraud victimization. The new screener version of the instrument is known as version 4. Because the version 4 screener asks about experiences with particular categories of fraud, it was possible to eliminate the follow-up items for certain types of fraud in which solely endorsing the screener item should be sufficient for classifying an individual as a fraud victim. For instance, if a victim donated money to a charity and later found out that the charity never actually existed, the victim experienced charity fraud and it is not necessary to ask whether he or she got or tried to get the money back. The version 4 screener, in addition to the version 3 screener, and their corresponding incident report questionnaire will be tested during the face-to-face interviews requested in this clearance.

Current Request for Cognitive Testing

In the current request, we are asking for clearance to conduct face-to-face cognitive interviewing with up to 90 respondents (up to ~45 respondents for version 3 and up to ~45 respondents for version 4). The cognitive testing will be used to test the full SFS instrument, the revised screener instrument and incident report questionnaire. All interviewing will take place in March of 2017 and will be completed in sufficient time to inform the full OMB clearance for the 2017 NCVS SFS. The version 3 and version 4 protocols are included as appendices.

Face-to-face cognitive interviews will be conducted with up to 90 respondents. The target population for the face-to-face cognitive interviewing is persons 18 years of age or older living in Raleigh/Durham, North Carolina; Charlotte, North Carolina; Washington, D.C, and Portland, Oregon. The only eligibility criteria are that the participant be 18 or older and speak English. All in-person interviews will take place either at an RTI office or another private location agreed upon by the interviewer and the participant. One-on-one interviews will last approximately 1 hour to allow time for the administration of the screener and incident report questionnaire, as well as cognitive probing. Participants will receive $40 as compensation for their time and to offset the cost of their participation, such as transportation, parking, and childcare. Prior cognitive testing experiences and research suggest that $40 is an effective incentive amount as it attracts a wide diversity of respondents.

The recruitment procedures for the face-to-face cognitive interviewing are designed to ensure that participants include victims of financial fraud (who have experienced individual financial fraud within the past 12 months).

Recruiters will advertise the study to solicit participation, using internal recruiting databases and Craigslist ads. Recruiters will adapt recruiting strategies as needed to ensure adequate participation. Persons selected to participate will be contacted by the recruiters and scheduled for their interview session. The target number of up to 90 respondents was identified as a number that allows for variation among respondents based on sex, race/ethnicity, age and different types of fraud experiences.

Language

The face-to-face cognitive interviews will be conducted in English.

Burden Hours for Cognitive Testing

The burden for this task consists of participants completing the SFS instrument via face-to-face interview with an RTI International interviewer. The burden associated with these activities is presented in the following table. BJS is requesting a maximum of 90 burden hours with this clearance.

Burden Associated with SFS Cognitive Testing Activities

	Number of Respondents	Administration Time (minutes)	Burden (hours)
Face-to-face Cognitive Interviewing	90	60	90.0

Cost to the Federal Government

Participants will be given a $40 incentive for their participation in the SFS interview. The costs for RTI to assist in the development of the interview protocol, to oversee and conduct the interviews, and to analyze and report on findings from this cognitive testing will be approximately $25,000. Thus, the total cost for this cognitive testing will be about $28,600 ($25,000 + ($40 incentive * 90 interviews)).

Reporting

Upon completion of all cognitive testing, a draft cognitive interviewing report will be delivered to BJS that will include recommendations for final revisions to the SFS survey instrument. These recommendations will provide detailed information on the cognitive testing methodology, basic characteristics of the respondents, average time needed to complete the screener instrument and narratives, and any issues with question comprehension noted by respondents. The report will also document changes made to the initial draft survey prior to the cognitive testing and all changes made during the cognitive interviewing process, if any.

Protection of Human Subjects

There is a slight risk of emotional distress for the respondents given the somewhat sensitive nature of the topic, since the questions are of a somewhat personal nature; however, appropriate safeguards are in place and the planned cognitive testing has been reviewed and approved by RTI’s Institutional Review Board (IRB), which has Federal-wide assurance.

Informed Consent, Data Confidentiality and Data Security

Informed Consent

At the beginning of the face-to-face cognitive interview appointment, the respondent will be handed a hard copy of the informed consent form and the interviewer will review the highlights of the informed consent form as the respondent follows along. If the respondent wants to proceed with the cognitive interview, they will "X" the appropriate lines to participate, allow for recording and, if applicable, allow for observers. If a participant does not want to consent to audio recording or observation, no recording device will be used and observers will be asked to leave. The interviewer will sign the RTI copy of the consent form and leave a blank hard copy with the respondent. We are not asking respondents to sign the consent form as a measure to protect their confidentiality.

Data Confidentiality and Security

BJS’ pledge of confidentiality is based on its governing statutes Title 42 USC, Section 3735 and 3789g, which establish the allowable use of data collected by BJS. Under these sections, data collected by BJS shall be used only for statistical or research purposes and shall be gathered in a manner that precludes their use for law enforcement or any purpose relating to a particular individual other than statistical or research purposes (Section 3735). BJS staff, other federal employees, and RTI International staff (the data collection agent) shall not use or reveal any research or statistical information identifiable to any specific private person for any purpose other than the research and statistical purposes for which it was obtained. Pursuant to 42 U.S.C. Sec. 3789g, BJS will not publish any data identifiable specific to a private person (including respondents and decedents). The cognitive interviewing methodology will not be collecting any personally identifying information from respondents.

5