FIST_OMB_Supporting_Statement_Part_B_Statistical_Methods_REVISED_FOR_2012_revised

2. Statistical Methods

UPDATE FOR 2012 COLLECTION

Supporting Statement B. for FIST ICR reference # 201010-1121-002 has been revised to document the non-substantive changes approved by OMB as outlined in the memorandum dated May 23, 2013. The current OMB approval for the FIST data collection expires in February 2014. BJS will be prepared to submit the requisite clearance package in advance of the expiration to avoid a delay to the data collection schedule for Calendar Year 2013 data.

1. Respondent Universe

Data for this collection will be collected through a voluntary survey administered to state and local reporting agencies that conduct background checks for firearm purchases and transfers. Currently, there are 30 state agencies and over 2,900 local agencies that conduct background checks on persons who apply to purchase a firearm or for a permit that may be used to make a purchase. As discussed previously, each state government determines the extent of its involvement in the NICS process. States may operate as a full POC that requests a NICS check on all firearm transfers originating in the state, as a partial POC that requests a NICS check on all handgun transfers (FFLs are required to contact the FBI for NICS checks for long gun transfers), or as a non-POC in which case FFLs are required to contact the FBI for NICS checks on all firearm transfers originating in the state. The data collection agent will maintain a list of state and local agencies that conduct background checks and their associated functions in the NICS process.

For this data collection, the only factor that would result in an agency’s ineligibility to participate in the survey is when it is no longer authorized to conduct background checks for firearm purchases. Agencies that are ineligible to participate in the survey due to changes in their background check reporting functions are removed from the sample and are not included in the nonresponse rate. The nonresponse rate is composed only of eligible agencies that have elected to not participate in the survey.

To estimate the application and rejection rates within a given area, state and local checking agencies will be stratified by size of the population served: state agencies that served an entire state population; local agencies that served a population greater than 100,000; local agencies that served a population between 10,000 and 100,000; and local agencies that served a population of less than 10,000. Population size is based the most current information from the Census Bureau, and the population categories were chosen to be consistent with those used by the FBI when conducting similar studies. To determine the agency population, the stratification classification of the county will be based on the size of the largest city within the county. If cities within a county were conducting their own background checks, their populations will be subtracted from the county population. If a municipal agency provided services for other selected municipalities, then populations from those municipalities will be added to the populations of the reporting municipalities. If an agency relied upon other jurisdictions to conduct background checks, they will be replaced by those other jurisdictions.

In 2009, the sample for the FIST survey was selected from the total population of 30 statewide agencies and over 2,900 local checking agencies across the nation that conducted background checks for firearm purchases and transfers. A total of 816 agencies were surveyed, including all 30 statewide agencies and a stratified random sample of local agencies. Overall, 559 agencies provided data for a response rate of 69%.

Background Checks for Firearm Transfers, 2009

Population Served by Agency	Total	Sample	Responses	Response Rate
Total	2,951	816	559	69%
Statewide	30	30	30	100%
Under 10,000	1,656	364	236	65
10,000 to 100,000	1,155	373	264	71
Over 100,000	110	49	29	59

UPDATE FOR 2012 COLLECTION

Revised Sampling Frame

The sample of local checking agencies will be redrawn for the 2012 FIST collection. REJIS spent a significant portion of time updating the FIST universe to ensure that all agencies that are authorized to conduct and are known to conduct background checks are accounted for in the universe (see Attachment II). BJS and REJIS worked extensively to develop the most appropriate sampling plan that will produce a reliable national estimate while staying within the confines of the budget allocated for the FIST data collection. The new sampling plan will also enable the creation of state-level estimates, which have become a source of increased public interest. After considering multiple methodologies, BJS has decided to employ an enumeration of local agencies in eight of the twelve states (DC, DE, IA, ID, MT, NC, NV, and NY) in which local checking agencies are responsible for conducting background checks. Agencies in the remaining four states with local checking agencies (GA, MN, NE, and WA) responsible for conducting background checks will be sampled due to the relatively greater number of agencies in these states compared to the other eight. In these four states, the decision was to create a stratified sample based on population size that roughly equates to Category 1) rural – places of less than 10,000 population; Category 2) small cities – places of between 10,000 and 99,999 population; Category 3) small metropolitan areas – places of between 100,000 and 199,999 population; and Category 4) large metropolitan areas – places of 200,000 or more population. These population categories were selected to be consistent with definitions of various census place levels. Sampling will be done for agencies in Categories 1 and 2, while the agencies in Categories 3 and 4 will be kept in the sample as certainty (see Attachment III).

A sample size of 791 is needed to implement this sampling design with the appropriate precision to calculate a reliable national estimate and improve the overall robustness of the sample. The sample size also stays within the budgetary parameters approved for the FIST data collection. BJS is requesting OMB approval to increase the number of respondents from the currently approved 653 to 791. The impact of the increased number of respondents on the overall burden hours is addressed in Supporting Statement A10.

2. Information Collection Procedures

The information collection will be administered through a survey to state agencies that serve an entire state population and a stratified sample of local checking agencies randomly selected from the total population. Recognizing that this is a voluntary survey, all reasonable efforts will be made to make the data collection process as seamless and convenient as possible for the respondents. The data collection agent will maintain a list of reporting agencies and their preferred data submission methods so that data requests and follow up efforts can be tailored to individual agencies. Respondents may choose to submit their data on paper (fax or regular mail), diskette, or electronically using a fillable pdf (email). The data collected will be summary statistics of an administrative nature, and will not allow for the identification of any individual.

Data from the state and local reporting agencies will be collected through various measures based on the individual agency’s characteristics and procedures, as well as its preferred reporting method in past collections. For both state and local checking agencies, regardless of the way the agency has elected to submit data in the past, a letter will be sent annually to explain the purpose of the information collection and uses of the data obtained, and request the agency’s ongoing participation in the program. All efforts will be made to personalize each letter, and the letters will be prepared for signature by the data collection agent and/or the BJS Program Manager. The letter will also clearly denote that participation in the information collection is voluntary and acknowledge that there are no assurances to confidentiality as the data collected are available in the public domain. The data collection agent will maintain a contact list for respondent agencies and verify the addresses each year to ensure the accuracy of contact information.

State agencies:

The data collection process for state reporting agencies will modified based on individual agency characteristics to make the process as convenient as possible for each respondent. Data will be collected from state agencies through one of three measures: reports, state websites, or survey. Some state agencies choose to regularly submit reports to the data collection agent, from which the agent will extract the relevant data. These reports are routine reports prepared by the state for legislative purposes and contain various data related to criminal justice statistics. Submitting reports to the FIST program is a voluntary decision made by the responding state agency because it determined this was the most efficient means to provide the data. Other state agencies publish data on background check activities on their state website. The data collection agent will maintain a list of state agencies that publish this information online and will make all efforts to collect the information directly from their websites in order to reduce the respondent burden. In these two circumstances, the data collection agent will complete the Excel survey spreadsheet (Attachment III) with the relevant figures compiled from the data published in state reports or online, and fax the completed forms and data request asking the respondent agencies to review and verify the numbers (Attachment IV). The respondents will be asked to revise figures or update missing information directly on the Excel spreadsheet, as applicable, and return the spreadsheet and signed form to confirm the accuracy of the information. The names will be cross-checked against the contact list to ensure that the individual verifying the data has the authority to do so. There will be no duplication of effort required for the state agencies that choose to submit information through these methods, and the associated burden will be the time spent emailing the reports to the data collection agent and/or verifying data accuracy at the end of the reporting period.

For state agencies that do not elect to submit monthly reports and do not publish data online, the data collection agent will fax the appropriate letter (Attachment V) and Excel survey spreadsheet to the reporting agency and request that the agency POC complete the totals for each category and return both the spreadsheet and signed review. Respondents will be encouraged to submit monthly totals, but cumulative aggregate totals by category will be requested if monthly data are not available. Faxing the data request has been the preferred mode of transmission, although the materials can also be mailed to the agency. When mailed, a self-addressed stamped return envelope will be included. Respondents will be asked to return the completed survey to the data collection agent via fax, diskette, mail, or electronically. The same practice will be in effect regarding cross-checking the name of the respondent against the contact list maintained by the data collection agent. The associated respondent burden will be the time spent completing and returning the form to the data collection agent.

In some states, one statewide agency conducts background checks for purchase and another agency (or division within an agency) issues ATF-approved permits. Controls will be put in place to ensure that state populations are not counted twice in the estimation process. This situation of dual agencies conducting background checks does not occur among local agencies.

Local agencies:

The collection methodology for the sample of local agencies will be collected through a tally sheet approved by OMB in 2007 and a survey cover letter explaining the purpose of the information collection (Attachment VI). Two tally sheets (the survey instruments) will be sent via fax or mail to local agencies selected for the survey based on the type of background check they are responsible for completing: one form collects background check data on carry and concealed permits (Attachment VII), and the other collects data on purchase permit statistics (Attachment VIII). The data collection agent will maintain a listing of local agencies and the type of background check each agency is responsible for conducting, and will send the appropriate form(s) based on the function of each individual agency. In all cases, the data collected will be summary counts and will not allow for the identification of any individual.

The content of the tally sheet to collect Purchase Permit statistics has not changed since receiving OMB approval in 2007. The presentation of the form will be modified slightly to accommodate the change to the data collection schedule from twice to once annually to enable respondents to report the annual aggregate totals instead of monthly totals. A second tally sheet will be sent to applicable reporting agencies to collect data on Concealed/Carry Permit statistics, as applicable. The presentation and reporting directions are the same for both tally sheets. Mailing the forms has been the preferred delivery method for local agencies in the past collection, but respondents will also be encouraged to submit the data via diskette or email if these methods are more convenient. The associated respondent burden will be the time spent completing and returning the form to the data collection agent.

FBI and ATF data:

Transaction data from the FBI NICS and data from the ATF Brady Operations on reasons for denial and appeal information, firearm retrievals, and referrals for investigation and prosecution will be collected and integrated with the data received from state and local agencies to complete a comprehensive source of background check data from application to denial to post-denial activities. The data collection agent will continue to extract the relevant data from FBI reports submitted to FIST throughout the collection period. The data collection agent will continue the process of contacting ATF directly to request and compile the relevant data on post-denial activities.

Future directions and Outreach Efforts:

As new technologies become more available to easily and safely transmit information electronically, a goal of the project will be to create and implement a web-based reporting form that enables respondents to enter data online to simplify the record tabulating functions of the agency and further reduce the reporting burden. The data collection agent will also follow up via phone or email email with reporting agencies to verify the accuracy of reported figures prior to submitting the information to BJS, and/or address any questionable figures. Outreach efforts will be made to the responding agencies to encourage participation, answer questions about the survey, and provide technical assistance as needed.

UPDATE FOR 2012 COLLECTION

Revised Survey

As part of BJS’ overall effort to improve the FIST methodology, BJS assessed the approved tally sheet that had been used previously to collect FIST data and decided that changes were necessary to better clarify what data are being requested and format the presentation to be more aligned with other BJS surveys. Copies of the approved tally sheet and proposed revised survey are appended to this memorandum for OMB’s review (Attachments V and VI). The following sections describe the changes that were made to the survey form and underlying rationale for the changes.

Survey format

The revised survey is designed to collect the same data on firearm background check activity for the reference year as the approved tally sheet. The revised survey includes screener questions to assist in maintaining the universe file of agencies that conduct firearm background checks, a section to obtain information about agency characteristics to better understand the sample population and assist efforts to conduct a nonresponse bias analysis if necessary, and a section to identify how agencies collect and report their data. The majority of these additions are in the form of a check box response, which will result in a minimal enhancement to the burden. One of the greatest benefits of the revised survey is that additional clarification and directions are provided to ensure that respondents are clear as to what data are being requested. BJS found that the lack of clear instructions and definitions in the tally sheet was problematic because it introduced potential data quality issues as it may have been unclear to respondents what information was being requested. Given the fact that the sample will be redrawn to include new agencies that have not participated in past FIST data collections, it is imperative that respondent agencies are provided with consistent guidance to improve the quality of data being collected and inform BJS about how state and local agencies collect and report data.

Instructions and definitions were added to each section in addition to screener questions to determine if the respondent agency tracked the requested data. A box was also added to applicable questions so the respondent could check if the response was an estimate to help BJS to better understand the accuracy of the data.

The following enhancements were also made to the survey instrument:

• Page 1: Questions have been added to determine if the respondent agency is currently authorized to and/or has conducted background checks during the reference year. These questions are designed to help maintain the FIST universe of local checking agencies.

• Section I: A section titled “Agency Characteristics” has been added to request information about the types of checks that the local agency conducted during the reference year. In the past, REJIS sent the tally sheet to respondent agencies based on what types of checks they were known to conduct based on state laws. These questions will help to verify the accuracy of and maintain the universe of local checking agencies and help to monitor changes in how agencies conduct background checks.

• Section II: A question (#7) was added to ask if the agency tracked the reason(s) why an application was denied. This question will help to determine which agencies are currently collecting this data. Two questions (#9 and #10) were added to determine how the agency records and reports on reasons for denial. This information is critical to inform BJS’s understanding of the tracking process and ensure that the data can be accurately reported.

• Section III: No new questions have been added apart from the screener questions.

• Section IV: A section has been added to address the reporting of partial year counts. Prior experience with the FIST survey has shown this to be a rare occurrence.

• Section VI: A section has been added to enable the respondent to provide a narrative response about its data collection procedures and/or reporting of estimated totals.

Feedback from testing

In accordance with approved OMB survey testing protocol, the survey was sent to five agencies with a request to complete the revised survey and provide feedback. Of the five agencies, four were local agencies and one was a state agency. The local agencies represented the different population strata (size A, B, and C) of those in the sample. Three local agencies responded: one size A and two size B. Additionally, the revised survey was presented to two subject matter experts who have significant experience in firearm background check activities and procedures for comment. See Attachment VII for a list of pre-test questions asked and feedback received.

Web-Based Form and Implementation Plan

As described in Supporting Statement A10, to satisfy the terms of the OMB clearance, BJS developed a web-based FIST survey that will be employed in the 2012 FIST data collection (Attachment IX).

The introduction of the web-based data collection instrument is intended to encourage participation by providing a secure, convenient mode of responding to the FIST survey and is intended to streamline the survey process by eliminating questions that the respondent may not need to see, thus also reducing response burden. It is also designed to reduce item nonresponse by requiring that a response to critical items is submitted before continuing to the next question. Each FIST participant will be provided a unique User ID and password that will allow secure access to the web form. In order to reduce any differences in response as a result of the mode by which a respondent chooses to participate in the survey (mode effects), particularly mode effects due to the different web and mail response modes, the web form has been designed to mimic the visual presentation of the paper survey as much as possible. The wording of the questions is consistent between the two modes. A detailed description of the web-based survey implementation plan can be found in Attachment X.

Revised correspondence

BJS updated the approved correspondence letters and phone script to include information about the web-based survey option. Copies of the updated communications can be found in Attachment XI.

Data collection strategy in 2011 (for 2010 data)

April to May – The data collection agent will research and update its comprehensive list of state contacts, and will research and confirm any changes in the background check responsibilities of respondent agencies to determine the population universe. The data collection agent will also collaborate with the FBI and ATF to receive relevant data on firearm background check activity and post-denial activities.

June

Week 1: A prenotice letter will be sent via mail, email, or fax based on the contact information available for the agency and the agency’s preferred mode of communication.

Week 2: A survey will be sent via mail, email, or fax based on the agency’s preferred mode of communication. A detailed cover letter will be included to explain the importance of a response and indicate alternate submission modes. A requested response deadline is provided (2 weeks after the survey is sent).

Weeks 3- 4: A thank you note will be sent via mail, email, or fax to express appreciation for responding to the survey, even if the survey has not yet been returned.

The data collection agent will enter data into the project databases as it is received and will continue to review state websites and FBI reports to extract published data. The data collection agent will update contact information for agencies as needed.

July

Week 2: A replacement survey will be sent via mail, email, or fax to agencies that have not yet responded. This will be sent 2-4 weeks after the initial survey is sent.

Week 3: A follow up phone call will be made to agencies who have not responded to the survey (if a telephone number is provided). Otherwise, a request will be made via mail, email, or fax.

The data collection agent will enter data into the project databases as it is received and will continue to review state websites and FBI reports to extract published data. Data verification efforts continue. The data collection agent will update contact information for agencies as needed.

August

Weeks 1-2: The data collection agent will make one final attempt to reach the reporting agency. The mode of outreach will vary depending on the history of past attempts made.

Week 4: Data entry concludes.

The data collection agent will enter data into the project databases as it is received and will continue to review state websites and FBI reports to extract published data. Data verification efforts continue. The data collection agent will update contact information for agencies as needed. Data entry will conclude at the end of August

September to October

Work to produce the estimates will begin. Data processing and analysis continue. Data verification will continue as needed.

November

Final reports and statistical tables will be completed and submitted to BJS for review. Efforts to maintain the master contact list and research state laws for changes in firearm background check procedures continue.

Throughout the data collection process, the data collection agent will maintain a comprehensive record of all follow up and reporting activity and log details of when data is received, from whom, by what means (fax, email, etc.) and applicable changes in address and other contact information. This will be done to ensure that duplicate requests are not made to agencies and that the agency’s preferred mode of submission is noted for subsequent years. Five attempted contacts will be made to each agency before it is considered to be nonresponsive. Specific dates will vary annually depending on holiday and staff schedules. The data collection agent will vary the modes of outreach so the reporting agency receives at least one phone call, one email (if an email address is available), and one fax or letter request.

December to March

BJS and the data collection agent will continue to evaluate the FIST program design and sampling plan. The sample will be redrawn to prepare for the 2011 data collection.

The data collection schedule may be revised and adjusted to accommodate data collection activities based on the selection of the data collection agent for FY 2011.

UPDATE FOR 2012 COLLECTION

Data Collection Schedule

The OMB clearance granted BJS approval to collect calendar year (CY) 2010 data using the same sample used in previous years with the understanding that the subsequent data collection would be conducted with a revised sampling frame. The FY 2011 FIST award to collect CY 2010 data was competitively awarded to REJIS in August 2011. The time associated with competing the award and making a funding decision, coupled with the delayed passage of the FY 2011 budget, resulted in significant delays in initiating the collection. Additionally, BJS spent a significant amount of time reviewing the FIST methodology to both improve the reporting of 2010 data and to determine the sampling plan for future collections. The Background Checks for Firearm Transfers

2010 Statistical Tables publication was published to the BJS website in February 2013.

Due to the delayed start of the 2010 data collection and time spent on addressing methodological issues, BJS was concerned that collecting CY 2011 data would create an unnecessary burden on respondent agencies by asking them to report data so soon after their last submission. Moreover, BJS has prioritized collecting CY 2012 data because it has been a notable year in firearm background check activity, as evidenced by the increase in transaction data reported by the FBI (up from 2010 and 2011), and a high demand for the most recent data on background check activities observed since the school shooting in Newtown, Connecticut, in December 2012.

For these reasons, BJS decided to forgo the collection of CY 2011 data and will collect the next round of data for CY 2012. Pending OMB approval of the changes outlined in this memo, the 2012 data collection is scheduled to be initiated in July 2013 (see Attachment I).

3. Development of Estimates

National estimates of applications and denials will be developed using population weighting factors. When an agency does not provide data for all months, a simple linear extrapolation or interpolation will be used to generate a 12-month total. Agencies with rejection rates over four standard deviations above the average standard rejection rate will be classified as outliers and their data will not be used for projection of estimates. In addition, rejection rates that cannot be determined with sufficient accuracy will not be used.

The accuracy of the estimates depends on two types of errors: nonsampling and sampling. For this data collection, nonsampling error may occur from the following: nonresponse; differences in the methods checking agencies use to process, code, store, and retrieve their information; differences in interpretation of the survey questions; and activities that delay personnel from completing the survey. In any sample survey, the full extent of nonsampling error is never known. However, continued steps will be taken to minimize the potential for error, such as periodic telephone follow-up calls and emails to encourage responses, answer questions about the request, and generally assist in reporting the information in a useable format. Upon receipt of the completed survey, the data collection agent will review the submission and follow up directly with the agency POC to discuss any questionable figures. Respondents will also be asked to return a signed review with their completed survey forms to verify the accuracy of the reported totals.

UPDATE FOR 2012 COLLECTION

National and state level estimates will be made for applications and denials. While most states’ checking agencies will be enumerated, four states (GA, MN, NE, WA) will be sampled with probabilities proportional to size in the first two size categories (with an enumeration of agencies in categories 3 and 4 (see Attachment III for complete details on the size categories and the distribution of agencies within each state). Weighting factors (inverse of an agency’s probability of selection) will be applied to agencies within the relevant sampled states and strata. Should the reserve samples be employed, a different series of weights will be applied to those agencies (initial weight x the probability of being selected into the reserve sample). Nonresponse weighting adjustments will be applied to the initial sampling weights to calculate a final weight for sampled agencies.

4. Procedures to Maximize Response Rates

The overall response rate in 2009 was 69 percent, which was consistent with the rate in 2008 but lower than rates seen in past collections (85 percent in 2005). While 100 percent of state agencies submitted responses in 2009, the number of local agency respondents dropped off from past years. This decrease was due in part to a reduction in the number of local agencies eligible for the survey as a result of changes in policies related to issuing permits. Additionally, as is a challenge associated with any voluntary surveys, there are no tangible incentives for the respondents to submit the data. The higher response rate in past years may also be attributed to efforts made by a marketing research firm previously supported under the grant to make follow up calls to reporting state and local agencies. Due to funding restrictions, the marketing research firm services are no longer employed and agencies are not being contacted as frequently about submitting responses for the FIST survey. Coupled with less consistent outreach by the marketing firm, the change in schedule to a once annual data collection also decreased contact between reporting agencies and the FIST data collection agent, which may have contributed to the lower response rate.

While the overall response rate has declined in previous years, it is important to note that the greatest volume of applications and rejections are transacted by the state agencies. As noted, the response rate for state agencies in the 2009 collection was 100 percent. The responses received by the state agencies drive the volume of responses received and the current response rate is considered to be well above the level needed to be considered statistically significant.

An ongoing priority of this data collection will be to increase the response rate to 80 percent or greater, and a continued focus will be to remain flexible in the ways that data are accepted by both the state and local reporting agencies in order to encourage participation and limit the burden on respondents. To achieve this goal, concerted efforts will be made by the data collection agent to encourage responses from participants, including identifying ways to engage in ongoing outreach to respondents and identifying ways to further decrease the reporting burden. BJS anticipates that moderate increases in the response rate will be realized over the next several years based upon the availability of funds to allocate to outreach techniques such as the marketing research firm services employed under previous awards. The FY2011 FIST solicitation has established maintaining a 100 percent response rate for state agencies and achieving an overall 80 percent response rate as performance benchmarks for the selected data collection agent. It is anticipated that increases in the response rate will in part be based on additional funding made available for concerted follow-up efforts.

UPDATE FOR 2012 COLLECTION

The overall response rate in 2010 was 85%. For the 2011 data collection, BJS will plan for unknown nonresponse by creating two reserve samples within each of the four sampled states with a 15% oversample for each. Agencies will be randomly assigned to one of the two reserve samples so that the probability of selection will equal the initial probability of selection times the probability of being assigned to a reserve sample. If a response rate within any state falls below 85%, the first reserve sample will be initiated. If the response rate falls below 75% after the first reserve sample is instituted, the second reserve sample will be employed. Appropriate sampling weights will be assigned such that state-level estimates may be made in addition to a national estimate. BJS has initiated and will continue to employ a rigorous follow up schedule based on Dillman, Smyth, and Christian’s work on mail and internet surveys to maximize the response rate (see Attachment IV).

5. Test of Procedures

No such tests are planned.

UPDATE FOR 2012 COLLECTION

BJS will submit to OMB for review the requisite analyses of precision after the data collection for CY 2012 employing the new sampling frame is complete.

UPDATE FOR 2012 COLLECTION

For information on statistical methodology, conducting the survey, and analyzing the data, contact:

Allina D. Lee

Bureau of Justice Statistics

810 Seventh Street, N.W.

Washington, D.C. 20531

(202) 307-0765 or [email protected]

6