SUPPORTING STATEMENT (PART B)
Survey of State Attorneys General Offices-Human Trafficking, 2018
B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS
1. Universe and Respondent selection
The Survey of State Attorneys General Offices (SSAGO) program intends to systematically assess and describe the organization, structure, responsibilities, and activities of all 56 state and territory attorney general offices.
2. Procedures for Collecting Information
The human trafficking survey will be launched with personalized email and mail survey packets sent to each attorney general office. A help desk for survey administration will be set up with a toll-free number at the Urban Institute for project staff to answer questions and provide any necessary assistance to respondents in completing the questionnaire. As described below, the data collection process will include a series of emails and follow-up activities with nonresponders, emphasizing questionnaire completion via the electronic fillable PDF version of the survey (though beginning in week two, hardcopy surveys will be mailed in addition to the electronic fillable PDF to some respondents). Responses completed over the phone, by mail, or fax will also be accepted, entered, and quality-checked by Urban Institute staff.
Week 1- Initial Contact (Mail)
In the first week of data collection, the initial point of contact at each attorney general office will be mailed a survey announcement letter on BJS letterhead and signed by the BJS Director (Attachment 8). This letter will discuss the purpose of the data collection, the type of information to be collected, and the benefits to the office in providing these data. The letter will include the BJS project manager’s contact information, and the SSAGO-HT toll-free phone number and email address so the offices may contact the BJS or the Urban Institute with any questions, in addition to providing a point of contact to handle the rest of the survey process. The survey announcement letter includes a quote about the SSAGO project from Connecticut Attorney General George Jepsen.
Week 2- Invitation with Electronic (Email) and Hard Copy (Mail) PDFs
Project consultant Ms. Cindy Lott, Esq. developed a list of initial contacts through her former work in a state attorneys general’s office and as current Executive Director and Lead Counsel to the National State Attorneys General Program (NSAGP) at Columbia University Law School. She has experience training attorneys general in a number of areas, and has developed relationships with many attorneys general offices. However, there are a few offices where Ms. Lott does not have a strong relationship through her work. The survey cover letter asks the initial contact to identify an expert coordinator. This will be helpful when the initial contact is unavailable or when there is another person in the office better suited to be the coordinator (i.e., Ms. Lott did not have enough of a relationship with that office to identify the correct initial contact). If the office does not identify an expert coordinator, all correspondence will be sent to the initial contact.
Approximately one week after the survey announcement letter mailing, a survey packet will be sent to all offices via email for those offices for which BJS has an email address for that person. It will be sent via mail to the remaining offices. Both the email and mail packets will include a cover letter on the Urban Institute’s letterhead signed by the Urban Institute’s Project Directors (Attachment 9). The emailed packet will contain the SSAGO-HT survey (Attachment 4) as an attachment, which can be completed as a fillable PDF or printed out, completed by hand, and returned. Based on responses to the cognitive test of the instrument regarding survey format and delivery, there are no plans to host a website where the PDF is available for download. However, the Urban Institute is prepared to offer this option if it is requested by the participants. The hardcopy packet will include a copy of the survey and a prepaid business reply envelope. The cover letter will include instructions to obtain the fillable copy of the survey. The initial packets will request a deadline to complete the survey within 30 days after the first mailing. All materials will have telephone and email contact information for the Urban Institute.
Week 4 – Initial Follow-up
Approximately two weeks after the survey mailing, the Urban Institute will send a reminder email (Attachment 10) to all nonresponding offices. If an email address is not available, the Urban Institute will mail a letter. This email or letter will continue to encourage nonresponding offices to complete the questionnaire as soon as possible, with a reminder of the survey deadline. It will contain the SSAGO-HT toll-free number and email address so that respondents may contact the Urban Institute with questions.
During this initial phase of data collection, project staff will monitor and respond to emails and telephone calls to the project toll-free number and email. All contacts via email and telephone will be documented and reviewed to assess persistent issues that may occur which need to be addressed.
Week 6 – Telephone Follow-up
The Urban Institute will begin telephoning nonresponding offices approximately two weeks after the reminder email (Attachment 11). This will serve to verify receipt of materials, answer questions, determine and attempt to resolve potential problems with timely submission, and prompt for questionnaire completion. Data collection specialists will also offer to complete the survey by telephone if the information requested is readily available and the office is not able to complete the electronic or hardcopy questionnaire (e.g., time constraints, preference of reporting over the telephone). Surveys completed over the telephone will be recorded by data collection specialists using the fillable PDF survey. Completed surveys will be emailed to the respondents for review and approval.
Weeks 7-9 – Targeted Follow-up
Lead consultant and attorney general liaison Ms. Cindy Lott, Esq., will phone attorney general offices who have not responded to the survey to emphasize its importance and offer individualized assistance in helping them complete the survey (Attachment 12). Ms. Lott is uniquely positioned to engage with state attorneys general offices from her work experience. Ms. Lott is in frequent communication with attorneys general throughout the country and has worked closely with state attorneys general offices on past National State Attorneys General Program research efforts. She has established long-standing relationships of trust with the attorney general community and attends the National Association of Attorneys General (NAAG) conferences every year, where she engages directly with the attorneys general and staff in attendance. Given her strong relationships and experience working with attorney general offices over time, Ms. Lott can exert influence with non-responders to foster their participation in the survey. This outreach will convey the importance of the office’s participation in the SSAGO-HT, encourage timely submission, and stress the importance of the information collection for the attorney general community. The outreach will provide instructions for submitting the completed survey via emailing the fillable PDF, or mailing or faxing the completed hardcopy survey back to the Urban Institute.
Week 9 – Direct Outreach by BJS
All non-respondents will be sent a letter by BJS directly to strongly encourage cooperation and reiterate the importance of responding to the survey (Attachment 13). The letter will be sent via mail or email, depending on whether the office identified a preferred method of contact. The emailed letter will include the fillable .PDF as an attachment, and the mailed letter will include a hardcopy with a prepaid business reply envelope.
Weeks 4-16 – Item-Specific Outreach
Project staff will review the surveys for completeness as they are received. Project staff will contact respondents that completed only part of the survey (electronic or hard copy) or provided illegible, illogical or confusing responses on the electronic or hard copy form as soon as possible after receipt of the surveys. Staff will attempt to obtain the missing or otherwise problematic information from respondents to reduce the number of missing items (Attachment 14).
Week 12 –“Last Chance” Email
During the final two weeks of the data collection, the Urban Institute will send a last chance email (Attachment 15) to nonresponding offices alerting them of the scheduled data collection end date. The customized email will contain the fillable PDF form as an attachment.
Week 16 – Data collection closes
The data collection period for this survey will conclude at the end of week 16.
Verifying and validating the submitted data
For offices that respond to the survey by filling in and mailing or faxing back the hardcopies, responses will be entered into the fillable PDFs by project staff at the Urban Institute and the following data quality assurance steps will be employed during the data collection and processing period:
Data Review. The Urban Institute will attempt to follow up by phone with attorney general offices that have missing or illogical data on particular survey items to obtain this information. The Urban Institute will manually review each questionnaire.
Data Retrieval: If additional data are required because of errors, inconsistencies, or missing data, the Urban Institute will phone the respondent for clarification as soon as possible after the date of submission of the questionnaire (Attachment 14). The Urban Institute will document the questions needing retrieval due to missing or inconsistent data, request clarification on the provided information if necessary, obtain values for missing data elements, and examine any other issues related to the submission. Data retrieval will occur through email and telephone outreach.
Data Conversion: Surveys that are deemed complete, those with no errors by the respondent, surveys reconciled after data retrieval, or after manual entry into the fillable PDF after the respondent replies via hardcopy or fax, will be extracted to a CSV file and imported directly into a SAS data file. The Urban Institute will perform 100% confirmation of the extracted data for all forms received using the fillable PDF format. The attorney general office identifier number for each record will be checked against the master list of identifier numbers assigned for each respondent (attorney general office) to confirm that all expected surveys have been extracted properly. Files will also be checked to confirm that all variables have been exported properly into SAS.
Following data collection, each fillable PDF will be extracted to a single SAS file. The data in that file will be reviewed to determine that the data extracted from the PDF is identical to the data in the SAS file. The attorney general office identifier numbers will be reviewed to ensure that all expected records from the fillable PDF forms are included in this merged file.
Any issues with extraction will be identified and corrected early on in the extraction and review process.
Each hardcopy response will be entered into the data file. The SAS file will be compared to the original hardcopy files to ensure that the data entered match the hardcopy responses. The attorney general office identifier numbers will be reviewed to ensure all expected records from the hardcopy forms are included in the merged files.
Second Data Review: As the survey data are imported into the SAS database, they will be made available to BJS via a secure file transfer site maintained by the Office of Justice Programs in the Department of Justice. The Urban Institute will make data available to BJS on a biweekly basis, or at other preferred intervals. The Urban Institute will run frequencies on the data on a biweekly basis to check for any issues or anomalies. Any issues uncovered will be investigated and resolved in a timely manner.
Thank you: Following verification of data received, each respondent will be sent an email to confirm the receipt of the data and to thank them for their participation (Attachment 16). The email serves to thank the respondent, inform them that their data have been included in the SSAGO-HT, and provide an approximate publication date of the final report. The thank you email will be converted and sent as a letter via U.S. mail in the event that the respondent’s email address becomes invalid or undeliverable.
Adjusting for Non-Response
Item-level missing data occur when an otherwise complete questionnaire is received but not all survey items are answered or the responses are illogical, such as when a response is inconsistent with an earlier response to a related survey question. Such losses of data could lead to item nonresponse bias in the statistics. The conventional approach to addressing item nonresponse is to either ignore it, if rates of missing data are exceedingly small, such as less than 2 percent for key items, or to impute plausible values in place of the missing data indicators. The Urban Institute will follow-up with respondents who provide incomplete responses; but it is expected that a small amount will remain.
Prior to the implementation of statistical imputation procedures, Urban will conduct logical edits when there is sufficient survey response and/or auxiliary information to deduce the value of the missing datum with a high degree of certainty. A trivial example is the imputation of a missing total when the individual components are recorded, and the respondent neglected to calculate and record the total.
After logical edits, statistical imputation will be used to ‘complete’ the data set. The methods used will depend on the amount and type of missing data. For example, missing values could be imputed to reflect the uncertainty associated with “predicting” the unobserved value. The addition of a stochastic random error term (with an analogous random assignment component for categorical variables) will avoid overstating the precision of the imputed values. Multiple imputations can readily be implemented using more sophisticated approaches with existing software. BJS could implement a stochastic approach for multiple imputation of item missing data. A regression with a random error term should represent the simplest form of imputation. If needed, BJS will develop Bayesian-type multiple stochastic imputations using an approach called data augmentation. Data augmentation may be preferable for imputing sets of core variables, while other, simpler methods may be employed for less essential variables. This combination of methods is a fiscally responsible, yet scientifically rigorous approach that tempers the expended resources with a focus on key variables.
Unit nonresponse occurs when a respondent fails to complete the survey. The SSAGO-HT is a census of all 56 state and territory attorney general offices. State attorneys general vary widely in their jurisdictional responsibilities, as do the state laws governing the criminal and civil handling of human trafficking. Typically, a weighting scheme is employed to adjust for unit nonresponse. In order to weight for nonresponse, every attempt is made to “match” the characteristics of the nonrespondents to respondents. During respondent follow-up, The Urban Institute will attempt to collect important matching characteristics, such as office size, and jurisdictional responsibility. BJS will attempt to use these characteristics, in addition to size of the state or territory population, to adjust for unit nonresponse, as needed.
3. Methods to Maximize Response Rates
As indicated in Part A of this application, in 2014, BJS collected data on white collar crime from state attorneys general offices through the State and Local White Collar Crime Program (SLWCCP). The survey received an 86% response rate (44/51) from the U.S. attorneys general. This response rate excludes the territorial offices (i.e., American Samoa, Guam, Northern Mariana Islands, Puerto Rico, and U.S. Virgin Islands). Based on the SLWCCP, the SSAGO-HT expects a response rate of 86% or higher from the 51 state attorneys general. Since the SLWCCP did not include the five territories, it is unknown whether the territory offices will respond to this survey. The data collection strategies are intended to encourage a high response rate. Ms. Lott is well-known in the attorneys general community, and has worked in previous research efforts to obtain survey responses. The feedback from the pilot surveys has been incorporated into the new survey instrument, including allowing respondents to estimate the total number of cases handled in the past year rather than to provide counts of the types of cases handled in the past year. It is expected that these changes will increase the response rate.
The use of paper and fillable PDF surveys were designed to facilitate response. Additionally, the Urban Institute is prepared to collect the data via telephone if the respondent prefers or is willing to complete the survey in that fashion. Last, attorney general offices will be able to contact the Urban Institute via email or a toll-free telephone number to ask questions or request assistance. Thus, respondents can contact the Urban Institute via their preferred contact method at no additional cost to them.
4. Testing of Procedures
Cognitive test methodology
The Urban Institute initially recruited four attorney general offices to participate in the cognitive test. These offices were located in Colorado, Maryland, Massachusetts, and Texas. Maryland dropped out of the pilot test when the human trafficking contact (Assistant Attorney General Elizabeth Embry, Chief of the Criminal Division) left to run for public office in the state of Maryland. Maryland was replaced with a territory attorney general office, Guam; however. Guam’s deputy attorney general responsible for human trafficking was on vacation during the cognitive testing period. As a result, the cognitive report does not include feedback from Guam. The Urban Institute sent a fillable PDF format of the pilot survey to 4 attorney general offices via email with a cover letter in the body of the email (Attachment 3, within the pilot report). Pilot respondents were asked to complete the survey within 3 weeks of the mailing.
During the cognitive test, The Urban Institute made reminder telephone calls and sent reminder emails to nonresponding offices to encourage return of the survey and answer questions. As completed surveys were returned to the Urban Institute, the data provider was contacted and a debriefing telephone call scheduled. During the 45-60 minute debriefing call, survey questions and response categories were reviewed for clarity and completeness (Attachment 3, within the pilot report). Pilot respondents were also asked about the availability of the requested data and burden. Three offices completed the pilot survey and the phone debriefing after the pilot survey. The responses culminated in a cognitive test report (Attachment 3). Guam’s deputy attorney general provided comments to the Urban Institute in September 2018. Guam indicated that there had been no human trafficking cases filed with the territory attorney general office in the past two years, therefore the survey only took 10 minutes to complete.
Use of cognitive test feedback
Based on feedback from the cognitive test, the draft instrument was modified by including options for “other” and “not applicable” responses, where appropriate. Additionally, language was clarified, as requested by the respondents. Last, an open-ended question was added, asking respondents about the biggest challenges to successfully prosecuting human trafficking cases. The pilot respondents indicated that the instructions and definitions in the survey instrument were clear.
Offices were asked if a due date might encourage faster response. Pilot respondents reported overwhelmingly that a due date would prevent the survey from being deprioritized by their office. Accordingly, the proposed data collection methodology includes a clear statement of a due date for the completion of the survey.
Offices were also asked about the best mode in which to contact them. Most offices reported that an emailed survey was favored, with a preference for follow up outreach by email as well. All offices reported that email would be the best method for reaching them. Therefore SSAGO-HT will use a multi-mode follow-up protocol that includes email, phone, and U.S. mail, in that order. As indicated earlier, the pilot respondents reported that they prefer a fillable PDF form to a hardcopy survey. Accordingly, a fillable PDF survey form will be the primary survey mode.
5. Contacts for Statistical Aspects and Data Collection
BJS contacts include –
Suzanne Strong
(202) 616-3666
The Urban Institute contacts include –
William Adams
Yasemin Irvin-Erickson
Attachments:
State Attorneys General Offices Website Review
Title 34 USC § 10132
SSAGO-HT Cognitive Report with Draft Instrument
SSAGO-HT Revised Instrument
60 Day Notice
30 Day Notice
IRB Approval
Survey Announcement Letter
Initial Survey Cover Letter
Reminder Email Text
Telephone Contacting and Voicemail Script
Targeted Telephone Contacting and Voicemail Script
BJS Outreach Letter
Telephone Retrieval Script
Last Chance Email
Thank You Email
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Strong, Suzanne |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |