October 22, 2013
Pretest findings
Westat conducted a pretest of the customer satisfaction survey in order to assess whether individuals who have attended past USCIS webinars would be able to answer the questions easily and accurately. In response to a personalized email from Westat and the attached USCIS letter of invitation to participate in the study, eight individuals who had participated in a USCIS webinar within a three-month period agreed to participate in the pretest. Effort was made to obtain the participation of respondents who had attended different types of these webinars, and there was at least one webinar participant from each of the following E-Verify topics:
Public E-Verify for Existing Users Webinar
Public Self Check Webinar
Customized TrueBlue Inc Webinar
Form I-9 Public Webinar
Public E-Verify FAR Webinar
Public E-Verify Overview Webinar
The respondents were asked to test the draft questionnaire and use an attached commentary guide to submit their feedback as they answered the questions. They were told specifically to pay attention to whether the questions were clearly worded, whether the questions could be answered easily, and whether the questions covered all aspects of the webinar. Of the eight respondents, five completed and returned the survey. While most of the respondents were current users of E-Verify, one respondent indicated that his company had never enrolled in E-Verify and never used it. About half of the respondents reported viewing the webinar as an employer at a company, with the other half viewing it from an employee’s perspective. Answers to the question about how many employees work at their companies ranged from around 350 to 3,000. Two of the respondents said they did not know how many employees work at their companies, and one of them indicated that this was the only question that was problematic.
Overall, the respondents reported finding the questionnaire to be clearly worded and easy to answer. None of the eight respondents reported any problem with a specific question although a few indicated that it was somewhat difficult to recall details of an event that occurred several months ago. Five pretest participants returned completed surveys even though they were not asked to do so.
Changes to survey questions
Overall, the pretest findings were very positive and none of the participants reported any problems with survey questions. As a final step to tease out any potential problems, Westat took a close look at the survey questions and responses to questions in the five completed surveys. Based on that examination, we added a cover page with instructions, dropped a few items that appeared to be somewhat redundant, and modified a few items to encourage survey respondents to read each item in grid format. Attached are two versions of the survey—a version with track changes and a version with accepted changes. Changes to the survey are summarized below.
Replacing prefilled information with a cover page. This first portion of the survey was replaced with a cover page that includes instructions for completing the survey and contacting Westat or USCIS in case of questions.
Minimizing response sets and overlapping items. To encourage respondents to read each item in grid questions before selecting an answer, questions 2 and 4 were modified to include both positive and negative statements about the webinar. In addition, some questions (3c, 6d, and 7d) were dropped because they overlapped with other items.
Number of people viewing the webinar from respondent’s computer (Question 1). We modified the question wording to clarify that this number should include the respondent.
Willingness to attend another USCIS E-Verify webinar (question 8). To elicit more information on why respondents may or may not be willing to attend another USCIS E-Verify webinar, this item was changed to a YES/NO response option. We also added a comment box for respondents to provide an explanation for the answer selected.
Whether respondent viewed webinar as an employer or employee (Question 12a). To account for respondents who may have multiple work roles, the wording of this question was modified to ask about the perspective from which the respondents were viewing the webinar.
Additional question to request respondent phone number (new question 14). A question was added at the end of the survey to request the respondent phone number in case we had follow-up questions.
In addition, a few minor edits were applied to improve precision and readability.
Sampling issues
Two respondents indicated that it was very difficult to recall events that took place months ago. The issue of recall has implications for the preparation of the participant list (sampling frame) from which the sample will be drawn for the webinar survey. Thus, we recommend restricting the participant list to two months prior to the survey, which means that it would be most efficient to prepare the list very close to the time of OMB clearance.
A second issue relates to the accuracy of email addresses in the participant list and the likelihood of a relatively high rate of email ‘bounce backs’ due to typos in email addresses. Since we do not have telephone numbers to attempt initial contact with these webinar participants, the loss of these cases will narrow the pool of potential respondents and suppress our response rates. This could be problematic since customer satisfaction surveys typically have low response rates. To address these known impediments to response rates and secure large enough numbers of completed surveys, we recommend targeting all individuals who have participated in USCIS webinars over the two months preceding the survey. We estimate that approximately 5,000 participants will be eligible for the survey after the list is cleaned.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Valerie Atkinson |
| File Modified | 0000-00-00 |
| File Created | 2021-02-01 |