Gallup CA Responses to OMB Questions

Department of State Response to

OMB Questions Regarding Renewal of the Land Border Survey

1. OMB Question:  In the 2007 emergency clearance, Gallup anticipated a 30-35 percent response rate with a 4-6 week data collection period and a 5x5 call design.  At OMB's urging, it agreed to perform additional efforts (a 7x7 call design and an additional 2 weeks of data collection) to increase anticipated response rates.  The summary report indicates that a 35 percent response rate was obtained and a 7x7 call design was used.  Was the collection period extended to 8 weeks? 

State Response: Yes. We agreed with OMB’s suggestions and adjusted the call design to 7x7, which required extendinged the collection period to eight weeks ((August 3 through October 3, 2007), and adjusted the call design.

2. OMB Question: What was the response rate increase from the additional 2 weeks of data collection?

State Response: Without the additional 7x7 call design and extended timeframe, Gallup estimates that the response rate would have fallen below 30%, the low end of the initial response estimate. Gallup estimates that the Due to the extended timeframe and the additional call design increased the , Gallup notes the ultimate response rate was enhanced to 35 percent. It is difficult to determine the effect of the additional time because the 7x7 call design, with a greater number of calls, requires an 8 week period of performance, rather than a 4-6 week period. It is problematic to attempt to estimate what the number of responses would have been if the data collection ended after 4-6 weeks.

3. OMB Question: What is Gallup's conclusion about why response rates were only at 35 percent despite the additional efforts?

 

State Response: As stated above, wWe believe that only the additional time and change to the design allowed us to reach the 35 percent response rate. . Had we not done so, Gallup concluded that the response would actually have been lower than the estimated 30-35 percent. The Gallup conducted a Non-Response Analysis, which has been provided to OMB, did not yield conclusive findings concerning the rate of response. 35% is actually high for surveys such as this. as required by OMB. State/CA provided that report to OMB. The findings do suggest that there was no indication of significant non-response bias, i.e., non-respondents during the primary survey did not have substantively different behavior or responses than respondents.

4. OMB Question:  Did Gallup have to "restrict data collection in the low-density stratum" as it suggested it might if "incidence is low?"  If so, how did this impact the representativeness of the results?

State Response: No. A restriction on the collection in the low density stratum was not necessary, as the overall incidence in that stratum was higher than expected – even for geographic areas not near land border crossings. While incidence was clearly higher in the pre-defined high density strata, the incidence in the low density areas did not fall below minimum thresholds that would have led to a restriction in those areas.

5. OMB Questions a-f:   Please provide to OMB the confidence intervals around the estimates provided in the Gallup report.

a.  What is the current status of the non-response bias analysis study?

 

bb.  How successful was the incentive in obtaining participation?

c.  What does the study indicate about bias?  OMB would like a copy of the report.

d.  Why did Gallup prepare a final report without this information included? 

e.  What was the plan to revise the estimates if needed?

f.  How will the findings influence the proposed new design?

 

State Responses to Questions a-f: The standard error and the confidence intervals depended upon the sample size and the estimates. Attached is the Executive Report – Consolidated Summary of National Surveys of Passport Demand. Exhibit 1 (on page 7) provides in Column 1 the estimates of Passport Demand. The following table presents the estimates and the corresponding confidence intervals. The standard errors for the confidence intervals were calculated using the software SUDAAN to ensure appropriate use of sampling weights and correct calculations of sampling variances.

Table 1: Adjusted Estimates of Passport Demand

Projected Travel Pattern	Estimate (Self-reports)	Lower Bound	Upper Bound
Neither 2007 nor 2008	9,177,575	8,450,903	9,904,247
2007 only	1,892,257	1,538,830	2,245,684
2008 only	10,044,929	9,302,897	10,786,962
2007 and 2008	7,743,221	7,067,588	8,418,855
Total Adult Citizens	28,857,982	27,591,997	30,123,968

1. The non-response analysis is complete and has been will be provided to OMB for review.

2. All potential participants in the non-response bias portion in the non-response bias portion of the study were offered the $20 incentive. T, (there was no control group, of potential participants who were offered no incentive ), so it is not possible to determine precisely what the response rate would have been in this study had an incentive not been offered. The overall response rate for the non-response follow-up phase was 11%. However, Gallup has conducted randomized field testing with and without incentives in the past on other studies and has found that offering the incentive for a difficult population with a low initial turnout rate or initial responders does results in a higher response rate.

3. A copy of the report has been provided to OMB. TIn the non-response bias study, the two groups of respondents and non-respondents were compared in terms of selected variables, and the findings did not indicate any pattern of significant non-response bias. Non-respondents during the primary survey did not have substantively different behavior or responses than respondents. A separate analysis was also conducted to compare “Early” and “Late” respondents for the main study and no significant differences were found between the two groups.

4. The non-response analysis data collection was conducted between October 11, 2007 and November 5, 2007, followed by approximately three weeks of data analysis. . The Bureau of Consular Affairs requested preliminary data and a draft report within two weeks of survey fielding. To meet this requirement, Gallup conducted analysis of the primary data collection effort in parallel, beginning immediately after that effort ended on October 3, 2007.

5. If the non-response analysis had indicated the potential of non-response bias, the plan was to examine appropriate ways to revise estimates depending on the specific findings. Based on its past experience in making adjustments duedue to non-response bias, Gallup would use would have recommended using relevant data from the completed cases of the non-response bias study to and exploreing the possibility use of using additional weighting adjustments. The final Land Border Crosser Survey report would have taken account of included the necessary adjustments with appropriate notations citing the Non-Response Analysis report.

6. The existing design will function effectively as a means for revising and updating the data in the short-term., based upon recent environmental and policy changes. In the longer term, CA may submit a new collection request that updates, and simplifies and/or redirects the sampling methodology, (based upon what was learned in the initial survey. about the land border crosser population), and which returns to a 5x5 call design as well as eliminates the non-response bias analysis which has been determined not be a factor.

6. OMB Question: The land border crossing survey estimates used the Gallup Panel study as a reference population in determining demand estimates.  Does State plan to repeat the panel study?  If not, why not, and how will that impact the production of future estimates?

 

State Response: Yes. CA needs to has determined the need to refine the study to forecast future workload. We plan to use the data in essentially the same way. The longitudinal component of the panel helps – to provide a macroscopic view of the demand population and to identify various subgroups of demand aside from that are separate from the LBC community. The longitudinal component of the panel can also be useful in helping to ground the propensity measures empirically and to recalibrate them.

7. OMB Question:   In the pending OMB clearance package, why does State use the same language about timing urgency that it used in 2007 to justify the truncated data collection period and inability to use some standard survey techniques?

State Response: CA has revised based tthe recent submission on the original, revising it to exclude language relating specifically to the previous emergency review processreflect the urgency of conducting this survey in 2008. We will require continuous reliable data input for several years and the LBC survey and other standard survey techniques areis an effective means of acquiring crucial data. The data collection period in 2007 was extended because of the 7x7 protocol, which also increased the response rate. There is no apparent need to shorten or truncate the 2008 data collection period. Last year was an Emergency Survey; this year is a Quick-Response Survey.

8. OMB Question:  What is the rationale for conducting up to two LBC surveys per year?  For how many years?  How will the decision whether to conduct additional surveys, and their timing, be decided?

 

State Department Response: The current request is for one LBCsurvey. We determined that two surveys per year for the next two years would be appropriate to monitor the natural environmental fluctuations (e.g., economy, security, natural disasters, etc), policy and rule changes, and State communication and marketing campaigns. CA would likely conduct one survey in the spring just prior to the summer season surge in travel, and one just following the summer driving season. CA is currently evaluating options for a macroscopic demand study schedule that will include future LBC surveys. Future The final decisions will be made by the Assistant Secretary of State for Consular Affairs, and will take into consideration feedback from Congress and other government agencies, including OMB.

9. OMB Question:  How many cases does State propose to include in its non-response bias study?  How many completed interviews does State anticipate with this design?

State Response: The 2008 Study will be modeled after the 2007 Study – the same strategy and design, and roughly the same number of cases. Gallup made nearly 10,000 call attempts, using more than 8,500 RDD phone numbers, to complete 200 interviews for the non-response analysis. A 7x7 call design was employed, with an average of 4.23 attempts for each completed interview. The overall response rate for the non-response follow-up phase was 11 percent. The follow-on study would almost have to be similar. For meaningful comparisons, as many factors and variables as possible should be held constant.

 

10. OMB Question:  Why would the letter with the incentive be sent from Gallup on Gallup letterhead rather than State?

State Response: Gallup is a well-known and respected survey organization that is recognizable by the public and more likely to result in a response to this type of survey of difficult to reach respondents. If OMB recommends or State decides that recommends letters should be printed on State Department letterhead for follow-up surveys, they will be. we will do so.

11. OMB Question: Is 50 households for the pre-testing the universe or the target completed cases?  Will the pretest also use RDD?

State Response: The pre-test will use RDD sampling. For the pre-test, the plan is to use approximately 50 screened households as the universe. Based on an anticipated overall eligibility rate of about 30 percent, the total number of completed cases is expected to range from nine to fifteen. The pre-test will use RDD sampling.

12. OMB Question: What does State mean in Part A, item 16, when it says that "The collection of information will not be published for statistical use?"

 

State Response: These surveys are intended to serve as internal government documents and not for public use. The We do not intend to publish the survey results will not be published. T; they will be used for planning and budgeting purposes.

13. OMB Question: Why isn't the information about the survey being voluntary and not collecting PII in the initial script (versus in an "if needed" portion of the script later on)?

State Response: Gallup has 70 years of survey and data collection experience. In order to achieve as high a response rate as possible, Gallup advised that survey introductions need to be as short and concise as possible. , providing only the most pertinent information. According to Gallup, the long statement of the Privacy Act for all respondents would likely turn people off, result in lower response rates and increased survey time. The Privacy Act/voluntary nature statement was available on every interviewer’s introduction screen and was employed if the respondent hesitated or indicated concern about the voluntary nature of the study.

14. OMB Question: CA and Gallup You originally currently

proposed 8 focus groups per quarter. Canyou reduce the number of focus groups be reduced to 1 focus group per quarter?

State Response: State agrees and has changed itsour request to reflect this.

15. OMB Question: Canyou include a non-response bias survey be included?

State Response: State agrees and has included a non-response bias survey in its ourrequest..