Justification for Non-substantive Change

Justification for Non-Substantive Change Request

Sample Design and Methodology. The LMI team understands the current design and methodology thoroughly. We will review the existing sample design and methodology and recommend improvements, if needed. We detail our sampling approach below.

Required sample size for this study. For large population surveys, the expected precision of the resulting estimates is a direct function of sample size. In general, the selected sample needs to be large enough to ensure margins of error (ε) no larger than ±ε with at least (1−α) percent confidence. Figure 2‑3 shows the required sample size for various values of ε when α is set at 5 percent, which is to say, when the level of confidence is set at 90 or 95 percent. Accordingly, under a 95 percent confidence scenario, the required sample size increases by more than 225 percent (from 1,067 to 2,401) when the error margin is to decrease from ±3 percent to ±2 percent. Conversely, for a fixed value of ε, a similar but less dramatic increase in the required sample size is observed when the level of confidence is increased from 90 percent to 95 percent (from 1,691 to 2,401). Our proposed sample sizes satisfy fully respond to program requirements by meeting the ±5 percent margin of error at a 95 percent level of confidence.

Figure 2‑3. Required sample size as a function of confidence level and margin of error.

On the basis of the required sample sizes in Table 2‑7, the LMI team proposes an alternative option for sample design. The current methodology outlines a desired target of 4,000 completed surveys from a sample of 36,000 persons. Our experience with this project indicate that the goal is not a feasible. In the past 12 months utilizing the current methodology, a gradually declining response rate of about 9 percent has been observed. Reduced response to surveys has become an ever-increasing problem for the industry. Using our knowledge of response rates typically seen in such populations, combined with actual historical observations, we compare the current option as well as the alternative option in Table 2‑7 and describe them in the following text.

Table 2‑7. The LMI team will execute and deliver the option that best meets the needs and intent of the government successfully.
Options	Completed surveys each month	Sample size drawn each month	Expected survey margin of error at national level	Effect on survey trends	Effect on final LMI model predictions for a 95%–99% accuracy
1. Current methodology	4,000	36,000	±1.5%	None	None
2. Proposed methodology	2,500	28,100	±2%	Low	None

Current Methodology. Sampling action plan for 4,000 completed surveys each month. Collecting 4,000 responses each month reflect an approximate ±1.5 percent margin of error at a 95 percent confidence interval. A nationally representative sample of 4,000 per month secures an effective sample size well above the needed 384 to keep the resulting margin of error below ±5 percent, with a 95 percent confidence interval. This design ensures subcategory-level analysis at the passport agency level. LMI stratifies by the 25 passport agencies with passport counter facilities across the country. Based on a response rate of 9 percent, the LMI team selects 36,000 Address-Based Sampling (ABS) records monthly, ensuring no duplicate addresses from previous months. The sample universe consists of all deliverable residential addresses in the United States (including Alaska and Hawaii), excluding the following types: vacants, seasonals, and PO Boxes. The only P.O. Boxes included are those for households that have no other way to get their mail. After the sample is drawn, Marketing Systems Group (MSG) carries out its enhanced match process using exact matches only. This results in approximately a 60 percent match rate by phone, a 30 percent match rate through email, and an 80 percent match rate on name. The sample file is delivered in a comma-separated values (CSV) format with a header record for easy import to Excel for a mail merge.

Alternative Sampling Action Plan—collect 2,500 completed surveys each month instead of 4,000:

• Does not impact final forecast estimates.

• Reduces the public burden and cost to the government.

• Ensures retention of original design through similar stratification.

Option 2. Sampling action plan for 2,500 completed surveys each month. The LMI team proposes an alternative to the current option as described above. We offer a plan for 2,500 completed surveys each month. We would draw a stratified sample of 100 addresses from each of the 25 passport offices, ensuring geographic representation. Our sampling team member, MSG, will provide at least 28,100 ABS records monthly, ensuring no duplicate addresses from any previous months. The sample universe will consist of all deliverable residential addresses in the United States (including Alaska and Hawaii), excluding the following types: vacants, seasonals, and PO Boxes. The only PO Boxes to be included will be those for households that have no other way to get their mail. After the sample is drawn, MSG will carry out its enhanced match process using exact matches only. This will result in approximately a 60 percent match rate by phone, a 30 percent match rate through email, and an 80 percent match rate on name. We will deliver the sample file in a comma-separated values (CSV) format with a header record for easy import to Excel for a mail merge. This reduction in sample size would still translate into an approximate ±2 percent margin of error at a 95 percent confidence level. This plan would enable stable estimates at the national level for passport demand, while keeping the same stratification. This option will have no impact on the overall LMI model predictions for this study at the national level.

Advantages of this alternative sampling plan include the following:

• No impact on the final forecast estimates presented to CA/PPT (at most, some trends in survey data may be affected at the subgroup level, but CA/PPT is mainly interested in national-level estimates)

• A 37 percent reduction in the public respondent burden by collecting 2,500 responses instead of 4,000

• A significant reduction in the cost to the government.