Document
A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey 2025
ICR 202607-0648-005 Β· OMB 0648-0652 Β· Object 170833900.
Document Viewer [pdf]
Status: Original and derived artifacts are available for this document.
Download: pdf
Loading document viewerβ¦
Document Metadata
| File Type | application/pdf |
|---|---|
| File Title | A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey 2025 |
| Subject | A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey 2025 |
| Keywords | A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey 2025, |
| Author | NOAA Fisheries |
| File Modified | 2026-02-24 |
| File Created | 2026-02-19 |
| Conversion State | complete |
Extracted Text
A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey 2025 February, 2026 To receive a Microsoft Word version of this document with screen-readable equations and notations, please email [email protected]. A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Abstract This study evaluates an experimental design for the Fishing Effort Survey (FES), comparing it to the current FES methodology to assess impacts on fishing activity estimates. The experimental design included monthly sampling and administration, a revised questionnaire with a reordered question sequence (12-month fishing activity question preceding the primary two-month reference period), and a split primary reference period presented as two individual months. Additionally, it incorporated composite estimation. Results indicate that experimental estimates of total angler trips were consistently lower than FES estimates for both private boat and shore fishing, with mean reductions of 22% and 9%, respectively. This difference was most pronounced during periods of lower fishing activity. The primary driver of these differences was identified as the revised question order, which provided a beneficial “bounding effect” that mitigated overreporting by satisfying respondents' desire to identify as anglers without biasing the primary survey measures. The format of the primary reference period (split into individual months) also influenced estimates of mean trips per fishing household, showing larger estimates for the split design, particularly for shore fishing. This study provides additional evidence that the current FES design is susceptible to measurement error and likely overestimates fishing activity. It also demonstrates the feasibility and benefits of monthly sampling and composite estimation for improved precision and stability of survey estimates. Future research should continue to explore the cognitive aspects of the questionnaire, especially regarding the split primary reference period and mode-specific reporting differences. U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 2 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Background The Fishing Effort Survey (FES) is a self-administered, household mail survey that was designed to estimate marine recreational shore and private boat fishing activity (National Marine Fisheries Service Office of Science and Technology, 2023). The survey is administered for discrete, two-month reference waves in the coastal states along the Atlantic and Gulf coasts, as well as Hawaii. For each fishing mode (shore and private boat), the FES questionnaire asks respondents to first report the number of days of recreational, saltwater fishing during a two-month reference wave, followed by the number of days fished during the previous 12 months. Only the information reported for the two-month reference period is used for the purpose of estimation—i.e. the survey estimates the number of shore and private boat fishing days for each two-month wave. Prior to implementing the FES, NOAA Fisheries completed several pilot studies to evaluate different data collection designs for estimating recreational fishing activity (Andrews et al. 2014). This included a series of cognitive interviews to evaluate different versions of mail survey instruments. A notable finding of cognitive interviewing was that anglers are eager to report fishing activity and be identified as an angler, regardless of whether or not the activity was within the scope of the survey. For example, interview participants reported charter fishing trips, trips that occurred prior to the two-month reference period, freshwater fishing trips and saltwater fishing trips in other states. Qualitative observations from cognitive interviews were supported by early FES questionnaire testing which demonstrated that anglers were more likely to report saltwater fishing during a fixed, two-month reference period when the questionnaire was limited to a single period than when it asked about fishing activity for multiple periods (Andrews 2023). This prompted the addition of the 12-month fishing activity question to the FES questionnaire to both enhance recall, for example by bounding the two-month period within a longer period (Sudman et al. 1984), as well as provide respondents with an additional opportunity to identify as an angler. The 12-month fishing activity question was placed after the two-month question because it was thought to require a greater cognitive burden than the two-month question. At the time, this seemed like a relatively minor consideration for a short mail survey design in which respondents can view the entire questionnaire prior to responding, and testing had demonstrated that adding the 12-month question after the two-month question resulted in fewer reports of fishing compared to only having the two-month question. Subsequent analysis (Andrews 2023) suggested that, despite the inclusion of the 12-month bounding question, the current FES questionnaire remains susceptible to bias resulting from measurement error and likely overestimates fishing activity, and that switching the order of the two-month and 12-month fishing activity questions would further reduce bias. Specifically, Andrews (2023) hypothesized that FES respondents are so eager to identify as an angler that they do so at the earliest opportunity, even if the reported information is inaccurate. Brenner and DeLamater (2016) suggest that surveys provide an easy opportunity for respondents to identify with a desired population (e.g. the population of anglers), and overreporting may occur because respondents interpret survey questions to be about identity rather U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 3 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 than behavior. In the context of the FES, a respondent may report a fishing trip because doing so reaffirms the belief that he or she is an angler, regardless of actual recent fishing activity. Such behavior would be similar to a social desirability bias, where survey respondents tend to report behaviors that will be viewed favorably by others, a phenomenon observed for a variety of other activities including voting (Bernstein, Chadha, and Montjoy 2001), church attendance (Hadaway et al. 1993, Hadaway et al. 1998) and exercise (Shephard 2003). Asking the 12-month fishing activity question first provides respondents with an opportunity to identify as an angler that is both more accurate (i.e. there is a greater probability that a respondent fished during a longer time period than a shorter period) and less likely to result in over-reporting for the two-month fishing questions. This study expands the work of Andrews (2023) by administering an experimental questionnaire with a revised question order over the course of an entire calendar year. In addition, the revised methodology was designed and administered to estimate fishing activity for one-month reference periods. The survey was administered in parallel to the FES for all of 2024 in all FES states (ME-MS and HI) to evaluate the combined effects on estimates of the shorter reference period and the revised question order. Specific differences between the FES and revised, pilot study include the following: 1. Monthly sampling: The pilot study was administered monthly and included the selection of independent address-based samples for each survey administration, 2. Revised question order: The FES questionnaire first asks respondents to report the number of days fished during the primary reference period (prior two months) followed by the number of days fished during the previous 12 months. The experimental questionnaire asks the number of days fished during the previous 12 months followed by the number of days fished during the primary reference period (two most recent individual months). 3. Asking about fishing during two separate months: The FES asks respondents to report the total number of days shore and private boat fishing during a two-month reference period (e.g. Number of days fished in January and February). The experimental questionnaire asks respondents to report the number of days shore and private boat fishing during two individual months (e.g. Number of days fished in January and the number of days fished in February) 4. Rolling survey administration: The FES is conducted for discrete and independent two month reference waves. The experimental design is a rolling administration that includes a one-month overlap between adjacent survey administrations. 5. Composite estimation: The FES produces a single estimate for each discrete, two-month reference period. As a result of the rolling, monthly design, the experimental design results in two independent estimates for each reference month that are combined using an optimized composite estimator. U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 4 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Methods The experimental survey was a rolling, monthly design, in which each survey administration asked about fishing activity during the two individual months immediately prior to the beginning of each monthly survey administration. Consequently, the reference period for each administration included a one-month overlap with the prior monthly administration. For example, both the February (referencing January and February) and March (referencing February and March) administrations included February as a reference month. As described in the sections below, the two measures for each overlapping month (e.g. February) varied in recall length (e.g. one-month recall for the February administration and two-month recall for the March administration) and the placement of the month in the two-month sequence (e.g. second month in the two-month sequence for the February administration and first month in the sequence for the March administration). Questionnaire The experimental questionnaire (QEXP) is included as Appendix A. The questionnaire was developed and qualitatively evaluated through a series of cognitive interviews administered prior to survey implementation (Brick et al. 2024). The questionnaire is identical to the current FES questionnaire (QFES, Appendix B), with the exception of questions 15 and 16, which ask respondents to report the number of days shore and boat fishing, respectively. Figure 1 presents questions 15 and 16 for the FES and experimental questionnaires. In the FES, respondents are first asked to report the number of days fished during a two-month reference period, followed by the number of days fished during the previous 12 months, including the reference period. The two months of the reference period are identified, but respondents are asked to report the aggregate number of days fished across both months. In the experimental questionnaire, respondents are first asked to report the number of days fished during the previous 12 months, followed by the number of days fished in each individual month of the most recent two-month period. The two individual months are presented in chronological order. In both the FES and experimental questionnaires, an “opt-out” checkbox is presented prior to the fishing days questions to allow respondents to indicate no fishing activity. U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 5 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Figure 1. Questions 15 and 16 from the FES (left) and experimental (right) questionnaires. The questionnaires differ in the order of the 12-month and primary reference period questions, as well as the format of the primary reference period - single (FES) two-month period vs. two-month period split into individual months (experimental). Sampling With the exception of the sampling frequency, the sampling design for the pilot study was identical to the FES, which is described in Papacostas and Foster (2018). Generally, both surveys utilize Address Based Samples (ABS) with sample frames derived from the United States Postal Service Computerized Delivery Sequence File (CDS) including all full-time (non-seasonal), residential addresses (AAPOR 2016). Within each coastal state, sampling is stratified by sub-state region, which is defined by geographic proximity to the coast - counties with borders that are within 25 miles of the coast are in the “coastal” stratum and all other counties are in the “non-coastal” stratum. The address frame is also matched by address to the National Saltwater Angler Registry (NSAR)1, a directory of all anglers licensed to participate in saltwater fishing, creating two additional strata within each state, NSAR matched and unmatched. A new sample frame is created prior to each sampling period, approximately two weeks prior to the initial survey mailing. Samples are allocated among states and reference periods with a goal 1 https://www.fisheries.noaa.gov/recreational-fishing-data/national-saltwater-angler-registry U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 6 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 of achieving equal precision across samples, and allocated among strata within each state and reference period using a Neyman allocation (Cochran 1977). During the pilot study period, a new sample frame was constructed and independent samples selected for each administration month. In the months in which FES and pilot study data collection activities overlapped (e.g. “even” months), a single sample, large enough to accommodate both surveys, was selected, and sample units were randomly assigned to either the FES or pilot study. This ensured that a single address was not assigned to both surveys for an overlapping data collection period. The sample sizes for each state and reference period are provided in table 1 and 2, for the FES and experimental design, respectively. Monthly pilot study sample sizes were identical to the FES sample size for the corresponding wave. For example, the pilot study sample sizes for the January administration and February administration were equal to each other, as well as the wave one (January/February) FES sample size. Pilot study samples were allocated among states, reference periods and strata in proportion to FES samples. U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 7 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Table 1. FES sample sizes by state and wave. Wave 1 Wave 2 Wave 3 Wave 4 Wave 5 Wave 6 AL 5,059 3,220 2,360 2,007 4,676 3,228 20,550 CT 0 8,481 2,510 2,496 2,587 6,585 22,659 DE 0 5,102 2,343 1,647 2,300 4,456 15,848 FL 1,799 2,004 2,092 2,132 2,438 2,023 12,488 GA 0 11,785 5,235 6,397 6,729 5,975 36,121 HI 5,048 4,818 2,436 3,086 3,594 3,641 22,623 ME 0 0 3,194 1,716 2,944 0 7,854 MD 0 5,021 2,709 2,629 3,098 4,114 17,571 MA 0 13,263 2,613 2,064 3,734 9,220 30,894 MS 5,715 4,162 3,600 3,151 4,057 4,812 25,497 NH 0 0 2,772 3,421 5,390 0 11,583 NJ 0 9,288 2,920 2,978 3,278 5,398 23,862 NY 0 10,735 4,938 3,359 5,906 9,776 34,714 NC 8,362 4,397 2,495 2,467 3,142 3,778 24,641 RI 0 7,302 2,727 2,037 2,913 4,711 19,690 SC 0 3,724 3,154 3,548 3,371 4,475 18,272 VA 0 7,409 2,898 2,432 3,316 3,519 19,574 25,983 100,711 50,996 47,567 63,473 75,711 364,441 Total Total U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 8 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Table 2. Experimental sample sizes by state and month. Wave 1 Wave 2 Wave 3 Wave 4 Wave 5 Wave 6 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Total AL 5,059 5,059 3,220 3,220 2,360 2,360 2,007 2,007 4,676 4,676 3,228 3,228 41,100 CT 0 0 8,481 8,481 2,510 2,510 2,496 2,496 2,587 2,587 6,585 6,585 45,318 DE 0 0 5,102 5,102 2,343 2,343 1,647 1,647 2,300 2,300 4,456 4,456 31,696 FL 1,799 1,799 2,004 2,004 2,092 2,092 2,132 2,132 2,512 2,438 2,023 2,023 25,050 GA 0 0 11,785 11,785 5,235 5,235 6,397 6,397 6,729 6,729 5,975 5,975 72,242 HI 5,048 5,048 4,818 4,818 2,436 2,436 3,086 3,086 3,594 3,594 3,641 3,641 45,246 ME 0 0 0 0 3,194 3,194 1,716 1,716 2,944 2,944 0 0 15,708 MD 0 0 5,021 5,021 2,709 2,709 2,629 2,629 3,098 3,098 4,114 4,114 35,142 MA 0 0 13,263 13,263 2,613 2,613 2,064 2,064 3,847 3,734 9,220 9,220 61,901 MS 5,715 5,715 4,162 4,162 3,600 3,600 3,151 3,151 4,057 4,057 4,812 4,812 50,994 NH 0 0 0 0 2,772 2,772 3,421 3,421 5,390 5,390 0 0 23,166 NJ 0 0 9,288 9,288 2,920 2,920 2,978 2,978 3,278 3,278 5,398 5,398 47,724 NY 0 0 10,735 10,735 4,938 4,938 3,359 3,359 5,906 5,906 9,776 9,776 69,428 NC 8,362 8,362 4,397 4,397 2,495 2,495 2,467 2,467 3,142 3,142 3,778 3,778 49,282 RI 0 0 7,302 7,302 2,727 2,727 2,037 2,037 2,913 2,913 4,711 4,711 39,380 SC 0 0 3,724 3,724 3,154 3,154 3,548 3,548 3,371 3,371 4,475 4,475 36,544 VA 0 0 7,409 7,409 2,898 2,898 2,432 2,432 3,316 3,316 3,519 3,519 39,148 25,983 25,983 100,711 100,711 50,996 50,996 47,567 47,567 63,660 63,473 75,711 75,711 729,069 Total U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 9 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Data Collection Procedures The data collection procedures for the pilot study followed a modified version of the Dillman method (Dillman et al., 2014), following a prescribed sequence and timing of survey mailings. For each survey administration, data collection began with an initial survey mailing, approximately one week prior to the end of the two-month reference period to ensure survey materials were received as close to the end of the reference period as possible. The initial mailing, delivered by regular first class mail, included a cover letter stating the purpose of the survey, a survey questionnaire, business reply envelope (BRE), and a $2 prepaid cash incentive. One week after the initial mailing, a follow-up thank you/reminder postcard was delivered via regular first class mail to all sampled addresses. Three to four weeks after the initial survey mailing, a final mailing, including a nonresponse conversion letter, a second questionnaire and a BRE, was delivered via first class mail to all addresses that had not yet responded to the survey. Data collection for each administration continued for 13 weeks after the initial survey mailing. The exact data collection schedule is provided in table 3. Table 3. Data collection schedule for the experimental design for each reference month. Jan Reference Period Begins First Mailing Reference Period ends Remi nder Postcard Second Mai li ng Reference Period Begins Reference Period ends Remi nder Postcard Second Mai li ng - Mar Apr May Jun 1/1/2024 2/1/2024 3/1/2024 4/1/2024 5/1/2024 6/1/2024 1/23/2024 2/21/2024 3/22/2024 4/22/2024 5/23/2024 6/21/2024 1/31/2024 2/29/2024 3/31/2024 4/30/2024 5/31/2024 6/30/2024 2/1/2024 3/1/2024 4/1/2024 5/1/2024 5/31/2024 7/1/2024 2/19/2024 3/18/2024 4/18/2024 5/20/2024 6/18/20024 7/18/2024 Jul First Mailing Feb Aug Sep Oct Nov Dec 7/1/2024 8/1/2024 9/1/2024 10/1/2024 11/ 1/2024 12/1/2024 7/23/2024 8/23/2024 9/23/2024 10/23/2024 11/22/2024 10/24/2024 7/31/2024 8/31/2024 9/30/2024 10/31/2024 11/30/2024 12/31/2024 8/1/2024 9/3/2024 10/1/2024 11/ 1/2024 12/2/2024 1/2/2025 8/19/2024 9/19/2024 10/18/2024 11/18/2024 12/18/2024 1/20/2025 Weighting and Estimation For each monthly survey administration, weighting was similar to the FES with sample weights calculated in stages (Papacostas and Foster 2018, FES Annual Report 2024). In the first stage, base weights for each sampled address within a given stratum were calculated as the inverse of the inclusion probabilities. In the second stage, base weights were adjusted to compensate for unit nonresponse (e.g., when households fail to mail back the completed survey). The sample was partitioned into nonresponse adjustment cells defined by stratum plus household boat registration (i.e., whether a sampled address could be matched by address to a state boat registration list). The base weights for responding households in each adjustment cell were then divided by the weighted response rate for that cell. Nonresponse weights were further adjusted through a raking procedure such that weighted distributions U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 10 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 for a series of auxiliary variables matched marginal, state-level control distributions derived from the American Community Survey (ACS), Current Population Survey (CPS)2 and National Health Interview Survey (NHIS)3. Auxiliary variables included the proportion of households with seniors, proportion of households with children, proportion of households that are wireless-only, household tenure and household size4. In a fourth stage, raked weights were post-stratified within geographic strata (state*coastal region) to the most recent one-year ACS estimates of the number of full-time, occupied, residential households. In a final weighting stage, an estimated mean squared error trimming procedure that balances the tradeoff between bias and variance was applied to post-stratified weights to mitigate the effects of extreme weights on estimated variance (Potter, 1990). As noted above, each administration of the experimental questionnaire asked respondents to report the number of days fished for each of two successive months, the month immediately preceding survey administration (m1) and the prior month (m2). The trimming procedure was specific to each survey measure—shore and private boat fishing days within each reference month - resulting in four final weights for each sample unit. For each state, monthly survey administration (Mi) reference month (mi), and fishing mode (shore and private boat), the total number of fishing days was estimated as a weighted sum using final survey weights (Papacostas and Foster, 2018). This results in two independent measures of fishing effort for each month - one measure from the survey administration immediately following the reference month (m1 in M1), and a second measure collected during the subsequent survey administration (m2 in M2). We used an inverse variance-weighted compositing approach to combine the independent measures into a single estimate for each month (Hartley 1962). The resulting estimate for each reference month is effectively a weighted average of the two independent estimates, with the weights optimized to minimize the variance of the composite estimate. 2 American Community Survey and Current Population Survey estimates are downloaded from https://data.census.gov/. 3 State-level estimates of wireless telephone coverage were obtained from the National Health Interview Survey Early Release Program, https://www.cdc.gov/nchs/nhis/early-release/index.html 4 The production FES uses 3 categories of household size (1,2 and 3 or more). For the pilot study, we utilized 5 categories of household size (1,2,3,4,5+). U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 11 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Let π2 λ= π(ππ2) π1 π2 (π(ππ1)+π(ππ2)) Then, π1 π2 ππ(λ) = λππ1 + (1 − λ) ππ2 With, 2 π1 2 π2 V[ππ(λ)] = λ π(ππ1) + (1 − λ) π(ππ2) Where m is the estimation domain (month) and M1 and M2 are the two consecutive survey administrations that collected data for domain m. Analysis All analyses were performed using SAS software Version 9.4 (SAS Institute, Cary, NC) and utilized appropriate survey procedures to account for complex survey designs. All estimates or estimated quantities were calculated using final sample weights unless otherwise noted. For the experimental questionnaire, we compared monthly shore and private boat fishing effort between successive survey administrations for each overlapping state*month domain by evaluating the differences between mean effort estimates using independent samples t-tests. This compared the combined effects on estimates of recall length (one or two months) and the position of the reference month in the two-month sequence (first or second month) for each estimation domain. Because this analysis compared estimates across survey administrations for the same reference month, the analysis weights were final, trimmed weights for each survey administration rather than final composited weights. Comparisons between FES and experimental estimates included descriptive comparisons of total shore and private boat fishing effort and ratios of effort estimates. Differences in estimated quantities (e.g. total effort, fishing prevalence, mean trips per fishing household) were tested using paired t-tests, with pairs defined by unique combinations of state and reference wave. Monthly experimental estimates of total fishing effort were summed to the two-month wave level to permit direct comparisons between FES and experimental estimates and in the forming of wave-level estimate pairs. Because of large geographic and temporal variability in fishing effort, significance tests for this statistic that spanned geographic and temporal domains were calculated using the ratios of paired measures rather than absolute differences between measures to remove any scaling effects. Comparisons between designs for fishing prevalence (percent of households that reported fishing during the reference period) and mean U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 12 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 trips per fishing household were limited to the survey administrations where the two-month periods directly overlapped between the FES and experimental designs. Results and Discussion Response Rates Table 4 provides FES and experimental response rates and the number of sampled addresses, overall and by state. The overall difference in response rate between the pilot study (23.34%) and FES (23.49%) was less than 0.2 percentage points, and differences among states were not systematic. FES, response rates ranged from 18.72% (GA) to 32.58% (HI), and pilot study response rates ranged from 18.23% (GA) to 32.17% (ME). Given the similarities in response rates between the FES and pilot study designs, differential bias resulting from nonresponse is not likely to be a contributor to any observed differences in estimates between the two designs. U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 13 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Table 4. FES and experimental sample sizes (N) and base-weighted response rates (AAPOR RR2) overall and by state. Pilot FES State N Response Rate (%) SE N Response Rate (%) SE AL 41,100 22.46 0.33 20,550 22.38 0.46 CT 45,318 25.45 0.25 22,659 25.58 0.36 DE 31,696 27.34 0.31 15,848 27.13 0.43 FL 25,050 23.33 0.29 12,488 23.43 0.41 GA 72,242 18.23 0.19 36,121 18.72 0.27 HI 45,246 31.83 0.23 22,623 32.58 0.32 ME 15,708 32.17 0.52 7,854 31.99 0.72 MD 35,142 24.14 0.28 17,571 23.87 0.39 MA 61,901 25.73 0.25 30,894 26.40 0.36 MS 50,994 20.85 0.30 25,497 20.64 0.42 NH 23,166 30.27 0.39 11,583 29.32 0.54 NJ 47,724 23.00 0.22 23,862 22.67 0.32 NY 69,428 21.79 0.23 34,714 21.55 0.32 NC 49,282 23.47 0.30 24,641 24.46 0.44 RI 39,380 27.25 0.27 19,690 27.53 0.38 SC 36,544 24.95 0.33 18,272 25.54 0.47 VA 39,148 26.66 0.34 19,574 26.37 0.48 Overall 729,069 23.34 0.09 364,441 23.49 0.13 Evaluating the Combined Effects of Recall Length and Question Position on Survey Measures U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 14 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 For each monthly survey administration (Mi), QEXP asked respondents to report the number of days shore and private boat fishing for two successive months, which were presented in chronological order. π1 π2 This resulted in two independent estimates for each reference month, ππ1 and ππ2 that varied in the length of recall (one month vs. two months) and the placement of the month in the question order (first month vs. second month). Ultimately, the independent estimates for each reference month were combined into a single estimate using a composite estimator. However, the study provided an opportunity to evaluate the combined effects of recall length and the position of the month in the question sequence (i.e. the net effect of successive survey administrations) on survey measures. A total of 155 state*month domains included two independent estimates of total private boat and shore fishing effort. Of these, differences between estimates were significant in 18 (11.6%) and 14 (9.0%) estimation domains for private boat and shore fishing, respectively. Figure 2 shows the distribution of estimate ratios (M2:M1) by fishing mode across state*month domains. Across all domains, the mean relative difference between M1 and M2 was approximately 26% (ratio=1.26) for both fishing modes (i.e. estimates derived from the second administration were, on average, 26% larger than estimates derived from the first administration). Figure 2 demonstrates a strong influence of outliers on mean values for both fishing modes. The respective median relative differences between M1 and M2 for private boat and shore fishing are 12% (median ratio=1.12) and 19% (median ratio=1.19), values that more closely align with the relative scale of significant differences between M1 and M2 estimates among individual state*month domains. U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 15 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Figure 2. Distribution of monthly effort ratios across all state*month estimation domains for fixed months resulting from successive administrations of the experimental design. Ratios are of the estimate derived from the second survey administration for a fixed reference month (M2) divided by the estimate from the first survey administration (M1). Estimates derived from the second survey administration (M2) for a given month tended to be larger than those from the first administration - M2 estimates were larger than M1 estimates in 108 of 155 (69.7%) and 97 of 155 (62.6%) domains for shore and private boat fishing, respectively. This result supports the hypothesis that respondents are eager to report fishing activity and tend to report trips at the first opportunity (Andrews 2023). In the experimental design, the reference months are presented in chronological order - i.e. the earlier month (m2) is presented before the more recent month (m1). The 12-month fishing activity question was placed prior to the one-month reference periods to absorb out-of-scope trips or provide an outlet for respondents to identify as anglers. The fact that experimental estimates for a given month tend to be larger for the second survey administration (M2) - i.e. when the reference month is presented first in the two-month sequence - despite the placement of the 12-month question, suggests some residual telescoping for the first month in the question sequence. However, the direction of differences in estimates is not entirely systematic, and differences are generally small and not significant, indicating the effect of any residual telescoping is minimal. U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 16 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Andrews et al. (2018)5 tested a rolling, monthly survey administration and questionnaire that was identical to QEXP, with the exception of the placement of the 12-month fishing activity question, which was positioned after the two individual reference months. That study, which was limited to MA, MD, GA and FL and administered from July through December, resulted in mean relative differences between M2 and M1 estimates of 158% (mean ratio=2.58, median=1.85) and 110% (mean ratio=2.10, median=1.48) for private boat and shore fishing, respectively. In contrast, the current study, when limited to these states and months, results in mean relative differences between M2 and M1 estimates of approximately 15% for both fishing modes, demonstrating that the 12-month fishing activity question, when placed prior to the individual reference months, absorbs a substantial amount of reported fishing activity that otherwise would have been reported for the initial month in the two-month sequence (m2 in M2 in this case). This result is evidence that the experimental question order (i.e. placing the 12-month fishing activity question prior to the primary reference period) substantially reduces measurement error. Comparisons Between FES and Experimental Estimates Figure 3 compares production FES (QFES) and experimental (QEXP) estimates of total fishing effort, summed across all states, by fishing mode and survey wave. For the purposes of comparison, monthly experimental estimates have been summed to the 2-month wave level. Estimate comparisons for all states and reference waves are provided in Appendix C. Overall, experimental estimates of total angler trips are lower than FES estimates by 6.0% and 22.3% for shore and private boat, respectively, and experimental estimates are lower than FES estimates in all wave comparisons with the exception of wave 4 shore fishing. 5 Andrews et al. (2018) compared monthly estimates between two experimental questionnaires, one that was limited to a single month (T2) and a second (T1) that included two successive individual months. Both T1 and T2 included a 12-month fishing activity question that followed question(s) for the reference month. For the comparisons described in the manuscript, T1 estimates were derived from the most recent month - i.e. the month immediately preceding survey administration. Andrews (2023) compared estimated monthly fishing prevalence between successive survey administrations that utilized both reference months from the T1 questionnaire. U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 17 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Figure 3. Estimated angler trips derived from the Fishing Effort Survey (QFES, dark blue bars) and experimental questionnaire (QEXP, light blue bars) by survey wave. Error bars represent 95% confidence intervals around the estimates. Figure 4 shows the distribution of estimate ratios (QEXP:QFES) across all state*wave estimation domains by fishing mode for total angler trips. Ratios less than one occur when experimental estimates are lower than FES estimates, and vice versa. Mean ratios are 0.78 and and 0.91 for private boat and shore fishing U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 18 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 effort, respectively, which corresponds to a mean decrease between FES and experimental estimates of 22% and 9% for the respective modes. Ratios are significantly different from 1.0 for both shore (p=0.013) and private boat fishing (p<0.001) indicating that overall experimental estimates are significantly lower than FES estimates for both fishing modes. Figure 4. Distribution of experimental (QEXP) to FES (QFES) estimate ratios across all state*wave estimation domains for total angler trips by fishing mode. Within each box plot, the diamond symbol and horizontal line represent the distribution means and medians, respectively. Figure 5 shows the distributions of estimate ratios (QEXP:QFES) across states for each two-month reference wave. The figure demonstrates a strong seasonal pattern in the differences between estimates which aligns closely with seasonal patterns in fishing activity. Differences between QEXP and QFES are more pronounced (ratios are further from one) during waves with relatively low fishing activity for both fishing modes, and less pronounced, or even in the opposite direction (ratios closer to or greater than one), during waves with more fishing activity. For private boat fishing, the mean difference between experimental and FES estimates ranges from a 47.4% decrease (mean ratio=0.53) in wave 6 to a 4.5% increase (mean ratio=1.04) during wave 4, and differences are significant (p<=0.018) for all waves with U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 19 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 the exception of wave 4, which is not significant (p=0.5571). For shore fishing, the mean difference between experimental and FES estimates across waves ranges from a 36.3% decrease (mean ratio=0.64) in wave 6 to an 18.6% increase (mean ratio=1.19) in wave 4. Differences between experimental and FES estimates for shore fishing are significant for wave 2 (p=0.019), wave 4 (p=0.040) and wave 6 (p<=0.019), and not significant for wave 1 (p=0.061), wave 3 (p=0.7391) and wave 5 (p=0.2243). As described below, the general direction of differences between QEXP and QFES estimates, as well as the seasonal pattern of differences support the hypothesis that FES respondents overreport fishing activity. The fact that differences are greatest during periods of low fishing activity suggests that the motivation for overreporting is a desire to be identified as an angler. Figure 5. Distributions of experimental (QEXP) to FES (QFES) estimate ratios for total fishing effort across state domains by fishing mode and survey wave. Among states (Figure 6 and Appendix C) experimental estimates are consistently lower than FES estimates for private boat fishing - across waves the mean relative difference between experimental and FES estimates ranges from approximately a 1% decrease in NH (mean ratio=0.99) to more than a 44% decrease in MS (mean ratio=0.56), and total annual experimental estimates of private boat fishing effort are lower than FES estimates in all states with the exceptions of CT and NH. Differences in estimates U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 20 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 between designs are more variable for shore fishing, although annual experimental estimates are lower than FES estimates in 12 of 17 states. The exceptions are AL, HI, MA, NH and NJ. Across waves, mean relative differences between experimental and FES shore fishing estimates range from a 28% increase in NH (i.e. the mean experimental estimate across waves was 28% larger than the FES estimate, mean ratio=1.28) to a 35.3% decrease in ME (mean ratio=0.65). Significance testing was not completed at the state level due to the small number of comparisons between experimental and FES estimates (n=3-6, d.f.=2-5). Figure 6. Distributions of experimental (QEXP) to FES (QFES) estimate ratios across wave domains by fishing mode and state. Ratios are for total angler trips. Comparison of Estimate Components Estimated total fishing effort (π) can be expressed as the product of the estimated fishing prevalence (π), or the proportion of households that report fishing, the estimated mean trips per fishing household (π¦) and the total household population (N). U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 21 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 π= π ·π·π¦ Differences between experimental and FES estimates of total effort are largely the result of differences in fishing prevalence. Figure 7 shows the distributions of state prevalence ratios (QEXP:QFES) by fishing mode among the survey administrations that overlap (experimental surveys administered during even months). To be comparable with the FES, experimental prevalence is calculated as the proportion of households that reported fishing activity in either or both of the two reference months. Across all states and two-month waves, mean ratios are 0.84 and 0.79 for shore and private boat fishing, respectively, corresponding to mean relative decreases of 16.2% and 20.9% between FES and experimental prevalence estimates. Ratios for both fishing modes are significantly different from 1.0 (p<0.001). In addition, differences in estimated prevalence between the FES and experimental design show the same strong seasonal pattern as total effort - differences are largest (i.e. ratios are smallest) during low activity waves and smallest during high activity periods. For private boat fishing, mean differences in prevalence between the FES and experimental designs range from a decrease of nearly 40% in wave 1 (ratio=0.60) to 13% in wave 4 (ratio=0.87) and differences are significant for all waves with the exception of wave 2 (p=0.188). For shore fishing, the differences range from a decrease of 35% in wave 1 (ratio=0.65) to 2% in wave 4 (ratio=0.98), and differences are significant for waves 2, 5 and 6. The lack of a significant difference between experimental and FES shore prevalence estimates during wave 1 (ratio=0.75) can likely be attributed to the low power of the significant test (D.F.=4) resulting from the limitation of the survey to five states during the wave. The net effect of the differences in estimated prevalence between the FES and experimental designs is that the experimental design estimates fewer fishing households (π · π) than the FES. U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 22 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Figure 7. Distributions of experimental (QEXP) to FES (QFES) estimate ratios for fishing prevalence across state domains by fishing mode and survey wave. Differences in fishing prevalence between the FES and experimental designs are offset slightly by differences in mean trips per fishing household, which are generally in the opposite direction experimental estimates are larger than FES estimates. Figure 8 shows the distributions of QEXP:QFES estimate ratios across states for mean trips per fishing household by fishing mode and reference wave. As with prevalence, mean trip estimates for the pilot study are for summed values across the two individual months referenced during each administration for the administrations that overlap. Across all states and waves, mean ratios are 1.25 and 1.10, corresponding to mean relative increases between the FES and experimental design of 24.7% and 10.4% for shore and private boat fishing, respectively. Differences between FES and experimental estimates are significant (ratios are significantly larger than 1.0) for both fishing modes (p<0.001). For shore fishing, the mean relative difference between designs ranges from an increase (QEXP>QFES) of 13% (ratio=1.13) in wave 1 to 31.8% (ratio=1.32) in wave 4, and differences are significant for waves 2-6. For private boat fishing, the mean difference between the FES and experimental design ranges from an increase of 4% (ratio=1.04) in wave 6 to 24% (ratio=1.24) in U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 23 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 wave 4, and differences are significant for wave 4 (p=0.028). For mean trips, differences between experimental and FES estimates are greatest during wave 4 for both shore and private boat fishing. However, the strong seasonal effect we observed for prevalence and total fishing effort is much less pronounced, or possibly even absent. Figure 8. Distributions of experimental (QEXP) to FES (QFES) estimate ratios for mean trips per fishing household across state domains by fishing mode and survey wave. The relative differences between the FES and experimental designs for prevalence and mean trips per fishing household are opposite in direction - prevalence is higher in the FES and mean trips per fishing household is higher in the pilot study. With respect to estimating total fishing effort, these two measures do not “cancel each other out” because the sizes of the populations to which each of the measures apply are different. Fishing prevalence is a measure that applies to the entire household population, while mean trips per fishing household applies only to the population of households that participate in fishing, a very small fraction of the total household population. U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 24 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Andrews (2023) hypothesized that differences in estimated fishing activity between the FES questionnaire and an experimental questionnaire, which switched the order of the 12-month and 2-month fishing questions, was the result of telescoping error, or the reporting of fishing activity that occurred prior to the intended reference period. The placement of the 12-month fishing question before the 2-month question was intended to reduce telescoping error, a technique commonly referred to as bounded recall (Sudman et al. 1984, Loftus et al. 1990). The mechanism by which bounding techniques improve recall is uncertain, but may include memory stimulation, satisfying a need to provide socially desirable information or identify with a desired population, or suggesting a need for more precise information (Loftus et al. 1990). The direction of differences between FES and experimental estimates is consistent with the results reported by Andrews (2023), supporting the proposition that the FES is susceptible to overreporting. The observed seasonal pattern of differences between FES and experimental estimates provides additional support. For example, the likelihood that a respondent participated in fishing is greater during periods of high fishing activity than periods of low activity so there is less need during high activity periods to report out-of-scope fishing trips in order to identify as an angler. In other words, there’s a greater probability that a respondent can identify as an angler by accurately reporting fishing activity that occurred during the reference period. Not coincidentally, experimental and FES estimates of total fishing effort and fishing prevalence are most similar during high activity waves when overreporting is likely to be at a minimum. The opposite is true during periods of low fishing activity - the probability that a respondent can identify as an angler by reporting an in-scope trip is lower resulting in inaccurate reporting of out-of-scope trips and larger differences between FES and experimental estimates. The fact that differences between FES and experimental estimates correlate with seasonal trends in fishing activity provides strong support for the telescoping hypothesis and suggests that the experimental questionnaire, which presents the 12-month activity question first, more effectively mitigates reporting error by providing an opportunity for respondents to identify as anglers without biasing the primary survey measure. The direction of differences between the FES and experimental designs for mean trips per fishing household also supports the telescoping hypothesis, particularly if the motivation for misreporting is a desire to identify as an angler. This can be accomplished by reporting a single trip, which would result in lower estimates of mean trips per household as we observed for the FES. Figure 9, which compares the distributions of weighted trip frequencies across state*wave estimation domains between the FES and experimental design, demonstrates that FES respondents, on average, were more likely than experimental respondents to report a single fishing trip during the reference period for both boat fishing (38.8% of anglers vs. 33.5%) and shore fishing (28.6% vs. 24.2%). As with prevalence and mean trips per fishing household, trip frequencies are for FES and experimental survey administrations that overlap and include total fishing activity across both months of the reference period. Differences between the FES and experimental design in the percentage of anglers that reported a single trip are significant for both shore (p=0.007) and private boat fishing (p=0.021) and are in the direction that supports the hypothesis that FES respondents are identifying as anglers by reporting a single trip (or a small number of trips) at the first opportunity, which in the FES is the primary survey measure. U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 25 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Figure 9. Distributions of weighted angler trip frequencies for the FES (QFES) and experimental (QEXP) designs. U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 26 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Effects of Design Elements Thus far, we have explored differences between experimental and FES estimates and attributed those differences to differential measurement errors. Previous sections describe in detail differences between the FES and current experimental designs. Differences include: 1) a modified questionnaire with both a revised question order (12-month fishing activity question precedes primary reference period), as well as a primary reference period split into two individual months, 2) monthly sampling and a rolling, monthly administration that includes a one-month overlap between adjacent survey administrations, and 3) as a result of the rolling, monthly design, a composited estimation design that combines data from adjacent survey administrations. In addition, the overall sample size for the experimental design was twice that of the FES. The following section explores the effects of these design elements on survey estimates and comparisons with FES estimates. We also consider how these design changes may have differential effects across states and fishing modes. Because the experimental and FES questionnaires differed in both the question order and splitting of the primary reference period, we were interested in evaluating the individual effects of these two design features on estimates. While the current study was not a factorial design that allowed for direct comparison between and among the effects of these design elements, over the course of the past several years we have tested all combinations of the two design features. Specifically, we tested: 1) Andrews et al. (2018) maintained the FES question order but split the two-month reference period into individual months, 2) Andrews (2023) maintained the single, two-month reference period but switched the question order, and 3) the current study (2024) switched both the question order and split the two-month reference period into individual months. Each of these studies was conducted in parallel to the production FES, providing a consistent reference point to evaluate the effects of the individual design features. Table 5 summarizes the differential questionnaire design features for the three studies and the FES. Table 5. Comparison of questionnaire design features for the current FES and three studies testing different combinations of question order and the format of the primary reference period. Question Order Primary Reference Period Format FES Primary / 12-mo Single Andrews et al (2018) Primary / 12-mo Split Andrews (2023) 12-mo / Primary Single Current exp (2024) 12-mo / Primary Split To normalize results across studies, we compared estimate ratios (QEXP:QFES) for mean trips per fishing household and fishing prevalence. This approach assumes that the relationship between FES and U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 27 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 experimental estimates is consistent across time and space. Figure 10 provides the distribution of estimate ratios for mean trips per fishing household across all state*wave estimation domains by fishing mode for the three studies noted above. As previously described, estimates are based upon the reported number of trips across both reference months for the survey administrations that overlap directly with the FES. Mean ratios are larger than 1.0 for all three studies, demonstrating that experimental estimates of mean trips per household are larger than FES estimates. Estimates derived from Andrews (2023) are most similar to FES estimates with mean ratios of 1.05 and 1.04 (mean relative difference of 5 and 4%) for shore and private boat fishing, respectively. The similarity between these experimental estimates and the FES suggests that question order alone has a relatively minor effect on the number of trips reported by fishing households. In this case, the difference in mean trips between the experimental design and FES may reflect a tendency of FES respondents to report a single trip (or small number of trips) in order to identify as an angler as discussed previously. Differences between experimental and FES estimates are larger for the two studies that split the two-month reference period into individual months (Andrews et al. 2018 and the present study). For private boat fishing, relative differences between experimental estimates and the FES are 13% (ratio=1.13) and 14% (ratio=1.14) for Andrews et al. (2018) and the present study, respectively. Differences between experimental and FES estimates are larger for shore fishing - 72% (ratio=1.72) and 27% (ratio=1.27) relative differences for the respective studies. For shore fishing, the mean ratio for Andrews et al. (2018) is skewed by a pair of outliers and a small sample size (n=12 estimation domains). However, these results suggest that the format of the primary reference period (a single two-month question vs two individual month questions) has a systematic effect on the number of trips reported by fishing households, which seems to be greater for shore fishing than private boat fishing. This likely contributes to the observed differential effects between fishing modes in comparisons between experimental and FES estimates of total fishing effort. U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 28 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Figure 10. Distribution of estimate ratios for mean trips per fishing household between experimental designs (QEXP) and the FES (QFES) for pilot studies that tested different questionnaire design features6. Figure 11 shows the distribution of estimate ratios for fishing prevalence across all state*wave estimation domains by fishing mode for the three studies. For private boat fishing, the difference between experimental and FES prevalence was greatest for the two designs in which the 12-month activity question preceded the 2-month activity question, regardless of whether the primary two-month reference period was a single period (Andrews 2023) or split (present study 2024). The mean relative difference between experimental and FES estimates for the respective studies was 26% (ratio=0.74) and 21% (ratio = 0.79). In contrast, the difference between experimental and FES estimates for Andrews et al. (2018), where the question order matched the FES, was negligible (ratio=1.01). This result suggests that question order has a larger effect on private boat fishing prevalence than the format of the primary reference period. For shore fishing, experimental estimates of prevalence are lower than FES estimates in all three studies (mean ratios=0.83, 0.72 and 0.84 for Andrews et al. (2018), Andrews (2023) and the present study (2024), respectively). Based upon this result, we can’t attribute differences between experimental and 6 To avoid differential sample size effects on the comparisons, estimates were derived from survey weights that do not include trimming adjustments. The effects are sample size and trimming on estimates are described below. U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 29 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 FES estimates of shore fishing prevalence to a single design attribute. However, given the results for boat fishing prevalence, as well as results described by Andrews (2023) from initial FES questionnaire testing, it seems likely that question order contributes to differences between experimental and FES prevalence estimates for both fishing modes. We cannot rule out the possibility that the split reference period also contributes to differences between designs for shore prevalence. Figure 11. Distribution of estimate ratios for fishing prevalence between experimental designs (QEXP) and the FES (QFES) for pilot studies that tested different questionnaire design features7. While not conclusive, these results suggest that question order (order of the 12-month and 2-month fishing activity questions) and the format of the 2-month activity question (single vs. split reference period) have differential effects on survey measures. Placement of the 12-month question prior to the 2-month question results in lower estimated prevalence for the two-month period for both fishing modes. This is consistent with the telescoping hypothesis (Andrews 2023), which suggests that respondents are eager to identify as anglers and report fishing activity at the first opportunity. Placing the 12-month question prior to the 2-month question provides an opportunity for respondents to 7 To avoid differential sample size effects on the comparisons, estimates were derived from survey weights that do not include trimming adjustments. The effects are sample size and trimming on estimates are described below. U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 30 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 identify as anglers that results in lower (and presumably more accurate) prevalence for the primary reference period. Splitting the reference period into individual months appears to have minimal effect on boat fishing prevalence. Based on the available information, we cannot rule out the possibility that the format of the 2-month activity question (single vs. split reference period) has an effect on shore fishing prevalence. Conversely, the split reference period results in larger estimates of mean trips per fishing household than the single reference period, and the effect is larger for shore fishing than private boat fishing. It’s not clear if differential fishing mode effects are due to the placement of the mode-specific questions on the questionnaire (e.g. shore fishing questions precede boat fishing questions) or differences in the nature of the respective fishing activities that might affect recall/reporting (e.g. differences related to expenses, time investment, etc.). Additionally, we cannot determine which format results in more accurate reporting. We believe it is likely that the split reference period enhances memory or effectively conveys a need for the respondent to provide more precise information (Loftus 1990). However, it’s also possible that the split reference period increases reporting error through some other mechanism such as increased respondent burden and satisficing (Krosnick et al. 1996). We cannot rule out the possibility that the differences among experiments for either prevalence or mean trips are the result of uncontrolled year or sample size effects. Additional study is needed to more definitively determine which question format results in more accurate reporting. However, it’s clear that the format of the questionnaire impacts survey estimates and contributes to observed differences between experimental and FES estimates in the current study. With respect to monthly sampling administration designs, Andrews et al. (2018) compared the FES to two experimental monthly designs: a questionnaire that asked about a single reference month, and a questionnaire that asked about the two months individually (the reference month of interest preceded by a bounding month). Results found that aggregated two-month wave estimates were systematically higher than FES estimates with the single month experimental questionnaire (T2 in Andrews et al. (2018)). However, the effect was eliminated for the experimental questionnaire where the reference month was bounded by the prior month (T1). In that study, differences between experimental and FES estimates were likely the result of an increased risk of telescoping error associated with the unbounded single one-month reference period rather than an artifact of the monthly sampling administration design (Andrews et al. 2018). The monthly design tested by Andrews et al. (2018) utilized a rolling survey administration but treated each sampling period as a discrete event and did not utilize composite estimation - estimates for the T1 treatment were derived from the month immediately preceding survey administration, which was the most recent month of the two-month sequence. For the current 2024 study, we previously noted relatively minor differences in monthly estimates between successive survey administrations (for a given month, the second survey administration, M2, tended to result in larger estimates than the first, M1) likely due to the ordering of the 12-month question before the two individual month questions. However, the effect of survey administration was generally small and not entirely systematic. In addition, composited estimates are generally more precise (i.e. less variable) because they are based upon larger U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 31 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 samples and are derived from survey weights that are optimized specifically to reduce variance. These design features reduce the effect of single, atypical samples on results and reduce the effects of highly variable estimates that could be misinterpreted as bias. It’s unlikely that either the monthly sampling administration design or composite estimation contribute to differences between experimental and FES estimates. One additional notable difference between the FES and current experimental design, as well as between the current and design tested in 2019 by Andrews (2023) is the sample size. The total sample size for the 2024 FES was 364,441 addresses, distributed among 17 states and six reference two-month waves, an average of 4,237 per state and wave. The total sample size for the 2019 pilot study was 41,606 addresses, distributed among 16 states (HI was not included in the 2019 study) and three reference waves, for an average sample size of 904 addresses per state and wave. In contrast, the total sample size for the current 2024 study was 729,069 addresses, distributed among 17 states and 12 reference months. The average sample size for each state and reference month was 4,239 addresses, and the average sample size for each two-month wave would be twice the monthly sample size (~8,500). Because the current experimental design includes monthly composite estimation, sample from three survey administrations contribute to each wave-level estimate. For example, experimental estimates for wave 2 (March and April) 2024 are derived from the March (Feb/Mar), April (Mar/Apr) and May (Apr/May) survey administrations. Consequently, the total experimental sample contributing to each wave estimate is actually larger than the sum of the two survey administrations that comprise each wave. Table 6 provides the number of completed surveys that contributed to wave estimates for the 2019 pilot study, the 2024 FES and the 2024 study. Table 6. Completed surveys contributing to estimates for each two-month reference wave. The 2019 experiment was not conducted during waves 1-3 and did not include HI. Survey Wave 2024 FES 2024EXP 2019 FES 2019 EXP 7,147 39,370 5,853 1 - 25,704 63,974 30,818 2 I• 13,237 38,865 17,323 - 3 - 12,340 41,173 12,908 3,541 4 I• 5 - 15,766 50,909 16,825 4,617 19,584 38,008 17,056 4,632 6 While a central focus of precision and sampling error considerations, sample size is rarely considered in discussions of non-sampling errors and bias. However, considering the weighting design of the FES and pilot studies, as well as the rare nature of the fishing population, it’s likely that differences in sample sizes across the studies contribute to differences in estimates, even if some of these effects are not U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 32 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 classified as traditional non-sampling errors within the context of probability-based survey designs and repeated sampling. First we consider the role of sample size on a particular estimation procedure (trimming), and then we discuss sample size and estimation of rare events. Trimming The final weighting step for the FES design is a trimming procedure that reduces the weights of extreme values with a goal of reducing the variance of estimates (Potter 1990). The procedure balances the tradeoff between variance and bias - i.e. variance is reduced at the cost of introducing bias. The specific goal of the FES trimming procedure is to identify a trimming threshold that minimizes the mean squared error (MSE) of the estimate. The MSE of an estimator (π) is equal to the sum of the variance and the square of the bias. 2 πππΈ(π) = πππ(π) + (π΅πππ (π)) In the MSE trimming procedure used for the study, MSE(ππ‘) is repeatedly estimated for different levels of trimming (t) until an optimal MSE is reached when the reduction in variance is expected to offset the increase in squared bias introduced by the trimming. The procedure assumes that the untrimmed estimate (π) is unbiased and thus ( ) ( ) ( πππΈ ππ‘ = πππ ππ‘ 2 ). + π − ππ‘ Of course, in practice the variance of the estimate and the bias squared must be estimated from the current sample. Estimates of variances (and of squared biases that are of the same order of magnitude) are very unstable in most surveys because the surveys are designed to produce precise estimates and the precision of the variance estimates are of less concern. As a result, the trimming thresholds derived from these calculations are also not very precise, especially when sample sizes are not very large. Because there is an inverse relationship between sample variance and sample size, estimates derived from smaller samples will have larger variances. In the context of MSE trimming, larger variances will result in larger trimming thresholds and more bias. We evaluated the effect of differential sample sizes between the 2019 and 2024 pilot studies, as well as the 2019 and 2024 FES, by comparing the estimated relative bias, or the relative difference between trimmed and untrimmed estimates of total fishing effort. Figure 12 shows that the estimated relative bias was greater for the 2019 study than any of the other data collections, reflecting a larger trimming threshold in the 2019 study. Across all estimation domains, the estimated mean relative bias for shore fishing estimates was 20.3%, 12.5%, 9.9% and 10.4% for the 2019 study, 2024 study, 2019 FES and 2024 FES, respectively. For private boat fishing, the estimated mean relative bias was 20.0%, 13.9%, 10.8% and 12.5% for the respective surveys. The difference in estimated relative bias between the 2019 pilot study and the 2019 FES, as well as between the 2019 pilot study and the 2024 FES and 2024 pilot study, reflects the large differences in sample sizes - the U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 33 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 smaller sample sizes in the 2019 pilot study resulted in larger variance estimates and consequently more trimming (and greater bias) to reduce variability. The differences in estimated relative bias between the 2024 FES and 2024 pilot study are relatively minor. Consequently, differences in estimates between the two designs reflect a true differential effect, likely resulting from differential measurement errors, rather than an artifact of sampling or estimation. The similarities in relative bias between the 2024 FES and experimental design may not be intuitive because the total sample size for the 2024 pilot study was twice that of the 2024 FES for each two-month wave. However, both compositing and summation of monthly estimates to the wave level occurs following trimming, so the sample to which trimming applies is approximately equivalent between the 2024 FES and pilot study. U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 34 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Figure 12. Distribution of estimated relative bias across state*wave estimation domains for the 2019 pilot study, 2024 pilot study, 2019 FES and 2024 FES. For each estimation domain, estimated relative bias is the difference between the untrimmed and trimmed estimate, divided by the untrimmed estimate. The 2019 pilot study was limited to waves 4-6 and excluded HI. For the purpose of comparison, estimated relative bias for the 2019 FES was limited to estimation domains that overlapped with the pilot study. The general conclusion from examining estimated relative bias is that much of the difference between the 2019 and 2024 pilot studies in the magnitude of differences between experimental and FES estimates can be attributed to weight trimming associated with sample sizes rather than differential nonsampling errors associated with design differences. The effect of trimming is a nonsampling error in the class of processing errors that is not often examined. In this situation, differences in sample sizes between the 2019 and 2024 pilot studies, as well as between the 2019 pilot study and 2019 FES resulted in a systematic effect leading to a directional bias. U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 35 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Rare Events Next, we explore the relationship between sample size and the ability to detect rare events, specifically households that reported large numbers of fishing trips. The issue of sampling and estimation for rare events is an important methodological one because it has potentially large effects on the reliability of estimates. Kalton and Anderson (1979) provide a review of the sampling and estimation issues, and Tourangeau et al. (2014) provide a more recent monograph. The mean estimated fishing prevalence (percent of households reporting a fishing trip during the two month wave) across all states and waves since 2018 is 4.1% (range of 1.2%-10.8%) for private boat fishing and 5.9% (1.8%-15.0%) for shore fishing. These estimates, derived from the FES (2018-2024), reveal that saltwater fishing is a relatively rare activity among the household population. Even rarer are fishing households reporting a large number of trips within a wave. Historical FES data demonstrate that, within the population of fishing households, a large majority (65% of shore fishing households and 71% of boat fishing households) reported five or fewer fishing trips during the two-month wave. At the other end of the distribution, approximately 3% of shore fishing households and 2% of private boat fishing households reported 30 or more days fishing during the reference wave. Relative to the full household population, the mean prevalences for households that reported 30 or more trips are 0.19% and 0.09% for shore and private boat fishing, respectively. Estimating a quantity such as the percent of households that have 30 or more trips per wave is a very rare event and faces all of the challenges associated with rare event estimation including having sufficient sample size to produce precise estimates. One way to examine the challenges associated with measuring rare events in the context of this study is to look at the expected number of households that would be included in the FES and experimental samples for different levels of fishing activity. This can be estimated using historical data from the 2018-2024 FES. We make a simplifying assumption that households are selected within the state with equal probability and that fishing activity is uniform across the state. As noted earlier, both the FES and experimental designs use auxiliary information and stratification to oversample households that are more likely to fish, but we ignore this design feature to illustrate the nature of the issue. Table 7 shows the expected number of responding households reporting 30 or more fishing trips for the experimental and FES samples by mode, state, and wave. The expected number is the product of the historical prevalence for the cell (estimated percentage of households reporting 30 or more fishing trips) and the observed number of respondents in the experimental and FES samples. Cells with an expected number of respondents less than 1 are highlighted. When an estimation domain has no respondents due to sampling, the result is called a sampling zero as opposed to a structural zero (a structural zero means that even if the entire population responded none would fall into that cell). When a sampling zero occurs, the estimate for the cell is, by default, zero. Conditionally, this estimate is biased since it is known that zero is not a feasible estimate (the population contains some households in the category). Unconditionally, over conceptual repeated samples, the estimate is unbiased because the zero estimate is averaged with non-zero estimates when one or more respondents is realized in the sample. While this U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 36 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 averaging over conceptual repeated samples is a useful construct, for the particular sample with a sampling zero the effect is under-estimation. Table 7 shows that the larger sample sizes in the experimental sample leads to far fewer expected sampling zeros (respondent samples of less than 1) than the FES sample. For shore fishing, 36% of the cells in the FES are sampling zeros compared to only 5% for the experimental sample. For private boat fishing, the frequency of domains with expected sampling zeros are 71% for the FES and 15% for the experimental samples. A sampling zero for households reporting relatively large numbers of trips is especially problematic for estimating total fishing effort. Based on historical FES data, households reporting 30 or more trips account for a disproportionate amount of the total fishing activity - approximately 20% of total shore trips and 15% of boat trips across all states and waves. Thus, exclusion due to a sampling zero in the category may lead to a substantial underestimate of total fishing effort. Considering actual sample sizes and the simplified assumptions used for this analysis, the effects of sampling zeros on estimates will be greater for the FES than the experimental design. As noted, this was a simplified example, which ignored key design elements, to more clearly demonstrate a sample size effect. In practice, the FES and experimental designs utilize stratification and oversample households that are both more likely to fish and report a large number of fishing trips. This design helps ensure reasonable probabilities of detecting households across the full spectrum of fishing activity, and also mitigates the effects of sampling zeros by limiting their potential effects to a subset of the estimation domain (e.g. to a stratum within a state*wave domain). However, it’s possible that sample size effects persist in some estimation domains, potentially confounding the analysis of questionnaire effects and contributing to differential effects among states. Any residual sample size effects are likely to reduce the magnitude of difference between FES and experimental estimates. Fishing activity is highly variable among states and throughout the year. Thus, quantifying the effects of sampling zeros requires a stratum-level analysis across all states and reference periods that is beyond the scope of this report. U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 37 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Table 7. Expected number of responding households under simple random sample using historical prevalence estimates (2018-2024) of households reporting 30 or more days of fishing, by mode, State, wave for experimental and FES sample sizes. Cells with less than 1 expected sampled household are highlighted. Shore Wave 1 Wave 2 Wave 3 Wave 4 Wave 5 Wave 6 State Exp FES Exp FES Exp FES Exp FES Exp FES Exp FES AL 1.99 0.77 1.87 0.65 1.65 0.6 4.59 1.05 3.3 1.24 1.18 0.61 CT 3.53 1.53 2.1 0.67 2.22 0.73 2.71 0.61 3.34 1.63 DE 7.19 3.01 7.15 2.67 7.29 2.08 5.66 1.42 4.83 2.43 7.65 2.47 5.3 1.71 5 1.53 4.26 1.51 2.79 1.53 2.25 0.92 1.45 0.45 1.18 0.39 1.16 0.4 0.58 0.32 19.17 8.11 23.41 7.27 27.29 8.49 29.12 9.92 16.44 8.44 MA 4.36 1.98 1.98 0.73 2.63 0.72 4.84 1.15 MD 2.37 0.96 2.35 0.75 3.41 1 4.55 1.36 0.96 0.53 6.18 2.44 3.91 1.08 2.91 1.4 FL 3.68 1.2 GA HI 30.23 10.77 ME MS 1.84 0.95 5.19 1.94 4.12 1.39 8.56 3.13 6.49 2.01 9.09 2.65 NC 4.94 2.51 4 1.56 2.86 1.19 1.5 0.48 2.6 0.83 4.03 1.24 5.72 2.88 0.62 0.2 2.79 0.76 NH NJ 0.8 0.38 7.95 3.35 1.91 0.65 6.33 2.01 1.35 0.34 NY 3.43 1.73 3.12 1.3 3.06 1.04 1.32 0.36 2.58 0.71 RI 2.51 1.31 10.11 4.58 4.69 1.8 5.97 1.76 8.42 2.2 SC 3.35 1.67 5.06 1.88 6.15 1.92 5.31 1.86 5.56 1.63 VA 1.79 0.95 4.88 2.01 1.7 0.6 1.76 0.53 2.33 0.72 U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 38 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Private Boat Wave 1 Wave 2 Wave 3 Wave 4 Wave 5 Wave 6 State Exp FES Exp FES Exp FES Exp FES Exp FES Exp FES AL 1.59 0.62 0.32 0.11 1.05 0.38 1.78 0.41 1.8 0.68 0.47 0.24 CT 1.05 0.45 0.72 0.23 1.52 0.5 1.78 0.4 2.63 1.28 DE 2.66 1.11 3.84 1.44 2.99 0.85 2.02 0.51 1.96 0.98 2.95 0.95 4.88 1.57 5.55 1.69 3.87 1.37 1.12 0.62 0.96 0.39 0.39 0.12 0.8 0.26 0.59 0.2 0.36 0.2 4.87 2.06 1.61 0.5 4.46 1.39 5.37 1.83 2.4 1.23 MA 2.95 1.34 1.8 0.66 1.89 0.52 3.36 0.8 2.58 1.34 MD 1.48 0.6 1.8 0.58 3.39 1 2.62 0.78 0.75 0.41 2.27 0.9 4.2 1.15 1.99 0.95 FL 1.61 0.53 GA HI 3.62 1.29 ME MS 2.23 0.84 2.38 0.8 4.14 1.51 2.22 0.69 3.67 1.07 2.42 1.22 NC 1.3 0.51 1.26 0.52 1.58 0.51 1.75 0.56 2.43 0.75 1.29 0.67 1.04 0.33 3.05 0.84 1.28 0.6 NH NJ 3.76 1.58 1.59 0.54 3.03 0.96 1.08 0.27 0.75 0.38 NY 1.97 0.82 1.97 0.67 3.25 0.89 2.36 0.65 1.87 0.95 RI 2.69 1.22 2.37 0.91 7.22 2.13 3.36 0.88 1.66 0.83 SC 1.81 0.67 3.47 1.08 5.17 1.81 3.1 0.91 2.3 1.23 VA 3.03 1.25 0.78 0.28 0.98 0.3 0.53 0.17 0.54 0.28 U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 39 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Summary and Conclusions The experimental design tested in this study differed in several fundamental ways from the current FES design. First, the experimental study included monthly sampling, survey administration, and estimation that increased the temporal resolution of the survey. In addition, the rolling monthly design, including a one-month overlap between adjacent survey administrations, allowed us to implement a composite estimation design that improves the precision of estimates and reduces potential effects of atypical periods (e.g. unusually high or low estimates with high variability) on final survey results. This results in more stable estimates and will potentially provide some relief in the event of reduced budgets. To accommodate the sampling and estimation design, the experimental questionnaire included a primary, two-month reference period that was split into the component months. In addition, the questionnaire built upon results from previous studies that recommended switching the order of the 12-month and two-month reference periods to reduce measurement error. As with previous studies (Andrews 2023) experimental estimates were lower than FES estimates for both private boat and shore fishing. This result is generally consistent across states and reference waves, although the largest differences between experimental and FES estimates are observed during periods of lower fishing activity. Across all states and waves, the mean difference between experimental and FES estimates is 22% and 9% for private boat and shore fishing, respectively. Of the differences between the experimental and FES designs, we found that the question order (i.e. placing the 12-month fishing activity question prior to the primary reference period) has the largest impact on survey estimates. The effect is especially apparent when comparing results between the present study and Andrews et al. (2018), who tested an identical design, with the exception of the placement of the 12-month fishing activity question, which was positioned after the two individual reference months. Andrews et al. (2018) observed that estimates derived from the second survey administration for a given reference month (when that month appeared first in the two-month sequence) were, on average, 158% and 110% larger than estimates derived from the first administration for the month (when the month appeared second in the two-month sequence), for private boat and shore fishing, respectively. For the current study, the first and second survey administrations for a fixed reference month differ in the position of the month in the two-month sequence and the recall period. We also observed that estimates derived from the second survey administration tended to be larger than those from the first administration in 2024, but differences were much smaller - approximately 26% for both fishing modes across all state*month domains, and approximately 15% for the states and waves included in Andrews et al. (2018). We suggest that the 12-month fishing activity question, when placed prior to the primary reference period, has a beneficial bounding effect and satisfies respondents’ desire to identify as an angler and reduces measurement error for the primary survey measures. When the 12-month fishing activity question is absent or placed after the primary reference period, respondents are more likely to U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 40 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 rely upon the primary reference period to satisfy this desire, and the bounding effect is reduced or absent, resulting in misreporting (and overestimation) of fishing activity. The effect of question order is manifested in estimated fishing prevalence - fewer anglers report fishing during the reference period (i.e. prevalence is lower) when the 12-month fishing activity question precedes the primary reference period. We also observed an effect of the format of the primary reference period question (e.g. single or split reference period) on estimates of mean trips per fishing household. Estimates are larger for the split design than the single design, and the effect is larger for shore fishing than private boat fishing. It’s unclear if the difference in magnitude of the effect between shore and private boat fishing is the result of the placement of the mode-specific questions on the questionnaire or differences in the nature of the respective fishing activities that may affect recall. Regardless, the differential effect of the split reference period between fishing modes on mean trips per fishing household contributes to the differential effect of the overall experimental design on total fishing effort and comparisons with the FES - differences between experimental and FES estimates are larger for private boat fishing than shore fishing. We are not able to determine from currently available information whether the single or split primary reference period provides more accurate information. We also noted that sample sizes could potentially impact estimate comparisons. In the current 2024 study, it’s unlikely that differences in sample sizes between the experimental and FES designs distort the comparisons of the questionnaire effects. Weight trimming reduces the variability of survey estimates but introduces bias. We observed that the magnitude of relative bias introduced as a result of weight trimming appears to be similar between the experimental design and the FES in 2024. Consequently, comparisons between FES and experimental estimates are likely to reflect true effects of differential design elements. Differences in sample size were much larger between the 2019 pilot and FES. Consequently, the estimated relative bias resulting from weight trimming was greater for the pilot study than the FES and subsequent comparisons between experimental and FES estimates likely overstated the true magnitude of differences. When using final, trimmed weights, the mean difference between experimental and FES estimates was approximately 30%. When using untrimmed weights, the mean relative difference between experimental and FES estimates is approximately 22%. This value is closer to the observed differences between experimental and FES estimates in the present study and is likely a more realistic measure of the relative effects of differential survey design features. Generally, results from this study are consistent with results reported by Andrews et al. (2018) and Andrews (2023). It provides additional support for the hypothesis that the current FES design, in which the primary reference period precedes the 12-month fishing activity question, is susceptible to measurement error and likely overestimates fishing activity. The study also demonstrates the feasibility of monthly sampling, as well as the benefits of the composite estimation design, including greater precision and stability of survey estimates. Future research should continue to evaluate the cognitive properties of the survey questionnaire, specifically with respect to the split primary reference period and differential measurement errors associated with different types of fishing activities U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 41 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 References AAPOR. 2016. Address-based sampling. Retrieved from: https://aapor.org/wp-content/uploads/2022/11/AAPOR_Report_1_7_16_CLEAN-COPY-FINAL-2.pdf. Andrews, R. 2023. Evaluating measurement error in the MRIP fishing effort survey. Retrieved from: https://apps-st.fisheries.noaa.gov/pims/main/public?method=DOWNLOAD_FR_RPTS&record_id=32268. Andrews, R., J.M. Brick and N. Mathiowetz. 2014. Development and testing of recreational fishing effort surveys: Testing a mail survey design. Retrieved from: https://apps-st.fisheries.noaa.gov/pims/main/public?method=DOWNLOAD_FR_PDF&record_id=1179. Andrews, W.R., K.J. Papacostas, and J. Foster. 2018. A comparison of recall error in recreational fisheries surveys with one- and two-month reference periods. North American Journal of Fisheries Management 38: 1284-1298. Bernstein, R., A. Chadha and R. Montjoy. 2001. Overreporting voting: Why It happens and why it matters. Public Opinion Quarterly 65:22–44. Brenner, P.S. and J. DeLamater. 2016. Lies, damned lies, and survey self-reports? Identity as a cause of measurement bias. Social Psychology Quarterly 79(4): 333–354. Brick, P.D., J. Arrue, and M. Brick. 2024. Findings from cognitive testing of the fishing effort survey. Retrieved from: https://apps-st.fisheries.noaa.gov/pims/main/public?method=DOWNLOAD_FR_RPTS&record_id=38281. Cochran, W.G. 1977. Sampling Techniques. 3rd Edition, John Wiley & Sons, New York. Dillman, D.A., J.D. Smyth and L.M. Christian. 2014. Internet, Phone, Mail and Mixed-Mode Surveys: The Tailored Design Method. 4th edition. John Wiley, Hoboken, NJ. FES Annual Report. 2024. Retrieved from: https://apps-st.fisheries.noaa.gov/pims/main/public?method=DOWNLOAD_FR_RPTS&record_id=43821. Hadaway, K.C., P.L. Marler and M. Chaves. 1998. Overreporting church attendance in America: Evidence that demands the same verdict. American Sociological Review 63:122–30. Hadaway, K.C., P.L. Marler and M. Chaves. 1993. What the polls don’t show: a closer look at U.S. church attendance. American Sociological Review 58:741-752. Kalton, G., and D.W. Anderson. 1986. Sampling rare populations. Journal of the Royal Statistical Society, Series A, 149(1): 65-82. U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 42 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Krosnick, J.A., S. Narayan and W.R. Smith. 1996. Satisficing in surveys: initial evidence. New Directions for Evaluation 70: 29-44. Loftus, E.F., M.R. Klinger, K.D. Smith, and J. Fielder. 1990. A tale of two questions: benefits of asking more than one question. Public Opinion Quarterly 54:330-345. National Marine Fisheries Service Office of Science and Technology. 2023. Marine Recreational Information Program Survey Design and Statistical Methods for Estimation of RecreationalFisheries Catch and Effort. Silver Spring, MD. Retrieved from https://www.fisheries.noaa.gov/resource/document/survey-design-and-statistical-methods-estimation-r ecreational-fisheries-catch-and. Potter, F.J. 1990. A study of procedures to control extreme sampling weights. Proceedings of the Section on Survey Research Methods. American Statistical Association. 225-230. Shephard R.J. 2003. Limits to the measurement of habitual physical activity by questionnaires. British Journal of Sports Medicine 37:197–206. Sudman, S., A. Finn, and L. Lannom. 1984. The use of bounded recall procedures in single interviews. Public Opinion Quarterly 48(2):520. Tourangeau, R., B. Edwards, T.P. Johnson, K.M. Wolter, K. M., and N. Bates (Eds.). 2014. Hard-to-Survey Populations. Cambridge: Cambridge University Press. U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 43 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Appendix A: Experimental Questionnaire U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 44 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 45 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Appendix B: FES Questionnaire U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 46 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 47 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 Appendix C: Estimate Comparisons U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 48 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 49 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 50 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 51 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 52 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 53 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 54 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 55 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 56 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 57 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 58 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 59 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 60 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 61 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 62 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 63 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 64 A Test of a Revised Design of the NOAA Fisheries Fishing Effort Survey—2025 U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service 65