Statistical Methodology Checklist

June 8, 2023
U.S. Department of Transportation
Checklist for Paperwork Reduction Act Submissions
Before forwarding a Paperwork Reduction Act (PRA) submissions to the Bureau of Transportation Statistics
for review, the OA Manager of the sponsoring program office of the proposed information collection or the OA
CIO should review the package, and then complete, sign and date the following checklist. Any responses of
“No” marked on this checklist should be accompanied by a written explanation of why the question is not
applicable to the package under consideration. The “Reference Sections” covered in the final column are
provided in the attached “Summary of Administrative and Technical Features for Paperwork Reduction Act
Submissions in the Department of Transportation.”
Yes No N/A

✔

✔
✔
✔
✔
✔
✔
✔
✔

1

Topic or ICR Name:
Lessons Learned

Reference
Sections

The PRA submission and instructions have been reviewed by the OA
Manager or OA CIO, and by the senior technical representative responsible
for the survey. In addition, if the agency intends to use a contractor to
prepare a PRA submission, the contract must specifically require that the
contractor “prepare all materials necessary for a PRA submission for this
study, with format and content that meets the standards required by the BTS
and OMB.”
Payments or gifts to respondents, if any, are carefully justified

I.1-I.2

Confidentiality and disclosure limitation methods are fully specified;
including citing the authority for any confidentially pledge.

II.3

and
II.1
II.2

OA Manager or OA CIO, and OA statistical technical representative have
II.4
reviewed and approved all statements of accounting fact in the proposed
package.
Population, parameters, scope of inference and frames are fully and
II.5
explicitly specified, with detailed mathematical formulas where appropriate.
Sample design, sample sizes and degree of accuracy are fully and explicitly
specified, with detailed mathematical formulas where appropriate.
Probability weights and design-based point estimators are fully specified; or
model-based estimation methods (including all model validation procedures
to be used) are fully specified
Design-based variance estimators and inference methods fully specified; or
model-based variance estimators and inference methods are fully specified.
Explicit justifications of anticipated response rates are included

II.6
II.7
II.8
II.9

June 8, 2023

✔
✔
✔

Methods of nonresponse weighting adjustment and/or imputation are
explicitly defined and justified.

II.10

The proposed package includes one copy of each technical report or
reference cited in the package, except for materials widely available in the
public domain.
(For reviews after the initial review):
The package submitted to the BTS includes an appendix with detailed
point-by-point responses to each question raised by the reviewer in the
previous review.

II.11

OA Statistical Technical Representative:

John Waters

JOHN WILLIAM
WATERS IV

Digitally signed by JOHN
WILLIAM WATERS IV
Date: 2024.06.27
11:58:19 -04'00'

_______________________
Name

___________________
Signature

[email protected]

________________
Telephone number

_______________________
E-mail address

6/27/24

________
Date

(202) 267-7755

OA Manager or OA Chief Information Officer:

Jennifer Audette

_______________________
Name
[email protected]
_______________________
E-mail address

2

signed by
JENNIFER Digitally
JENNIFER J AUDETTE
2024.06.27
J AUDETTE Date:
10:51:43 -07'00'
___________________

Signature

(202) 702-1457
___________________
Telephone number

6/27/24

________
Date

II.12

June 8, 2023
Summary of Administrative and Technical Features for Paperwork Reduction Act Submissions in the
Department of Transportation
I. Administrative Features of the PRA Process in the Department of Transportation
I.1. Background
Under the Paperwork Reduction Act (44 U.S.C. chapter 35) and related regulations issued by the Office of
Management and Budget (5 CFR §1320), OMB approval is required for essentially all surveys conducted for
federal agencies. The OMB has established a process, for Paperwork Reduction Act submissions and related
questionnaires (“Request for OMB Clearance of Information Collection under the Paperwork Reduction Act of
1980. Part A: Justification. Part B: Collections of Information Involving Statistical Methods”) to be completed
by the agency that proposes to conduct the survey. This includes all surveys used for statistical purposes, and
also includes any census (100% sample) that collects data used to make inferences to a larger population.
Copies of OMB’s instructions for PRA submissions are available at:
http://www.whitehouse.gov/omb/inforeg/83i-fill.pdf 1
Somewhat shorter questionnaires are provided for customer satisfaction surveys.
In addition, pursuant to 5 CFR § 1320, the Department of Transportation has established a policy that before a
PRA submission is transmitted to the OMB, it must first be reviewed and approved by the DOT Clearance
Officer. For data collection work that will involve statistical methods (see section I.2.c below), the PRA
submission must also be reviewed and approved by the Bureau of Transportation Statistics.
The materials provided in this summary are intended for studies in which the lead agency is an agency in the
Department of Transportation other than the Bureau of Transportation Statistics.
I.2. Assignment of Responsibilities.
I.2.a. Lead Agency. The lead agency is responsible for specification of the purpose and scope of the study;
general management of the study; qualifications, selection, management and performance of the government or
contractor personnel assigned to the study; budgeting and statements of fact regarding accounting or respondent
burden; and inferences, interpretations or policy statements based on results of the study. Much of this work
requires a high degree of specialized technical and managerial training and experience, and it is the
responsibility of the lead agency to ensure that the personnel assigned to this work have the necessary training
and experience. Important evaluation criteria include a record of strong training and performance in similar
previous studies, and demonstrated capacity for continued strong performance in the proposed study. This
includes work in the following areas.
i.

Construction of probability sample designs appropriate for the proposed study. This generally involves
definition of target populations and parameters; frame development; stratification; multistage selection
and unequal-probability sampling.

N/A
Please note, the Form OMB 83-I is obsolete as of July 2006; however, OMB instructions for completing both Parts A and B of the
Justification for a Paperwork Reduction Act Submission are attached to the Form OMB 83-I and remain in effect.

1

3

June 8, 2023
ii.

If the agency proposes to use a census (100% sample), justification of the decision not to collect data on
a sample basis.

iii.

Development and implementation of appropriate statistical formulas for point estimation, variance
estimation and inference.

iv.

Design, testing and implementation of data collection instruments and field procedures.

v.

Data collection and management of field staff.

vi.

Management, cleaning, editing and nonresponse adjustment of data; and production of valid survey
estimates.

vii.

Interpretation of inferential results and production of appropriate reports.

viii.

Management of projects, personnel, information technology and finances; and compliance with all
applicable laws, regulations and contractual requirements.

ix.

In addition, if the agency intends to use a contractor to prepare a PRA submission, the contract must
specifically require that the contractor “prepare all materials necessary for a PRA submission for this
study, with format and content that meets the standards required by the BTS and OMB.”

I.2.b. DOT Clearance Officer. The DOT Clearance Officer manages the review and submission process for
PRA submissions; and is the primary point of contact in the DOT for questions regarding the said submissions.
I.2.c. Bureau of Transportation Statistics. For studies in the Department of Transportation for which the Bureau
of Transportation Statistics is not the lead agency, the role of the BTS centers on review of technical material in
the PRA submission, such as specification of the sample design, estimation methods, and related statistical
topics. For the PRA submissions, there are two distinct cases.

4

i.

Studies based on a 100% sample and limited inferential goals. In some cases, data collection by a DOT
agency will satisfy both of the following conditions. (A) Data collection is attempted for all units in the
inference population. (This often is referred to as a census or a 100% sample.) (B) No attempt will be
made to draw inferences to any population other than the set of units that responded to the data
collection effort in (A). (C) The response rate is expected to be close to 100%. If each of (A), (B) and
(C) are true, then there is no need for a review by the BTS. Instead, the lead agency should forward the
PRA submission to the DOT Clearance Officer, and should state in the Justification Part B, “No
statistical methods will be used. All data collection will be based on a 100% sample of the inference
population. In all reports and other publications and statements resulting from this work, no attempt will
be made to draw inferences to any population other than the set of units that responded to the data
collection effort.”

ii.

All other studies involving statistical methods. In keeping with the Paperwork Reduction Act and
related regulations for the PRA submission process, the Bureau of Transportation Statistics is asked to
review technical features of the sample design, and related statistical estimation and inference methods,
covered in a proposed PRA submission package. Issues regarding the purpose and scope of the study;

June 8, 2023
general management of the study; qualifications, selection, management and performance of the
government or contractor personnel assigned to the study; budgeting and statements of fact regarding
accounting or respondent burden; inferences, interpretations or policy statements based on results of the
study; or decisions by the DOT Clearance Officer, are outside the scope of BTS responsibility.
I.2.d. Office of Management and Budget. The Office of Management and Budget has final authority over
approval or disapproval of PRA submission packages submitted to it for review.

5

June 8, 2023
II. Technical Features of the PRA Submission Process
A copy of the “Request for OMB Clearance of Information Collection Under the Paperwork Reduction Act of
1980. Part A: Justification. Part B: Collections of Information Involving Statistical Methods” is attached to the
end of this summary. Parts A and B cover several important statistical topics.
For PRA submission packages handled through the Department of Transportation, the Bureau of Transportation
Statistics has developed a “Checklist for PRA Submissions” (attached).
II.1. Before the start of work on the study, the PRA submission; including, Parts A and B should be reviewed
by the OA Manager or OA CIO, and the OA statistical technical representative in the lead agency, and (if
applicable) by the responsible senior management and senior technical personnel in each contractor and
subcontractor. Questions A.1-A.8, A.11, A.15-A.18, B.4 and B.5 are largely self-explanatory. The remaining
questions in Parts A and B are covered in additional lines of the “Checklist for PRA Submissions” as follows.
II.2. Incentives. Question A.9 covers payments or gifts to respondents, other than remuneration of contractors
or grantees. The OMB generally discourages the use of gifts or payments to respondents in federal surveys. If
the lead agency considers such gifts or payments to be necessary, it is advisable to respond to Question A.9 with
a careful justification of these gifts of payments, with appropriate citations of relevant empirical literature on
respondent incentives, response rates and related methodological issues.
II.3. Respondent confidentiality. Question A.10 covers assurances of confidentiality given to respondents. The
agency must state whether the respondents will receive an explicit pledge of confidentiality in the collection
instrument or in related field procedures. If the respondents are given assurances of confidentiality, then the
answer to Question A.10 question should include explanation of these assurances, and the basis of these
assurances in statute, regulation, or agency policy. As part of the “agency policy” component, the response also
should include an explanation of disclosure-limitation procedures that will be used in dissemination of results
from this survey. This generally will involve a description of the level of aggregation at which results will be
published; and a description of specific methods that will be used to ensure that inadvertent disclosure will not
take place through, e.g., publication of tables at excessively fine levels of aggregation. For any cases involving
public-use microdata files, the answer to Question A.10 should also include an explanation of the microdata
disclosure-limitation procedures that will be implemented.
II.4. Statements of accounting fact. Items A.12, A.13 and A.14 cover, respectively, the respondent hour burden,
the respondent cost burden and the annualized cost to the federal government. As with any statement of
accounting fact, it is very important for the lead agency’s responses to items A.12, A.13 and A.14 be complete,
accurate and transparent. The OMB has stated that under the “total cost to the government” response to item
A.14, the lead agency should include full cost of the entire study, including the cost of sample and questionnaire
design; data collection; data management; editing, processing, analyzing, reviewing, and interpreting the
information; preparation of reports and documentation; reviewing reports; and (if applicable) preparing public
use files. If a contractor is doing some or all of the abovementioned tasks, then the cost of the contract should
be included, but agency staff time devoted to managing or doing the project should be included as well. Some
components, such as contractor costs, will be relatively easy to know precisely. Other components, such as the
costs of government employee time, will need to be a good-faith estimate. The costs reported in A.14 should be
broken down into major categories of expense.

6

June 8, 2023
II.5. Population, parameters, scope of inference and frames. Question B.1 covers several important statistical
issues. The response to Question B.1 should cover all of the issues raised in this question, either through direct
answers or through referral to a more detailed answer under Questions B.2 or B.3. In addition, the initial part of
the response to Question B.1 should include a clear and explicit statement of the inference population; an
explicit mathematical definition of the parameters of primary importance for the study; and a clear and explicit
statement of the frame(s) that will be used in the survey.
II.6. Specification of the sample design, sample sizes and degree of accuracy. The response to Question B.2
should cover all of the issues raised in this question. This should include explicit statements (with detailed
mathematical formulas as appropriate) of the specific methods that will be used for formation of strata,
definition of units at each stage of sample selection, calculation of selection probabilities. In addition, the
response to Question B.2 should include clear and explicit statements (including mathematical formulas where
appropriate) regarding: the degree of accuracy needed for the purpose described in Part A; assumed design
effects (with citation of appropriate technical reports or other references to support these design-effect
assumptions); and the resulting calculations of sample sizes that will ensure that the needed degree of accuracy
is satisfied.
II.7. Point estimation methods. The response to Question B.2 should also state clearly and explicitly whether
the proposed work will use design-based or model-based estimation and inference methods, and give a clear
justification for this choice. For design-based work, the response to Question B.2 should include clear and
explicit statements (including detailed mathematical formulas) for probability weights and design-based point
estimators for each of the principal parameters of interest, as specified in the response to Question B.1. For
model-based work, the response to Question B.2 should include clear and explicit statements (including detailed
mathematical formulas) of the statistical models being used; point estimators for each of the principal
parameters of interest, as specified in the response to Question B.1; and specific empirical procedures that will
be used to check the validity of the models under consideration.
II.8. Variance estimation and inference methods. The response to Question B.2 should also include clear and
explicit statements (including detailed mathematical formulas) for each of variance estimators and inference
methods (e.g., test statistics or confidence intervals) that will be used for the parameter estimators specified
under (II.7). The statements on variance estimation and inference methods should be clearly and explicitly
justified within the framework of the design-based or model-based approaches specified under (II.7). Any
additional approximations employed in this variance estimation work should be clearly and explicitly stated and
justified with detailed mathematical formulas and literature references, as appropriate.
II.9. Justification of anticipated response rates. The response to Question B.3 should cover all of the issues
raised in this question. In addition, the anticipated response rates in B.1, and the methods to maximize response
rates should be justified through careful citation of empirical and methodological literature that is relevant to the
proposed study.
II.10. Nonresponse adjustments. The response to Question B.3 also should include clear and explicit statements
(including mathematical formulas as appropriate) of the specific weighting adjustment or imputation procedures
that will be used. In addition, the underlying assumptions (e.g., homogeneity of response rates or mean values)
for these weighting adjustment and imputation procedures should be stated explicitly.
II.11. Reference materials. PRA submission packages generally include a bibliography that lists references
cited in the text of the package. Some of these references are textbooks, journal articles, web pages or other
7

June 8, 2023
materials that are widely accessible to the public. However, some of the references may include agency
technical reports or other materials that are not readily available to the BTS and OMB reviewers. For any cited
references in the latter category, one paper or electronic copy of the reference should be included with the PRA
submission package submitted to the BTS.
II.12. Point-by-point responses to issues raised in previous reviews. If the BTS does not approve a PRA
submission package on its first review, then the BTS generally will provide a report that lists specific questions
or areas in which the lead agency will need to do additional work before the package is ready for submission to
the OMB. When the lead agency submits a revised package to the BTS for further review, the lead agency
should include an appendix that provides point-by-point responses to the issues identified by the BTS in its
previous review.

8