Checklist for Paperwork Reduction Act Submissions OEG

U.S. Department of Transportation

Checklist for Paperwork Reduction Act Submissions

Before forwarding a Paperwork Reduction Act (PRA) submissions to the Bureau of Transportation Statistics for review, the OA Manager of the sponsoring program office of the proposed information collection or the OA CIO should review the package, and then complete, sign and date the following checklist. Any responses of “No” marked on this checklist should be accompanied by a written explanation of why the question is not applicable to the package under consideration. The “Reference Sections” covered in the final column are provided in the attached “Summary of Administrative and Technical Features for Paperwork Reduction Act Submissions in the Department of Transportation.”

Yes	No	Topic	Reference Sections
X		The PRA submission and instructions have been reviewed by the OA Manager or OA CIO, and by the senior technical representative responsible for the survey. In addition, if the agency intends to use a contractor to prepare a PRA submission, the contract must specifically require that the contractor “prepare all materials necessary for a PRA submission for this study, with format and content that meets the standards required by the BTS and OMB.”	I.1-I.2 and II.1
	X	Payments or gifts to respondents, if any, are carefully justified NOTE: No payments or gifts to respondents are planned or funded.	II.2
	X	Confidentiality and disclosure limitation methods are fully specified; including citing the authority for any confidentially pledge. Explanation: While the feedback portal does not collect any personally identifiable information, there has been no assurance or explicit pledge of confidentiality in the collection instrument or in related procedure.	II.3
X		OA Manager or OA CIO, and OA statistical technical representative have reviewed and approved all statements of accounting fact in the proposed package.	II.4
	X	Population, parameters, scope of inference and frames are fully and explicitly specified, with detailed mathematical formulas where appropriate. Explanation: No statistical methods will be used. In all reports and other publications and statements resulting from this work, no attempt will be made to draw inferences to any population other than the set of unites that responded to the data collection effort. Submissions to OEG are voluntary and individuals self-select to participate in feedback portal.	II.5
	X	Sample design, sample sizes and degree of accuracy are fully and explicitly specified, with detailed mathematical formulas where appropriate. Explanation: No statistical methods used. In all reports and other publications and statements resulting from this work, there will be no attempt made to draw inferences to any population other than the set of units responded to the data collection effort. Submissions to OEG are voluntary and individuals self-select to participate in the feedback portal.	II.6
	X	Probability weights and design-based point estimators are fully specified; or model-based estimation methods (including all model validation procedures to be used) are fully specified. Explanation: No statistical methods used. In all reports and other publications and statements resulting from this work, no attempt to draw inferences to any population other than the set of units that responded to the data collection effort. Submissions to OEG are voluntary and individuals self-select to participate in the feedback portal.	II.7
	X	Design-based variance estimators and inference methods fully specified; or model-based variance estimators and inference methods are fully specified. Explanation: No statistical methods used. In all reports and other publications and statements resulting from this work, no attempt to draw inferences to any population other than the set of units that responded to the data collection effort. Submissions to OEG are voluntary and individuals self-select to participate in the feedback portal.	II.8
	X	Explicit justifications of anticipated response rates are included Explanation: OEG is not designed to be a statistically valid data collection. OEG has not reviewed measures to maximum the response rate. OEG estimates approximately 8500 annual submissions based on the number of email.	II.9
	X	Methods of nonresponse weighting adjustment and/or imputation are explicitly defined and justified Explanation: No statistical methods used. In all reports and other publications and statements resulting from this work, no attempt to draw inferences to any population other than the set of units that responded to the data collection effort. Submissions to OEG are voluntary and individuals self-select to participate in the feedback portal.	II.10
X		The proposed package includes one copy of each technical report or reference cited in the package, except for materials widely available in the public domain. Note: A copy of the feedback portal is included. There are no technical reports to include.	II.11
		(For reviews after the initial review): The package submitted to the BTS includes an appendix with detailed point-by-point responses to each question raised by the reviewer in the previous review.	II.12

OA Statistical Technical Representative:

Fonda Woodard __________________ April 15, 2020 202-267-6501

Name Signature Date Telephone number

[email protected]

E-mail address

OA Manager or OA Chief Information Officer:

Michael Helvey __________________ April 15, 2020 202-267-9354

Name Signature Date Telephone number

[email protected]

E-mail address

Summary of Administrative and Technical Features for Paperwork Reduction Act Submissions in the Department of Transportation

I. Administrative Features of the PRA Process in the Department of Transportation

I.1. Background

Under the Paperwork Reduction Act (44 U.S.C. chapter 35) and related regulations issued by the Office of Management and Budget (5 CFR §1320), OMB approval is required for essentially all surveys conducted for federal agencies. The OMB has established a process, for Paperwork Reduction Act submissions and related questionnaires (“Request for OMB Clearance of Information Collection under the Paperwork Reduction Act of 1995. Part A: Justification. Part B: Collections of Information Involving Statistical Methods”) to be completed by the agency that proposes to conduct the survey. This includes all surveys used for statistical purposes, and also includes any census (100% sample) that collects data used to make inferences to a larger population. Copies of OMB’s instructions for PRA submissions are available at:

http://www.whitehouse.gov/omb/inforeg/83i-fill.pdf¹

Somewhat shorter questionnaires are provided for customer satisfaction surveys.

In addition, pursuant to 5 CFR § 1320, the Department of Transportation has established a policy that before a PRA submission is transmitted to the OMB, it must first be reviewed and approved by the DOT Clearance Officer. For data collection work that will involve statistical methods (see section I.2.c below), the PRA submission must also be reviewed and approved by the Bureau of Transportation Statistics.

The materials provided in this summary are intended for studies in which the lead agency is an agency in the Department of Transportation other than the Bureau of Transportation Statistics.

I.2. Assignment of Responsibilities.

I.2.a. Lead Agency. The lead agency is responsible for specification of the purpose and scope of the study; general management of the study; qualifications, selection, management and performance of the government or contractor personnel assigned to the study; budgeting and statements of fact regarding accounting or respondent burden; and inferences, interpretations or policy statements based on results of the study. Much of this work requires a high degree of specialized technical and managerial training and experience, and it is the responsibility of the lead agency to ensure that the personnel assigned to this work have the necessary training and experience. Important evaluation criteria include a record of strong training and performance in similar previous studies, and demonstrated capacity for continued strong performance in the proposed study. This includes work in the following areas.

1. Construction of probability sample designs appropriate for the proposed study. This generally involves definition of target populations and parameters; frame development; stratification; multistage selection and unequal-probability sampling.

2. If the agency proposes to use a census (100% sample), justification of the decision not to collect data on a sample basis.

3. Development and implementation of appropriate statistical formulas for point estimation, variance estimation and inference.

4. Design, testing and implementation of data collection instruments and field procedures.

5. Data collection and management of field staff.

6. Management, cleaning, editing and nonresponse adjustment of data; and production of valid survey estimates.

7. Interpretation of inferential results and production of appropriate reports.

8. Management of projects, personnel, information technology and finances; and compliance with all applicable laws, regulations and contractual requirements.

9. In addition, if the agency intends to use a contractor to prepare a PRA submission, the contract must specifically require that the contractor “prepare all materials necessary for a PRA submission for this study, with format and content that meets the standards required by the BTS and OMB.”

I.2.b. DOT Clearance Officer. The DOT Clearance Officer manages the review and submission process for PRA submissions; and is the primary point of contact in the DOT for questions regarding the said submissions.

I.2.c. Bureau of Transportation Statistics. For studies in the Department of Transportation for which the Bureau of Transportation Statistics is not the lead agency, the role of the BTS centers on review of technical material in the PRA submission, such as specification of the sample design, estimation methods, and related statistical topics. For the PRA submissions, there are two distinct cases.

1. Studies based on a 100% sample and limited inferential goals. In some cases, data collection by a DOT agency will satisfy both of the following conditions. (A) Data collection is attempted for all units in the inference population. (This often is referred to as a census or a 100% sample.) (B) No attempt will be made to draw inferences to any population other than the set of units that responded to the data collection effort in (A). (C) The response rate is expected to be close to 100%. If each of (A), (B) and (C) are true, then there is no need for a review by the BTS. Instead, the lead agency should forward the PRA submission to the DOT Clearance Officer, and should state in the Justification Part B, “No statistical methods will be used. All data collection will be based on a 100% sample of the inference population. In all reports and other publications and statements resulting from this work, no attempt will be made to draw inferences to any population other than the set of units that responded to the data collection effort.”

2. All other studies involving statistical methods. In keeping with the Paperwork Reduction Act and related regulations for the PRA submission process, the Bureau of Transportation Statistics is asked to review technical features of the sample design, and related statistical estimation and inference methods, covered in a proposed PRA submission package. Issues regarding the purpose and scope of the study; general management of the study; qualifications, selection, management and performance of the government or contractor personnel assigned to the study; budgeting and statements of fact regarding accounting or respondent burden; inferences, interpretations or policy statements based on results of the study; or decisions by the DOT Clearance Officer, are outside the scope of BTS responsibility.

I.2.d. Office of Management and Budget. The Office of Management and Budget has final authority over approval or disapproval of PRA submission packages submitted to it for review.

II. Technical Features of the PRA Submission Process

A copy of the “Request for OMB Clearance of Information Collection Under the Paperwork Reduction Act of 1995. Part A: Justification. Part B: Collections of Information Involving Statistical Methods” is attached to the end of this summary. Parts A and B cover several important statistical topics.

For PRA submission packages handled through the Department of Transportation, the Bureau of Transportation Statistics has developed a “Checklist for PRA Submissions” (attached).

II.1. Before the start of work on the study, the PRA submission; including, Parts A and B should be reviewed by the OA Manager or OA CIO, and the OA statistical technical representative in the lead agency, and (if applicable) by the responsible senior management and senior technical personnel in each contractor and subcontractor. Questions A.1-A.8, A.11, A.15-A.18, B.4 and B.5 are largely self-explanatory. The remaining questions in Parts A and B are covered in additional lines of the “Checklist for PRA Submissions” as follows.

II.2. Incentives. Question A.9 covers payments or gifts to respondents, other than remuneration of contractors or grantees. The OMB generally discourages the use of gifts or payments to respondents in federal surveys. If the lead agency considers such gifts or payments to be necessary, it is advisable to respond to Question A.9 with a careful justification of these gifts of payments, with appropriate citations of relevant empirical literature on respondent incentives, response rates and related methodological issues.

II.3. Respondent confidentiality. Question A.10 covers assurances of confidentiality given to respondents. The agency must state whether the respondents will receive an explicit pledge of confidentiality in the collection instrument or in related field procedures. If the respondents are given assurances of confidentiality, then the answer to Question A.10 question should include explanation of these assurances, and the basis of these assurances in statute, regulation, or agency policy. As part of the “agency policy” component, the response also should include an explanation of disclosure-limitation procedures that will be used in dissemination of results from this survey. This generally will involve a description of the level of aggregation at which results will be published; and a description of specific methods that will be used to ensure that inadvertent disclosure will not take place through, e.g., publication of tables at excessively fine levels of aggregation. For any cases involving public-use microdata files, the answer to Question A.10 should also include an explanation of the microdata disclosure-limitation procedures that will be implemented.

II.4. Statements of accounting fact. Items A.12, A.13 and A.14 cover, respectively, the respondent hour burden, the respondent cost burden and the annualized cost to the federal government. As with any statement of accounting fact, it is very important for the lead agency’s responses to items A.12, A.13 and A.14 be complete, accurate and transparent. The OMB has stated that under the “total cost to the government” response to item A.14, the lead agency should include full cost of the entire study, including the cost of sample and questionnaire design; data collection; data management; editing, processing, analyzing, reviewing, and interpreting the information; preparation of reports and documentation; reviewing reports; and (if applicable) preparing public use files. If a contractor is doing some or all of the abovementioned tasks, then the cost of the contract should be included, but agency staff time devoted to managing or doing the project should be included as well. Some components, such as contractor costs, will be relatively easy to know precisely. Other components, such as the costs of government employee time, will need to be a good-faith estimate. The costs reported in A.14 should be broken down into major categories of expense.

II.5. Population, parameters, scope of inference and frames. Question B.1 covers several important statistical issues. The response to Question B.1 should cover all of the issues raised in this question, either through direct answers or through referral to a more detailed answer under Questions B.2 or B.3. In addition, the initial part of the response to Question B.1 should include a clear and explicit statement of the inference population; an explicit mathematical definition of the parameters of primary importance for the study; and a clear and explicit statement of the frame(s) that will be used in the survey.

II.6. Specification of the sample design, sample sizes and degree of accuracy. The response to Question B.2 should cover all of the issues raised in this question. This should include explicit statements (with detailed mathematical formulas as appropriate) of the specific methods that will be used for formation of strata, definition of units at each stage of sample selection, calculation of selection probabilities. In addition, the response to Question B.2 should include clear and explicit statements (including mathematical formulas where appropriate) regarding: the degree of accuracy needed for the purpose described in Part A; assumed design effects (with citation of appropriate technical reports or other references to support these design-effect assumptions); and the resulting calculations of sample sizes that will ensure that the needed degree of accuracy is satisfied.

II.7. Point estimation methods. The response to Question B.2 should also state clearly and explicitly whether the proposed work will use design-based or model-based estimation and inference methods, and give a clear justification for this choice. For design-based work, the response to Question B.2 should include clear and explicit statements (including detailed mathematical formulas) for probability weights and design-based point estimators for each of the principal parameters of interest, as specified in the response to Question B.1. For model-based work, the response to Question B.2 should include clear and explicit statements (including detailed mathematical formulas) of the statistical models being used; point estimators for each of the principal parameters of interest, as specified in the response to Question B.1; and specific empirical procedures that will be used to check the validity of the models under consideration.

II.8. Variance estimation and inference methods. The response to Question B.2 should also include clear and explicit statements (including detailed mathematical formulas) for each of variance estimators and inference methods (e.g., test statistics or confidence intervals) that will be used for the parameter estimators specified under (II.7). The statements on variance estimation and inference methods should be clearly and explicitly justified within the framework of the design-based or model-based approaches specified under (II.7). Any additional approximations employed in this variance estimation work should be clearly and explicitly stated and justified with detailed mathematical formulas and literature references, as appropriate.

II.9. Justification of anticipated response rates. The response to Question B.3 should cover all of the issues raised in this question. In addition, the anticipated response rates in B.1, and the methods to maximize response rates should be justified through careful citation of empirical and methodological literature that is relevant to the proposed study.

II.10. Nonresponse adjustments. The response to Question B.3 also should include clear and explicit statements (including mathematical formulas as appropriate) of the specific weighting adjustment or imputation procedures that will be used. In addition, the underlying assumptions (e.g., homogeneity of response rates or mean values) for these weighting adjustment and imputation procedures should be stated explicitly.

II.11. Reference materials. PRA submission packages generally include a bibliography that lists references cited in the text of the package. Some of these references are textbooks, journal articles, web pages or other materials that are widely accessible to the public. However, some of the references may include agency technical reports or other materials that are not readily available to the BTS and OMB reviewers. For any cited references in the latter category, one paper or electronic copy of the reference should be included with the PRA submission package submitted to the BTS.

II.12. Point-by-point responses to issues raised in previous reviews. If the BTS does not approve a PRA submission package on its first review, then the BTS generally will provide a report that lists specific questions or areas in which the lead agency will need to do additional work before the package is ready for submission to the OMB. When the lead agency submits a revised package to the BTS for further review, the lead agency should include an appendix that provides point-by-point responses to the issues identified by the BTS in its previous review.

1 Please note, the Form OMB 83-I is obsolete as of July 2006; however, OMB instructions for completing both Parts A and B of the Justification for a Paperwork Reduction Act Submission are attached to the Form OMB 83-I and remain in effect.

8