Title of Information Collection
OMB Control Number, if applicable
Justification – Part B a statistical survey
Part B must be completed for any information collection that is a statistical survey, that is, a collection in which one of the sponsoring agency’s purposes is to use the information for the description, estimation, or analysis of the characteristics of groups, segments, activities, or geographic areas. Surveys include both censuses (that is, all members of the group of interest are asked to submit information) and sample surveys (that is, only a subset of all members are asked to submit information but the intention is to describe, estimate, or analyze the characteristics of larger groups, segments, activities, or geographic areas). For more information, see OMB’s Standards and Guidelines for Statistical Surveys.
Development of Concepts, Methods, and Design: Describe the following:
Key survey estimates and the precision required of the estimates.
The target population and its estimated size. Also:
The frame used,
Why the frame is appropriate for the target population including any coverage issues,
How the frame is updated, and
How current the frame is (for example, includes housing units as of December 31, 2004).
For a census, explain why a sample survey cannot be used to reduce the cost and burden of the information collection.
For a sample survey, describe the sampling methodology and the estimated sample size for each stratum. Also, if nonprobability sampling methods (for example, cut-off or model-based samples) will be used, provide a statistical justification for the methodology chosen.
Any pretests (including cognitive, usability, and field tests) conducted or planned on any component of the survey and provide results, if available. (Note that pretests that involve collecting information from ten or more persons must be approved by OMB.)
How the survey design identifies and minimizes measurement error including planned data evaluation activities such as interviews and response analysis surveys.
The USA Hire assessment satisfaction survey, based on the above definitions, would be considered an optional census which is administered immediately to all applicants upon completion of online assessments. This ensures the applicant’s responses are recent to the user experience and overall assessment process they are asked to evaluate. It is administered to all applicants who complete the assessment to ensure the feedback received is diverse and representative of the entire population completing assessments. Dissemination is not targeted to any description, estimation, or analysis of the characteristics of groups, segments, activities, or geographic areas. The census consists of three questions and is expected to take less than a minute to complete. It measures feedback regarding location and sentiment pertaining to the assessment process and platform. The census is optional and does not need to be completed and failure to complete or skipping the survey has no impact on the applicant and their assessment. By making it optional, the public burden is reduced. To administer it in another less-consistent manner would increase the burden to the public by increase costs to the Federal government and public.
Given the survey is optional, those who complete it become a representative sample of all test takers who completed the assessments. The number of individuals presented with the survey and who complete it will consistently fluctuate. Factors driving this fluctuation are:
The amount of job announcements agencies post using USA Hire assessments
Then, the number of eligible applicants who apply to those vacancies
Then, the number of eligible applicants who choose to complete USA Hire assessments
Finally, the number which choose to complete the optional survey
Based on past studies, we estimate 62% of the applicants who are invited to take a USA Hire assessment will complete the assessments. Of those who complete assessments, 48% are expected to complete the optional survey. Based on this, we estimate approximately 200,000 respondents annually will complete the 1-minute assessment satisfaction survey.
Collection of Data: Describe the following:
Describe how the data will be collected and the appropriateness of the methods given the objectives of the data collection, the target population and the planned uses of the data.
How respondents will be informed (see 5 CFR 1320.8(b)(3)) about:
The reasons for the data collection,
The planned use of the data,
Whether the collection is voluntary, mandatory, or required for benefits,
The nature and extent of confidentiality to be provided, and
The estimated respondent burden.
How data collection activities will be monitored for quality assurance.
Response rates:
Formulas to be used for calculating unit and item response rates.
For on-going or periodic surveys, provide unit response rates since the last OMB approval.
Estimates of expected response rates and the basis for the expectation (for example, estimates from similar studies or pretests).
Methods that will be used to attain the expected response rates.
Procedures for nonresponse (for example, imputation, weight adjustment) when creating survey estimates.
Plans for nonresponse bias analyses.
OPM delivers USA Hire assessments through an online platform. OPM will offer the assessment satisfaction survey to all applicants upon completion of the online assessments. This survey is optional; the instructions for the survey clearly state that it is optional and that responses have no impact on the applicant’s opportunity for employment. Applicants who choose to complete the satisfaction survey submit their responses electronically. By administering the survey immediately upon assessment completion, OPM ensures the applicant is responding to the questions while their sentiment and perceptions are recent. This electronic method of collection reduces applicant and agency administrative burden by eliminating time, labor and material costs typically associated with a pen-and-paper or batch administration.
OPM uses assessment satisfaction survey results in aggregate to measure the performance of the USA Hire program. OPM analyzes this data on a monthly basis to identify trends in applicant satisfaction. OPM reviews the data monthly and annually in aggregate; OPM does not view individual applicant responses. OPM uses this information to evaluate impact of recent technology and assessment content changes on the USA Hire applicant experience.
The USA Hire Program Management Office makes use of this data through monthly reporting where the aggregated response data is transformed into charts to compare average monthly question responses with average responses collected since the inception of the survey. This data is shared with the OPM stakeholders responsible for developing and administering the online assessments. The administration of the USA Hire assessment satisfaction survey allows OPM to continually measure applicant satisfaction and its effectiveness in delivering and presenting online competency-based assessments. Without this information collection, OPM will be unable to internally evaluate the impact of its technology and content changes to USA Hire assessments on the public. Additionally, it will be unable to establish if USA Hire effectively delivers on mission, experience and taxpayer stewardship requirements outlined in current and future regulations, policies, and directives. Collecting the data less frequently would not be technically feasible within the platform today, would increase the potential for sample bias in the response data, and eliminate OPM’s capability to immediately identify, track, and resolve unexpected negative trends in the applicant’s assessment experience.
Regarding applicant burden and notification of the burden, they will be provided the following information on-screen when presented with the survey:
Public Burden Statement
We think providing this information takes an average of 1 minute to complete, including the time for readding the questions, choosing answers, and reviewing the completed survey. Send comments regarding our time estimate or any other aspect of this survey, including suggestions for reducing completion time, to the Office of Personnel Management (OPM), OPM Forms Manager (3206-XXX), 1900 E Street NW, Washington, DC 20415-7900. The OMB number 3206-XXXX is currently valid. OPM may not collect this information, and you are not required to respond, unless this number is displayed.
Monthly response rates and responses are exported, transformed, and presented to OPM internal stakeholders. Monthly responses to each question are displayed in charts which compare overall monthly percentages with percentages since the inception of the survey. Two questions have a 5-poing rating scale. For reporting purposes, this the 5-point rating scale is condensed to a “% positive” metric, calculated as the % of respondents for each question who selected “Strongly Agree/Agree” or “Very Satisfied/Satisfied”. Applicants may skip or choose not to complete the survey. However, if one question is answered, all three questions must be answered to submit the survey. Therefore, there’s 100% response rate to all three questions if the survey is submitted.
Response rates are calculated simply based on which assessment completion record does or does not have survey responses. Based on past studies, we estimate response rates to fluctuate between 40 and 55% of applicants completing assessments. There are no expected response rate goals.
The only filtering of responses planned is at the vacancy level, prior to aggregation, to remove vacancies used for product development testing or which never had USA Hire activity. Since survey data is provided at the vacancy level, we currently have no way to reliably differentiate applicants who skip the survey; hence, no actions are anticipated to record/report this info.
Processing and Editing of Data: Describe the following:
Procedures for editing to mitigate or correct detectable errors.
Procedures to minimize processing errors.
Whether standardized codes (for example, Federal Information Processing System, statistical areas, race and ethnicity, North American industrial classification, and standard occupational classification) are applicable and the justification for the use of any non-standard codes.
Safeguards in the survey process to ensure the protection of the confidentiality discussed in Part A, section 10.
Aside from the above filtering of data to exclude certain vacancies and transform data into % positive, no other data manipulation occurs. Therefore, there is minimal to no standard for error. Safeguards are in place to prevent attribution of any assessment information to any applicant’s personably identifiable information.
Production of Estimates and Projections: Describe the following:
Estimation methods that will be applied to the survey data to develop target population estimates for use internally and/or for dissemination. Also, the use of any biased estimators must be identified and justified.
As applicable, the methodology to measure sampling error and estimation error.
The weights applied to the survey data to calculate target population estimates; or if an alternative method of sample design is utilized (for example, ratio estimation), any evaluations to ensure the method results in population estimates of high quality.
Other activities (for example, use of auxiliary data) that will be utilized in conjunction with the survey data to improve the quality of the estimates.
If the survey data are collected with an intention of developing model-based estimates of target population characteristics or projections of future values, explain the methods/models that will be applied to the survey data.
No estimation methods will be used. No models will be created based on this data. This survey is used internally for understanding changes in applicant sentiment at different points in time. OPM uses assessment satisfaction survey results in aggregate to measure the performance of the USA Hire program.
Data Analysis: Describe the plans for analyzing the data including methods to be used for statistical tests to address the needs and uses of the information as described in Part A, section 2.
Response data is exported monthly, transformed in a manner to allow aggregated reporting in the form of charts and graphs used to measure changes in response values over time. Data is qualitative in nature and therefore not appropriate for most statistical tests. However, no statistical tests, models or estimation will be generated from this data. Qualitative data is sufficient in measuring applicant sentiment and testing location for the purposes of administering, measuring, evaluating, and communicating the impact of changes to the USA Hire platform.
Review and Evaluation Procedures: Describe the content/subject matter, statistical, and methodological reviews, as well as evaluation procedures instituted to comply with OMB and agency Information Quality Guidelines.
Only three questions are being asked related to content/subject matter which is necessary for OPM research and evaluation purposes. The question content and responses are static and not subjective to technological or environmental changes over time. Monthly review and interpretation of the data occurs by OPM Business Analyst and, when warranted, Industrial Occupational Psychologists, to investigate unexpected fluctuations in responses over time. No statistical or methodological reviews are necessary, and evaluation procedures comply with OMB and OPM guidelines.
Data Dissemination: Describe the following:
Procedures for releasing and disseminating statistics and microdata (that is, detailed responses for individual respondents) to the public. As applicable, also describe the disclosure limitation methods to be used to achieve the confidentiality discussed in Part A, section 10.
Documentation that will be released to the public to improve understanding of how to properly interpret, analyze, and evaluate information from the survey.
Data is not released to the public. Aggregated data is reported in the form of charts to OPM internal stakeholders.
Contact Person(s): Provide the name, telephone number and e-mail address of the lead individual(s) who can answer questions about the statistical aspects of the survey and the name(s) of the agency unit(s), contractor(s), grantee(s),
Jeff Cain, Office of Personnel Management, Human Resources Solutions, Federal Staffing Center, USA Hire Product Development Branch. Email: [email protected]. Phone: 202–897–8829
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |