O*NET Data Collection Program
OMB Control Number 1205-0421
OMB Expiration Date: November 30, 2021
SUPPORTING
STATEMENT FOR
O*NET DATA COLLECTION PROGRAM
OMB CONTROL NO. 1205-0421
This Information Collection Request (ICR) requests a revision to the Occupational Information Network (O*NET®) data collection program, to collect updated occupational characteristics and requirements information on an ongoing basis. Selected sample sizes of business establishments have been increased to offset declining response rates and the potentially lasting impacts of COVID-19 on business eligibility (see Section 15). Also, Appendix G in this ICR package differs from the 2018 Appendix A submission: the questions related to association membership have been removed from the Background questionnaire because this information is available through other existing sources; and numerous level scale anchors have been updated as part of efforts to modernize and lend clarity to the Knowledge and Work Activities questionnaire descriptors.
The O*NET Data Collection Program continually operates to populate and maintain a current database on the detailed characteristics of workers, occupations, and skills. The program uses an occupational taxonomy, the O*NET-Standard Occupational Classification (SOC), which has been updated to reflect the 2018 SOC, mandated by the Office of Management and Budget (OMB) for use by all federal agencies collecting occupational and labor market information. Previously, the O*NET-SOC was based on the 2010 SOC taxonomy. The O*NET-SOC comprises occupations at the most detailed level of the SOC and includes additional occupational specificity as needed. In addition, new and emerging occupations in high-growth sectors of the economy have been identified and added to the taxonomy. Data have been published for 821 O*NET-SOC occupations, 701 of which have undergone more than one update.1 The O*NET Data Collection Program received initial OMB clearance in 1999 for a pretest and seven subsequent clearances that have allowed data collection to continue without interruption since June 2001. The current clearance expires November 30, 2021. This request is to continue to update occupations that reflect older data as well as to collect data on new and changing occupations included in the 2018 SOC for 3 more years, subject to annual budget levels.
O*NET information is the foundational common language for fulfilling the statutory and regulatory responsibilities regarding information on occupational skills discussed in the Workforce Innovation and Opportunity Act (WIOA) legislation and regulations.
Section 308 of the WIOA (Public Law 113-128) amended section 15 of the Wagner- Peyser Act to require the Secretary of Labor to oversee the “development, maintenance, and continuous improvement of a nationwide workforce and labor market information system,” which shall include, among other components, “skill trends by occupation and industry.” (See 29 U.S.C. 49l-2.) The O*NET program is the primary means for collecting skills information across all occupations in the economy. Updating the entire O*NET database is a critical component of the nationwide workforce and labor market information system to support employer, workforce, and education information needs.
The WIOA contains numerous references that identify the need for information on the skill requirements of jobs. The word “skill” is used more than 120 times, in addition to 3 references to “knowledge, skills, and abilities” and 2 to “knowledge, skills, and competencies.” For example, WIOA Section 102 requires the Unified State Plan to include an economic analysis of “(i) existing and emerging in-demand industry sectors and occupations and (ii) the employment needs of employers, including a description of the knowledge, skills, and abilities, needed in those industries and occupations.” (See 29 U.S.C. § 3112.) Section 134 requires the provision of “information on job skills necessary” and on “skill requirements” for obtaining jobs listed for the local occupations in demand. (See 28 U.S.C. 3174 § (c)(2)(A)(vi)(ii)).
Other WIOA references address the need for information on the skills of individuals. For example, Section 134 provides for assessment services to identify “the skill levels and service needs of adults and dislocated workers.” Section 129 allows funds to be used to provide youth with an assessment that “shall include a review of basic skills, occupational skills, prior work experience, employability, interests, [and] aptitudes.” (See 29 U.S.C. § 3164(c)(1)(A)). The O*NET Career Exploration Tools, including the O*NET Interest Profiler and O*NET Work Importance Locator, are such assessment tools, designed specifically to relate a person’s interests and work values to the information on education and skill requirements for occupations in the O*NET database and Web sites.
WIOA Section 303, on the Public Labor Exchange Services System, amends the Wagner- Peyser Act to add “The Secretary, in consultation with States, is authorized to assist the States in the development of national electronic tools that may be used to improve access to workforce information for individuals….” The suite of O*NET Web sites (such as O*NET OnLine and My Next Move) and O*NET Web Services used to disseminate O*NET occupational information and related workforce and labor market information are such national electronic tools designed to improve access to information for individuals.
Although the O*NET name is not referenced specifically in statutes, it is cited twice in the U.S. Code of Federal Regulations (C.F.R.). The WIOA regulations include definitions for O*NET and for O*NET-SOC, the taxonomy used for classifying O*NET data, as quoted below:
PART 651—GENERAL PROVISIONS GOVERNING THE WAGNER-PEYSER ACT EMPLOYMENT SERVICE
Sec. 651.10 Definitions of terms used in this part and parts 652, 653, 654, and 658 of this chapter. Authority: 29 U.S.C. 49a; 38 U.S.C. part III, 4101, 4211; Secs. 503, 3, 189, Pub. L
Occupational Information Network (O*NET) system means the online reference database which contains detailed descriptions of U.S. occupations, distinguishing characteristics, classification codes, and information on tasks, knowledge, skills, abilities, and work activities as well as information on interests, work styles, and work values.
O*NET–SOC means the occupational codes and titles used in the O*NET system, based on and grounded in the Standard Occupational Classification (SOC), which are the titles and codes by Federal statistical agencies to classify workers into occupational categories for the purpose of collecting, calculating, and disseminating data. The SOC system is issued by the Office of Management and Budget, and the Department of Labor is authorized to develop additional detailed O*NET occupations within existing SOC categories. The Department uses O*NET–SOC titles and codes for the purposes of collecting descriptive occupational information and for State reporting of data on training, credential attainment, and placement in employment by occupation.
Finally, the predecessor to the O*NET database, the Dictionary of Occupational Titles (DOT), is still cited in older regulations as a source of occupational information in support of federal programs. The DOT—which O*NET replaces— is cited 23 times in the C.F.R. (the DOT was formerly cited in statutes, but there are no current statutory references to it). These 23 citations include references to determining disability; administering Department of Labor (DOL) programs; and administering immigration, civil rights, and labor standards law. DOL officials responsible for the O*NET Program work with various federal users of the DOT, some of whom have made the transition either in regulatory changes or in practices and procedures, while others are exploring potential transition to O*NET information or uses of O*NET data. These include State Department officials responsible for visas, the Social Security Administration, and the Office of Apprenticeship, and the Office of Foreign Labor Certification within the U.S. Department of Labor.
Copies of the WIOA and 29 U.S.C. 49l-2, as well as the specific O*NET and DOT citations in the C.F.R., appear in Appendix A.
The continued population of the O*NET database is important because the O*NET database is the most current and comprehensive standard source of descriptive occupational information in the United States. The O*NET Data Collection Program remains at the center of an extensive network of occupational and skill information used by a wide variety of audiences, including individuals making career decisions; public agencies, such as workforce boards and American Job Centers, making training investment decisions; educational institutions preparing a future workforce; and employers making staffing and training decisions. With new workforce challenges due to the coronavirus (COVID-19) pandemic, O*NET data are an essential tool in assisting students, job seekers, employers, government agencies, and researchers in taking important steps to ensure a productive, successful economy.
This program provides a common language and framework to facilitate communication about industry skill needs among business, education, and the workforce investment system. The resulting O*NET database also is used to develop industry competency models and occupational competency profiles. O*NET data include information about transferable skills that are used for skills gap analysis, facilitating a mobile workforce responsive to changing economic needs. The O*NET database and companion O*NET Career Exploration Tools are used by many private companies and public organizations to tailor applications to their needs and those of their customers. The broad utility of the O*NET tools plays an important role in developing and maintaining a skilled workforce and contributes to U.S. competitiveness in a global, 21st-century economy. Additional information (e.g., content model and database history) regarding the O*NET Program can be found in Appendix B.2
The O*NET Program provides essential tools and services for numerous critical federal and state workforce investment functions. These tools also serve job seekers, businesses, and educational institutions across the country. The common language used in O*NET occupational and skill descriptions facilitates communication among the various user groups. This facilitation contributes to our nation’s talent development and promotes U.S. competitiveness in the global 21st-century economy.
Users of the O*NET database can access and use the information in various ways. The data are available to the public free of charge through the O*NET OnLine, My Next Move, and My Next Move for Veterans Web sites; through the O*NET Web Services application programming interface (API); or by downloading the database, which is done by developers who provide applications targeted to specific communities or audiences. The O*NET database is now the foundation for many programs serving the American workforce, providing information to build transferable skills, skills gap analyses, and competency profiles and to facilitate cross-functional team building. These examples are just a few of the ways the O*NET Program supports activities critical to maintaining a mobile workforce responsive to changing regional and national economic needs.
With the November 2020 version of the database, 821 O*NET-SOC 2019 occupations have been comprehensively updated with information collected from job incumbents and occupation experts through the O*NET Data Collection Program; 701 of these occupations have had more than one update. The O*NET database currently provides
detailed occupational information for more than 900 occupations3;
information on standardized descriptors of skills, abilities, interests, knowledge, work values, education, training, work context, and work styles; and
occupational coding based on the 2018 SOC.
The O*NET database has been enhanced with new occupational information that provides more coverage of the occupations and will enable users to more effectively use the database. These enhancements include (1) more comprehensive detailed work activity information; (2) updated military-to-civilian occupational crosswalks; (3) the identification of technologies employers most frequently include in job postings (e.g., hot technologies); and (4) the development of new alternate titles that enable users to more easily find occupations of interest in O*NET Web applications.
Easy-to-use competency frameworks for organizing workforce competencies, credentials, and other work-relevant information are now available, including Work Activities, Technology Skills, Abilities, Cross-Functional Skills, Basic Skills, and Knowledge. The frameworks are published in Excel format and also as JSON-LD: machine-readable Linked Data described using the CTDL-ASN schema (Credential Transparency Description Language Profile of Achievement Standards Network Description Language) developed by the Credential Engine project.
A summary of O*NET machine-discoverable information can be found at the O*NET Resource Center’s Machine-Readable Data page.
The O*NET database is provided free of charge to the public through O*NET OnLine, a Web-based application at https://www.onetonline.org. O*NET OnLine offers users multiple ways to search for O*NET occupations and related data. O*NET OnLine includes
regularly updated occupational information;
tiered search algorithms that maximize successful results of searches by keyword (title), occupational code, or partial code;
O*NET equivalents of occupations listed in other occupational systems, such as the Military Occupational Classification or the Classification of Instructional Programs;
the ability to browse by O*NET descriptor or variable (this search enables users to make cross-occupational comparisons by viewing an occupation’s rank order on a selected knowledge, skill, ability, work activity, interest, work value, tool or technology, or task);
the ability to browse groups of similar occupations, including Bright Outlook4; career clusters; industry type; or science, technology, engineering, and mathematics (STEM) disciplines;
a sample of reported job titles for each occupational report, providing the user with a broader understanding of the world of work;
report display options (in addition to the default Summary Report, users can choose to view a comprehensive Details Report or build a Custom Report).
Information below can be tailored to the local community that the user resides in or may be thinking of relocating to:
wage and employment trends information (derived from Bureau of Labor Statistics [BLS] data) and links to Web sites of professional associations, giving users access to additional information on related specialties, industries, and education and training resources; and
occupation-specific links to training, certification, licensing, and apprenticeship information, as well as to actual job openings.
My Next Move (https://www.mynextmove.org/) is an easy-to-use, easy-to-read Web-based interactive tool for new job seekers, students, and other career explorers to learn more about their career options. Users can explore more than 900 different careers and see important information about them, including skills, tasks, salaries, employment outlook, and related career videos. They can also look at related salary credentials and training, as well as search actual job openings. They can find careers by searching keywords, by browsing industries, or by using the O*NET Interest Profiler (see below). Additionally, users can browse occupations by Bright Outlook, interests, and amount of job preparation required.
Career reports in My Next Move feature the most important knowledge, skills, and abilities needed to perform the work, explained in language that is easy to understand. Outlook and education sections let users find salary information, job postings, and training opportunities. The visual design enables users to identify a career’s key points or to explore a career in depth.
Also, at the site is a Web-based version of the popular O*NET Interest Profiler, a tool designed to assess an individual’s vocational interests. The Web-based version of the Interest Profiler features 60 items that are scored and, along with information about the user’s education and work experience, guide users to careers they may enjoy. More information about the O*NET Interest Profiler–Short Form can be found at https://www.onetcenter.org/IP.html.
My Next Move for Veterans (https://www.mynextmove.org/vets/) is a Web-based interactive tool for U.S. veterans to learn more about their career options. This tool has all the functionality of My Next Move as described above.
Unique to My Next Move for Veterans is a feature that allows veterans to search civilian occupations related to their military training. Veterans indicate their branch of service and either the military code or title of their training. They then receive a list of civilian occupations that best match their military training. Where available, the results returned will also indicate how closely their military duties match the civilian occupations, along with the minimum military rank and length of military service associated with the civilian occupation.
The site’s Career Reports also offer rich information about military classifications through direct links into Careers in the Military (https://www.careersinthemilitary.com/). In addition, a new, interactive salary graphic is included within the Career Reports to aid veterans in understanding the often-wide range of salaries associated with civilian careers.
Mi Próximo Paso (https://www.miproximopaso.org/) is a Web-based interactive tool for Spanish-speaking job seekers, students, and other career explorers to learn more about their career options. Mi Próximo Paso includes all the features of the English-language site, My Next Move.
The Career Exploration Tools are based on a “whole person” concept and are designed for career counseling, career planning, and career exploration. They include the Ability Profiler, the Interest Profiler, and the Work Importance Locator. Within the My Next Move Web site, users can choose to complete a 60-item Web-based version of the Interest Profiler. A “mini” mobile-friendly version of the Interest Profiler was created and released for developers to leverage (https://www.onetcenter.org/reports/Mini-IP.html). A short, paper-and-pencil version of the O*NET Interest Profiler is now available for download and self-print, free of charge. The new tool is ideal for settings without computers or with very limited time for career assessment, such as correctional facilities and schools. The tool also has an updated Score Report and Career Worksheet, streamlined Career Listings, and a technical development report.
Individuals exploring careers may use the Ability Profiler to discover what they do well, the Interest Profiler to identify the types of work they may like to perform, and the Work Importance Locator to determine which occupations will likely be satisfying according to their values and needs. The tools enable users to discover important information about themselves and use the information to explore the world of work. Workers may use these tools as aids in exploring career options, in considering career transitions, and in preparing for career change. The assessments also are designed for use by students exploring a school-to-work transition.
The O*NET Resource Center (https://www.onetcenter.org/) serves as a central point of information about the O*NET program and is the main source for O*NET products, such as the O*NET database, Career Exploration Tools, and development reports. It also provides the latest news on O*NET developments and product releases, as well as important links to related O*NET sites.
The Code Connector (https://www.onetcodeconnector.org/) was developed to assist workforce professionals needing to code jobs. American Job Centers, other government workforce agencies, and college career services offices are the most prevalent users of the Code Connector. The Code Connector uses the O*NET data to help users determine the correct occupational code for their job orders. To access an occupation, the user may type in a keyword or select an occupational group from the home page. On subsequent pages the user is able to refine the search to select a specific occupation. The final report contains information to help determine whether the selected occupational code is the best match for the particular job order. This information includes the O*NET-SOC description, Tasks, Related Occupations, Occupation Family, and Detailed Work Activities.
O*NET Web Services was introduced to O*NET customers in January 2014. As of June 2020, there are more than 2,000 user accounts and more than 30 million user requests per month. O*NET Web Services (https://services.onetcenter.org/) is an API that provides access to the full O*NET Database, along with the most popular features and tools from each of the O*NET Web sites. For developers, published APIs are available to connect vendor systems to key features of O*NET Web applications. Through O*NET Web Services, developers can integrate O*NET tools such as the following into their own Web sites or Web-enabled applications:
Keyword Search—both the My Next Move search and the OnLine occupation search are available for use in career sites. The REST Web Services API returns occupations matching a word, phrase, title, or full or partial O*NET-SOC code. The results include the code and title of each matching occupation.
My Next Move Career Reports—concise, easy-to-read overviews for over 900 occupations. APIs also provide Bright Outlook information, job outlook, and more.
Summary and Details Occupation Reports—detailed information from O*NET OnLine for more than 900 occupations. User applications can include an occupation’s most important or all tasks, knowledge, skills, abilities, technology skills, and more.
Military Search—the military transition search used in My Next Move for Veterans is also available through the Web Services API. The search returns relevant O*NET-SOC occupations based on full or partial codes and titles from the Army, Navy, Air Force, Marine Corps, and Coast Guard classification systems.
Spanish Keyword Search—the Spanish-language keyword search used in Mi Próximo Paso is part of the Web Services API. Occupation titles are returned, in Spanish, matching a Spanish word or phrase. A wide assortment of features from Mi Próximo Paso, including detailed career reports and Interest Profiler questions and scoring, is also available.
Interest Profiler—this assessment tool can be included in customer career tool sites using the IFrame Widget. After a simple block of HTML code is added, users can take the O*NET Interest Profiler without leaving their career resources Web site. For tighter integration, a REST Web Services API is offered. It provides scoring services and career results from the range of O*NET-SOC occupations. This tool is provided in both English and Spanish.
Recent additions include the following:
Browse by Career Cluster—career clusters contain occupations in the same field of work that require similar skills.
Browse STEM Occupations—STEM occupations require education in science, technology, engineering, or mathematics disciplines.
Browse by Industry—industries are broad groups of businesses or organizations with similar activities, products, or services. Occupations are considered related based on their employment within an industry.
Database Services—the API provides real-time, up-to-date access to all of the information in the downloadable O*NET Database. Users can retrieve data from any part of the Database and customize results by filtering and sorting based on multiple criteria.
Organizations using Web Services include federal, state, and local government agencies; military services; educational institutions; assessment and career information delivery systems; public workforce investment systems; private organizations and corporations; and international users. O*NET Web Services significantly reduce the cost and effort necessary for developers to update their applications with O*NET products and tools. With Web Services, O*NET data updates are seamlessly incorporated; thus, no new programing is required by developers. Additionally, as new features are added to O*NET Web applications, new Web Services are designed so developers can have immediate access to them and update their applications in an efficient and timely manner.
Use of O*NET products has remained strong over the past few years. O*NET OnLine currently averages more than 2 million visits per month. The O*NET Resource Center (https://www.onetcenter.org) averages 300,000 visits per month. The My Next Move sites average over 1 million visits per month. O*NET Web Services averages over 30 million user requests per month. In addition, use of career information systems, Web site linkages, user certifications, and O*NET product downloads is widespread; by design, the primary dissemination strategy of the O*NET Program is for the private sector to build O*NET-based products that are tailored to specific audiences or user needs. More information regarding Web site linkages, user certifications, and O*NET product downloads is available in Appendix Section C.1.
The O*NET Program provides comprehensive, up-to-date occupational information used directly by the public through access to the O*NET Web sites, as well as indirectly through value-added products delivered by both governmental and private developers. O*NET-based products benefit the public through product and service development by
federal and state government agencies,
public workforce investment systems and workforce boards,
educational and research institutions,
U.S. armed forces,
private companies and commercial product developers, and
international users.
Appendix Exhibits C-3 through C-8 list recent examples of O*NET uses by user category.
Presented here are two examples of references to the O*NET Program in publications. For an extensive list of research articles, books, book chapters, technical reports, and presentations referencing the O*NET Program, see Appendix D.
To assess the economic impact of “social distancing” measures taken to arrest the spread of the COVID-19 pandemic, the National Bureau of Economic Research used O*NET work context and work activities data to study the feasibility of performing work at home. Researchers used data on items addressing outdoor work and operation of mechanized equipment and motor vehicles to assign each work-from-home rating. Results suggested that 34% of U.S. jobs could be performed at home and that these jobs account for 44% of all wages. Using the same data, the researchers also rank-ordered O*NET job families by the percentage of jobs within them that meet the work-at-home criterion (Dingel and Neiman 2020).
A study in AEA Papers and Proceedings—“What Can Machines Learn, and What Does It Mean for Occupations and the Economy?”(Brynjolfsson, Mitchell et al. 2018)—used the O*NET taxonomy and task and work activities data to build measures of suitability for machine learning (SML). The authors found that, although most occupations across industries have at least some SML tasks, few, if any, are entirely conducive to machine learning. They further concluded that widespread application of machine learning would require significant job task redesign. The authors recommend a shift in the debate about the effects of artificial intelligence on work from wholesale automation of jobs to job and business process redesign.
The O*NET Data Collection Program is key to the continued effort to update the O*NET database to reflect changing skills requirements of occupations with the advent of new technologies and the changing world of work. In the research leading to the O*NET Data Collection Program, various sources and methods for collecting occupational information were examined, including collection of data from job incumbents and supervisors and development of ratings by occupation experts and occupational analysts. On the basis of this work, the O*NET team determined that the preferred source of data for most domains (Generalized Work Activities, Work Context, Knowledge, Education and Training, and Work Styles) is job incumbents. Other occupation experts, such as supervisors and trainers, may be used where access to job incumbents proves difficult or where the sampling of business establishments is inefficient.
By contrast, occupational analysts, who are provided with updated information from job incumbents, are preferred for the Abilities domain, which tends to be more abstract than the other domains. The Skills domain, whose variables are somewhat abstract, is a strong candidate for either source of collection; it is now updated by occupational analysts. A study conducted in 2006 found no clear evidence that one source of raters provides more valid or accurate data than the other for the Skills domain (Tsacoumis and Van Iddekinge 2006).5 Consequently, considerations of relative practicality, such as cost, informed the decision to proceed with analyst ratings of both Abilities and Skills.
As part of a random sample, workers selected to participate in the O*NET Data Collection Program are asked to rate the requirements of their own jobs as defined by the O*NET questionnaire items. The responses are tabulated into statistics, such as mean ratings for each scale. Collecting information from job incumbents presents many challenges; among them is determining the best method for identifying a representative sample of job incumbents in each occupation. Sampling allows an estimate of the population. The O*NET Program is concerned, in particular, with identifying sampling approaches that minimize burden on employers and the public, achieving broad coverage of the workers in each occupation, ensuring acceptable response rates, and supporting overall cost-efficiency.
Three types of sampling frames are available for identifying samples of workers in each occupation: (1) lists of individual workers identified through professional and trade associations, licensing agencies, and unions; (2) households; and (3) employer establishments.
Identifying sampling frames of workers through professional and trade associations and unions retains the advantage of lower response burden because contacts with a sample of employers are replaced with contact with one or a few associations. Although it adds the cost of soliciting and maintaining association cooperation, this procedure also removes the cost of soliciting and maintaining employer cooperation.
The Establishment Method (using employer establishments to identify occupational samples, as described further under Sample Design below, as well as in Supporting Statement Part B, Section 2.1) provides the advantages of lower response burden and cost than a household survey, as well as good coverage for the large majority of occupations. Response burden and costs are lower for two primary reasons: (1) there are more workers per employer than per household, so fewer contacts are required to identify workers; and (2) employer contacts can be minimized by focusing on those most likely to employ workers in each occupation for which the sample is required. Efficient sample design is possible because the distribution of employment in an occupation is usually a function of the industry of the employer. Employment by occupation by industry is measured by the federal-state Occupational Employment Statistics (OES) program national estimates provided by BLS.
The Establishment Method provides good coverage of wage and salary employment so long as an acceptable employer sampling frame is available. Coverage of self-employment is more difficult, although the sampling frame used in O*NET surveys includes many establishments operated by self-employed workers.
The Establishment Method, pretested in 1999 and 2000,6 remains the primary way to update the O*NET database; most data are currently collected this way. Achieving high response rates with the Establishment Method can be challenging, however, because the method requires cooperation at two levels: the employer and the sampled worker. Nonetheless, acceptable levels of cooperation have been attained to date, and this method has proved successful. Although the resulting response rates have been acceptable, the O*NET team continually works to enhance response rates.7
An alternative method for collecting occupational information, involving occupation experts, is used to optimize the use of burden hours and resources because some occupations are difficult to sample efficiently. This situation occurs when it is difficult to locate industries or establishments with occupation incumbents; when employment is low; or when employment data are not available, as is the case for many new and emerging occupations. With the Occupation Expert (OE) Method, persons considered experts in the target occupation are surveyed. These experts include supervisors and trainers, as well as experienced job incumbents. The limitation of the OE Method is that locating experts can be difficult. For some occupations, identifying a professional association proves difficult; in other cases, the association may lack membership information sufficient to identify experts for a specific occupation.
Using the most appropriate sources of information (e.g., workers, occupation experts, analysts) and a multiple-method approach, the O*NET Data Collection Program efficiently collects and yields high-quality occupational data.
The O*NET Program sampling approaches are designed to create and update the O*NET database in a highly cost-efficient, timely manner while maximizing the reliability of the information in it. The primary method for collecting this information is the Establishment Method, which entails a survey of workers employed in a national probability sample of establishments. Data collection for approximately 75% of occupations is completed by the Establishment Method. The method uses a stratified two-stage design: (1) businesses (the primary stage) are selected from a national database, provided by Dun & Bradstreet (D&B), of roughly 17 million establishments, with probability proportional to the expected number of employed workers in the specific occupations being surveyed, and (2) a sample of workers (the secondary stage) is selected in the occupations within the sampled businesses. For selected occupations that are difficult to complete and for which additional observations are required, a special frame, such as a professional or trade association membership list, is sometimes used to supplement the D&B sample. The sample selection procedures vary across associations, depending on the type of information available on association members. In general, association lists are sampled with a single-stage, stratified, simple random sampling approach. Stratification by geographic location and occupation subspecialty is considered if it is appropriate for the occupation.8
The OE Method is considered for use when the Establishment Method would likely be problematic because the target occupations have very low employment rates, are new or emerging occupations lacking industry employment data, or are populated by incumbents in remote or difficult-to-access locations. The OE Method can be used only if the occupation is well represented by one or more professional or trade associations that are willing and able to identify experts in the target occupation. For this method, stratified samples of experts are selected from lists of potential respondents. These potential experts are questioned to determine whether they meet the program-specified criteria to serve as occupation experts for their respective occupations. Data collection for approximately 25% of the O*NET-SOC occupations is completed by the OE Method.9
The O*NET Data Collection Program employs the latest information technology systems and procedures to enhance the quality of the data, minimize burden on the responding establishments and questionnaire recipients, and reduce the overall cost of the data collection effort.
Data collection operations are conducted by RTI International at its O*NET Operations Center in Raleigh, North Carolina, and its Research Operations Center, also in Raleigh. For the Establishment Method, the Operations Center’s Business Liaisons (BLs) contact sampled business establishments, secure the participation of points of contact (POCs), and work with the POCs to carry out data collection in the target occupations. The data are provided by randomly selected employees in the occupations of interest. All within-establishment data collection is coordinated by the POCs; the BLs do not contact employees directly. After a POC agrees to participate, informational materials and questionnaires are provided to the POC, who distributes the questionnaires to the sampled employees. Completed questionnaires are returned directly to RTI for processing. Respondents also have the option of completing the survey online.
As noted above, for difficult-to-complete occupations, the D&B sample may be supplemented with a sample of workers selected from a professional or trade association membership list. Similarly, when the OE Method is used, occupation experts are also selected from professional or trade association lists. In both situations, the workers or occupation experts are contacted directly by the BLs, without involvement of a sampled establishment or a POC.
Three domain questionnaires are used to collect data from sampled workers: Knowledge (including Work Styles and Education and Training), Generalized Work Activities, and Work Context. Each sampled worker is randomly assigned one of the three questionnaires. The workers are also asked to provide basic demographic information and to complete a brief task inventory for their specific occupations. By contrast, the occupation experts are asked to complete all three domain questionnaires, as well as basic demographic questions and a task inventory for the occupation of interest.
Workers may complete the paper questionnaire and return it by mail, or they may choose to complete the questionnaire online at the project Web site. Occupation experts have the same options for completing their questionnaires. Questionnaires are available in Spanish for selected O*NET-SOC occupations. Data for two domains, Abilities and Skills, are provided by trained analysts because of the more abstract nature of the questions.10
Electronic versions of the O*NET questionnaires are available via the Internet to sampled job incumbents and occupation experts. Many of the benefits of the paper questionnaires are replicated in the electronic questionnaires. Specifically, users are able to start and stop multiple times without losing data. They can return to a partially completed questionnaire at any time during the survey period and resume where they stopped. A respondent may also review and edit previous answers as necessary. In addition, an on-screen progress meter keeps respondents informed of their movement through the questionnaire.
Advances in Web technologies and security, as well as the increasing prevalence of establishments’ and employees’ access to Web browsers, have made Internet-based data collection both feasible and practical. Internet use continues to accelerate, and the use of the O*NET Web questionnaire has increased with it. For 2019, 42.0% of job incumbents and 71.9% of occupation experts used the online survey. The paper questionnaire cover and informational materials mailed to respondents continue to highlight and encourage the online option.
The paper and Web versions of the questionnaires were designed to be optimal for their respective modes of administration. The questionnaire design literature suggests that this approach is essential to reduce mode effects. That is, if each questionnaire is designed to minimize measurement error in its particular mode of interview, differences between the two types of questionnaires (Web and paper) are also minimized. For example, in the paper version, multiple questions appear on a single page of the questionnaire. However, in the Web version, the domain questionnaires display only one question per screen (although the respondent can navigate at will through the instrument). The literature on Web survey design (see, e.g., Couper 2008) suggests that one question per screen for Web surveys reduces measurement error and therefore the effects of administration mode. This difference was the only important one necessary for the Web version because both instruments are self-administered. In fact, to ensure comparability between the paper and Web responses, the formats and wordings of the questions and response categories for the two versions remain identical.
Section 508 of the Rehabilitation Act of 1973, as amended, specifies that persons with disabilities shall have access to and use of the same information that persons without disabilities have. To comply with this section, the O*NET Data Collection Program designed its DOL-sponsored Web applications, including O*NET OnLine, My Next Move, My Next Move for Veterans, Mi Próximo Paso, and O*NET Code Connector, to ensure that the data and information are accessible to the widest possible audience, including people with disabilities. The sites also provide links to several accommodation and disability resources on the Internet. Site accessibility remains an important design component in the ongoing maintenance and development of DOL-sponsored sites, with close adherence to the guidelines of the Web Accessibility Initiative (WAI) from the World Wide Web Consortium (W3C).
The O*NET Program uses the Internet to gather additional occupational information, such as alternate (lay) titles and high-demand technologies. This use expands the O*NET database, providing easily maintained current information and enhancing users’ ability to find occupations relevant to their training and expertise by giving them a wider range of search terms. This enhancement is accomplished without additional burden to the public and at less cost than other means of data collection.
“Web scraping” is employed to gather information on tools and technologies, with particular emphasis on the technologies that are most in demand by employers. Information from this data mining approach is used to compile a list of “hot technologies.” This list, which is updated quarterly, includes programming and software technologies such as Python and Meditech that appear frequently in lists of occupational or job requirements. This technique is also used to evaluate and augment the lists of tools and technologies developed and maintained for each occupation in the SOC taxonomy. Tools and technologies linked to specific occupations are found by using job advertisement data mining software and by searching job posting, career education, professional association, and other Web sites. Occupational analysts link the tools and technologies to the United Nations Standard Products and Services Code, an online classification system for tools and services (http://www.unspsc.org/). Before the information is published in O*NET OnLine, rigorous reviews are performed to ensure the quality and usefulness of the data. Information collected from the Web in this way represents a significant enhancement to the data available to O*NET users, and it ensures that real-time labor market information is being used to maintain currency of the O*NET database.
A multimethod data collection approach is used to populate the O*NET alternate titles. Data sources include incumbent/occupational expert data, employer job postings, Occupational Code Assignment submissions, transactional analyses (e.g., unmatched search terms submitted to CareerOneStop.org, a DOL-sponsored online resource for job seekers), customer and professional group requests, additional classification systems, and other miscellaneous sources. Each title undergoes an extensive multistep review process and is reviewed by multiple occupational analysts. Deliverables include (1) reported job titles that appear in O*NET OnLine, My Next Move, and Web Services and (2) a published Alternate Titles database that includes alternate job titles appearing in the downloadable database, keyword search, and Web Services. Alternate titles greatly enhance the keyword search functions in the O*NET Web sites. Alternate titles are also incorporated into a number of public and private keyword searches through the O*NET Web Services.
Use of Internet Web sites further enhances the O*NET database, provides greater search capabilities to users, and reduces burden. It also allows for rapid update of the data through user input and at minimal cost.
To avoid duplication and reduce cost, several
portions of the O*NET Content Model are provided from existing data
sources. Specifically, as described in Appendix B, the domain of
Workforce Characteristics—including information on industries,
job opportunities, and pay—is obtained through links to
existing labor market information databases. Information about
occupational licensing, certifications, related instructional
programs, and associations is provided from existing sources and
several Web sites, including the CareerOneStop (COS) Certification
Finder at
https://www.careeronestop.org/toolkit/training/find-certifications.aspx,
the COS Licensed Occupations Database at
https://www.careeronestop.org/Toolkit/Training/find-licenses.aspx,
and the COS Professional Associations page at
https://www.careeronestop.org/
Developers/Data/professional-associations.aspx.
Wage and employment projections information is provided through work
conducted by the BLS, including by the OES program
(https://www.bls
.gov/oes/)
and the Employment Projections program (https://www.bls.gov/emp/).
The exhaustive reviews of existing labor market and occupational information conducted by the DOL’s review staff, as well as subsequent research, identified no other comprehensive, valid, and reliable sources that could be used for the data items included in the O*NET database. The development of the O*NET Program has involved staff and advisors who have many years of experience in labor market and occupational information and who are familiar with existing data sources. In fact, as discussed in Section 2, many existing systems that provide detailed occupational information are actually using information based on O*NET data or the predecessor DOT.
The few existing sources with similar measures are too limited to be used in the O*NET database. Some existing sources are valid and reliable—for example, information from the U.S. Office of Personnel Management and the U.S. Department of Defense—but are not comprehensive, because they represent only those jobs in federal civilian employment or the military. Some private sources of job analysis information exist; however, they are based on job analyses conducted for particular purposes or settings rather than on a representative sample of employers and workers. They are therefore limited in their coverage and not representative of the entire workforce. Furthermore, these analyses are not comparable because they do not use the prescribed O*NET common language to describe occupational requirements; it is not practical to combine them, because they include dissimilar components. Finally, these private data sources are not available to the general public.
The selected samples for most occupations represent all sizes of establishments. Occasionally, the targeting strategy used in selecting an efficient sample may lead us to omit some small establishments from the sampling frame, but this omission occurs for few occupations. The omission is allowed only when it is clear that sampling small establishments will greatly reduce the efficiency of the data collection or that incumbents from small establishments are not working in the mainstream of the occupation.
Because establishments of all sizes should be represented in the samples for most occupations, specific design provisions have been undertaken to avoid overly burdening small establishments. For example, Exhibit A-1 shows the distribution of establishments by number of employees on the D&B list used as the sampling frame for O*NET data. It illustrates how O*NET sampling selects small establishments at a much lower rate than that at which they occur in the population. For example, although 93.8% of establishments employ fewer than 25 employees (represented in the first three rows of the exhibit), only 41.2% of the O*NET sample will consist of such small establishments. On the other hand, large establishments (with 250 or more employees, as represented in the 7th to 9th rows of the exhibit) will make up 22.7% of the O*NET sample but only 0.4% of all establishments. Thus, to reduce the burden on small establishments with few employees, the O*NET sample relies more heavily on large establishments. The disproportionate sampling of large and small establishments is properly accounted for in the analysis weighting, resulting in statistically consistent estimates.
In addition, data collection procedures place lower burden on small establishments than on large establishments. When a small establishment is selected, it likely employs fewer of the targeted occupations and has fewer employees working in the occupations. Thus, a POC at a small establishment generally spends less time preparing sampling lists and distributing questionnaires than a POC at a large establishment, which is more likely to employ several of the targeted occupations and to have a large number of employees working in the occupations.
Exhibit A-1. Distribution of Frame and Sample Establishments by Employment Size
Number of Employees |
Total Number of Frame Establishmentsa |
Frame Distribution (Percent) |
Actual Distribution of O*NET Sampled Establishmentsb (Percent) |
1–4 |
12,791,682 |
73.8 |
16.6 |
5–9 |
2,174,659 |
12.5 |
6.3 |
10–24 |
1,302,021 |
7.5 |
18.3 |
25–49 |
487,160 |
2.8 |
8.6 |
50–99 |
299,079 |
1.7 |
15.1 |
100–249 |
159,572 |
0.9 |
9.4 |
250–499 |
40,234 |
0.2 |
13.0 |
500–999 |
15,482 |
0.1 |
5.6 |
1,000+ |
9,809 |
0.1 |
4.1 |
Unknown |
50,280 |
0.3 |
3.0 |
Total |
17,329,978 |
100% |
100% |
Note: Percentages may not sum to 100% because of rounding.
a Data based on June 2020 Dun & Bradstreet (D&B) frame of establishments.
b Data based on distribution of prior O*NET samples that used the D&B frame. Future O*NET samples will be similarly designed.
The O*NET database is the most comprehensive source of occupational information in the United States. No other similarly comprehensive, reliable, and valid source exists. If O*NET data are not collected, U.S. citizens, industry, business establishments, military, government and educational institutions, and the workforce investment system will have few options that meet their needs for descriptive occupational information. O*NET data are used to develop industry competency models and occupational competency profiles (i.e., industry- or occupation-specific job or work analyses). O*NET data also include information on transferable skills and are used for skills gap analysis, promoting the development of a mobile workforce responsive to changing economic needs. “Students, jobseekers and workers need up-to-date information on required job skills for specific occupations. O*NET provides the best resource for detailed descriptions of the knowledge, skills, abilities, work-related tasks, and tools and technologies used by specific occupations (974 occupations are covered)” (Workforce Information Advisory Council (WIAC) 2017, February).
The use of O*NET data by industry, employers,
software developers, job seekers, students, educators, and workforce
development specialists supports a well-functioning U.S. labor market
and workforce investment system—both essential to U.S.
competitiveness in the global 21st-century economy. O*NET-SOC
occupations conform to the SOC, permitting O*NET data to be linked to
and analyzed with sources of information on current occupational
employment and trends, wages, and demographic data. Its electronic
format is freely accessible through DOL-sponsored Web sites,
including O*NET OnLine, My Next Move, My Next Move for Veterans, Mi
Próximo Paso, and O*NET Code Connector. A wide variety of
database versions are also available to download free of charge from
the O*NET Resource Center (see https://www.onetcenter.org/).
Over 2,000 customers are signed up for O*NET Web Services, including
private, nonprofit, government, and military organizations, allowing
them to take advantage of the ability to access and seamlessly
incorporate O*NET data and applications within their systems via the
extensive Web Service offerings (see
https://services
.onetcenter.org).The
O*NET data and structure are also being incorporated into a number of
open data initiatives, including the Data at Work initiative,
OpenGovernmentdata.org, the Credential Engine/Credential Registry,
and the Competency and Skills System.
The initial 3.1 version of the O*NET database has been updated 18 times as new data have been collected and analyzed. Additional data releases are planned through 2024 allowing for the continued update of occupations and release of data on new and emerging occupations. The consequences of discontinuing data collection would be that the millions of users who rely on O*NET data for business and career decisions, for educational programming, and for work in human resources or workforce development would instead be using portions of information that are out of date and incomplete. If data were collected less frequently, the currency of some data would become questionable, especially for occupations that are changing as a result of new technologies. The focus of data collection on high-growth and new and emerging occupations reflects the need to provide current information in a rapidly changing, demand-driven economy.
A 3-year extension of the O*NET Data Collection Program is being requested for the period of December 2021 through November 2024. This extension will provide for the updating of selected high-growth occupations and for data collection activities for new and emerging occupations. The recently released 2018 Standard Occupation Classification system identified many new occupations (such as data scientists), as well as a number of modifications to existing occupation titles, definitions, and classifications (https://www.bls.gov/soc/2018/home.htm). A dynamic and progressive U.S. economy requires continuous improvement to the data on which so many decisions are based. Millions of people are currently using O*NET information, and the numbers continue to expand as public agencies and private developers integrate O*NET data into their systems and products. The O*NET database provides valid, reliable, and current occupational information crucial to a strong U.S. workforce. O*NET database updates are scheduled to occur once a year to incorporate newly collected information on recently surveyed occupations. A schedule for data analysis is provided in Exhibit A-5 in Section 16; schedules for data collection and analysis are subject to annual appropriations.
Requiring respondents to report information to the agency more often than quarterly;
Requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;
Requiring respondents to submit more than an original and two copies of any document;
Requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records, for more than three years;
In connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study;
Requiring the use of a statistical data classification that has not been reviewed and approved by OMB;
That includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; or
Requiring respondents to submit proprietary, trade secrets, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information's confidentiality to the extent permitted by law.
The ICR requires no special circumstances nor record keeping requirements.
In accordance with the Paperwork Reduction Act of 1995, the public was allowed 60 days to comment through the Federal Register Notice posted on March 29, 2021 (86 FR 16392). The Department of Labor received comments from three sources in response to this notice. No changes were made to the proposed O*NET data collection in response to these comments.
One individual sought clarification on the nature of the indicated revisions. The Department believes that the response in Question 15 below clarifies the nature of the changes to the ICR.
One individual submitted two comments. The first comment requested clarification of the definition on the O*NET survey questionnaire regarding First Professional Degree: “includes a total of at least 6 academic years of work to complete.” We clarify that the definition includes the typical 4 years of an undergraduate degree with the addition of subsequent graduate-level work, such as 2 years for an M.B.A. or 3 years for a J.D. In the second comment, this individual also suggested that it would be useful to have estimated time/effort allocations for the various work activities measures. DOL notes that Tasks do have Frequency ratings and that Tasks are rolled up into various levels of Work Activities, which are not directly measured. As this individual is from another federal statistical agency, DOL plans to engage in follow-up informal communication.
One association submitted a letter containing four comments: (1) suggestions for increasing response rates; (2) a suggestion to shorten the questionnaires to reduce burden and encourage response; (3) a suggestion to prioritize updating of in-demand occupations; and (4) a suggestion to update the five Job Zone category definitions to better reflect certain concerns regarding classification of supervisors and managers. These suggestions will be taken into consideration as part of O*NET’s ongoing efforts for continued improvement of survey quality and efficiency.
The OMB clearance package was reviewed by an expert consultant, Dr. James B. Rounds. Dr. Rounds is a professor in the Psychology Department at the University of Illinois at Urbana-Champaign. He is a widely published industrial/organizational psychologist who is intimately familiar with the O*NET program and the types and uses of the data collected. Revisions responsive to his comments have been incorporated, as appropriate, in the clearance package.
The data collection contractor, RTI International, has several mechanisms in place to obtain ongoing feedback from study participants. The BLs have multiple telephone contacts with POCs and occupation experts; they are careful to document in their call notes any significant comments or suggestions they receive. Subject matter experts at the professional associations that we contact to help identify occupation experts frequently provide feedback that the National Center for O*NET Development uses to refine occupation descriptions and tasks. Letters and brochures sent to POCs and occupation experts offer a toll-free number they can call with questions or comments. In addition, the project Web site, https://onet.rti.org/, which both study participants and the public can access, has a “contact us” tab that offers both a toll-free number and an e-mail link. Finally, both the paper and online versions of the questionnaires provide respondents an opportunity to submit comments with their questionnaire responses. All feedback received from these sources is promptly reviewed by project management staff. Of course, the agency will comply with all Paperwork Reduction Act requirements should comments warrant changing the information collection.
Since the origins of the survey, incentives have been offered to POCs, establishments, and employees to encourage their participation in the O*NET Data Collection Program. Although the procedures are designed to minimize respondent burden, the effort for the company and the POC participation is not insignificant.
The POC’s responsibilities include:
reading the introductory package to become familiar with the purpose of the O*NET Data Collection Program and the role of a POC;
seeking permission within the company, as necessary, to participate in the O*NET Data Collection Program;
making a roster of all employees at the location who work in as many as five different occupations;
participating in a sampling process that selects as many as 20 total employees from these occupations, and maintaining this sample roster for future reference;
distributing questionnaires to the sampled persons within the company and addressing their questions and concerns about the survey; and
distributing follow-up materials to employees, including thank you/reminder cards and replacement questionnaires, and following up with nonrespondents to encourage participation.
Because POCs are the only link with the O*NET respondents, they must be fully committed to the data collection process. The POC is the O*NET Program’s representative within the establishment who communicates the importance of the O*NET Program.
The employer is also essential because he or she is being asked to:
support the O*NET Data Collection Program by agreeing to the company’s participation in data collection,
support and encourage the POC in carrying out his or her responsibilities,
allow the POC to provide information regarding the number of persons employed in the establishment in the occupations of interest, and
support the participation of the sampled employees.
Incentives for both POCs and employers are essential to encourage the highest response rates possible. POCs who agree to participate receive a framed Certificate of Appreciation from DOL. The Certificate of Appreciation is printed in color on card stock, bears the DOL Seal and O*NET logo, displays the POC’s name, and is signed by a high-ranking DOL official. It has a solid oak frame and Plexiglas cover and is suitable for displaying on an office wall.
Employers who agree to participate receive the O*NET Toolkit for Business. The toolkit is an O*NET Program information packet, including a guide for writing job descriptions, that managers can use for human resource planning. These materials are attractively organized in colored, glossy folders.
The continuation of these incentives is planned for both the POC and the employer because they seem to be working quite well, as evidenced by O*NET’s competitive employer response rate (see Appendix E). An experiment was conducted from 2002 to 2004 to evaluate an additional incentive of $20 to the POC. Essentially, the incentive had no effect on POC cooperation rates or employee response rates, but it significantly increased data collection costs. The experimental findings suggested that the current POC and employer incentives are adequate for maximizing response rates at a reduced cost (Biemer, Ellis et al. 2006).11
In keeping with what has been done since 2001, each employee is offered a prepaid incentive of $10 to ensure that a high percentage of the job incumbents respond by completing the questionnaire.
Monetary incentives have the greatest potential impact when the respondent has to exert some special effort, such as taking a test or filling out a multi-item questionnaire. The incentive encourages respondents in a task requiring higher levels of involvement and commitment than the typical one-time, face-to-face interview. Although the O*NET questionnaires are not tests, the cognitive demands they place on respondents resemble test-taking demands in that the respondents must assess the requirements of their jobs. The monetary incentive is instrumental in impressing upon the respondent the importance of this rating task. Respondents who perceive the rating task as important will likely make thoughtful, carefully considered assessments rather than hasty ones, thus improving the reliability of the data.
In addition, the monetary incentive is important because respondents are encouraged to complete the questionnaire, which takes about 30 minutes, on their own time rather than on the job. This encouragement minimizes the burden on employers and also improves the quality of the data; without it, busy workers might be underrepresented in the data, which would bias the estimates for job performance.
The monetary incentive may at least partially offset its inherent cost through efficiencies created in the data collection process as a result of higher response rates (Statistics Research Division 2000, October). For the job incumbent respondents especially—although they are not viewed as a difficult-to-reach population in the usual sense—considerable effort and resources are expended to identify and reach them through the sampling process. Each one represents a worker in a specific occupation in a specific establishment in a specific industry. The expense of reaching that particular respondent justifies the cost of a monetary payment to ensure a high rate of response.
With regard to the size of the employee incentive, payment amounts were evaluated in the pretest to determine the optimal means to maximize the response rate. On the basis of these data, a $10 prepaid cash incentive has been used since the initial wave of data collection in 2001. Because the employee response rate continues to compare favorably with those of other voluntary establishment surveys, an increased incentive does not seem to be justified at this time.12 However, we will continue to monitor the employee response rate and will reassess the amount of the incentive if it appears that the benefits of an increased incentive would outweigh the cost.
Occupation experts provide data for approximately 25% of the O*NET-SOC occupations. Each occupation expert who agrees to participate receives a prepaid incentive of $40 and a framed Certificate of Appreciation from DOL. Unlike job incumbents, who complete only one domain questionnaire, occupation experts are asked to complete all three domain questionnaires; thus, the $40 incentive is about $13.33 per questionnaire. This incentive is slightly higher than the $10 offered to Establishment Method respondents for completing one domain questionnaire. The increased incentive and the Certificate of Appreciation are necessary to gain cooperation from what is often a rare group of experts for an occupation. Moreover, the additional incentives seem commensurate with the effort involved in responding to multiple questionnaires, given that occupation experts are supervisors and trainers in the occupation and, as such, earn a higher salary than the average employee.
Respondents are informed that their individual responses will be kept private to the extent permitted by law. A system privacy statement is located on the system website at https://www.onetonline.org/help/privacy/. Survey data are collected from job incumbents (Establishment Method) and from occupation experts (OE Method). Informational materials provided to potential respondents contain essential program information and assurances of privacy that enable the person to make an informed decision about his or her voluntary participation in the data collection effort. Examples of informational materials provided to survey participants appear in Appendix F.
Employees sampled at establishments are asked to complete their questionnaires on their personal time, not company time. This stipulation enables the employee to select a comfortable and private setting, if desired, in which to complete the questionnaire. All respondents have a choice of completing paper questionnaires or completing the questionnaires online at the project Web site. Paper questionnaires are mailed directly to RTI in a stamped reply envelope provided by RTI. The individual responses are processed according to a study ID number. All O*NET Data Collection Program staff are required to sign a privacy pledge that assures each respondent that the privacy of responses to the questionnaire will be maintained. Only authorized staff have access to the completed instruments and data files. The completed and processed questionnaires are stored in a secure document-control area until federal authorization has been granted to destroy them. All computer files, including those associated with the control system, are password protected.
The Internet-based system that allows respondents to provide their survey responses electronically has restricted access, including a user ID and password authentication protocol for respondents. The Web server includes an SSL certificate, the same technology used by e‑commerce sites, to allow encrypted transmission of all information over the Internet. The database containing the survey data is not accessible via the Internet; it resides on a server inside the RTI firewall. Only O*NET Data Collection Program staff have access to the master survey database.
The O*NET questionnaires (see Appendix G) collect very little personal information about the respondent, and what is collected contains no identifiers, such as personal name or place of employment. No individual-level data are published, nor are they accessible or provided to anyone except the O*NET Data Collection Program staff. Published results are made available only in aggregate, as one set of estimates for an entire occupation. Furthermore, no demographic data (e.g., sex, race) are released, even in aggregate form. Finally, estimates are not produced for any subpopulations within an occupation, such as geographic region or sociodemographic group, as these details could allow the identification of individuals.
Before publishing the O*NET tables on the public Web site, the O*NET Program team thoroughly examines the tables for any risk of disclosure of private information. In particular, each table is analyzed to identify any “sensitive” cells (i.e., cells that may reveal too much information about an individual employee). Willenborg and De Waal (1996) have recommended using an (n,k)-dominance rule that a cell be regarded as sensitive if the sum of the largest n contributions account for more than k% of the total cell value. Willenborg and De Waal further recommend that n = 5 and k = 80. Because every sampled employee contributes only one response, these recommendations translate into a minimum cell size of 5/0.80, or about 6. In fact, the minimum cell size for the O*NET tables is 10 respondents; any sensitive cells with fewer than 10 are suppressed. In addition, the (n,k)-dominance rule assumes a complete census. As Willenborg and Waal note, when applied to tables based on samples and where the cell entries are weighted averages, the (5,0.80) rule affords even greater disclosure control. This extra control ensures that these O*NET tabular data pose no disclosure risks to any individual respondent.
Only four questions in the O*NET questionnaires may be considered to be of a sensitive nature. In the Background Questionnaire, the survey uses the four disability questions developed for the Census Bureau’s American Community Survey. The first two questions ask respondents about serious hearing and sight difficulties. The next question has subparts that ask about difficulties with concentration, memory, or decision making; mobility; and self-care. The last question asks whether respondents’ physical or mental health makes it difficult to do errands alone. Completion rates for these items indicate that the great majority of participants (97%) elect to answer them.
The O*NET sampling strategy is to randomly select participants at the individual level. The disability questions, together with the demographic questions, provide descriptive information about the sample of respondents.
Respondents to the O*NET survey are informed that responding to all questions, which includes disability status and the other demographic characteristics, is voluntary and that the data will be kept private to the extent permitted by law. They complete the survey on their own time, in a private setting if they choose. No identifying information, such as the respondent’s name or place of employment, is recorded on the questionnaire.
Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. Unless directed to do so, agencies should not conduct special surveys to obtain information on which to base hour burden estimates. Consultation with a sample (fewer than 10) of potential respondents is desirable. If the hour burden on respondents is expected to vary widely because of differences in activity, size, or complexity, show the range of estimated hour burden, and explain the reasons for the variance. Generally, estimates should not include burden hours for customary and usual business practices.
If this request for approval covers more than one form, provide separate hour burden estimates for each form.
Provide estimates of annualized cost to respondents for the hour burdens for collections of information, identifying and using appropriate wage rate categories. The cost of contracting out or paying outside parties for information collection activities should not be included here. Instead, this cost should be included in Item 14.
As described in Section 3.1, there are two methods for O*NET data collection: the Establishment Method and the OE Method. The Establishment Method uses a two-stage sample, with establishments selected at the first stage and employees at the second stage. Thus, there are burden hours associated with both establishments and employees. For the OE Method, burden is incurred only by sampled occupational experts who return questionnaires.
Exhibit A-2 presents the estimated annualized cost and hour burden across participating establishments and individuals. The burden estimates are driven by the average annual number of respondents and the average time per response. The eligibility and participation rates assumed to calculate the average annual number of respondents are based on prior years of O*NET data collection and assume that 100 occupations will be completed annually: 40 via the OE Method and 60 via the Establishment Method.13 For more details on the calculation of the burden estimates, see Appendix H.
The time per response for a sampled establishment depends on the number of activities completed among those listed below. Most establishments do not complete all activities because of ineligibility and nonresponse. A POC is identified within participating establishments to coordinate the following data collection activities:
Verification call to confirm the establishment is in business (2 minutes);
Screening call to identify whether any occupations of interest are present (3 minutes);
Recruiting call to gain participation in O*NET (12 minutes);
Creating lists of employees in the selected occupations for sampling (20 minutes);
Sampling call to select employees for the survey (10 minutes);
Distribution of questionnaire packets (15 minutes); and
Up to four follow-up calls to POC (2 minutes per call).
The total time per response for an establishment can range from 2 to 70 minutes, depending on how many of these activities are completed. Past years of O*NET data collection experience show that the average time per response is 18.8 minutes, or 0.313 hours.
The average time per response for sampled individuals is 30 minutes per domain questionnaire. Employees sampled under the Establishment Method are asked to complete one domain questionnaire (0.5 hour per response), and occupational experts sampled under the OE Method are asked to complete three domain questionnaires (1.5 hours per response).
All O*NET activities are voluntary and contribute only reporting or processing burden. No record-keeping burden is incurred.
There are no respondent costs for capital or start-up or for operations, maintenance, and purchase of services. There are no costs to the employers, POCs, or sampled employees other than the time it takes them to participate in the survey.
Exhibit A-2. Estimated Annualized Cost and Hour Burden
Activity |
Number of Respondents |
Number of Responses per Respondent |
Total |
Average Burden Hours |
Total Annual Burden Hours (Rounded) |
Hourly Ratea |
Total Burden Cost |
Individuals |
|
|
|
|
|
|
|
[1] Establishment Method employee respondents |
13,068 |
1 |
13,068 |
0.5 |
6,534 |
$38.67 |
$252,670 |
[2] Occupation Expert Method respondents |
1,000 |
1 |
1,000 |
1.5 |
1,500 |
$38.67 |
$58,005 |
Unduplicated Totals |
14,068 |
1 |
14,068 |
2 |
8,034 |
$38.67 |
$310,675 |
Establishments |
|
|
|
|
|
|
|
[3] State/local/tribal governments |
578 |
1 |
578 |
0.313 |
181 |
$70.00 |
$12,670 |
[4] Federal government |
1,706 |
1 |
1,706 |
0.313 |
534 |
$70.00 |
$37,380 |
[5] Private sector |
24,590 |
1 |
24,590 |
0.313 |
7,697 |
$70.00 |
$538,790 |
Unduplicated Totals |
26,874 |
1 |
26,874 |
0.313 |
8,412 |
$70.00 |
$588,840 |
Grand Totals |
40,942 |
— |
40,942 |
— |
16,446 |
— |
$899,515 |
a The average annual hourly rate was estimated using the March 2020 Employer Costs for Employee Compensation Summary issued by BLS on June 18, 2020. The average total compensation per hour for private industry was $35.34, which was inflated based on the Employment Cost Index to a median hourly total compensation of $37.59 for December 2021–November 2022, $38.66 for December 2022–November 2023, and $39.75 for December 2023–November 2024. The average of the three inflation-adjusted rates ($38.67) is used for estimating the employee/individual cost burden. Given that the establishment POC will often be a human resources manager, the March 2020 total compensation rate of $63.98 for the Management, Professional and Related category was inflated to a median hourly total compensation of $68.06 for December 2021–November 2022, $69.98 for December 2022–November 2023, and $71.96 for December 2023–November 2024. The average of the three inflation-adjusted rates ($70.00) is used for estimating the establishment cost burden.
Source: Bureau of Labor Statistics, U.S. Department of Labor. (2020, June). Employer costs for employee compensation summary. Available from Databases, tables and calculators by subject, https://data.bls.gov/timeseries/CIS2010000000000Q
The estimated annual cost to the government for the O*NET Data Collection Program for the period December 2021 through November 2024 is approximately $7.6 million. This estimate includes all direct and indirect costs of conducting the sampling, data collection, and analysis activities of the O*NET Data Collection Program. In Exhibit A-3, personnel and fringe benefit costs are for the grantee (N.C. Department of Commerce) personnel who manage the O*NET Data Collection Program. The grantee subcontracts certain program activities (e.g., survey operations and data analysis); those costs are listed on the contractual line.
Exhibit A-3. Annualized Costs to the Federal Government
Item |
Amount |
|
|
Personnel |
$502,686 |
Fringe benefits |
$167,786 |
Travel |
$15,000 |
Equipment |
$4,000 |
Supplies |
$41,400 |
Contractual |
$6,518,552 |
Costs for incentives |
$150,600 |
Total direct costs |
$7,469,269 |
Indirect charges |
$109,887 |
Grand Total |
7,509,911 |
As described in Question 1, Appendix G in this ICR package differs from the 2018 Appendix A submission: the questions related to association membership have been removed from the Background questionnaire because this information is available through other existing sources; and numerous level scale anchors have been updated as part of efforts to modernize and lend clarity to the Knowledge and Work Activities questionnaire descriptors. The Department does not believe that these changes in requirements impact the burden calculation because the burden is already accounted for as an average.
Exhibit A-4 compares the burden projected for the December 2021–November 2024 period with the burden requested for the previous 3-year period, October 2018–November 2021. Estimates for the previous 3-year period are available from the OMB Supporting Statement, Part A, dated September 2018. The average annual burden hours projected for December 2021–November 2024 are 16,446, which represents an increase from the previous 3-year period (2018–2021), when 14,292 hours were requested. Similarly, the annual average number of responses for the December 2021–November 2024 period is 40,942, whereas 28,494 responses were estimated for the 2018–2021 period (U.S. Department of Labor 2018).
The increase in the estimated burden is due to two factors. First, we have increased the assumed number of establishments fielded per month from approximately 1,360 to 3,000 to offset declining response rates and potentially lasting impacts of the COVID-19 pandemic. The AAPOR, the Census Bureau, and others have widely noted the general downward trend in survey participation (Tourangeau 2017, May, Office of Survey Methods Research 2020, American Association of Public Opinion Research n.d.). Over the past decade, O*NET has also experienced a gradual decline in response rates, particularly for the Establishment Method (see Appendix E for a summary of response rate experience to date). We further expect that the pandemic may have lasting impacts on participation and eligibility, as U.S. businesses are challenged to remain open and operate at full capacity. Given these uncertainties, we have assumed lower participation and eligibility rates and increased the estimated sample size requirements for the 2021–2024 burden period. Although the potential increase in the monthly establishment sample size appears large, the sample sizes may be smaller in practice. If production is higher than expected for the Establishment Method, O*NET’s Model-Aided Sampling (discussed in Supporting Statement Part B, Section 2.1) paradigm permits early termination of data collection to prevent overproduction and safeguard against burden.
The second factor leading to an increase in the estimated burden is that we have increased the assumed number of OE Method occupations fielded per year from 25 to 40. This adjustment is designed to help ease some of the establishment burden. Like the Establishment Method, the OE Method has experienced some decline in participation but to a lesser extent; this is likely because individuals who are sampled via the OE Method are contacted directly, unlike the Establishment Method, for which selected individuals remain anonymous and questionnaire distribution is coordinated through a POC. Thus, the OE Method provides an efficient approach for collecting and updating occupations that are increasingly difficult to survey under the Establishment Method.
Last, Exhibit A-4 compares the estimated annual average cost to respondents for December 2021–November 2024 with the average annual cost estimated for 2018–2021. The increased annual costs since 2018–2021 are due to the increased sample sizes described above and to inflation in the benefits portion of employee compensation.
More detailed comparisons of the hour and cost burden between the 2021–2024 and 2018–2021 burden periods may be found in Appendix H.
Exhibit A-4. Comparison of the Average Annual Hour and Cost
Burden
Between the 2018–2021 and 2021–2024 Burden
Periods
|
Annual
Average |
|
Annual
Average |
||||
Activity |
Total |
Total |
Monetized |
|
Total |
Total |
Monetized |
Individuals |
|
|
|
|
|
|
|
[1] Establishment Method employee respondents |
14,263 |
7,132 |
$255,540 |
|
13,068 |
6,534 |
$252,670 |
[2] Occupation Expert Method respondents |
675 |
1,013 |
$36,316 |
|
1,000 |
1,500 |
$58,005 |
Establishments |
|
|
|
|
|
|
|
[3] State/local/tribal governments |
291 |
132 |
$8,377 |
|
578 |
181 |
$12,670 |
[4] Federal government |
861 |
390 |
$24,749 |
|
1,706 |
534 |
$37,380 |
[5] Private sector |
12,404 |
5,625 |
$356,963 |
|
24,590 |
7,697 |
$538,790 |
Unduplicated
Totals |
28,494 |
14,292 |
$681,945 |
|
40,942 |
16,446 |
$899,515 |
a From the OMB Supporting Statement, Part A, dated September 2018.
b From Appendix Exhibit H-2.
The major components of the O*NET Data Collection Program include sampling, data collection operations, and analysis. Exhibit A-5 shows the expected schedule for annual data analysis cycles and data publications for the next few years. Ranges for the estimated number of published occupations presented for Analysis Cycles 23 and 24 demonstrate the uncertainty of the impacts of COVID-19 on participation and business eligibility rates. It is assumed that the impacts of COVID-19 will gradually dissipate, thereby increasing the number of occupations available for publication each subsequent year.
Exhibit A-5. Data Analysis and Publication Schedule
Analysis Cycle |
Analysis Cycle Start Date |
Analysis Cycle End Date |
Estimated Establishment Method O*NET-SOC Occupations |
Estimated Occupation Expert Method O*NET-SOC Occupations |
Estimated O*NET-SOC Occupations Published |
Publication Date |
23 |
July 2021 |
July 2022 |
50–60 |
30–40 |
80–100 |
July 2022 |
24 |
July 2022 |
July 2023 |
55–60 |
35–40 |
90–100 |
July 2023 |
25 |
July 2023 |
July 2024 |
60 |
40 |
100 |
July 2024 |
Described here is the approach used for data cleaning and editing, as well as the analyses that are performed annually.
Paper questionnaires are manually and machine edited so that completely blank questionnaires are removed; responses to items that should have been skipped are blanked out; multiple responses are blanked out; and codes indicating missing data, multiple responses, and legitimate skips are inserted. Codes for legitimate skips and missing responses are also inserted in the records for questionnaires obtained through the Web. In addition, an electronic check is conducted to detect duplicate questionnaires from the same respondent.
Analyses of the questionnaire ratings are based on the assumption that raters were qualified, willing, and able to engage in the rating tasks. This task is accomplished by requiring each case to first pass through a series of machine edits using prescribed eligibility criteria, including having at least one task rated important and having at least 50% of the domain questionnaire items completed. Cases not meeting these criteria are excluded from the analysis file. Cases with certain questionable characteristics are flagged in this editing process, and analysts review these cases to determine their completion status. Flagged for review are all cases in which the respondent (1) indicated in the “global match” item that the target O*NET-SOC occupation description did not at all describe his or her own job and (2) rated fewer than one third of the tasks as important. An analyst reviews the self-reported job titles of these cases to determine if they are at all likely to belong in the O*NET-SOC occupation. If a case does not belong, it is removed from further analysis. If, in the judgment of the analyst, there is a reasonable chance that the case belongs in the O*NET-SOC occupation, it is sent to the next stage of review.
Finally, cases that pass the machine edits and the analyst review are subjected to a deviance analysis designed to identify cases that are outliers relative to other cases in their occupation. The deviance analysis involves two procedures: (1) a statistical procedure to quantitatively identify potential outliers and (2) an analyst review of these potential outliers to make the final decision for each case. Cases that do not pass the analyst review are deemed deviant within their occupation and are removed from the data set. On average, these activities eliminate about 9% of all returned questionnaires. The cases passing all data cleaning criteria are used to create the estimates for publication.
Basic sampling weights are applied to the data to make inferences about the population of incumbents for each occupation.14 These weights are computed as the inverses of the overall selection frequencies and the selection probabilities for each selected establishment and each individual participant. The analysis weights for the eligible sample units are adjusted to compensate for unit nonresponse for both establishments and employees, multiple subwaves of sampling, and sample adjustments. In addition, when variation in the weights is large, the weights are trimmed to reduce the variation. To maximize comparability of O*NET estimates to estimates from other federal sources, the final sample weights are also ratio adjusted to occupation estimates obtained from the OES survey.
Sampling errors are computed. The analysis weights used in the sampling error computations, as noted, have been adjusted for nonresponse and are consistent with the complex sampling design.
For each occupation, the sample size, mean, standard deviation, and standard error of the ratings for each descriptor are calculated, together with the 95% confidence interval around the mean. Estimates with questionable precision are flagged and recommended for suppression if any of the following conditions is true:
The sample size (i.e., number of respondents who answered the question) is less than 10.
The variance is zero and the sample size is less than 15.
The relative standard error (RSE) is greater than 50%.15
On average, 2–3% of the estimates are flagged or suppressed.16
Interrater reliability and agreement are assessed with three different analyses. The first two measure reliability in terms of the covariation among ratings, and the third reflects rater agreement. The intraclass correlation coefficients (ICCs) for each questionnaire item across all occupations are computed. These results allow one to compare respondent rating variance within an occupation with respondent rating variance across occupations. In addition to the ICCs, the mean interrater correlations (Pearson’s r) are calculated for all pairs of raters within each occupation. Finally, to assess the absolute difference among ratings of each item within each occupation, an average deviation index is calculated for each item within each occupation. For any given item and occupation, the average deviation index measures the average extent to which each individual rating deviates from the item mean. Some differences in ratings within occupations are expected because O*NET-SOC occupations comprise a variety of different jobs in most cases.
Each of the reliability analyses conducted (rater, standard errors) is influenced by the number of respondents. The O*NET data collection methods include a sufficient number of respondents in each occupation to ensure reliability (Peterson, Mumford et al. 1997).
The O*NET database is scheduled to be updated annually. Each update will include data for those occupations collected and analyzed during the previous 12-month period. Consequently, a database update includes occupations from multiple data collection waves, depending on the number of prior waves for which analysis was completed that year. For each occupation collected, the newly calculated means data replace existing analyst-based or incumbent-based means data in the database. Users are provided with metadata regarding when the data were collected and other pertinent information that will assist the users in interpreting the data.
The O*NET database is designated with a version number denoting each update (e.g., from O*NET 25.0 to O*NET 25.1). The database is developed and administered with the MySQL database management system. Once the MySQL database is updated, it is used to generate the database for public release as a series of flat text files. It is accessible to the public on the O*NET Program Web site at https://www.onetcenter.org/.
The expiration date will be displayed on the cover of the survey questionnaires.
There are no exceptions.
1 One hundred forty-five updated occupations not included in the 2018 SOC were not published in the 25.1 O*NET database. In addition to the 821 occupations published, 53 “rolled-up” 2018 SOC occupations are published with aggregate data from more detailed 2010 SOC occupations. New data will be collected for these 53 occupations at the 2018 SOC taxonomy level.
2 More information about the O*NET Data Collection Program can be found at the National O*NET Program’s public Web site, https://www.onetcenter.org/ (the O*NET portal page that links to several O*NET-related Web sites), and at http://www.doleta.gov/programs/onet (Web site of the Employment and Training Administration at the U.S. Department of Labor) Employment and Training Administration, U.S. Department of Labor (2009). O*NET OnLine in transition assistance programs at Offutt Air Force Base. Lincoln, NE,
3 Of the 923 occupations in the O*NET data collection plan, 821 occupations have been comprehensively updated, and 102 occupations have been partially updated in the November 2020 database version. The 102 partially updated occupations will be comprehensively updated in subsequent database versions.
4 Bright Outlook occupations are those expected to grow rapidly in the next several years or those expected to have large numbers of job openings.
5 The goal of the study was to compare the psychometric quality of incumbents’ Skills ratings with that of analysts’ Skills ratings across a large sample of O*NET-SOCs. Although some mean differences between incumbents’ and analysts’ ratings were observed, the results yielded only minimal differences between the two systems of obtaining Skills information.
6 For a description of the pretest, see Supporting Statement Part B, Section 4.1.
7 For a discussion of current and future efforts to improve response rates, see Supporting Statement Part B, Section 4.
8 For additional information about sampling with the Establishment Method, see Supporting Statement Part B, Section 2.1.
9 For additional information about sampling with the OE Method, see Supporting Statement Part B, Section 2.2.
10 See Appendix Exhibit B-2 for the list of questionnaires, number of items and scales, and data sources.
11 For a more detailed discussion of the Biemer et al. Biemer, P., C. Ellis, A. Pitt and K. Robbins (2006). Effects on response rates and costs of a monetary incentive for the point of contact in an establishment survey. Proceedings of the American Association of Public Opinion Research, Montreal, Canada. results, see Supporting Statement Part B, Section 5.
12 See Appendix E for a discussion of O*NET’s response rate experience.
13 Historically, it has been assumed that of the 100 occupations completed annually, 75 would be completed via the Establishment Method and 25 via the OE Method. The number of OE occupations has been increased from 25 to 40 for the 2021–2024 period to help offset the burden and unknown impacts of COVID-19 on establishments (see Section 15 for further discussion). Note that this change does not affect the overall proportions of Establishment and OE occupations in the O*NET database as described in Section 3.1; instead, occupations that have historically been OE have been accelerated in the data collection schedule.
14 For details about the weighting and estimation procedures, see Supporting Statement Part B, Section 1.1.
15 RSE = the standard error of the mean divided by the mean.
16 O*NET suppression criteria are based on substantive expert recommendations Peterson, N. G., M. D. Mumford, W. C. Borman, P. R. Jeanneret, E. A. Fleishman and K. Y. Levin, Eds. (1997). O*NET final technical report. Salt Lake City, UT, Utah Department of Workforce Services., best practices Willenborg, L. and T. De Waal (1996). Statistical disclosure control in practice series: Lecture notes in statistics. New York, NY, Springer., and other large government surveys Klein, R. J., S. E. Proctor, M. A. Boudreault and K. M. Tuczyn (2002). Healthy People 2010 criteria for data suppression. Hyattsville, MD, National Center for Health Statistics..
A-
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Bogus, Michelle |
| File Modified | 0000-00-00 |
| File Created | 2021-10-19 |