1205-0421 Supporting Statement Part A_FINAL 9.25.18

This ICR seeks to renew and extend the Occupational Information Network (O*NET) data collection program, with no changes, to collect updated occupational characteristics and requirements information, on an ongoing basis. The appendices in this ICR package differs from the 2015 submission as there are no changes to the questionnaires (Appendix A) in this package and the Advisory Panel for the Dictionary of Occupational Titles: Final Report is not included; Appendix C in this submission is now Publications Referencing the O*NET Data Collection Program.

Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.
1. Overview

This Supporting Statement is a request and justification for a 3-year clearance from the Office of Management and Budget (OMB) to continue, with no changes, the Occupational Information Network (O*NET®) Data Collection Program. The O*NET Data Collection Program continually operates to populate and maintain a current database on the detailed characteristics of workers, occupations, and skills. The program uses an occupational taxonomy, the O*NET-SOC, which is currently based on the 2010 version of the Standard Occupational Classification (SOC) mandated by OMB for use by all federal agencies collecting occupational and labor market information (LMI). The 2018 SOC will be adopted by O*NET as corresponding wage data and as sampling frame data become available from the U.S. Bureau of Labor Statistics, Occupational Employment Statistics (OES) program. The O*NET-SOC comprises occupations at the most detailed level of the SOC and includes additional occupational specificity as needed. In addition, new and emerging occupations in high-growth sectors of the economy have been identified and added to the taxonomy. Data have been published for 966 O*NET-SOC occupations, 723 of which have undergone more than one update. The O*NET Data Collection Program received initial OMB clearance in 1999 for a pretest and 6 subsequent clearances that have allowed main study data collection to continue without interruption since June 2001. Our current clearance expires September 30, 2018. This request is to continue to update occupations that reflect older data as well as to collect data on new and changing occupations included in the 2018 SOC for 3 more years (October 1, 2018–September 30, 2021), subject to annual budget levels.

The continued population of the O*NET database is important because the O*NET database is the most current and comprehensive standard source of descriptive occupational information in the United States. The O*NET Data Collection Program remains at the center of an extensive network of occupational and skill information used by a wide variety of audiences, including individuals making career decisions; public agencies, such as workforce boards and American Job Centers, making training investment decisions; educational institutions preparing a future workforce; and employers making staffing and training decisions.

This program provides a common language and framework to facilitate communication about industry skill needs among business, education, and the workforce investment system. The resulting O*NET database also is used to develop industry competency models and occupational competency profiles. O*NET data include information about transferable skills that are used for skills gap analysis, facilitating a mobile workforce responsive to changing economic needs. The O*NET database and companion O*NET Career Exploration Tools are used by many private companies and public organizations to tailor applications to their needs and those of their customers. The broad utility of the O*NET tools plays an important role in developing and maintaining a skilled workforce and contributes to U.S. competitiveness in a global, 21st-century economy.¹

The O*NET Data Collection Program employs a multiple-method approach to updating the O*NET database. The primary method involves a two-stage sample design to survey establishments and workers in those establishments. When necessary, this method may be supplemented with a sample selected from additional sources, such as professional and trade association membership lists, resulting in a dual-frame approach. An alternative method, based on sampling from lists of identified occupation experts, is used for occupations for which the primary method is inefficient. This method is reserved for selected occupations, such as those with small employment scattered among many industries and those for which no employment data currently exist on which to base a sample, such as new and emerging occupations. The O*NET survey instruments are used with all methods.² The rest of Part A describes the O*NET Program and reviews statutory and regulatory information.

What Is the O*NET Program?

The O*NET Program is a comprehensive system for collecting and disseminating information on occupational and worker requirements.

As shown in Exhibit A-1, the O*NET Program uses a data structure, the Content Model, to organize occupational information and to provide a common language of standardized and defined occupation descriptors and measures for use by all audiences. The O*NET Content Model is the result of extensive research, and its development is fully documented (Peterson, Mumford, Borman, Jeanneret, & Fleishman, 1995, pp. 2–6; Peterson, Mumford, Borman, et al., 1997; Peterson et al., 2001). It comprises worker-oriented and job-oriented characteristics at both an occupation-specific level and across occupations, as the exhibit illustrates.

Each of the six domains of the Content Model groups information hierarchically. For example, the Worker Characteristics domain contains four types of information: Abilities, Occupational Interests, Work Values, and Work Styles. From these four, the Abilities domain, in turn, contains four types of abilities: Cognitive, Psychomotor, Physical, and Sensory. Each of these types of abilities contains further levels of detail. For example, the Psychomotor type includes Fine Manipulative, Control Movement, and Reaction Time and Speed. Finally, Fine Manipulative contains three specific descriptors: Arm-Hand Steadiness, Manual Dexterity, and Finger Dexterity. Hierarchies are a useful means of both organizing occupational information and allowing for its access at different levels of specificity. By organizing worker- and job-oriented characteristics hierarchically, the O*NET Content Model provides a flexible, common-language-based system to describe the world of work.

Exhibit A-1. O*NET Content Model

The descriptors and rating scales for O*NET data were developed through extensive research, drawing primarily from job analysis in industrial/organizational psychology and human resource management (Peterson et al., 1995). The descriptors in the O*NET Program are meant to be comprehensive. The primary sources of data are job incumbents and occupation experts. The SOC system is used as the basis for classifying occupations. The use of questionnaires and rating scales reflects the most widely accepted approach to job analyses conducted across settings, occupations, or positions (Guion, 2011). The scales used for the O*NET ratings are Importance, Level, and Frequency. Each descriptor in the O*NET questionnaires may use one or more scales. For example, the O*NET Work Activities Descriptor—Monitoring and Controlling Resources is rated on both a 5‑point Importance scale and a 7-point Level scale. For the complete set of O*NET questionnaires, which include O*NET descriptors, see Appendix A.

Exhibit A-2 summarizes the number of descriptors and scales in the O*NET Data Collection Program questionnaires. Descriptors are identified from O*NET Content Model domains. Data are collected by means of 239 descriptors that include 400 scales (e.g., Importance, Level, and Frequency). To collect ratings for the Abilities and Skills domains, trained occupational analysts review updated information (e.g., Tasks, Generalized Work Activities) provided by job incumbents.³ No data collection is planned for the Workforce Characteristics domain. Information for it is provided through links to the employment, wage, and long-term projections databases produced by the U.S. Bureau of Labor Statistics (BLS), the state employment security agencies, and other agencies.

Exhibit A-2. O*NET Data Collection Program Questionnaires

*ONET Data Collection Program Questionnaire**	Number of Descriptors	Number of Scales per Descriptor	Total Number of Scales	Data Source
Skills	35	2	70	Analysts
Knowledge	33	2	66	Job incumbents
Work Styles^a	16	1	16	Job incumbents
Education and Training^a	5	1	5	Job incumbents
Generalized Work Activities	41	2	82	Job incumbents
Work Context	57	1	57	Job incumbents
Abilities	52	2	104	Analysts
Tasks^b	Varies	2	Varies	Job incumbents
Total (not including Tasks)	239	NA	400	NA

Notes: Occupation experts use the same questionnaires as job incumbents for those occupations whose data collection is by the Occupation Expert Method. NA = not applicable.

^a The Knowledge Questionnaire packet also contains the Work Styles Questionnaire and the Education and Training Questionnaire.

^b All job incumbents are asked to complete a Task Questionnaire in addition to the domain questionnaire.

Versions of the O*NET Database

The first version of the O*NET database released to the public was O*NET 98. The O*NET 98 database contained 306 descriptors and 684 scales. A review of O*NET 98–specific scales and descriptors during the preparation for pretest data collection led to some consolidation and refinement of descriptors and scales to reduce burden on the public and to increase employee response rate.⁴

The O*NET 98 database was first replaced with the O*NET 3.1 database and has been updated 18 times as new data have been collected and analyzed. The current database, O*NET version 22.0, contains the same descriptors used in O*NET 98; however, the occupations have been restructured and coded to encompass the most detailed level of the 2010 SOC, with more occupational specificity added as needed. Research is ongoing to identify additional new and emerging occupations in high-growth industries. New occupations emerge because of changes in technology, society, law, business practices, and markets. As these new and emerging occupations are identified and their data are collected, they will be integrated into the O*NET-SOC occupation classification and database.

O*NET 22.0 has a Web-based accessing application, O*NET OnLine, which is available to the public at no cost at https://online.onetcenter.org/. An electronic version of the 22.0 database can be downloaded at https://www.onetcenter.org/. The data can also be accessed/incorporated via O*NET Web Services. (See https://services.onetcenter.org.) The O*NET 22.0 database has been restructured to incorporate improvements made to the O*NET data collection instruments and is the structure currently being offered to developers.

Data in the O*NET database include the mean ratings on each of the items (or descriptors) in the O*NET questionnaires. Ratings have been standardized to facilitate interpretation and comparison across occupations. In addition to mean rating data on Level and Importance for various questionnaire items, text information is also included on occupational definitions, descriptor definitions, scale anchors, and task descriptions.

The O*NET Data Collection Approach

The O*NET Data Collection Program is key to the continued effort to update the O*NET database to reflect changing skills requirements of occupations with the advent of new technologies and the changing world of work. In the research leading to the O*NET Data Collection Program, various sources and methods for collecting occupational information were examined, including collection of data from job incumbents and supervisors and development of ratings by occupation experts and occupational analysts. On the basis of this work, the O*NET team determined that the preferred source of data for most domains (Generalized Work Activities, Work Context, Knowledge, Education and Training, and Work Styles) is job incumbents. Other occupation experts, such as supervisors and trainers, may be used where access to job incumbents proves difficult or where the sampling of business establishments is inefficient.

Previous studies comparing various sources of job analysis ratings suggest that incumbents are able to provide information across a variety of descriptor domains (Fleishman & Mumford, 1988; Peterson, Owens-Kurtz, Hoffman, Arabian, & Whetzel, 1990; Schumacher, Kleinmann, & Konig, 2012). In addition, “large samples of knowledgeable job incumbents are available, which should contribute to the reliability of the resulting descriptive system” (Peterson, Mumford, Levin, Green, & Waksberg, 1999, pp. 2–6). Furthermore, the world of work is constantly changing, and technological advancements are occurring so rapidly that an efficient, effective way to remain current and accurate is to obtain the information directly from those performing the work.

By contrast, occupational analysts, who are provided with updated information from job incumbents, are preferred for the Abilities domain, which tends to be more abstract than the other domains. The Skills domain, whose variables are somewhat abstract, is a strong candidate for either source of collection; it is now updated by occupational analysts. A study conducted in 2006 found no clear evidence that one source of raters provides more valid or accurate data than the other for the Skills domain (Tsacoumis & Van Iddekinge, 2006).⁵ Consequently, considerations of relative practicality, such as cost, informed the decision to proceed with analyst ratings of both Abilities and Skills.

As part of a random sample, workers selected to participate in the O*NET Data Collection Program are asked to rate the requirements of their own jobs as defined by the O*NET questionnaire items. The responses are tabulated into statistics, such as mean ratings for each scale. Collecting information from job incumbents presents many challenges; among them is determining the best method for identifying a representative sample of job incumbents in each occupation. Sampling allows an estimate of the population. The O*NET Program is concerned, in particular, with identifying sampling approaches that minimize burden on employers and the public, achieving broad coverage of the workers in each occupation, ensuring acceptable response rates, and supporting overall cost-efficiency.

Three types of sampling frames are available for identifying samples of workers in each occupation: (1) lists of individual workers identified through professional and trade associations, licensing agencies, and unions; (2) households; and (3) employer establishments.

Identifying sampling frames of workers through professional and trade associations and unions retains the advantage of lower response burden because contacts with a sample of employers are replaced with contact with one or a few associations. Although it adds the cost of soliciting and maintaining association cooperation, this procedure also removes the cost of soliciting and maintaining employer cooperation. However, a major limitation of using special sampling frames of professional and trade associations and unions is coverage: rarely does association membership encompass a broad coverage of employment in the occupation. In addition, the membership of many associations consists of people in multiple occupations, retired individuals, and other interested parties. Moreover, few associations keep occupational information on their membership, causing the identification of job incumbents in a specific occupation to be problematic. Where coverage of employment in an occupation by association membership is significant and members of the occupation in the association can be identified, a special frame can sometimes be used to supplement the use of employer establishments in a dual-frame sample design.

Persons in various occupations could also be identified and interviewed using a sample of households. The primary disadvantage of this approach is cost. Because it is impossible to know what occupations would be represented by the employed members of a household, many households would have to be sampled and screened in order to identify and interview persons in specific occupations. As an example, consider that there are approximately 68,000 veterinarians in the United States and about 126 million households. This means that more than 1,700 households, on average, must be screened to find just one veterinarian. Some occupations are much rarer and would require screening thousands of households to locate and interview the required number for the O*NET survey. The cost of this approach would be prohibitive, and other, more economical options are available.

The Establishment Method (using employer establishments to identify occupational samples, as described further in Section A.1.4) provides the advantages of lower response burden and cost than a household survey, as well as good coverage for the large majority of occupations. Response burden and costs are lower for two primary reasons: (1) there are more workers per employer than per household, so fewer contacts are required to identify workers; and (2) employer contacts can be minimized by focusing on those most likely to employ workers in each occupation for which the sample is required. Efficient sample design is possible because the distribution of employment in an occupation is usually a function of the industry of the employer. Employment by occupation by industry is measured by the federal-state Occupational Employment Statistics (OES) program national estimates provided by BLS.

The Establishment Method provides good coverage of wage and salary employment so long as an acceptable employer sampling frame is available. Coverage of self-employment is more difficult, although the sampling frame used in O*NET surveys includes many establishments operated by self-employed workers.

The Establishment Method, pretested in 1999 and 2000,⁶ remains the primary way to update the O*NET database; most data are currently collected this way. Achieving high response rates with the Establishment Method can be challenging, however, because the method requires cooperation at two levels: the employer and the sampled worker. Nonetheless, acceptable levels of cooperation have been attained to date, and this method has proved successful. Although the resulting response rates have been acceptable, the O*NET team continually works to enhance response rates.⁷

An alternative method for collecting occupational information, involving occupation experts, is used to optimize the use of burden hours and resources because some occupations are difficult to sample efficiently. This situation occurs when it is difficult to locate industries or establishments with occupation incumbents; when employment is low; or when employment data are not available, as is the case for many new and emerging occupations. With the Occupation Expert (OE) Method, persons considered experts in the target occupation are surveyed. These experts include supervisors and trainers, as well as experienced job incumbents. The limitation of the OE Method is that locating experts can be difficult. For some occupations, identifying a professional association proves difficult; in other cases, the association may lack membership information sufficient to identify experts for a specific occupation.

Using the most appropriate sources of information (e.g., workers, occupation experts, analysts) and a multiple-method approach, the O*NET Data Collection Program efficiently collects and yields high-quality occupational data.

Summary of the O*NET Data Collection Process

The O*NET data collection process is broadly summarized here and detailed later in separate sections of this Supporting Statement.

Sample Design

The O*NET Program sampling approaches are designed to create and update the O*NET database in a highly cost-efficient and timely manner while maximizing the reliability of the information. The primary method for collecting this information is the Establishment Method, a survey of workers employed in a national probability sample of establishments. Data collection for approximately 75% of occupations is completed by the Establishment Method. The method uses a stratified two-stage design. At the first stage, a sample of businesses is selected from a national database, provided by Dun & Bradstreet (D&B), of nearly 17 million establishments. The sample is selected with probability proportional to the expected number of employed workers in the specific occupations being surveyed. At the second stage, a sample of workers is selected in the occupations within the sampled businesses.

For selected occupations that are difficult to complete and for which additional observations are required, a special frame, such as a professional or trade association membership list, is sometimes used to supplement the D&B sample. The sample selection procedures vary across associations, depending on the type of information available on association members. In general, association lists are sampled with a single-stage, stratified, simple random sampling approach. Stratification by geographic location and occupation subspecialty is considered if it is appropriate for the occupation.⁸

The OE Method is considered for use when the Establishment Method would likely be problematic because the target occupations have very low employment rates, are new or emerging, lack industry employment data, or are populated by incumbents in remote or difficult-to-access locations. The OE Method can be used only if the occupation is well represented by one or more professional or trade associations that are willing and able to identify experts in the target occupation. For this method, stratified samples of experts are selected from lists of potential respondents. These potential experts are questioned to determine whether they meet the program-specified criteria to serve as occupation experts for their respective occupations. Data collection for approximately 25% of the O*NET-SOC occupations is completed by the OE Method.⁹

Data Collection

Data collection operations are conducted by RTI International at its Operations Center in Raleigh, North Carolina, and at its Survey Support Department, also located in Raleigh. For the Establishment Method, the Operations Center’s Business Liaisons (BLs) contact sampled business establishments, secure the participation of points of contact (POCs), and work with the POCs to carry out data collection in the target occupations. The data are provided by randomly selected employees in the occupations of interest. All within-establishment data collection is coordinated by the POCs; the BLs do not contact employees directly. After a POC agrees to participate, informational materials and questionnaires are mailed to the POC, who distributes the questionnaires to the sampled employees.

As noted above, for difficult-to-complete occupations, the D&B sample may be supplemented with a sample of workers selected from a professional or trade association membership list. Similarly, when the OE Method is used, occupation experts are also selected from professional or trade association lists. In both situations, the workers or occupation experts are contacted directly by the BLs, without involvement of a sampled establishment or a POC.

Survey support staff mail materials to POCs, job incumbents, and occupation experts, and they receive and process completed questionnaires returned by respondents. Both the telephone operations of the BLs and the mailing and questionnaire-receipt operations of the survey support staff are supported by a Case Management System (CMS). Data-entry software supports the keying and verification of incoming survey data.

Three domain questionnaires are used to collect data from sampled workers: Knowledge (including Work Styles and Education and Training), Generalized Work Activities, and Work Context. Each sampled worker is randomly assigned one of the three questionnaires. The workers are also asked to provide basic demographic information and to complete a brief task inventory for their specific occupations. By contrast, the occupation experts are asked to complete all three domain questionnaires, as well as basic demographic questions and a task inventory for the occupation of interest.

Workers may complete the paper questionnaire and return it by mail, or they may choose to complete the questionnaire online at the project Web site. Occupation experts have the same options for completing their questionnaires. Questionnaires are available in Spanish for selected O*NET-SOC occupations. Data for two domains, Abilities and Skills, are provided by trained analysts because of the more abstract nature of the questions.^¹⁰

Data Cleaning; Identification and Evaluation of Anomalous Cases

Data cleaning procedures eliminate completely blank questionnaires and insert consistent analysis codes for legitimate skips, blank items, and invalid responses. Anomalous cases are identified so respondents may be removed if their responses either suggest that they are not working in the occupation of interest or are highly inconsistent with those of the others responding for the occupation. Unusable cases are identified according to prescribed eligibility criteria, such as percentage of items completed. Cases with certain questionable characteristics are flagged for further analysis. These include cases with response patterns deviating from those of other cases in the occupation and cases with write-in job titles that do not appear to match the occupation. Responses judged invalid by expert reviewers are excluded from the analysis file.¹¹

Weighting and Estimation

Estimates generated from O*NET survey data are computed with sampling weights that compensate for the unequal probabilities of selecting establishments, occupations within establishments, and employees within each selected occupation. In addition, these base weights are adjusted to further compensate for multiple subwaves of sampling, sample adjustment, population under- and overcoverage caused by frame imperfections, and nonresponse at both the establishment and the employee levels.

These weight adjustments can lead to weights that are very large or very small compared with the weights for other sample units. Such weight variability may increase the standard error estimates. When the variation in the weights is large, it is desirable to trim the weights to reduce the variation. For the O*NET estimates, the weighting process involves a weight trimming procedure in which extremely large or small weights are truncated to fall within a specified range. Although trimming weights can introduce bias in the estimates, the variance reduction it achieves usually offsets the potential bias, resulting in estimates with smaller net mean squared errors.

Based on a pooled sample of all completed waves, final estimates are produced. Estimates are computed by summing the weighted observations and dividing by the sum of the weights. Standard errors are estimated with the first-order Taylor series approximation of deviations of estimates from their expected values. These design-based variance estimates are computed with SUDAAN^® software (RTI International, 2013). These estimates properly account for the combined effects of clustering, stratification, and unequal weighting—all of which are present in the O*NET data. In addition, estimates with questionable precision are flagged “recommended for suppression” in the O*NET database.¹²

Nonresponse Analysis

Nonresponse is analyzed at multiple levels. Establishment-level nonresponse in the O*NET Data Collection Program can occur at the verification, screening, recruiting, and sampling stages of selection. Employee-level nonresponse occurs when a selected employee fails to complete and return a questionnaire. Item-level nonresponse occurs when an employee who returns a questionnaire skips one or more items in the questionnaire.

Nonresponse analyses are conducted annually as part of the processing of the data in each analysis cycle. Respondents and nonrespondents at the establishment and employee levels are compared across a variety of common attributes to determine the representativeness of the net sample. Generally, few significant differences are found, indicating a low potential for nonrespondent bias. Even when significant differences are found, the potential bias due to nonresponse is unlikely to diminish the utility of the O*NET estimates of occupational characteristics because these same characteristics are used to adjust the analysis weights to compensate for the bias. Moreover, item response rates are high, generally higher than 90% for Likert-scale items, with most exceeding 95%, so the risk of erroneous inferences due to item nonresponse is low.¹³

Interrater Reliability and Agreement

For each O*NET-SOC occupation, the degree of interrater reliability (the co-variation among ratings) and the level of interrater agreement (the absolute difference among ratings) are calculated annually for the data in each analysis cycle. The results of the analyses are used to examine the potential sources of variability across respondents in a specific occupation. As part of a continuous improvement process, these results also inform an evaluation of the O*NET-SOC occupational taxonomy, Content Model descriptors, and scales.¹⁴

A.1.5 Summary of Response Rate Experience to Date

Establishment Method

Data collection for the O*NET Data Collection Program began in June 2001 and has been in continuous operation since then. Exhibit A-3 shows our cumulative response rate experience as of December 31, 2017. As indicated, 180,153 establishments and 213,603 employees have responded to the survey request, resulting in an establishment response rate of 75% and an employee response rate of 64%.

Exhibit A-3. Establishment Method Data Collection Results

Sampled establishments	308,642
Eligible establishments	241,346
Participating establishments	180,153
Establishment response rate (participating establishments/eligible establishments)	75%
Sampled employees	333,566
Participating employees	213,603
Employee response rate (participating employees/eligible employees)	64%

Comparisons of the O*NET response rates with those of other business surveys are complicated by several unusual design characteristics of the O*NET survey, including the following:

Voluntary rather than mandatory participation. The literature indicates that response rates on mandatory surveys are typically higher than those on comparable voluntary surveys (Navarro, King, & Starsinic, 2011; Tulp, Hoy, Kusch, & Cole, 1991; Worden & Hoy, 1992). Because the O*NET survey is voluntary, its response rates would be expected to be lower than those for the average federally mandated survey.
No direct personal contact with the survey respondents by the survey organization conducting the data collection. The O*NET requirement of respondent anonymity means that participation at the employee level relies exclusively on the interactions between the POC and the employee. The survey organization is not able to speak to the employee to respond to questions, motivate responses, or follow up on noncompliance. In their review of establishment mail survey response rates, Paxon, Dillman, and Tarnai (1995) found that establishment surveys featuring anonymous mailings typically have lower response rates (by as many as 30 percentage points) than surveys featuring direct personal contact with the respondents.
Participation required at three stages of response—establishment level, point of contact level, and employee level. The typical establishment survey requires participation at only one or two levels: the establishment level and, in some cases, the POC level. By contrast, three often distinct entities must agree to participate in the O*NET Program: the establishment administration, the POC identified in the screening interview, and the employee who is asked to complete the questionnaire. Because very few surveys incorporate such a design, the survey methods literature is essentially devoid of examples on which to base a reasonable response rate expectation for the O*NET Data Collection Program. However, it is possible to compare O*NET response rates at each stage with other establishment surveys that incorporate these stages either separately or in combination. For example, the O*NET establishment-level response rate can be compared with other mail establishment surveys having only one response stage at the establishment level. In addition, the O*NET employee-level response rate can be compared with the response rate of other establishments’ self-conducted employee surveys.

The literature indicates that voluntary business surveys typically experience relatively low response rates. Cycyota and Harrison (2006) analyzed response rate data from 231 surveys of business executives conducted from 1992 to 2003 and found the overall average rate to be 32%. Tarnai & Paxon (2004) obtained a response rate of 48.6% in their survey of 2,626 businesses on survey mode preference; they concluded that the typical establishment survey achieves a response rate of between 40% and 50%. Baruch and Holtom (2008) analyzed the response rates for 463 studies published across 17 first- and second-tier journals in 2000 and 2005. Among the studies examined, 117 of them were organization-level surveys whose average response rate was 35.7%.

In a more recent review of four major, voluntary, establishment-based surveys conducted by BLS, initial unweighted data collection response rates (defined as the percentage of sampled establishments that agreed to provide any of the requested data) ranged from 66% to 87% (Petroni, Sigman, Willimack, Cohen, & Tucker, 2004). Additionally, the Census Bureau’s Monthly Retail Sales Survey and Monthly Wholesale Survey, both of which are voluntary mail surveys with a telephone follow-up, show cooperation rates ranging between 66% and 80% (W. Davie, personal communication, March 23, 2011). The 2001 Survey of Respirator Use in Private Sector Firms, a voluntary mail survey of 40,002 establishments sponsored by the National Institute of Occupational Safety and Health, used a two-tiered sampling process similar to the one used on O*NET and achieved a business-level response rate of 75.5% (U.S. Department of Labor, Bureau of Labor Statistics, and U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health, 2003).

The above results for establishment surveys, summarized in Exhibit A-4, suggest that the O*NET establishment response rate of 75% is comparable to those reported for similar surveys.

The O*NET employee response rates can be compared with those of surveys that directly sample employees within an establishment. Because federally sponsored surveys of employees within organizations are rare, the literature on their response rates is sparse. Most surveys of this type are employee satisfaction surveys. For example, one well-documented, government-sponsored survey of employees is the Public Service Employee Survey, administered to about 258,000 employees of the Public Service of Canada and conducted by Statistics Canada in 2008

Exhibit A-4. Studies of Establishment-Level Response Rates

Study	Response Rate
Cycyota & Harrison (2006)	32%
Tarnai & Paxton (2004)	49%
Baruch & Holtom (2008)	36%
Petroni et al. (2004)	66%–87%
W. Davie (2011)	66%–80%
National Institute of Occupational Safety and Health (2003)	76%
*ONET**	75%

(Treasury Board of Canada Secretariat, 2009). A questionnaire was delivered to each employee by a government agent who personally requested that the employee complete the questionnaire and return it by mail. Multiple follow-ups of nonrespondents were made by e-mail and interoffice mail to maximize the response rate. No incentive was used, however; because all sample members were also employees of the organization conducting the survey and could fill out the survey on government time, the use of an incentive was thought to be unnecessary. The final overall response rate for the survey was 66%.

The Office of Personnel Management (OPM) developed the Organizational Assessment Survey (OAS) and has encouraged all federal agencies to survey their employees in order to evaluate organizational performance, benchmark best practices, and align performance with important and measurable outcomes. The experience OPM has had in implementing these surveys in numerous federal agencies provides some evidence of response rates for employee surveys conducted by the U.S. government. The OAS design closely resembles that of the Canadian Public Service Employee Survey. The surveys are self-administered and are conducted by each agency for its own employees. Furthermore, like the Public Service Employee Survey, the OAS request to participate is personalized and made directly to the employee by his or her employer. The features of the design offer a significant advantage over the O*NET survey design, as previously noted.

Although the results of the OAS surveys are not publicly available, an official at OPM was able to provide some general information regarding OAS response rates (C. Simons, personal communication, March 21, 2002). According to OPM, response rates for OAS surveys vary considerably by agency, from 30% to 80%. However, the average response across all agencies is approximately 57%.

The Employee Viewpoint Survey, an annual census administered to hundreds of thousands of full-time federal employees across U.S. government agencies, has yielded response rates ranging from 46% to 50% during the period 2011–2016 (U.S. Office of Personnel Management, 2016).

Additionally, Anseel, Lievens, Schollaert, and Choragwicka (2010), who conducted a meta-analysis of 2,037 surveys in the field of industrial and organizational psychology, management, and marketing from 1995 through 2008, found that surveys implementing many of the response-rate-enhancing features of the O*NET survey had an average response rate of 52%. In a study examining implications of employees’ mode preference in completing a survey, Cole, Bedeian, and Feild (2006) sampled 8,598 employees across 50 countries who worked in the manufacturing industry and who varied in age, job tenure, and job functions. The study offered targeted respondents the same survey mode options as are given in O*NET: a paper-and-pencil version or a Web-based option. The overall response rate was 57%.

The above results for employee surveys are summarized in Exhibit A-5. Again, the data indicate that the O*NET employee response rate of 64% is comparable to that of similar surveys.

Exhibit A-5. Studies of Employee-Level Response Rates

Study	Response Rate
Treasury Board of Canada Secretariat (2009)	66%
OPM (C. Simons, personal communication, 2002)	57%
OPM (2016)	46%–50%
Anseel et al. (2010)	52%
Cole et al. (2006)	57%
O*NET	64%

Occupation Expert Method

The OE Method is a much smaller but still important component of the O*NET Data Collection Program protocol. Exhibit A-6 shows our response rate experience with this method as of December 31, 2017. As indicated, 11,184 of 15,134 eligible OEs have participated, for a response rate of 74%. This response rate is higher than the employee response rate for the Establishment Method, likely because of factors such as personalized direct contact, greater total incentive, and generally higher education levels of the eligible population.

Exhibit A-6. Occupation Expert Method Data Collection Results

Sampled occupation experts	20,247
Eligible occupation experts	15,134
Participating occupation experts	11,184
Occupation expert response rate (participating occupation experts/eligible occupation experts)	74%

A.1.6 Statutory and Regulatory Information

Although the O*NET name is not referenced specifically in statutes, it is cited twice in the U.S. Code of Federal Regulations (C.F.R.), and the DOT—which O*NET largely replaced—is cited 23 times in the C.F.R. (the DOT was formerly cited in the statutes, but there are no current statutory references to it). Furthermore, O*NET information is the foundational common language for fulfilling the statutory and regulatory responsibilities regarding information on skills discussed in the Workforce Innovation and Opportunity Act (WIOA) legislation and regulations.

Section 308 of the WIOA (Public Law 113-128) amended section 15 of the Wagner-Peyser Act to require the Secretary of Labor to oversee the “development, maintenance, and continuous improvement of a nationwide workforce and labor market information system,” which shall include, among other components, “skill trends by occupation and industry.” (See 29 U.S.C. 491-1.) The O*NET program is the primary means for collecting skills information across all occupations in the economy. Updating the entire O*NET database is a critical component of the nationwide workforce and labor market information system to support employer, workforce, and education information needs.

The WIOA contains numerous references that identify the need for information on the skill requirements of jobs. The word “skill” is used alone more than 120 times, in addition to 3 references to “knowledge, skills, and abilities” and 2 to “knowledge, skills, and competencies.” For example, Section 102 requires the Unified State Plan to include an economic analysis of “(i) existing and emerging in-demand industry sectors and occupations and (ii) the employment needs of employers, including a description of the knowledge, skills, and abilities, needed in those industries and occupations.” (See 29 U.S.C. § 3112.) Section 134 requires the provision of “information on job skills necessary” and on “skill requirements” for obtaining jobs listed for the local occupations in demand. (See 28 U.S.C. 3174 § (c)(2)(A)(vi)(ii)).

Other WIOA references address the need for information on the skills of individuals. For example, Section 134 provides for assessment services to identify “the skill levels and service needs of adults and dislocated workers.” Section 129 allows funds to be used to provide youth with an assessment that “shall include a review of basic skills, occupational skills, prior work experience, employability, interests, [and] aptitudes.” (See 29 U.S.C. § 3164(c)(1)(A)). The O*NET Career Exploration Tools, including the O*NET Interest Profiler and O*NET Work Importance Locator, are such assessment tools, designed specifically to relate a person’s interests and work values to the information on education and skill requirements for occupations in the O*NET database.

WIOA Section 303, on the Public Labor Exchange Services System, amends the Wagner-Peyser Act to add “The Secretary, in consultation with States, is authorized to assist the States in the development of national electronic tools that may be used to improve access to workforce information for individuals….” The suite of O*NET Web sites (such as O*NET OnLine and My Next Move) and O*NET Web Services used to disseminate O*NET occupational information and related workforce and labor market information are such national electronic tools designed to improve access to information for individuals.

The WIOA regulations also include definitions for O*NET and for O*NET-SOC, the taxonomy used for classifying O*NET data:

PART 651—GENERAL PROVISIONS GOVERNING THE WAGNER-PEYSER ACT EMPLOYMENT SERVICE Sec. 651.10 Definitions of terms used in this part and parts 652, 653, 654, and 658 of this chapter. Authority: 29 U.S.C. 49a; 38 U.S.C. part III, 4101, 4211; Secs. 503, 3, 189, Pub. L

§ 651.10 Definitions of terms used in this part and parts 652, 653, 654, and 658 of this chapter.

Occupational Information Network (O*NET) system means the online reference database which contains detailed descriptions of U.S. occupations, distinguishing characteristics, classification codes, and information on tasks, knowledge, skills, abilities, and work activities as well as information on interests, work styles, and work values.

O*NET–SOC means the occupational codes and titles used in the O*NET system, based on and grounded in the Standard Occupational Classification (SOC), which are the titles and codes by Federal statistical agencies to classify workers into occupational categories for the purpose of collecting, calculating, and disseminating data. The SOC system is issued by the Office of Management and Budget and the Department of Labor is authorized to develop additional detailed O*NET occupations within existing SOC categories. The Department uses O*NET–SOC titles and codes for the purposes of collecting descriptive occupational information and for State reporting of data on training, credential attainment, and placement in employment by occupation.

Finally, the predecessor to the O*NET database, the DOT, is frequently cited as a source of occupational information in support of federal programs. The 23 citations of the DOT in the C.F.R. include references to determining disability; administering DOL programs; and administering immigration, civil rights, and labor standards law. DOL officials responsible for the O*NET Program work with various federal users of the DOT, some of whom have made the transition either in regulatory changes or in practices and procedures, while others are exploring potential transition to O*NET information or uses of O*NET data. These include State Department officials responsible for visas, the U.S. Office of Apprenticeship, the Office of Foreign Labor Certification, and the Social Security Administration.

The specific O*NET and DOT citations in the C.F.R. appear in Exhibit A-7.

Exhibit A-7. O*NET Citations in Code of Federal Regulations

1	20 CFR § 651.10 - Definitions of terms used in parts 651-658. [PDF 101 KB] Code of Federal Regulations (annual edition). Title 20: Employees' Benefits. Part 651: GENERAL PROVISIONS GOVERNING THE FEDERAL-STATE EMPLOYMENT SERVICE SYSTEM. Friday, April 1, 2016. ... seasonal farmworker. Occupational Information Network (O*NET) means the online reference database which... More Information Historical editions have been hidden from results. Show all editions.
2	34 CFR § 600.2 - Definitions. [PDF 96 KB] Code of Federal Regulations (annual edition). Title 34: Education. Subpart A: General. Friday, July 1, 2016. ... Office of Management and Budget (OMB) or an Occupational Information Network O*NET-SOC code established... More Information
3	41 CFR § 60-3.15 - Documentation of impact and validity evidence. [PDF 112 KB] Code of Federal Regulations (annual edition). Title 41: Public Contracts and Property Management. Subjgrp: Documentation of Impact and Validity Evidence. Friday, July 1, 2016. ... 41 Public Contracts and Property Management 1 2016-07-01 2016-07-01 false Documentation of impact and validity evidence. Â§ 60-3.15 Section Â§ 60-3.15 Public Contracts and Property Management Other Provisions Relating to Public Contracts OFFICE OF FEDERAL CONTRACT COMPLIANCE PROGRAMS, EQUAL EMPLOYMENT OPPORTUNITY, DEPARTMENT OF LABOR 3-UNIFORM... More Information
4	29 CFR § 1607.15 - Documentation of impact and validity evidence. [PDF 109 KB] Code of Federal Regulations (annual edition). Title 29: Labor. Subjgrp: Documentation of Impact and Validity Evidence. Friday, July 1, 2016. ... 29 Labor 4 2016-07-01 2016-07-01 false Documentation of impact and validity evidence. Â§ 1607.15 Section Â§ 1607.15 Labor Regulations Relating to Labor (Continued) EQUAL EMPLOYMENT OPPORTUNITY COMMISSION UNIFORM GUIDELINES ON EMPLOYEE SELECTION PROCEDURES (1978) Documentation of Impact and Validity Evidence § 1607.15 Documentation of impact and... More Information
5	20 CFR § 220.13 - Establishment of permanent disability for work in regular railroad occupation. [PDF 90 KB] Code of Federal Regulations (annual edition). Title 20: Employees' Benefits. Subpart C: Disability Under the Railroad Retirement Act for Work in an Employee's Regular Railroad Occupation. Saturday, April 1, 2017. ... 20 Employees' Benefits 1 2017-04-01 2017-04-01 false Establishment of permanent disability for work in regular railroad occupation. Â§ 220.13 Section Â§ 220.13 Employees' Benefits RAILROAD RETIREMENT BOARD REGULATIONS UNDER THE RAILROAD RETIREMENT ACT DETERMINING DISABILITY Under the Railroad Retirement Act for Work in an Employee's... More Information

(continued)

Exhibit A-7. O*NET Citations in Code of Federal Regulations (continued)

6	20 CFR § 220.135 - Exertional and nonexertional limitations. [PDF 90 KB] Code of Federal Regulations (annual edition). Title 20: Employees' Benefits. Subpart K: Vocational Considerations. Saturday, April 1, 2017. ... 20 Employees' Benefits 1 2017-04-01 2017-04-01 false Exertional and nonexertional limitations. Â§ 220.135 Section Â§ 220.135 Employees' Benefits RAILROAD RETIREMENT BOARD REGULATIONS UNDER THE RAILROAD RETIREMENT ACT DETERMINING DISABILITY Vocational Considerations § 220.135 Exertional and nonexertional limitations. (a) General. The claimant's... More Information
7	20 CFR § 404.1569a - Exertional and nonexertional limitations. [PDF 90 KB] Code of Federal Regulations (annual edition). Title 20: Employees' Benefits. Subjgrp: Vocational Considerations. Friday, April 1, 2016. ... 20 Employees' Benefits 2 2016-04-01 2016-04-01 false Exertional and nonexertional limitations. Â§ 404.1569a Section Â§ 404.1569a Employees' Benefits SOCIAL SECURITY ADMINISTRATION FEDERAL OLD-AGE, SURVIVORS AND DISABILITY INSURANCE (1950- ) Determining Disability and Blindness Vocational Considerations § 404.1569a Exertional and nonexertional... More Information
8	20 CFR § 416.969a - Exertional and nonexertional limitations. [PDF 90 KB] Code of Federal Regulations (annual edition). Title 20: Employees' Benefits. Subjgrp: Vocational Considerations. Friday, April 1, 2016. ... 20 Employees' Benefits 2 2016-04-01 2016-04-01 false Exertional and nonexertional limitations. Â§ 416.969a Section Â§ 416.969a Employees' Benefits SOCIAL SECURITY ADMINISTRATION SUPPLEMENTAL SECURITY INCOME FOR THE AGED, BLIND, AND DISABLED Determining Disability and Blindness Vocational Considerations § 416.969a Exertional and nonexertional... More Information
9	28 CFR § 50.14 - Guidelines on employee selection procedures. [PDF 172 KB] Code of Federal Regulations (annual edition). Title 28: Judicial Administration. Part 50: STATEMENTS OF POLICY. Friday, July 1, 2016. ... 28 Judicial Administration 2 2016-07-01 2016-07-01 false Guidelines on employee selection procedures. Â§ 50.14 Section Â§ 50.14 Judicial Administration DEPARTMENT OF JUSTICE (CONTINUED) STATEMENTS OF POLICY § 50.14 Guidelines on employee selection procedures. The guidelines set forth below are intended as a statement of policy of the Department... More Information
10	20 CFR § 416.969 - Listing of Medical-Vocational Guidelines in appendix 2 of subpart P of part 404 of this chapter. [PDF 87 KB] Code of Federal Regulations (annual edition). Title 20: Employees' Benefits. Subjgrp: Vocational Considerations. Friday, April 1, 2016. ... 20 Employees' Benefits 2 2016-04-01 2016-04-01 false Listing of Medical-Vocational Guidelines in appendix 2 of subpart P of part 404 of this chapter. Â§ 416.969 Section Â§ 416.969 Employees' Benefits SOCIAL SECURITY ADMINISTRATION SUPPLEMENTAL SECURITY INCOME FOR THE AGED, BLIND, AND DISABLED Determining Disability and Blindness Vocational... More Information

(continued)

Exhibit A-7. O*NET Citations in Code of Federal Regulations (continued)

11	20 CFR § 404.1569 - Listing of Medical-Vocational Guidelines in appendix 2. [PDF 87 KB] Code of Federal Regulations (annual edition). Title 20: Employees' Benefits. Subjgrp: Vocational Considerations. Friday, April 1, 2016. ... 20 Employees' Benefits 2 2016-04-01 2016-04-01 false Listing of Medical-Vocational Guidelines in appendix 2. Â§ 404.1569 Section Â§ 404.1569 Employees' Benefits SOCIAL SECURITY ADMINISTRATION FEDERAL OLD-AGE, SURVIVORS AND DISABILITY INSURANCE (1950- ) Determining Disability and Blindness Vocational Considerations § 404.1569 Listing of... More Information
12	20 CFR Appendix 2 to Part 220 - Medical-Vocational Guidelines [PDF 113 KB] Code of Federal Regulations (annual edition). Title 20: Employees' Benefits. Part 220: DETERMINING DISABILITY. Saturday, April 1, 2017. ... 20 Employees' Benefits 1 2017-04-01 2017-04-01 false Medical-Vocational Guidelines 2 Appendix 2 to Part 220 Employees' Benefits RAILROAD RETIREMENT BOARD REGULATIONS UNDER THE RAILROAD RETIREMENT ACT DETERMINING DISABILITY Pt. 220, App. 2 Appendix 2 to Part 220—Medical-Vocational Guidelines Sec. 200.00 Introduction. 201.00 Maximum sustained work... More Information
13	20 CFR Appendix 2 to Subpart P of... - Medical-Vocational Guidelines [PDF 132 KB] Code of Federal Regulations (annual edition). Title 20: Employees' Benefits. Subpart P: Determining Disability and Blindness. Friday, April 1, 2016. ... 20 Employees' Benefits 2 2016-04-01 2016-04-01 false Medical-Vocational Guidelines 2 Appendix 2 to Subpart P of Part 404 Employees' Benefits SOCIAL SECURITY ADMINISTRATION FEDERAL OLD-AGE, SURVIVORS AND DISABILITY INSURANCE (1950- ) Determining Disability and Blindness Pt. 404, Subpt. P, App. 2 Appendix 2 to Subpart P of Part 404... More Information
14	20 CFR § 220.134 - Medical-vocational guidelines in appendix 2 of this part. [PDF 87 KB] Code of Federal Regulations (annual edition). Title 20: Employees' Benefits. Subpart K: Vocational Considerations. Saturday, April 1, 2017. ... 20 Employees' Benefits 1 2017-04-01 2017-04-01 false Medical-vocational guidelines in appendix 2 of this part. Â§ 220.134 Section Â§ 220.134 Employees' Benefits RAILROAD RETIREMENT BOARD REGULATIONS UNDER THE RAILROAD RETIREMENT ACT DETERMINING DISABILITY Vocational Considerations § 220.134 Medical-vocational guidelines in appendix 2 of this par... More Information
15	29 CFR § 553.30 - Occasional or sporadic employment-section 7(p)(2). [PDF 94 KB] Code of Federal Regulations (annual edition). Title 29: Labor. Subjgrp: Other Exemptions. Friday, July 1, 2016. ... 29 Labor 3 2016-07-01 2016-07-01 false Occasional or sporadic employment-section 7(p)(2). Â§ 553.30 Section Â§ 553.30 Labor Regulations Relating to Labor (Continued) WAGE AND HOUR DIVISION, DEPARTMENT OF LABOR REGULATIONS APPLICATION OF THE FAIR LABOR STANDARDS ACT TO EMPLOYEES OF STATE AND LOCAL GOVERNMENTS General Other Exemptions § 553.30...More Information

(continued)

Exhibit A-7. O*NET Citations in Code of Federal Regulations (continued)

16	20 CFR § 220.132 - Physical exertion requirements. [PDF 90 KB] Code of Federal Regulations (annual edition). Title 20: Employees' Benefits. Subpart K: Vocational Considerations. Saturday, April 1, 2017. ... 20 Employees' Benefits 1 2017-04-01 2017-04-01 false Physical exertion requirements. Â§ 220.132 Section Â§ 220.132 Employees' Benefits RAILROAD RETIREMENT BOARD REGULATIONS UNDER THE RAILROAD RETIREMENT ACT DETERMINING DISABILITY Vocational Considerations § 220.132 Physical exertion requirements. To determine the physical exertion requirements o... More Information
17	20 CFR § 404.1567 - Physical exertion requirements. [PDF 91 KB] Code of Federal Regulations (annual edition). Title 20: Employees' Benefits. Subjgrp: Vocational Considerations. Friday, April 1, 2016. ... 20 Employees' Benefits 2 2016-04-01 2016-04-01 false Physical exertion requirements. Â§ 404.1567 Section Â§ 404.1567 Employees' Benefits SOCIAL SECURITY ADMINISTRATION FEDERAL OLD-AGE, SURVIVORS AND DISABILITY INSURANCE (1950- ) Determining Disability and Blindness Vocational Considerations § 404.1567 Physical exertion requirements. To determine... More Information
18	20 CFR § 416.967 - Physical exertion requirements. [PDF 91 KB] Code of Federal Regulations (annual edition). Title 20: Employees' Benefits. Subjgrp: Vocational Considerations. Friday, April 1, 2016. ... 20 Employees' Benefits 2 2016-04-01 2016-04-01 false Physical exertion requirements. Â§ 416.967 Section Â§ 416.967 Employees' Benefits SOCIAL SECURITY ADMINISTRATION SUPPLEMENTAL SECURITY INCOME FOR THE AGED, BLIND, AND DISABLED Determining Disability and Blindness Vocational Considerations § 416.967 Physical exertion requirements. To determine... More Information
19	20 CFR § 655.730 - What is the process for filing a labor condition application? [PDF 97 KB] Code of Federal Regulations (annual edition). Title 20: Employees' Benefits. Subpart H: Labor Condition Applications and Requirements for Employers Seeking To Employ Nonimmigrants on H-1b Visas in Specialty Occupations and as Fashion Models, and Requirements for Employers Seeking To Employ Nonimmigrants on H-1b1 and E-3 Visas in Specialty Occupations. Friday, April 1, 2016. ... 20 Employees' Benefits 3 2016-04-01 2016-04-01 false What is the process for filing a labor condition application? Â§ 655.730 Section Â§ 655.730 Employees' Benefits EMPLOYMENT AND TRAINING ADMINISTRATION, DEPARTMENT OF LABOR TEMPORARY EMPLOYMENT OF FOREIGN WORKERS IN THE UNITED STATES Labor Condition Applications and Requirements for Employers... More Information
20	20 CFR § 404.1560 - When we will consider your vocational background. [PDF 91 KB] Code of Federal Regulations (annual edition). Title 20: Employees' Benefits. Subjgrp: Vocational Considerations. Friday, April 1, 2016. ... 20 Employees' Benefits 2 2016-04-01 2016-04-01 false When we will consider your vocational background. Â§ 404.1560 Section Â§ 404.1560 Employees' Benefits SOCIAL SECURITY ADMINISTRATION FEDERAL OLD-AGE, SURVIVORS AND DISABILITY INSURANCE (1950- ) Determining Disability and Blindness Vocational Considerations § 404.1560 When we will consider your... More Information

(continued)

Exhibit A-7. O*NET Citations in Code of Federal Regulations (continued)

21	20 CFR § 416.960 - When we will consider your vocational background. [PDF 90 KB] Code of Federal Regulations (annual edition). Title 20: Employees' Benefits. Subjgrp: Vocational Considerations. Friday, April 1, 2016. ... 20 Employees' Benefits 2 2016-04-01 2016-04-01 false When we will consider your vocational background. Â§ 416.960 Section Â§ 416.960 Employees' Benefits SOCIAL SECURITY ADMINISTRATION SUPPLEMENTAL SECURITY INCOME FOR THE AGED, BLIND, AND DISABLED Determining Disability and Blindness Vocational Considerations § 416.960 When we will consider your... More Information
22	20 CFR § 220.131 - Work which exists in the national economy. [PDF 90 KB] Code of Federal Regulations (annual edition). Title 20: Employees' Benefits. Subpart K: Vocational Considerations. Saturday, April 1, 2017. ... 20 Employees' Benefits 1 2017-04-01 2017-04-01 false Work which exists in the national economy. Â§ 220.131 Section Â§ 220.131 Employees' Benefits RAILROAD RETIREMENT BOARD REGULATIONS UNDER THE RAILROAD RETIREMENT ACT DETERMINING DISABILITY Vocational Considerations § 220.131 Work which exists in the national economy. (a) General. The Board... More Information
23	20 CFR § 404.1566 - Work which exists in the national economy. [PDF 91 KB] Code of Federal Regulations (annual edition). Title 20: Employees' Benefits. Subjgrp: Vocational Considerations. Friday, April 1, 2016. ... 20 Employees' Benefits 2 2016-04-01 2016-04-01 false Work which exists in the national economy. Â§ 404.1566 Section Â§ 404.1566 Employees' Benefits SOCIAL SECURITY ADMINISTRATION FEDERAL OLD-AGE, SURVIVORS AND DISABILITY INSURANCE (1950- ) Determining Disability and Blindness Vocational Considerations § 404.1566 Work which exists in the national... More Information Historical editions have been hidden from results. Show all editions.
24	20 CFR § 416.966 - Work which exists in the national economy. [PDF 87 KB] Code of Federal Regulations (annual edition). Title 20: Employees' Benefits. Subjgrp: Vocational Considerations. Friday, April 1, 2016. ... 20 Employees' Benefits 2 2016-04-01 2016-04-01 false Work which exists in the national economy. Â§ 416.966 Section Â§ 416.966 Employees' Benefits SOCIAL SECURITY ADMINISTRATION SUPPLEMENTAL SECURITY INCOME FOR THE AGED, BLIND, AND DISABLED Determining Disability and Blindness Vocational Considerations § 416.966 Work which exists in the national... More Information
25	29 CFR § 553.103 - “Same type of services” defined. [PDF 90 KB] Code of Federal Regulations (annual edition). Title 29: Labor. Subpart B: Volunteers. Friday, July 1, 2016. ... 29 Labor 3 2016-07-01 2016-07-01 false âSame type of servicesâ defined. Â§ 553.103 Section Â§ 553.103 Labor Regulations Relating to Labor (Continued) WAGE AND HOUR DIVISION, DEPARTMENT OF LABOR REGULATIONS APPLICATION OF THE FAIR LABOR STANDARDS ACT TO EMPLOYEES OF STATE AND LOCAL GOVERNMENTS Volunteers § 553.103 “Same type of services” defined....More Information

Note: The search for O*NET and the DOT in the most recent editions of the C.F.R. was performed on August 4, 2017, using the U.S. Government Printing Office’s Federal Digital System at www.gpo.gov/fdsys/search/home.action. Two O*NET citations (the first two rows of the table) were identified in the C.F.R., along with 23 DOT citations (the remaining rows).

Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.

The O*NET Program provides essential tools and services for numerous critical federal and state workforce investment functions. These tools also serve job seekers, businesses, and educational institutions across the country. The common language used in O*NET occupational and skill descriptions facilitates communication among the various user groups. This facilitation contributes to our nation’s talent development and promotes U.S. competitiveness in the global 21st-century economy.

Users of the O*NET database can access and use the information in various ways. The data are available to the public free of charge through the O*NET OnLine, My Next Move, and My Next Move for Veterans Web sites; through the O*NET Web Services application programming interface (API); or by downloading the database, which is done by developers who provide applications targeted to specific communities or audiences. The O*NET database is now the foundation for many programs serving the American workforce, providing information to build transferable skills, skills gap analyses, and competency profiles and to facilitate cross-functional team building. These examples are just a few of the ways the O*NET Program supports activities critical to maintaining a mobile workforce responsive to changing regional and national economic needs.

The O*NET Database, O*NET OnLine, My Next Move, O*NET Career Tools, O*NET Training Academy, and O*NET Code Connector

The O*NET Database

With the August 2017 version of the database, 966 occupations have been comprehensively updated through the O*NET Data Collection Program; 723 of these occupations have had more than one update. The O*NET database currently provides

detailed occupational and skill information for more than 966 occupations;
information on standardized descriptors of skills, abilities, interests, knowledge, work values, education, training, work context, and work styles; and
occupational coding based on the 2010 SOC.

Since September 2007, the database has been updated 10 times (Exhibit A-8).

Exhibit A-8. Database Updates

Update	Number of Occupations Updated
June 2008	108
June 2009	117
June 2010	120
July 2011	107
July 2012	108
July 2013	105
July 2014	126
August 2015	102
August 2016	116
August 2017	100

The O*NET database has been enhanced with new occupational information that provides more coverage of the occupations and will enable users to more effectively use the database. These enhancements include (1) more comprehensive detailed work activity information; (2) updated military-to-civilian occupational crosswalks; (3) the identification of technologies employers most frequently include in job postings (e.g., hot technologies); and (4) the development of new alternate titles that enable users to more easily find occupations of interest in O*NET Web applications.

O*NET OnLine

The O*NET database is provided free of charge to the public through O*NET OnLine, a Web-based application at https://www.onetonline.org/. O*NET OnLine offers users multiple ways to search for O*NET occupations and related data. O*NET OnLine includes

regularly updated occupational information;
tiered search algorithms that maximize successful results of searches by keyword (title), occupational code, or partial code;
O*NET equivalents of occupations listed in other occupational systems, such as the Military Occupational Classification or the Classification of Instructional Programs;
the ability to browse by O*NET descriptor or variable (this search enables users to make cross-occupational comparisons by viewing an occupation’s rank order on a selected knowledge, skill, ability, work activity, interest, work value, tool or technology, or task);
the ability to browse groups of similar occupations, including Bright Outlook¹⁵; career clusters; Green economy sectors; industry type; or science, technology, engineering, and mathematics (STEM) disciplines;
a sample of reported job titles for each occupational report, providing the user with a broader understanding of the O*NET-SOC;
report display options (in addition to the default Summary Report, users can choose to view a comprehensive Details Report or build a Custom Report);
wage and employment trends information (derived from BLS data) and links to Web sites of professional associations, giving users access to additional information on related specialties, industries, and education and training resources; and
occupation-specific links to training, certification, licensing and apprenticeship information, and job openings.

My Next Move

My Next Move (https://www.mynextmove.org/) is a new, easy-to-use, easy-to-read Web-based interactive tool for new job seekers, students, and other career explorers to learn more about their career options. Users can explore more than 900 different careers and see important information about them, including skills, tasks, salaries, and employment outlook. They can also look at related apprenticeships, credentials, and training, as well as search actual job openings. They can find careers by searching keywords, by browsing industries, or by using the O*NET Interest Profiler (see below). Additionally, users can browse occupations by Bright Outlook, apprenticeship availability, and amount of job preparation required.

Career reports in My Next Move feature the most important knowledge, skills, and abilities needed to perform the work, explained in language that is easy to understand. Outlook and education sections let users find salary information, job postings, and training opportunities. The visual design enables users to identify a career’s key points or to explore a career in depth.

Visitors interested in specific careers can start exploring quickly with an intuitive keyword search; more than 900 career options are only a few keystrokes away. Users looking for a broader range of opportunities can explore more than a dozen different industries; each features a range of careers to choose from, including those with a Bright Outlook for job opportunities.

Also at the site is a Web-based version of the popular O*NET Interest Profiler, a tool designed to assess an individual’s vocational interests. The Web-based version of the Interest Profiler features 60 items that are scored and, along with information about the user’s education and work experience, guide users to careers they may enjoy. More information about the O*NET Interest Profiler–Short Form can be found at https://www.onetcenter.org/IPSF.html.

My Next Move for Veterans

My Next Move for Veterans (https://www.mynextmove.org/vets/) is a Web-based interactive tool for U.S. veterans to learn more about their career options. This tool has all the functionality of My Next Move as described above.

Unique to the My Next Move site for veterans is a feature that allows veterans to search civilian occupations related to their military training. Veterans indicate their branch of service and either the military code or title of their training. They then receive a list of civilian occupations that best match their military training. Where available, the results returned will also indicate how closely their military duties match the civilian occupations, along with the minimum military rank and length of military service associated with the civilian occupation.

Mi Próximo Paso

Mi Próximo Paso (https://www.miproximopaso.org/) is a Web-based interactive tool for Spanish-speaking job seekers, students, and other career explorers to learn more about their career options. Mi Próximo Paso includes all the features of the English-language site, My Next Move.

O*NET Career Exploration Tools

The Career Exploration Tools are based on a “whole person” concept and are designed for career counseling, career planning, and career exploration. They include the Ability Profiler, the Interest Profiler, and the Work Importance Locator. They also include electronic versions of the Interest and Work Importance assessments—the Computerized Interest Profiler and the Work Importance Profiler. These electronic versions can be downloaded onto a computer system and used at no cost by schools, American Job Centers, and others. Within the My Next Move Web site, users can choose to complete a 60-item Web-based version of the Interest Profiler. Recently, a “mini” mobile-friendly version of the Interest Profiler was created and released for developers to leverage (https://www.onetcenter.org/reports/Mini-IP.html). Individuals exploring careers may use the Ability Profiler to discover what they do well, the Interest Profiler to identify the types of work they may like to perform, and the Work Importance Locator to determine which occupations will likely be satisfying according to their values and needs. The tools enable users to discover important information about themselves and use the information to explore the world of work. Workers may use these tools as aids in exploring career options, in considering career transitions, and in preparing for career change. The assessments also are designed for use by students exploring a school-to-work transition.

O*NET Academy

Training support for the O*NET system can be found on the Web at http://www.onetacademy.org. The O*NET Training Academy provides various user communities (workforce development professionals, employers, educators, students, workforce boards, job seekers, etc.) with access to O*NET support materials tailored to their needs. Through podcasts and O*NET user spotlights, customers can gain new insights into user needs and real-world applications of O*NET data. Through the O*NET Academy, O*NET users gain electronic access to recorded webinars, self-paced training courses, and best-practices tutorials on using the O*NET system and applying it on the job. The Academy site also provides access to an array of O*NET tools (e.g., O*NET OnLine, O*NET questionnaires).

O*NET Code Connector

The Code Connector (https://www.onetcodeconnector.org/) was developed to assist workforce professionals needing to code jobs. American Job Centers, other government workforce agencies, and college career services offices are the most prevalent users of the Code Connector. The Code Connector uses the O*NET database to help users determine the correct occupational code for their job orders. To access an occupation, the user may type in a keyword or select an occupational group from the home page. On subsequent pages the user is able to refine the search to select a specific occupation. The final report contains information to help determine whether the selected occupational code is the best match for the particular job order. This information includes the O*NET-SOC description, Tasks, Related Occupations, Occupation Family, and Detailed Work Activities.

O*NET Web Services

O*NET Web Services was introduced to O*NET customers in January 2014. As of 2017, there are more than 1,000 user accounts and more than 4.3 million user requests per month. O*NET Web Services (https://www.onetcenter.org/dev_web.html) is an API that provides a set of subroutine definitions, protocols, and tools that help developers display O*NET information in their applications. An intuitive screen interface and comprehensive contextual information help make the application easy to use without training and support. For developers, published APIs are available to connect vendor systems to key features of O*NET Web applications. Through O*NET Web Services, developers can integrate O*NET tools such as the following into their own Web sites or Web-enabled applications:

Keyword Search—both the My Next Move search and the OnLine occupation search are available for use in career sites. The REST Web Services API returns occupations matching a word, phrase, title, or full or partial O*NET-SOC code. The results include the code and title of each matching occupation.
My Next Move Career Reports—concise, easy-to-read overviews for over 900 occupations. APIs also provide Bright Outlook and Green information, job outlook, and more.
Summary and Details Occupation Reports—detailed information from O*NET OnLine for more than 900 occupations. User applications can include an occupation’s most important or all tasks, knowledge, skills, abilities, tools and technology, and more.
Military Search—the military transition search used in My Next Move for Veterans is also available through the Web Services API. The search returns relevant O*NET-SOC occupations based on full or partial codes and titles from the Army, Navy, Air Force, Marine Corps, and Coast Guard classification systems.
Spanish Keyword Search—the Spanish-language keyword search used in Mi Próximo Paso is part of the Web Services API. Occupation titles are returned, in Spanish, matching a Spanish word or phrase. A wide assortment of features from Mi Próximo Paso, including detailed career reports and Interest Profiler questions and scoring, is also available.
Interest Profiler—this assessment tool can be included in customer career tool sites using the IFrame Widget. After a simple block of HTML code is added, users can take the O*NET Interest Profiler without leaving their career resources Web site. For tighter integration, a REST Web Services API is offered. It provides scoring services and career results from the range of O*NET-SOC occupations. This tool is provided in both English and Spanish.

Organizations using Web Services include federal, state, and local government agencies; military services; educational institutions; assessment and career information delivery systems; public workforce investment systems; private organizations and corporations; and international users. O*NET Web Services significantly reduce the cost and effort for developers to update their applications with O*NET products and tools. With Web Services, O*NET data updates are seamlessly incorporated; thus, no new programing is required by developers. Additionally, as new features are added to O*NET Web applications, new Web Services are designed so developers can have immediate access to them and update their applications in an efficient and timely manner.

O*NET Web Site Statistics

Use of O*NET products has increased dramatically over the past few years. O*NET OnLine currently averages more than 3.4 million visits per month, 3 times as many as the reported average 3 years ago. The O*NET Resource Center (https://www.onetcenter.org) averages 2,750,000 visits per month—nearly 4 times the number of visitors from 3 years ago. The more recently developed My Next Move sites average over 1,000,000 visits per month. In addition, use of career information systems, Web site linkages, user certifications, and O*NET product downloads is widespread; by design, the primary dissemination strategy of the O*NET Program is for the private sector to build O*NET-based products that are tailored to specific audiences or user needs.

Career Information Systems

Private and governmental online career information systems using O*NET data and career tools reach millions of customers annually. For example, XAP Corporation (http://www.xap.com/), an industry leader in developing and providing students and adults with tools to explore careers and explore post-secondary education options, embeds O*NET data into its products. XAP’s tools are available in over 8,000 middle and high schools in North America. XAP also uses O*NET products and tools in its Virtual Career Network (https://vcn.org/index.php), a DOL-sponsored Web site aimed at linking workers to in-demand careers. This Web site is available to the public and is used in American Job Centers across the country. IntoCareers (http://intocareers.org), another career information systems provider, includes O*NET occupational data that provide O*NET-driven career exploration. IntoCareers powers more than 35% of the nation's state-sponsored career information systems. Nationally, career information systems programs are accessed at more than 87,000 sites by more than 37 million users a year, according to a 2009 survey done by the National Career Development Association. The O*NET Program is designed to be accessible to multiple users. The O*NET Program encourages these and other service providers and product developers to create applications that deliver O*NET information to the public.

Internet Linkages

According to an exploratory search conducted in April 2017,

more than 19,100 sites link to O*NET OnLine,
more than 800 sites link to the O*NET Code Connector,
more than 5,000 sites link to the O*NET Resource Center,
more than 4,900 sites link to My Next Move,
more than 1,200 sites link to My Next Move for Veterans, and
more than 700 sites link to Mi Próximo Paso.

The number of linkages has increased almost 30% since 2014. Many different types of organizations and professionals are linked to the O*NET Web sites, including

libraries and career centers based in higher education;
higher education institutions’ schools of business, labor and industrial relations, psychology, education, and counseling;
federal, state, and local government agencies;
public libraries (especially those offering career and job search assistance programs);
career counselors, coaches, and recruiters (mostly private-sector vendors);
providers of career exploration or job search assistance (both private and public sectors);
public school systems, educational associations, and secondary schools (often recommending the O*NET database as a resource for faculty, parents, and high school juniors and seniors);
human resources management organizations;
law firms specializing in immigration law;
vocational rehabilitation or occupational medicine and health centers; and
international sites in Turkey, Canada, the United Kingdom, Australia, New Zealand, Holland, Japan, Bangladesh, and elsewhere.

User Certifications

When individuals or organizations download the O*NET database or intend to use all, some, or even part of one of the O*NET Career Exploration Tools to develop value-added products, they are asked to voluntarily register their use by completing a certification form. As of March 31, 2017, the certification database holds 4,663 entries. Because registration is voluntary, this number likely reflects only a portion of those actually using O*NET products for their own applications. An overview of those user groups that have submitted certification forms for their use of O*NET products appears as Exhibit A-9.

Exhibit A-9. Main Organization Types Submitting O*NET Certifications

Organization Type	Percentage of Registrations
Education services	23
Government/public administration	10
Computer systems design; programming services	9
Employment services	8
Human resources and executive search consulting	4
Internet publishing	4
Vocational rehabilitation services	4
Health care	3
Software publishers	3
Individual and family services	2
Research and development, social services, and the humanities	2
Legal services	1
Military	1
Temporary health services	1
Other	25
Total	100

O*NET Product Downloads

From January 2002 through March 2017, downloads of O*NET products totaled 1,494,229 (Exhibit A-10). The use of O*NET products and tools continues to increase. The O*NET Program, through continuous improvement efforts based on user needs and advancing technology, works to efficiently develop products that meet customer demands in both the public and private sectors.

Exhibit A-10. O*NET Product Downloads

Product	Number of Downloads
Database		174,373
Career Exploration Tools
Ability Profiler	270,429
Interest Profiler	403,373
Work Importance Locator	181,251
Computerized Interest Profiler, Work Importance Profiler software	243,103
Total Career Exploration Tools		1,098,156
Other (e.g., Toolkit for Business)		221,700
*Total ONET Products**		1,494,229

Examples of O*NET Data and Products in Use

The O*NET Program provides comprehensive, up-to-date occupational information used directly by the public through access to the O*NET Web sites, as well as indirectly through value-added products delivered by both governmental and private developers. O*NET-based products benefit the public through product and service development by

federal and state government agencies,
public workforce investment systems and workforce boards,
assessment and career information systems
educational and research institutions,
U.S. armed forces,
private companies and commercial product developers, and
international users.

The following recent examples of O*NET uses are taken from the publication O*NET Products at Work (https://www.onetcenter.org/paw.html).

Federal and State Government Agencies

Federal and state agencies are using O*NET products to achieve their goals of service to employers and the public. Business development specialists, human resources personnel, and others rely on the common language of O*NET products to build connections required for a strong workforce. Provided below and in Exhibit A-11 are examples of how O*NET products are being used by federal and state agencies.

The United States Department of Labor (DOL) integrates O*NET data in its online tools to assist individuals and businesses toward a variety of career and workforce development objectives:

CareerOneStop (http://www.careeronestop.org) is an online resource for assistance in career exploration and preparation; job searches; talent acquisition, development, and retention; and disaster recovery assistance relating to employment. Its career exploration interface uses the O*NET occupational taxonomy, data, and assessment tools to match users’ interests, skills, experience, and work values to jobs.

CareerOneStop Toolkit (https://www.careeronestop.org/Toolkit/toolkit.aspx), incorporates O*NET knowledge, skills, abilities, and task data in the occupation profiles presented in its enhanced job search tool.
Job Description Writer (http://www.careeronestop.org/businesscenter/jdw/gettingstarted.aspx) supplies eight categories of occupation-specific O*NET data, which the user may customize in building a functional job description.
Competency Model Clearinghouse (http://www.careeronestop.org/competencymodel/) provides two interactive online tools: Build a Competency Model and Build a Career Ladder/Lattice. Both incorporate O*NET occupations’ titles, tasks, vocational preparation levels, and Job Zones at the models’ highest levels of specificity.

The U. S. Department of Commerce, Economics and Statistics Administration, used data from the O*NET database for a report on the growing importance of data in the economy. The report identifies occupations where data analysis and processing are central to the work performed, and it measures the size of employment and earnings in these occupations, as well as in the industries that have the highest concentration of these data occupations (http://www.esa.doc.gov/reports/importance-data-occupations-us-economy).

The California CareerZone (www.cacareerzone.org) includes O*NET assessment tools and the O*NET occupational taxonomy to assist students contemplating college or a career. Users assess their interests, skills, and work values with the O*NET Interest Profiler, Skills Search, and Work Importance Profiler. They explore and compare occupations by browsing O*NET Job Families and selecting occupations to compare on Job Zone and primary interest area, as well as on salary, projected growth, and possible college majors. The California CareerZone, part of the California Career Resource Network, is linked from the sites of American Job Centers throughout the state.

Discover Arkansas, the state’s LMI delivery system, uses O*NET data in its skills-matching program to match job seekers to occupations in the state’s growth industries. The system provides O*NET skills, work activities, tasks, work values, and interests within occupational profiles linked to targeted jobs in the state’s 10 local workforce investment areas. Within each area, a job seeker begins by selecting an industry and education career cluster, an occupational group or “pathway,” and then a specific occupation. The occupation page furnishes the top O*NET skills, tasks, work activities, work values, and interests, as well as alternate job titles and the Job Zone-based level of preparation associated with the occupation. With the aid of O*NET data, the Discover Arkansas system enables job seekers to match their education levels to employers in their chosen geographic areas and industries and to determine the worker and job characteristics associated with jobs offered by those employers (http://discoverarkansas.net/).

Exhibit A-11. Federal and State Government Users

Organization	Description and URL
Social Security Administration	Uses O*NET information (e.g., cognitive descriptors, tasks, lay titles, technology skills, tools) to develop disability determination procedures: https://www.ssa.gov/disability/step4and5.htm#&a0=2
Connecticut Department of Labor	Used O*NET information to investigate skills requirements of current and future jobs in the state: https://www.doleta.gov/programs/ONET/ct.cfm
Oklahoma Career Connection Center	O*NET Job titles and skills were used in a survey to determine employer workforce requirements: https://www.doleta.gov/programs/ONET/okla.cfm
West Virginia Rehabilitation Center	Individuals with disabilities use O*NET assessment tools and Web sites to explore new employment opportunities: https://www.doleta.gov/programs/ONET/WVRehabCenter.cfm
Maine Department of Labor	Used ONET to attract new business to the state by matching skills requirements of the prospective jobs, as defined by ONET data, to the skills of current workers in the state: https://www.1maine.gov/labor/
U.S. Department of Labor	ETA’s Projections Managing Partnership uses ONET data to help develop state and local occupational employment projections. ONET data are also used to identify skill requirements and possible skills gaps: http://creconline.org/projects/state-lmi-improvement-through-projections-training-modernization/.

Public Workforce Investment Systems and Workforce Investment Boards

State workforce investment systems have always been among the primary users of O*NET products. Serving employers and the public through state-sponsored online career information systems and American Job Centers, O*NET products are responding to the demand for information about high-growth demand industry sectors and occupations, and they are helping to build the connections needed for a strong workforce. Below and in Exhibit A-12 are some specific examples of workforce investment systems using O*NET products.

Charlotte Works conducts a workshop for job seekers titled “Using O*NET and mySkills myFuture in your Re-employment Campaign.” The workshop helps job seekers use O*NET Online for resume development and career planning. Participants also use O*NET Career Exploration Tools to help them focus their job searches, and they use the Web site to identify transferable skills. CareerOneStop’s partner Web site, mySkills myFuture, is used to help job seekers find employment opportunities (http://www.charlotteworks.com/).

The Pennsylvania Department of Labor and Industry has used O*NET skills data to create the Job Skills Currency Calculator. The calculator finds the estimated monetary value of a job skill in given occupations. Skills are categorized into four useful groups that can aid in training and in career planning and transitions: Knowledge Areas, General Work Activities, Detailed Work Activities, and Tools & Technologies (http://www.workstats.dli.pa.gov/Products/JobSkills/Pages/default.aspx).

Exhibit A-12. Public Workforce Investment Systems and Workforce Investment Boards

Organization	Description and URL
Indiana Workforce Development	Uses the O*NET database to conduct skills requirement and skills gap analyses: http://www.in.gov/dwd/
Florida Workforce Investment Board	As part of Florida’s Transition Assistance Program, O*NET’s My Next Move for Veterans is used to help veterans learn about occupations, develop resumes, improve their interview skills, and find civilian employment opportunities: http://careersourcenortheastflorida.com/home.aspx
Fairbanks Job Center	Uses ONET in its weekly career planning workshop. Clients include students, veterans, senior citizens, and dislocated workers. ONET career exploration tools are administered, and clients use their results to find occupations to explore. The emphasis is on skills transferability for employment: http://jobs.alaska.gov/offices/Workshops/Fairbanks.pdf.
Iowa Data Dissemination Bureau	Uses O*NET education, knowledge, skills, and abilities information in occupational publications and Web reports for employers, job seekers, educators, and economic developers. The publications provide information on the top skill sets, high-demand and high-wage occupations, and education/training requirements for the fastest growing jobs: https://www.iowaworkforcedevelopment.gov/labor-market-information-division.

Assessment and Career Information Systems

Organizations routinely rely on O*NET products as the basis for their assessment and career information systems. Community colleges, university career centers, and many higher education Web sites also have integrated O*NET products into their career services to students. From not-for-profit organizations to high-powered consulting firms, specific examples provided below and in Exhibit A-13 indicate the variety of organizations using O*NET products.

The American Foundation for the Blind (AFB) uses O*NET data in CareerConnect, a free employment planning resource for people who are blind or visually impaired. Sponsored by the AFB, the program helps these individuals learn about the varied occupations available in the labor market. It also provides mentors and information about assistive technology that can help with specific work. This practical, user-friendly resource incorporates O*NET data to supply essential information for career exploration and to expand the universe of jobs for individuals with visual impairments (http://www.afb.org/Section.asp?SectionID=7).

The Pennsylvania State University has incorporated O*NET Career Exploration Tools and O*NET OnLine into its academic counseling program. Students can use their assessment results to identify occupations to pursue. Occupational choices are linked to academic choices. The system enables students to select possible careers and academic pursuits based on course preferences, education-level preferences, personal styles, or values (Igou, 2012).

The Ohio Career Information System (OCIS) is geared toward middle and high school students, college students, and adult job seekers. The Web site has tailored sections for each of these groups. OCIS takes advantage of O*NET Career Exploration Tools and occupational information. It includes Spanish translations of the O*NET Interest and Work Values assessments, special sections on new and emerging and Bright Outlook occupations, and live interviews on how occupations are going Green. O*NET products and tools help ensure that the OCIS provides accurate, current, and comprehensive information on the world of work to its users (https://portal.ocis.intocareers.org/).

Exhibit A-13. Assessment and Career Information Systems

Organization	Description and URL
Crown Financial Industries	Career Direct Complete Guidance Systems is a self-administered personality, skills, abilities, interests, and work/life values career guidance system. Because many of the system’s users have job titles with a religious element, Career Direct created a crosswalk that translates religious job titles to the closest ONET occupations. The system guides users into exploring ONET occupations aligning with their talents and personal goals: http://www.careerdirectonline.org/.
National Institutes of Health	LifeWorks enables users to explore health and medical science careers. Driven by ONET data, the system offers an array of information on more than 100 health and medical science careers. Users select occupations on the basis of their interests as assessed by the ONET Interest Profiler, along with skills they wish to acquire. They are then given a customized list of health-related careers to explore through summary occupational reports populated with O*NET information: http://nihlifeworks.org/feature/index.htm.
Temple University Center for Professional Development in Career and Technical Education	Incorporates O*NET OnLine in its courses on program planning and evaluation, curriculum development, and cooperative education. It is also used as an aid in structuring occupational competency assessment committee reviews: http://education.temple.edu/cte/career-technical-education.

Higher education institutions are among the most prevalent types of users of O*NET products for assessment and career information systems. A Web search shows O*NET products at work in most state educational systems. A few examples include AlabamaMentor.org, Arizona State University CRESMET, CaliforniaColleges.edu, Cascadia Community College (Washington), University of Central Florida, CTMentor.org (Connecticut), IllinoisMentor, Middle Tennessee State University, MississippiMentor, Normandale Community College (Minnesota), PennsylvaniaMentor, TexasMentor, and Texas State University.

Educational and Research Institutions

With the help of the occupational information contained in the O*NET database, colleges are developing educational curricula, and research organizations are conducting a broad array of research projects. Examples are presented below and in Exhibit A-14.

The National Bureau of Economic Research and Harvard University used O*NET to study changes in the task content of occupations, with a specific goal of determining how workplaces have changed in their demand for social skills. They found that workplace social skills demand has increased significantly in recent years, with implications for employment numbers and wage growth (http://sites.nationalacademies.org/cs/groups/dbassesite/documents/
webpage/dbasse_170239.pdf).

In a study titled The Hidden STEM Economy (Rothwell, 2013), the Metropolitan Policy Program at the Brookings Institution used O*NET knowledge data to redefine the STEM economy and its constituent workforce. Among the study’s key findings were that about 20% of all U.S. jobs require a high level of knowledge in at least one STEM field, and that only about 20% of all federal STEM funding supports training at the associate’s degree level and below. To define STEM fields, the researchers grouped six O*NET knowledge descriptors into four fields. The O*NET knowledge descriptors biology, chemistry, and physics were chosen to represent the science field; computers and electronics to represent the technology field; engineering and technology to represent the technology field; and mathematics to represent the mathematics field. For each occupation, O*NET descriptor-level ratings were used to arrive at an average level score per STEM field. The author concluded that greater federal support was needed for training in STEM jobs requiring an associate’s degree or less and that greater coordination was needed between workforce development and state and local education resources (https://www.brookings.edu/research/the-hidden-stem-economy).

In “Workforce Skills and the Changing Knowledge Economy in Massachusetts,” Renski and Wallace (2012) at the University of Massachusetts describe a study using O*NET data to investigate the hybrid skill sets needed in the changing knowledge economy in Massachusetts. The study, conducted jointly with the Federal Reserve Bank of Boston, uses O*NET occupational data on education, experience, training, and skill requirements. The study observes that, although the Massachusetts economy has become more technology intensive, success in growing industries like health care depends on combining technical skills with social, communication, and learning skills. With the aid of O*NET skills data, the study concludes that skills related to the acquisition, processing, and dissemination of new knowledge are essential to the Massachusetts economy’s ability to meet the changing demands of the 21st century (http://www.massbenchmarks.org/publications/issues/vol14i1/5.pdf).

Exhibit A-14. Educational and Research Institutions

Organization	Description and URL
The Research Institute of the Finnish Economy	O*NET occupations and tasks were used as the starting point of a study to determine which work tasks are most likely to be taken over by technology: https://nettsteder.regjeringen.no/fremtidensskole/files/2014/05/Computerization-and-the-Future-of-Jobs-in-Norway.pdf
Center for Employability Outcomes, Texas State Technical College	Researchers developed a Skills Engine application to help analyze and apply occupational information. They used O*NET data structures, including Detailed Work Activities (DWAs), in the development of a common language that drives the application: http://www.c4eo.org/profile-builder.
Educational Testing Service	Used O*NET resources to develop a data-driven competency model to address discrepancies between job-seeker attributes and employer needs: http://www.ets.org/research/policy_research_reports/publications/report/2013/jrkj
University of Illinois; Center for Regional Economic Competitiveness	Web tool uses O*NET knowledge areas, training, and education information in a process that allows comparison of economic data across more than 3,100 U.S. counties: http://economicdiversityinappalachia.creconline.org/
Journal of Management Education	In “Using the Department of Labor’s ‘My Next Move’ to Improve Career Preparedness,” Koys (2016) looked at treatment and control groups to determine the impact of the My Next Move tool on occupational knowledge acquisition and career preparedness status. Both variables were increased significantly by use of the tool, and the author suggests ways in which classroom instructors can use the tool to assist students with job search processes.

U.S. Armed Forces

The U.S. military has recognized the value of O*NET data and career tools in its various transition programs, recruiting activities, and human systems development projects. Presented here and in Exhibit A-15 are a few examples of the variety of O*NET products being put to work in the armed forces.

As described in Section A.2.1, My Next Move for Veterans is designed for U.S. veterans who are current job seekers. This interactive tool helps veterans learn about their career options. The site has tasks, skills, salary information, job listings, and more for over 900 different careers. Veterans can find careers through searching keywords, by browsing industries that employ different types of workers, or by discovering civilian careers that are similar to their jobs in the military. Veterans can also take advantage of the O*NET Interest Profiler, a tool that offers personalized career suggestions based on a person’s interests and level of work experience.

Nebraska Workforce Development worked with a transition assistance program at Offutt Air Force Base in which O*NET OnLine was used to help individuals leaving military service determine how their skills and military experience could relate to occupations and career possibilities in the civilian sector. A crosswalk was used to link military occupations to corresponding O*NET SOC occupations, and participants received instruction in the use of O*NET as not just a career identification and job search tool, but also a source of language to use in resume writing (https://www.doleta.gov/Programs/onet/ne-offutt.cfm).

The Army Study to Assess Risk and Resilience in Servicemembers (ARMY STARRS) used data from the 246 descriptive dimensions for worker requirements, occupational requirements, and worker characteristics reported by O*NET to derive summary dimensions of military occupations. These dimensions are used to classify military occupations in terms of their job conditions. Future Army research will use these dimensions to assess the relationship between job characteristics and soldier outcomes. This research includes studies identifying the risk and resilience factors related to service member suicides (http://starrs-ls.org/#/list/
publications).

The North Carolina Military Foundation teamed with the North Carolina Military Business Center to create a database and interactive Web site that enables businesses to link their needs to the competencies of troops exiting the military. One of the challenges faced by troops and business leaders alike is identifying the knowledge, skills, and abilities shared by military and civilian jobs. Using a keyword related to a job opening, employers are able to search for related military occupations and information on how many military personnel in these occupations are returning annually to civilian jobs. They can view additional information about these occupations, including a list of related civilian job titles. Further exploration is available through a link to the related occupations in O*NET OnLine. This Web site helps employers and transitioning military personnel come together through the common language of the O*NET system (http://www.ncmbc.us/).

The Office of the Secretary of Defense enlisted the RAND National Defense Research Institute to convene a panel of experts to provide assistance in refining the implementation of the Department of Defense human capital strategy. The goal of the strategy is to develop a foundation for military personnel management. A major component of this goal is a competency-based occupational analysis system. In the Final Report of the Panel on the Department of Defense Human Capital Strategy (Hanser et al., 2008), the panel members concluded that O*NET “has the potential to provide a framework for developing much of the common language and functionality desired in a new DoD system.”

Exhibit A-15. U.S. Armed Forces

Organization	Description and URL
Navy Manpower Analysis Center	Uses ONET data in its work to develop occupational standards that serve as a basis for training and career development. The ONET skills taxonomy is used to categorize task statements as part of the process to develop the standards: http://www.public.navy.mil/bupers-npc/organization/navmac/Pages/default3.aspx.
U.S. Department of Defense	The Armed Services Vocational Aptitude Battery (ASVAB) uses O*NET data to broaden occupational choices for nearly 600,000 ASVAB participants annually at more than 17,000 American high schools: https://www.asvabprogram.com
U.S. Navy Transition Assistance Program	As part of a program for service members transitioning to civilian life, the Navy uses O*NET's Career Assessment Tool to help counsel those wanting to make a career change that requires additional education: http://www.navy.mil/ah_online/documents/TGPS.pdf
Air Force Personnel Center	ONET is used in a program that identifies those most likely to succeed as operators of remotely piloted aircraft or sensors. ONET’s Content Model is part of the framework for selection batteries for these two positions: http://www.dtic.mil/dtic/tr/fulltext/u2/a554209.pdf.

Private Companies and Commercial Products

Private companies are using O*NET information for both in-house purposes (human resources functions such as job description writing and employee development) and for commercial product development. The number of products with “O*NET in-it” continues to grow. Some specific examples are presented below and in Exhibit A-16.

HRTMS, a developer of job information and description management software, uses the O*NET occupational taxonomy and Lay Titles database in its collaborative job description tool. A human resources professional begins the job description process by matching the job’s title to an O*NET Lay Title. The system then compiles other O*NET information (e.g., tasks, skills, technology) to develop a job description that is tailored to a company’s needs (http://www.hrtms.com/).

Profiles International (PI) is one of many assessment companies that incorporate O*NET data in customized workforce development tools for public- and private-sector use. PI’s products are designed to help businesses improve their hiring practices, reduce turnover rates and costs, and enhance workforce harmony and performance. In one tool, O*NET information is used for a job description application that is especially helpful to small and midsize companies without large human resources departments. Another PI career development application uses the O*NET Interest Profiler as part of cadre of tools that job seekers take to assess soft skills. Their results are matched to O*NET occupations and presented in a Career Compatibility Report, which displays “good fit” occupations. Users go to O*NET OnLine to obtain more information about the occupations (http://www.profilesinternational.com/).

Manpower, a worldwide provider of staffing services with nearly 1,100 offices in North America and 4,500 offices in 80 countries, provides jobs to 5 million people every year and serves more than 400,000 clients worldwide. The O*NET occupational and skills taxonomy helps Manpower match the right person to the right job. The O*NET system also offers a systematic structure that enhances Manpower’s analysis of the employment marketplace and its tracking of staffing trends. By incorporating O*NET structure into its procedures, Manpower has benefited by being able to

accurately identify the types of placements each field office makes;
locate field offices where the highest need exists;
more accurately consolidate information for various types of analysis, including marketing analysis; and
begin the process of having Manpower offices in other countries map their occupations to O*NET, enabling Manpower to more efficiently consolidate information for global reporting (https://www.doleta.gov/programs/ONET/Manpower.cfm).

Exhibit A-16. Private Companies and Commercial Products

Organization	Description and URL
Piedmont Natural Gas	Use of ONET tools to better match job applicants to job openings has helped reduce turnover of entry-level employees. Job descriptions were improved by using information from ONET occupational reports, enabling the company to better identify the skills necessary for successful job performance: https://www.piedmontng.com/.
Trustmark Companies	Uses O*NET OnLine to collect information on job titles, tasks, and skills and to access salary data. This information is used to compare occupational requirements across jobs and industries, as well as to help develop compensation benchmarks: http://www.trustmarkcompanies.com/.
Fors Marsh Group, LLC	Used ONET data as the foundation of a career mapping program for the Department of Veterans Affairs (Mycareer@VA). By using ONET job analytic data, this new system, used throughout the VA, was developed in a timely fashion and presents valid information to its users: http://www.forsmarshgroup.com/.
Assessment Associates International	Used ONET data to develop the Work Behavior Inventory (WBI), an assessment measuring employee work styles. The WBI assesses Work Styles as defined in the ONET system and provides users with information that can be used to guide their leadership development, identify strengths that can lead to career advancement, identify training needs, and help guide career choice and transition decisions: http://aai-assessment.com/.

International Users

O*NET data and career tools have quickly gained prominence in government and private industry products around the globe. One example is Australia’s government, which uses the O*NET database as a behind-the-scenes data set linked to its own Standard Classification of Occupations. Human resource professionals in Japan have adopted the O*NET Career Exploration Tools. Chinese researchers have relied on the O*NET database for their occupational studies. European and Central American countries are translating O*NET products for their own populations. O*NET OnLine has received visits from users in over 190 countries. Countries logging hundreds of thousands of hits include Australia, Canada, China, Egypt, the Netherlands, New Zealand, Singapore, Germany, Great Britain, Hong Kong, India, Indonesia, South Africa, South Korea, and Taiwan. Each year, the O*NET Web sites log millions of visitors from virtually every geographic region in the world. Here and in Exhibit A-17 we present some specific examples of international users of O*NET.

The Italian Integrated System of Occupations was created based on the O*NET model. The Institute for Development and Vocational Training of Workers (ISFOL) and the Italian National Institute of Statistics (ISTAT) collaborated to create this classification of job holder characteristics on behalf of the Ministry of Labour. This Italian system uses tools from O*NET’s survey to create its own classification and description of occupational units in Italy (https://www.researchgate.net/publication/264897587_Skills_and_occupational_needs_labour_
market_forecasting_systems_in_Italy).

The International Labour Organization (ILO) used crosswalks from the O*NET occupational taxonomy to the International Standard Classification of Occupations (ISCO) classification system to link O*NET data to occupations within the 10 major groups of the ISCO-88 and ISCO-08 systems. This off-the-shelf statistical utility includes a comprehensive step-by-step mapping of O*NET descriptors and data points, as well as a many-to-many matching of O*NET classifications to ISCO classifications.

Exhibit A-17. International Users

Organization	Description and URL
Economic Modeling Specialists International (EMSI) (Slane, 2013)	Uses ONET data to provide information to professionals in workforce development and education. They supply data to clients in the United States, the United Kingdom, Canada, and Australia. EMSI connected ONET descriptors to jobs in the United Kingdom. Matching skill sets found in the O*NET system to U.K. jobs permits valid cross-job comparison of required knowledge and abilities, helping, in turn, to create career pathways for learners as well as those already in the workforce: https://www.economicmodelling.co.uk/2013/06/11/how-onet-classification-helps-us-match-jobs-and-skills/
Aedo, Hentschel, Javier, & Moreno (2013)	Used O*NET skills to compare skill requirements of occupations across 30 countries
Balasubramanian (2014)	Used O*NET skill information to examine the relationship between the geographical location of skilled workers and employment growth in India
Tijdens, De Ruijter, & De Ruijter (2013)	Used O*NET task data to compare tasks completed by people with the same job title across eight European countries

Foster and Gloss (2015) described their collaboration with representatives of the Professional Qualification Authority (PAQ) of the People’s Republic of China. Several O*NET database components, such as task statements and clusters of detailed work activities, provided information that advanced the mission of the PAQ to support the development of workers’ skills. Similar applications of O*NET data in other countries could facilitate economic growth (http://www.siop.org/tip/jan15/pdf/HWP.pdf).

Examples of the O*NET Program in Published Literature

Presented here are some examples of references to the O*NET Program in publications. For an extensive list of research articles, books, book chapters, technical reports, and presentations referencing the O*NET Program, see Appendix C.

An article in the Journal of Dynamic Decision Making describes the use of O*NET knowledge, skills, and abilities to develop a competency model for complex problem solving. The model was shown to demonstrate the differences across complex problems in the specific knowledge, skills, abilities, and other characteristics demanded of the problem solver. The authors emphasized the model’s usefulness to research on training and assessment of complex problem-solving competencies (Fischer & Neubert, 2015).

A study reported in the IZA Journal of Labor Economics used O*NET skill and work context descriptors to determine how job quality differences between a worker’s primary and secondary jobs vary with the business cycle. The researchers found that the difference in occupational job quality between one’s primary and second jobs tends to narrow slightly during recessions and widen in economic expansions, with the overall conclusion that individual households vary their job holding to benefit them financially and enhance well-being (Hirsch, Husain, & Winters, 2016).

A study reported in the Journal of Family Issues used O*NET work values data to determine how work characteristics differ for men and women in their decision to marry. The researchers found that worker autonomy on the job was positively related to women’s decision to marry. For men, no relationship was found. Use of O*NET data enabled these researchers to identify aspects of work beyond earnings as factors relating to the decision to marry (Kuo & Raley, 2016).

Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also, describe any consideration of using information technology to reduce burden.

The O*NET Data Collection Program employs the latest information technology systems and procedures to enhance the quality of the data, minimize burden on the responding establishments and questionnaire recipients, and reduce the overall cost of the data collection effort.

Web Questionnaires

Electronic versions of the O*NET questionnaires are available via the Internet to sampled job incumbents and occupation experts. Many of the benefits of the paper questionnaires are replicated in the electronic questionnaires. Specifically, users are able to start and stop multiple times without losing data. They can return to a partially completed questionnaire at any time during the survey period and resume where they stopped. A respondent may also review and edit previous answers as necessary. In addition, an on-screen progress meter keeps respondents informed of their movement through the questionnaire.

Advances in Web technologies and security, as well as the increasing prevalence of establishments’ and employees’ access to Web browsers, have made Internet-based data collection both feasible and practical. Internet use continues to accelerate, and the use of the O*NET Web questionnaire has increased with it. In 2013, 19.8% of job incumbents and 47.8% of occupation experts completed the survey online. For 2016, 38.9% of job incumbents and 65.3% of occupation experts used the online survey. The paper questionnaire cover and informational materials mailed to respondents continue to highlight and encourage the online option.

The paper and Web versions of the questionnaires were designed to be optimal for their respective modes of administration. The questionnaire design literature suggests that this approach is essential to reduce mode effects. That is, if each questionnaire is designed to minimize measurement error in its particular mode of interview, mode effects are also minimized. For example, in the paper version, multiple questions appear on a single page of the questionnaire. However, in the Web version, the domain questionnaires display only one question per screen (although the respondent can navigate at will through the instrument). The literature on Web survey design (see, e.g., Couper, 2008) suggests that one question per screen for Web surveys reduces measurement error and therefore the effects of administration mode. This difference was the only important one necessary for the Web version because both instruments are self-administered. In fact, to ensure comparability between the paper and Web responses, the formats and wordings of the questions and response categories for the two versions remain identical.

Project Web Site

An O*NET data collection Web page application has been developed to support the O*NET Data Collection Program: https://onet.rti.org. This site is divided into two major sections: the public and the restricted-access sections. The restricted-access section is further subdivided into two areas: the online questionnaires area and the project management area.

The goal for the public section is to support the establishment-recruiting process. This section is accessible to the public, without restrictions. The public section includes O*NET background information, endorsement letters, FAQs, copies of the questionnaires, and links to other O*NET Program–related Web sites. The purpose of this section is to provide establishments, sampled workers, and occupation experts with readily accessible information about the data collection effort and uses of the data.

The restricted-access section contains sensitive information available only to certain groups, such as survey respondents and project managers. This section is controlled by a user ID and password authentication protocol. The Web server includes a Secure Socket Layer (SSL) certificate to allow encrypted transmission of all information over the Internet. This same technology is used by e-commerce Web sites to secure credit card numbers. No cookies are used. (A cookie is data given to a user’s Web browser so that the browser will return the data to the server or Web site during subsequent requests.) Some Internet users distrust Web sites that deposit cookies on their computers and may even configure their computers to prohibit cookies, so the “cookie-less” techniques ensure that the site will perform as expected, whether or not a user has disabled cookies.

The online questionnaire area of the restricted-access section provides sample members an alternative to completing the paper questionnaire. Only persons who have been selected to participate in the survey have access to this area. Unique user IDs and passwords are assigned to each job incumbent and occupation expert by a central office computer system at the time of selection into the sample; names and other personally identifiable information are not obtained. The ID and password, along with other survey materials, are provided to the sample member. Before allowing access to the online questionnaire area, the Web site confirms the validity of the ID and password and verifies that a completed paper survey form has not already been received.

Having entered this portion of the site, respondents are:

informed that participation in the survey is voluntary,
assured that their survey responses will remain private to the extent permitted by law,
limited to seeing only the questionnaire they have been asked to complete,
permitted to stop at any point and continue responding later,
permitted to skip questions they choose not to answer, and
permitted to review and change previous responses.

On the last page of the survey, respondents confirm that they have completed the questionnaire and are given the option to enter comments about the survey; then they exit the questionnaire area of the site and are thanked for their participation. Their user ID is automatically deactivated at this time. Any further attempts to log in will not be possible because the system recognizes these users as having completed the survey.

The database containing the survey data resides on a server inside RTI’s firewall. The Web data collection application encrypts and transmits data from the respondent’s computer into the survey database. Only authorized project staff, operating from inside the firewall, have access to the survey database.

The project management area of the restricted-access section contains data collection management reports and information. Login credentials for this area are created for managers of the O*NET Data Collection Program. This portion of the Web site serves as an intranet for the O*NET Program, facilitating communication among RTI staff, program staff at the National Center for O*NET Development, and DOL. Production reports, posted nightly, include summaries of the progress of establishment recruiting, questionnaire shipment and receipt, and overall data collection status. Additional applications include a secure, centralized data file repository used by statisticians, analysts, and others to share results of specialized, nonroutine analyses and reports. The project management area of the Web site reduces the cost of the data collection effort by centralizing and streamlining features used by the project team.

The Case Management System and Data Collection Utilities

The O*NET CMS is a Web-based control system that supports and monitors the data collection activities of the BLs, the mailing of informational materials and questionnaires, and the receipt of completed paper and Web questionnaires. Enhancements have been made to the CMS that allow greater flexibility and effectiveness in the communications between the BLs and the POCs in sample establishments. For example, visual cues (icons) in the CMS allow the BLs to prioritize and customize their approach for certain types of establishments that require special procedures. Another enhancement is the ability to do ad hoc package modifications, such as the inclusion of special endorsement letters, based on O*NET-SOC occupations. Because the packages for specific O*NET-SOC occupations are customized, the POCs receive targeted materials that help communicate the data collection mission. In addition, tools have been developed to help operations management load-balance the number of cases assigned across the team of BLs. Another feature, for use with multisite organizations, permits shipping of questionnaires to more than one POC in an establishment.

In an effort to help manage supplies (envelopes, brochures, questionnaires, etc.) associated with data collection, an inventory tracking system integrated with the CMS has been developed and deployed. As the BLs place orders for informational materials and questionnaires to be shipped to survey participants, the inventory system updates reports to show the expected remaining inventory. As stocks run low, operations staff are alerted to replenish supplies. The system provides a means to reconcile physical and expected inventory. As a result, the system has improved the efficiency of ordering, storing, and shipping data collection materials.

Questionnaires are prepared for each respondent by an automated order-fulfillment system. The system detects the questionnaire domain type and occupation assigned to the respondent, and it prints the appropriate pages, including the individually customized and labeled questionnaire, ready to pack and ship.

The CMS automatically assigns questionnaire domain types sequentially as new respondents are added to the sample. Depending on how many of each type are returned by respondents, it is possible to receive enough responses of a particular questionnaire type for an occupation before receiving the total desired number of questionnaires of all types. The system continually monitors the number of returned questionnaires by type and overrides the sequential assignment of questionnaires as appropriate so that only the questionnaire types that are still required are sent. This feature enhances efficiency by eliminating the shipping of questionnaires that are no longer necessary to complete the data for an occupation. Moreover, by continually focusing resources only on the remaining questionnaire types required, it reduces the total time required to complete any occupation.

Section 508

Section 508 of the Rehabilitation Act of 1973, as amended, specifies that persons with disabilities shall have access to and use of the same information that persons without disabilities have. To comply with this section, the O*NET Data Collection Program designed its DOL-sponsored Web applications, including O*NET OnLine, My Next Move, My Next Move for Veterans, Mi Próximo Paso, and O*NET Code Connector, to ensure that the data and information are accessible to the widest possible audience, including people with disabilities. The sites also provide links to several accommodation and disability resources on the Internet. Site accessibility remains an important design component in the ongoing maintenance and development of DOL-sponsored sites, with close adherence to the guidelines of the Web Accessibility Initiative (WAI) from the World Wide Web Consortium (W3C).

Additional Uses of the Internet for Data Collection

The O*NET Program uses the Internet to gather additional occupational information, such as alternate (lay) titles and high-demand technologies. This use expands the O*NET database, providing easily maintained current information and enhancing users’ ability to find occupations relevant to their training and expertise by giving them a wider range of search terms. This enhancement is accomplished without additional burden to the public and at less cost than other means of data collection.

“Web scraping” is employed to gather information on tools and technologies, with particular emphasis on the technologies that are most in demand by employers. Information from this data mining approach is used to compile a list of “hot technologies.” This list, which is updated quarterly, includes programming and software technologies such as Python and Meditech that appear frequently in lists of occupational or job requirements. This technique is also used to evaluate and augment the lists of tools and technologies developed and maintained for each occupation in the SOC taxonomy. Tools and technologies linked to specific occupations are found by using job advertisement data mining software and by searching job posting, career education, professional association, and other Web sites. Occupational analysts link the tools and technologies to the United Nations Standard Products and Services Code, an online classification system for tools and services (http://www.unspsc.org/). Before the information is published in O*NET OnLine, rigorous reviews are performed to ensure the quality and usefulness of the data. Information collected from the Web in this way represents a significant enhancement to the data available to O*NET users, and it ensures that real-time labor market information is being used to maintain currency of the O*NET database.

A multimethod data collection approach is used to populate the O*NET alternate titles. Data sources include incumbent/occupational expert data, employer job postings, Occupational Code Assignment (OCA) submissions, transactional analyses (e.g., unmatched search terms submitted to CareerOneStop.org, a DOL-sponsored online resource for job seekers), customer and professional group requests, additional classification systems, and other miscellaneous sources. Each title undergoes an extensive multistep review process and is reviewed by multiple occupational analysts. Deliverables include (1) reported job titles that appear in O*NET OnLine, My Next Move, and Web Services and (2) a published Alternate Titles database that includes alternate job titles appearing in the downloadable database, keyword search, and Web Services. Alternate titles greatly enhance the keyword search functions in the O*NET websites. Alternate titles are also incorporated into a number of public and private keyword searches through the O*NET Web Services.

Use of Web sites further enhances the O*NET database, provides greater search capabilities to users, and reduces burden. It also allows for rapid update of the data through user input and at minimal cost.

Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in item A.2 above.

To avoid duplication and reduce cost, several portions of the O*NET Content Model are provided from existing data sources. Specifically, as discussed in Section ‎A.1, the domain of Workforce Characteristics—including information on industries, job opportunities, and pay—is obtained through links to existing LMI databases. Information about occupational licensing, certifications, and related instructional programs is provided from existing sources and several Web sites, including the CareerOneStop (COS) Certification Finder at http://www.careeronestop.org/EducationTraining/Find/certification-finder.aspx and the COS Licensed Occupations Database at http://careerinfonet.org/licensedoccupations/
?ES=Y&EST=Licensed+Occupations. Wage and employment projections information is provided by work conducted by the Bureau of Labor Statistics, including by the OES program (https://www.bls.gov/oes/) and the Occupational Projections program (https://www.bls.gov/emp/).

The exhaustive reviews of existing labor market and occupational information conducted by the DOL’s review staff, as well as subsequent research, identified no other comprehensive, valid, and reliable sources that could be used for the data items included in the O*NET database. The development of the O*NET Program has involved staff and advisors who have many years of experience in labor market and occupational information and who are familiar with existing data sources. In fact, as discussed in Section ‎A.2, many existing systems that provide detailed occupational information are actually using information based on O*NET data or the predecessor DOT.

The few existing sources with similar measures are too limited to be used in the O*NET database. Some existing sources are valid and reliable—for example, information from the OPM and the U.S. Department of Defense—but are not comprehensive, because they represent only those jobs in federal civilian employment or the military. Some private sources of job analysis information exist; however, they are based on job analyses conducted for particular purposes or settings rather than on a representative sample of employers and workers. They are therefore limited in their coverage and not representative of the entire workforce. Furthermore, these analyses are not comparable because they do not use the prescribed O*NET common language to describe occupational requirements; it is not practical to combine them, because they include dissimilar components. Finally, these private data sources are not available to the general public.

If the collection of information impacts small businesses or other small entities, describe any methods used minimize burden.

All sizes of establishments are represented in the O*NET estimates for most occupations. For some occupations, the targeting strategy used in selecting an efficient sample may lead us to omit some small establishments from the sampling frame, but this omission occurs for few occupations. The omission is allowed when it is clear that sampling small establishments will greatly reduce the efficiency of the data collection or that incumbents from small establishments are not working in the mainstream of the occupation.

Given that establishments of all sizes should be represented in the samples for most occupations, specific design provisions have been undertaken to avoid overly burdening small establishments. For example, Exhibit A-18 shows the distribution of establishments by number of employees on the D&B list used as the sampling frame for O*NET data. Exhibit A-18 illustrates how O*NET sampling selects small establishments at a much lower rate than that at which they occur in the population. For example, although 93.4% of establishments employ fewer than 25 employees (represented in the first three rows of the exhibit), only 39.8% of the O*NET sample will consist of such small establishments. On the other hand, large establishments (with 250 or more employees, as represented in the 7th to 9th rows of the exhibit) will make up 23.3% of the O*NET sample but only 0.4% of all establishments. Thus, to reduce the burden on small establishments with few employees, the O*NET sample relies more heavily on large establishments. The disproportionate sampling of large and small establishments is properly accounted for in the analysis weighting, resulting in statistically consistent estimates.

In addition, data collection procedures place lower burden on small establishments than on large establishments. When a small establishment is selected, it likely employs fewer of the targeted occupations and has fewer employees working in the occupations. Thus, a POC at a small establishment generally spends less time preparing sampling lists and distributing questionnaires than a POC at a large establishment, which is more likely to employ several of the targeted occupations and to have a large number of employees working in the occupations.

Exhibit A-18. Distribution of Frame and Sample Establishments by Employment Size

Number of Employees	Total Number of Frame Establishments^a	Frame Distribution (Percent)	*Actual Distribution of ONET Sampled Establishments^b (Percent)**
1–4	12,312,516	72.6	15.9
5–9	2,136,048	12.6	5.9
10–24	1,395,949	8.2	18.0
25–49	528,016	3.1	8.5
50–99	318,662	1.9	15.0
100–249	169,064	1.0	9.5
250–499	41,342	0.2	13.4
500–999	15,700	0.1	5.7
1,000+	10,359	0.1	4.2
Unknown	36,939	0.2	3.9
Total	16,964,595	100%	100%

^aData based on June 2017 Dun & Bradstreet (D&B) frame of establishments.

^bData based on distribution of prior O*NET samples that used the D&B frame. Future O*NET samples will be similarly designed.

Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

The O*NET database is the most comprehensive source of occupational information in the United States. No other similarly comprehensive, reliable, and valid source exists. If O*NET data are not collected, U.S. citizens, industry, business establishments, military, government and educational institutions, and the workforce investment system will have few options that meet their needs for occupational information. O*NET data are used to develop industry competency models and occupational competency profiles (i.e., industry- or occupation-specific job or work analyses). O*NET data also include information on transferable skills and are used for skills gap analysis, promoting the development of a mobile workforce responsive to changing economic needs. “Students, jobseekers and workers need up-to-date information on required job skills for specific occupations. O*NET provides the best resource for detailed descriptions of the knowledge, skills, abilities, work-related tasks, and tools and technologies used by specific occupations (974 occupations are covered)” (Workforce Information Advisory Council, 2017).

The use of O*NET data by industry, employers, software developers, job seekers, students, educators, and workforce development specialists supports a well-functioning U.S. labor market and workforce investment system—both essential to U.S. competitiveness in the global 21st-century economy. O*NET-SOC occupations conform to the SOC, permitting O*NET data to be linked to and analyzed with sources of information on current occupational employment and trends, wages, and demographic data. Its electronic format is freely accessible through DOL-sponsored Web sites, including O*NET OnLine, My Next Move, My Next Move for Veterans, Mi Próximo Paso, and O*NET Code Connector. A wide variety of database versions are also available to download free of charge from the O*NET Resource Center (see https://www.onetcenter.org/). Over a thousand customers are signed up for O*NET Web Services, including private, nonprofit, government, and military organizations, allowing them to take advantage of the ability to access and seamlessly incorporate O*NET data and applications within their systems via the extensive Web Service offerings (see https://services
.onetcenter.org).The O*NET data and structure are also being incorporated into a number of open data initiatives, including the Data at Work initiative, OpenGovernmentdata.org, the Credential Engine/Credential Registry, and the Competency and Skills System (CASS).

The initial 3.1 version of the O*NET database has been updated 18 times as new data have been collected and analyzed. Additional data releases are planned through 2021, allowing for the continued update of occupations and release of data on new and emerging occupations. The consequences of discontinuing data collection would be that the millions of users who rely on O*NET data for business and career decisions, for educational programming, and for work in human resources or workforce development would instead be using portions of information that are out of date and incomplete. If data were collected less frequently, the currency of some data would become questionable, especially for occupations that are changing as a result of new technologies. The focus of data collection on high-growth and new and emerging occupations reflects the need to provide current information in a rapidly changing, demand-driven economy.

A 3-year extension of the O*NET Data Collection Program is being requested for the period October 2018 through September 2021. This extension will provide for the updating of selected high-growth occupations and for data collection activities for new and emerging occupations. The recently released 2018 Standard Occupation Classification system identified many new occupations (such as data scientists), as well as a number of modifications to existing occupation titles, definitions, and classifications (https://www.bls.gov/soc/2018/home.htm). A dynamic and progressive U.S. economy requires continuous improvement to the data on which so many decisions are based. Millions of people are currently using O*NET information, and the numbers continue to expand as public agencies and private developers integrate O*NET data into their systems and products. The O*NET database provides valid, reliable, and current occupational information crucial to a strong U.S. workforce. O*NET database updates are scheduled to occur once a year to incorporate newly collected information on recently surveyed occupations. A schedule for data analysis is provided in Section ‎A.16.1; schedules for data collection and analysis are subject to annual appropriations.

Explain any special circumstances that would cause an information collection to be conducted in a manner that requires further explanation pursuant to regulations 5 CFR 1320.5.

There are no special circumstances that might require deviation from the guidelines.

If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the agency’s notice, required by 5 CFR 1320.8(d), soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments. Specifically address comments received on cost and hour burden.

Describe efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or reported.

Consultation with representatives of those from whom information is to be obtained or those who must compile records should occur at least once every 3 years—even if the collection of information activity is the same as in prior periods. There may be circumstances that may preclude consultation in a specific situation. These circumstances should be explained.

In accordance with the Paperwork Reduction Act of 1995, the public was allowed 60 days to comment through the Federal Register Notice posted on March 30, 2018 (83 FR 13787).

DOL received only one response during the 60-day public comment period. That comment indicated support for continued O*NET data collections efforts based on the value and usefulness of the O*NET database for workforce development and analysis. The commenter recommended that the O*NET survey, rather than the Occupational Employment Statistics (OES) program, collect and produce wage information. DOL notes that the OES program is an established, high-quality, reliable federal-state cooperative program that produces national, state, and local employment and wage estimates on an annual basis. The O*NET survey is national in scope and produces qualitative information; the O*NET survey sample as designed could not produce such quantitative estimates comparable to the OES program. The commenter also indicated lack of support for using public funds to develop further Web sites to disseminate O*NET information. DOL gives careful consideration whenever there is an apparent need to better address certain populations or economic dislocations through provision of a customized Web site. As noted in the comment, many private, public, and nonprofit career information tools are built using O*NET data. Indeed, much of the O*NET information and tools are available via download as well as through extensive O*NET Web services and APIs. Hundreds of organizations already make use of these open resources, and additional users are continually registering for them and using O*NET in a wide variety of applications. No changes have been made to this Information Collection Request on the basis of public comments.

The OMB clearance package was reviewed by an expert consultant, Dr. James B. Rounds. Dr. Rounds is a professor in the Psychology Department at the University of Illinois at Urbana-Champaign. He is a widely published industrial/organizational psychologist who is intimately familiar with the O*NET program and the types and uses of the data collected. Revisions responsive to his comments have been incorporated, as appropriate, in the clearance package.

The data collection contractor, RTI International, has several mechanisms in place to obtain ongoing feedback from study participants. The BLs have multiple telephone contacts with POCs and occupation experts; they are careful to document in their call notes significant comments or suggestions they receive. Subject matter experts at the professional associations that we contact to help identify occupation experts frequently provide feedback that the National Center for O*NET Development uses to refine occupation descriptions and tasks. Letters and brochures sent to POCs and occupation experts offer a toll-free number they can call with questions or comments. In addition, the project Web site, https://onet.rti.org/, which both study participants and the general public can access, has a “contact us” tab that offers both a toll-free number and an e-mail link. Furthermore, both the paper and online versions of the questionnaires provide respondents an opportunity to submit comments with their questionnaire responses. All feedback received from these sources is promptly reviewed by project management staff. Of course, the agency will comply with all Paperwork Reduction Act requirements should comments warrant changing the information collection.

Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.

Since the origins of the survey, incentives have been offered to POCs, establishments, and employees to encourage their participation in the O*NET Data Collection Program. Although the procedures are designed to minimize respondent burden, the effort for the company and the POC participation is not insignificant.

Incentives for the Point of Contact and the Employer

The POC’s responsibilities include

reading the introductory package to become familiar with the purpose of the O*NET Data Collection Program and the role of a POC;
seeking permission within the company, as necessary, to participate in the O*NET Data Collection Program;
making a roster of all employees at the location who work in as many as five different occupations;
participating in a sampling process that selects as many as 20 total employees from these occupations, and maintaining this sample roster for future reference;
distributing questionnaires to the sampled persons within the company and addressing their questions and concerns about the survey; and
distributing follow-up materials to employees, including thank you/reminder cards and replacement questionnaires, and following up with nonrespondents to encourage participation.

Because POCs are the only link with the O*NET respondents, they must be fully committed to the data collection process. The POC is the O*NET Program’s representative within the establishment who communicates the importance of the O*NET Program.

The employer is also essential because he or she is being asked to

support the O*NET Data Collection Program by agreeing to the company’s participation in data collection,
support and encourage the POC in carrying out his or her responsibilities,
allow the POC to provide information regarding the number of persons employed in the establishment in the occupations of interest, and
support the participation of the sampled employees.

Incentives for both POCs and employers are essential to encourage the highest response rates possible. POCs who agree to participate receive a framed Certificate of Appreciation from DOL. The Certificate of Appreciation is printed in color on card stock, bears the DOL Seal and O*NET logo, displays the POC’s name, and is signed by a high-ranking DOL official. It has a solid oak frame and Plexiglas cover and is suitable for displaying on an office wall.

Employers who agree to participate receive the O*NET Toolkit for Business. The toolkit is an O*NET Program information packet, including a guide for writing job descriptions, that managers can use for human resource planning. These materials are attractively organized in colored, glossy folders.

The continuation of these incentives is planned for both the POC and the employer because they seem to be working quite well, as evidenced by O*NET’s competitive employer response rate (see Section A.1.5). An experiment was conducted from 2002 to 2004 to evaluate an additional incentive of $20 to the POC. Essentially, the incentive had no effect on POC cooperation rates or employee response rates, but it significantly increased data collection costs. The experimental findings suggested that the current POC and employer incentives are adequate for maximizing response rates at a reduced cost (Biemer, Ellis, Pitt, & Robbins, 2006).¹⁶

Incentives for the Employee

In keeping with what has been done since 2001, each employee is offered a prepaid incentive of $10 to ensure that a high percentage of the job incumbents respond by completing the questionnaire.

Monetary incentives have the greatest potential impact when the respondent has to exert some special effort, such as taking a test or filling out a multi-item questionnaire. The incentive encourages respondents in a task requiring higher levels of involvement and commitment than the typical one-time, face-to-face interview. Although the O*NET questionnaires are not tests, the cognitive demands they place on respondents resemble test-taking demands in that the respondents must assess the requirements of their jobs. The monetary incentive is instrumental in impressing upon the respondent the importance of this rating task. Respondents who perceive the rating task as important will likely make thoughtful, carefully considered assessments rather than hasty ones, thus improving the reliability of the data.

In addition, the monetary incentive is important because respondents are encouraged to complete the questionnaire, which takes about 30 minutes, on their own time rather than on the job. This encouragement minimizes the burden on employers and also improves the quality of the data; without it, busy workers might be underrepresented in the data, which would bias the estimates for job performance.

The monetary incentive may at least partially offset its inherent cost through efficiencies created in the data collection process as a result of higher response rates (Statistics Research Division, 2000, October). For the job incumbent respondents especially—although they are not viewed as a difficult-to-reach population in the usual sense—considerable effort and resources are expended to identify and reach them through the sampling process. Each one represents a worker in a specific occupation in a specific establishment in a specific industry. The expense of reaching that particular respondent justifies the cost of a monetary payment to ensure a high rate of response.

With regard to the size of the employee incentive, payment amounts were evaluated in the pretest to determine the optimal means to maximize the response rate. On the basis of these data, a $10 prepaid cash incentive has been used since the initial wave of data collection in 2001. Because the employee response rate continues to compare favorably with those of other voluntary establishment surveys, an increased incentive does not seem to be justified at this time.¹⁷ However, we will continue to monitor the employee response rate and will reassess the amount of the incentive if it appears that the benefits of an increased incentive would outweigh the cost.

Incentives for Occupation Experts

Occupation experts provide data for approximately 25% of the O*NET-SOC occupations. Each occupation expert who agrees to participate receives a prepaid incentive of $40 and a framed Certificate of Appreciation from DOL. Unlike job incumbents, who complete only one domain questionnaire, occupation experts are asked to complete all three domain questionnaires; thus, the $40 incentive is about $13.33 per questionnaire. This incentive is slightly higher than the $10 offered to Establishment Method respondents for completing one domain questionnaire. The increased incentive and the Certificate of Appreciation are necessary to gain cooperation from what is often a rare group of experts for an occupation. Moreover, the additional incentives seem commensurate with the effort involved in responding to multiple questionnaires, given that occupation experts are supervisors and trainers in the occupation and, as such, earn a higher salary than the average employee.

Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.

RTI has extensive experience in protecting and maintaining the privacy of respondent data collected from surveys. To ensure privacy, RTI has drawn from its experience in designing the data collection procedures incorporated in this program. In addition, RTI’s institutional review board is bound by institute policy, the O*NET contract, and federal regulations to review and approve the research protocol to ensure compliance with federal regulations (45 C.F.R. § 46) concerning data privacy and the protection of human subjects from research risks.

Respondents are informed that their individual responses will be kept private to the extent permitted by law. Survey data are collected from job incumbents (Establishment Method) and from occupation experts (OE Method). Informational materials mailed to potential respondents contain essential program information and assurances of privacy that enable the person to make an informed decision about his or her voluntary participation in the data collection effort. Examples of informational materials provided to survey participants appear in Appendix B.

Employees sampled at establishments are asked to complete their questionnaires on their personal time, not company time. This stipulation enables the employee to select a comfortable and private setting, if desired, in which to complete the questionnaire. All respondents have a choice of completing paper questionnaires or completing the questionnaires online at the project Web site. Paper questionnaires are mailed directly to RTI in a stamped reply envelope provided by RTI. The individual responses are processed according to a study ID number. All O*NET Data Collection Program staff are required to sign a privacy pledge that assures each respondent that the privacy of responses to the questionnaire will be maintained. Only authorized staff have access to the completed instruments and data files. The completed and processed questionnaires are stored in a secure document-control area until federal authorization has been granted to destroy them. All computer files, including those associated with the control system, are password protected.

The Internet-based system that allows respondents to provide their survey responses electronically has restricted access, including a user ID and password authentication protocol for respondents. The Web server includes an SSL certificate, the same technology used by e‑commerce sites, to allow encrypted transmission of all information over the Internet. The database containing the survey data is not accessible via the Internet; it resides on a server inside the RTI firewall. Only O*NET Data Collection Program staff have access to the master survey database.

The O*NET questionnaires (see Appendix A) collect very little personal information about the respondent, and what is collected contains no identifiers, such as personal name or place of employment. No individual-level data are published, nor are they accessible or provided to anyone except the O*NET Data Collection Program staff. Published results are made available only in aggregate, as one set of estimates for an entire occupation. Furthermore, no demographic data (e.g., sex, race) are released, even in aggregate form. Finally, estimates are not produced for any subpopulations within an occupation, such as geographic region or sociodemographic group, as these details could allow the identification of individuals.

Before publishing the O*NET tables on the public Web site, the O*NET Program team thoroughly examines the tables for any risk of disclosure of private information. In particular, each table is analyzed to identify any “sensitive” cells (i.e., cells that may reveal too much information about an individual employee). Willenborg and De Waal (1996) have recommended using an (n,k)-dominance rule that a cell be regarded as sensitive if the sum of the largest n contributions account for more than k% of the total cell value. Willenborg and De Waal further recommend that n = 5 and k = 80. Because every sampled employee contributes only one response, these recommendations translate into a minimum cell size of 5/0.80, or about 6. In fact, the minimum cell size for the O*NET tables is 10 respondents; any sensitive cells with fewer than 10 are suppressed. In addition, the (n,k)-dominance rule assumes a complete census. As Willenborg and Waal note, when applied to tables based on samples and where the cell entries are weighted averages, the (5,0.80) rule affords even greater disclosure control. This extra control ensures that these O*NET tabular data pose no disclosure risks to any individual respondent.

Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

Only four questions in the O*NET questionnaires may be considered to be of a sensitive nature. In the Background Questionnaire, the survey uses the four disability questions developed for the Census Bureau’s American Community Survey. The first two questions ask respondents about serious hearing and sight difficulties. The next question has subparts that ask about difficulties with concentration, memory, or decision making; mobility; and self-care. The last question asks whether respondents’ physical or mental health makes it difficult to do errands alone. Completion rates for these items indicate that the great majority of participants (97%) elect to answer them.

The O*NET sampling strategy is to randomly select participants at the individual level. The disability questions, together with the demographic questions, provide descriptive information about the sample of respondents.

Respondents to the O*NET survey are informed that responding to all questions, which includes disability status and the other demographic characteristics, is voluntary and that the data will be kept private to the extent permitted by law. They complete the survey on their own time, in a private setting if they choose. No identifying information, such as the respondent’s name or place of employment, is recorded on the questionnaire.

Provide estimates of the hour burden of the collection of information.

As described in Section ‎A.1.4, there are two protocols for O*NET data collection—the Establishment Method and the OE Method. An estimated 75% of occupations are completed by the Establishment Method. The OE Method is used for occupations as necessary to improve sampling efficiency and avoid excessive burden, as when it is difficult to locate industries or establishments with occupation incumbents; employment is low; or employment data are not available, as is the case for many new and emerging occupations.

The Establishment Method uses a two-stage sample, with establishments selected at the first stage and employees selected at the second stage. Thus, there are burden hours associated with both establishments and employees. For each participating establishment, a POC is identified to coordinate data collection activities in the establishment. In Exhibit A-19, the first and second columns of the Establishment Activity section present the number of responses and assumed average burden per response for the POC’s activities; these averages were obtained from previous years of O*NET data collection experience. When the total establishment burden was estimated, the estimated number of establishments that will complete each activity was multiplied by the average burden and summed across the activities.

The Employee Activity section of Exhibit A-19 is based on the fact that selected employees under the Establishment Method will complete one domain questionnaire, requiring an average of 30 minutes of effort, whereas occupation experts will complete all three domain questionnaires, requiring an average of 90 minutes of effort.

Exhibit A-19 also displays the estimated number of sampling units and the estimated burden hours. As shown in Exhibit A-21 in Section ‎A.16, we assume that 75 occupations will be completed under the Establishment Method and 25 will be completed under the OE Method each year. From October 2018 through September 2019, establishments are estimated to expend 6,211 burden hours, and employees are estimated to expend 8,390 burden hours, for a total of 14,601 burden hours. From October 2019 through September 2020, establishments are estimated to expend 6,211 burden hours, and employees are estimated to expend 7,991 burden hours, for a total of 14,202 burden hours. From October 2020 through September 2021, establishments are estimated to expend 6,022 burden hours, and employees are estimated to expend 8,053 burden hours are estimated for employees, for a total of 14,075 burden hours. The slight decline in total burden hours across the 3-year period October 2018 through September 2021 (14,601; 14,202; 14,075) results from minor differences in the data collection schedule assumed for each year of the burden period. The data collection schedule for each year of the burden period is determined by the sampling characteristics of the specific set of occupations being studied.¹⁸

The Total Respondents section of Exhibit A-19 shows the estimated annual number of respondents by category and overall. The respondent totals include the number of POCs (the row name is “Screening call to POC”), the number of Establishment Method employee respondents, and the number of OE Method respondents. The category totals are based on prior sampling experience. The total number of respondents across all 3 years is 85,483.

The Total Burden Hours section of Exhibit A-19 shows the annual number of burden hours by category and overall. The burden hour totals include both establishment burden and employee burden. The category totals are based on the estimated number of respondents for each category, the number of responses, and the average burden per response. The total number of burden hours across all 3 years is 42,878. This 3-year total reflects a slight decrease in burden compared with the previous 3-year period of 2015 to 2018, during which 43,610 burden hours were requested (U.S. Department of Labor, Employment and Training Administration, 2015).

O*NET Establishment Method data collection has been designed to minimize the burden on the selected establishments:

Establishments are asked about no more than 10 occupations each, with questioning terminated once 5, or sometimes fewer, occupations are identified at an establishment.
Establishments are asked to complete rosters of employees only for the 5 or fewer occupations identified.
Establishments are selected no more than once within a 12-month period.
No more than 20 employees are selected from an establishment across all of its selected occupations.
If an occupation proves difficult to complete under the Establishment Method, the dual-frame approach may be used to supplement it. For occupations that are difficult to sample, the alternative OE Method may be used.

Finally, the last row of Exhibit A-19 shows that the combined establishment and employee total cost burden is $679,108 for October 2018–September 2019, $680,547 for October 2019–September 2020, and $686,312 for October 2020–September 2021. The slight increase in total cost burden across the 3-year period October 2018–September 2021 results from minor variations in the data collection schedule for each year as well as from the inflation adjustment applied to the assumed average total compensation per hour.

Exhibit A-19. Estimate of Hour and Cost Burden by Year

	Number of Responses per Sample Unit	Average Burden per Response (Minutes)	Oct 2018 – Sept 2019		Oct 2019 – Sept 2020		Oct 2020 – Sept 2021
	Number of Responses per Sample Unit	Average Burden per Response (Minutes)	Sampling Units	Burden Hours	Sampling Units	Burden Hours	Sampling Units	Burden Hours
Establishment Activity
Verification calls to initial contact at establishment	1	2	16,500	550	16,500	550	16,000	533
Screening call to POC	1	3	13,695	685	13,695	685	13,280	664
Initial recruitment call to POC	1	12	7,491	1,498	7,491	1,498	7,264	1,453
POC creation of occupation lists for sampling	1	20	4,098	1,366	4,098	1,366	3,973	1,324
Call to POC to sample workers	1	10	4,098	683	4,098	683	3,973	662
POC’s distribution of questionnaire packets	1	15	3,729	932	3,729	932	3,616	904
Follow-up calls to POC	4	2	3,729	497	3,729	497	3,616	482
Total, establishment^a	NA	NA	NA	6,211	NA	6,211	NA	6,022
Employee Activity
Establishment Method employee respondents	1	30	14,753	7,377	13,956	6,978	14,079	7,040
Occupation Expert Method respondents	3	30	675	1,013	675	1,013	675	1,013
Total, employee^b	NA	NA	NA	8,390	N/A	7,991	NA	8,053
Total Respondents
Private sector	NA	NA	12,531	NA	12,531	NA	12,151	NA
Federal government	NA	NA	870	NA	870	NA	843	NA
State/local/tribal governments	NA	NA	294	NA	294	NA	286	NA
Subtotal, establishment respondents	NA	NA	13,695	NA	13,695	NA	13,280	NA
Individuals/households	NA	NA	15,428	NA	14,631	NA	14,754	NA
Total, all respondents^c,d	NA	NA	29,123	NA	28,326	NA	28,034	NA
Total Burden Hours
Private sector	NA	NA	NA	5,683	NA	5,683	NA	5,510
Federal government	NA	NA	NA	394	NA	394	NA	383

(continued)

Exhibit A-19. Estimate of Hour and Cost Burden by Year (continued)

Number of Responses per Sample Unit

Average Burden per Response (Minutes)

Oct 2018 – Sept 2019

Oct 2019 – Sept 2020

Oct 2020 – Sept 2021

Sampling Units

Burden
Hours

Sampling Units

Burden Hours

Sampling Units

Burden
Hours

State/local/tribal governments

134

129

Subtotal, establishment burden hours

6,211

6,022

Individuals/households

8,390

7,991

8,053

Total, all burden hours^e

14,601

14,202

14,075

Oct 2018 – Sept 2019

Oct 2019 – Sept 2020

Oct 2020 – Sept 2021

Establishments^f

$385,206

$394,150

$391,008

Employees^g

$293,902

$286,397

$295,304

Total

$679,108

$680,547

$686,312

Note: NA = not applicable.

Exhibit A-19 displays the estimated annualized cost to respondents for burden hours by year. The cost burden was estimated with use of average total compensation rates obtained from the March 2017 Employer Costs for Employee Compensation Summary issued by BLS on June 9, 2017 . The average total compensation per hour for private industry was $33.11, which was inflated based on the Employment Cost Index to a median hourly total compensation of $35.03 for October 2018–September 2019, $35.84 for October 2019–September 2020, and $36.67 for October 2020–September 2021. These are the total compensation rates used for estimating the employee cost burden. Given that the POC will often be a human resources manager, the March 2017 total compensation rate of $58.62 for the Management, Professional and Related category was inflated to a median hourly total compensation of $62.02 for October 2018–September 2019, $63.46 for October 2019–September 2020, and $64.93 for October 2020–September 2021 for estimating the establishment cost burden.

^aIncludes total burden time for all establishments (private sector, federal government, and state/local/tribal governments).

^bIncludes total burden time for individuals/households.

^cThe total number of respondents across all 3 years = 85,483.

^dThe total number of respondents was derived by summing the number of POCs (row name is “Screening call to POCs”), Establishment employees (row name is “Establishment Method employee respondents”), and occupation experts (row name is “Occupation Expert Method respondents”).

^eThe total number of burden hours across all 3 years = 42,878.

^fAssumed hourly total compensation rates: $62.02 for October 2018–September 2019; $63.46 for October 2019–September 2020; $64.93 for October 2020–September 2021.

^gAssumed hourly total compensation rates: $35.03 for October 2018–September 2019; $35.84 for October 2019–September 2020; $36.67 for October 2020–September 2021.

Source: Bureau of Labor Statistics, U.S. Department of Labor. (2017, June). Employer costs for employee compensation summary. Available from Databases, tables and calculators by subject, https://data.bls.gov/timeseries/CIS2010000000000Q

Exhibit A-19. Estimate of Hour and Cost Burden, Annual Averages (continued)

Activity	Annual Number of Respondents	Frequency	Total Annual Responses	Time Per Response (Hours)	Total Annual Burden (Hours)	Average Annual Hourly Rate	Monetized Annual Value of Respondent Time
Individuals/households
Establishment Method employee respondents	14,263	1	14,263	0.500	7,132	$35.83	$255,540
Occupation Expert Method respondents	675	1	675	1.500	1,013	$35.85	$36,316
State/local/tribal governments	291	1	291	0.4545	132	$63.46	$8,377
Federal government	861	1	861	0.4535	390	$63.46	$24,749
Private Sector	12,404	1	12,404	0.4535	5,625	$63.46	$356,963
Unduplicated Totals	28,494	Once	28,494	Varies	14,292	Varies	$681,945

Provide an estimate for the total annual cost burden to respondents or record keepers resulting from the collection of information.

There are no respondent costs for capital or start-up or for operations, maintenance, and purchase of services. There are no costs to the employers, POCs, or sampled employees other than the time it takes them to participate in the survey.

Provide estimates of annualized costs to the Federal government. Also provide a description of the method used to estimate cost, which should include quantification of hours, operational expenses (such as equipment, overhead, printing, and support staff), and any other expense that would not have been incurred without this collection of information.

The estimated annual cost to the government for the O*NET Data Collection Program for the period October 2018 through September 2021 is approximately $7.3 million. This estimate includes all direct and indirect costs of conducting the sampling, data collection, and analysis activities of the O*NET Data Collection Program. In the chart below, Personnel and Fringe Benefit costs are for the grantee (N.C. Department of Commerce) personnel who manage the O*NET Data Collection Program. The grantee subcontracts certain program activities (e.g., survey operations and data analysis); those costs are listed on the Contractual line.

Federal Staff Costs (GS 14 Step 4 [$118,057] + GS 13 Step 4 [$99,905] x .5 [part-time])	$ 108,981
Personnel	$ 528,000
Fringe Benefits	$ 160,665
Travel	$ 8,000
Equipment	$ 4,000
Supplies	$ 56,800
Contractual	$6,187,762
Costs for Incentives	$ 158,000
Total Direct Costs	$7,212,208
Indirect Charges	$113,995
Total	$7,326,203

Explain the reasons for any program changes or adjustments reported on the burden worksheet.

Exhibit A-20 compares the projected burden hours for the period October 2018–September 2021 with the average annual burden as estimated for the period July 2015–June 2018 in the OMB Supporting Statement, Part A, dated August 2015. The projected total annual burden hours for October 2018–September 2021 range from 14,075 to 14,601. The average annual burden is 14,293 hours, compared with an average annual burden of 14,537 hours requested for the previous 3-year period (2015–2018). In addition, as indicated in Exhibit A-19, the total burden hours for the October 2018–September 2021 period, 42,878, reflect a slight decrease in burden compared with the 2015–2018 period, for which a total 43,610 hours were requested (U.S. Department of Labor, Employment and Training Administration, 2015). The reduction in burden hours is due to minor differences in the assumed data collection schedule for the 2018–2021 period. The data collection schedule is dependent on the sampling characteristics of the specific occupations being studied.¹⁹

Exhibit A-20 also compares the estimated costs to respondents for October 2018–September 2021 with the average annual cost estimated for 2015–2018. The increased annual costs since 2015–2018 are primarily because of inflation in the benefits portion of employee compensation.

Exhibit A-20. Comparison of Hour and Cost Burden Between 2015–2018 and October 2018–September 2021

	Annual Average 2015–2018^a					Oct 2018–Sept 2019			Oct 2019–Sept 2020			Oct 2020–Sept 2021
Establishment Activity	Sampling Units		Burden Hours			Sampling Units	Burden Hours		Sampling Units	Burden Hours		Sampling Units	Burden Hours
Verification calls to initial contact at establishment	16,556		552			16,500	550		16,500	550		16,000	533
Screening call to POC	13,735		687			13,695	685		13,695	685		13,280	664
Initial recruitment call to POC	7,554		1,511			7,491	1,498		7,491	1,498		7,264	1,453
POC creates occupation lists for sampling	4,155		1,385			4,098	1,366		4,098	1,366		3,973	1,324
Call to POC to sample workers	4,155		696			4,098	683		4,098	683		3,973	662
POC distributes questionnaire packets	3,822		956			3,729	932		3,729	932		3,616	904
Follow-up calls to POC	3,822		510			3,729	497		3,729	497		3,616	482
Total establishment	NA		6,297			NA	6,211		NA	6,211		NA	6,022
Employee Activity
Establishment Method employee respondents	14,456		7,228			14,753	7,377		13,956	6,978		14,079	7,040
Occupation Expert Method respondents	675		1,013			675	1,013		675	1,013		675	1,013
Total, employee	NA		8,241			NA	8,390		NA	7,991		NA	8,053

Total			14,537				14,601			14,202			14,075
Respondent Type				Cost Burden
		Annual Average 2015–2018^a			Oct 2018–Sept 2019			Oct 2019–Sept 2020			Oct 2020–Sept 2021
Establishments^b		$343,021			$385,206			$394,150			$391,008
Employees^c		$259,007			$293,902			$286,397			$295,304
Total		$602,028			$679,108			$680,547			$686,312

^a From the OMB Supporting Statement, Part A, dated August 2015.

^b Assumed hourly total compensation rates: $62.02 for October 2018–September 2019; $63.46 for October 2019–September 2020; $64.93 for October 2020–September 2021.

^c Assumed hourly total compensation rates: $35.03 for October 2018–September 2019; $35.84 for October 2019–September 2020; $36.67 for October 2020–September 2021.

For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.

The major components of the O*NET Data Collection Program include sampling, data collection operations, and analysis. Exhibit A-21 shows the expected schedule for annual data analysis cycles and data publications for the next few years.

Exhibit A-21. Data Analysis and Publication Schedule

Analysis Cycle	Analysis Cycle Start Date	Analysis Cycle End Date	*Estimated Establishment Method ONET-SOC Occupations**	*Estimated Occupation Expert Method ONET-SOC Occupations**	*Estimated ONET-SOC Occupations Published**	Publication Date
20	July 2018	July 2019	75	25	100	July 2019
21	July 2019	July 2020	75	25	100	July 2020
22	July 2020	July 2021	75	25	100	July 2021

Data Analysis Tasks Conducted for Each Cycle

Described here is the approach used for data cleaning and editing, as well as the analyses that are performed annually.

Data Cleaning

Paper questionnaires are manually and machine edited so that completely blank questionnaires are removed; responses to items that should have been skipped are blanked out; multiple responses are blanked out; and codes indicating missing data, multiple responses, and legitimate skips are inserted. Codes for legitimate skips and missing responses are also inserted in the records for questionnaires obtained through the Web. In addition, an electronic check is conducted to detect duplicate questionnaires from the same respondent.

Identification and Evaluation of Anomalous Cases

Analyses of the questionnaire ratings are based on the assumption that raters were qualified, willing, and able to engage in the rating tasks. This task is accomplished by requiring each case to first pass through a series of machine edits using prescribed eligibility criteria, including having at least one task rated important and having at least 50% of the domain questionnaire items completed. Cases not meeting these criteria are excluded from the analysis file. Cases with certain questionable characteristics are flagged in this editing process, and analysts review these cases to determine their completion status. Flagged for review are all cases in which the respondent (1) indicated in the “global match” item that the target O*NET-SOC occupation description did not at all describe his or her own job and (2) rated fewer than one third of the tasks as important. An analyst reviews the self-reported job titles of these cases to determine if they are at all likely to belong in the O*NET-SOC occupation. If a case does not belong, it is removed from further analysis. If, in the judgment of the analyst, there is a reasonable chance that the case belongs in the O*NET-SOC occupation, it is sent to the next stage of review.

Finally, cases that pass the machine edits and the analyst review are subjected to a deviance analysis designed to identify cases that are outliers relative to other cases in their occupation. The deviance analysis involves two procedures: (1) a statistical procedure to quantitatively identify potential outliers and (2) an analyst review of these potential outliers to make the final decision for each case. Cases that do not pass the analyst review are deemed deviant within their occupation and are removed from the data set. On average, these activities eliminate about 9% of all returned questionnaires. The cases passing all data cleaning criteria are used to create the estimates for publication.

Computation of Sampling Statistics

Basic sampling weights are applied to the data to make inferences about the population of incumbents for each occupation.²⁰ These weights are computed as the inverses of the overall selection frequencies and the selection probabilities for each selected establishment and each individual participant. The analysis weights for the eligible sample units are adjusted to compensate for unit nonresponse for both establishments and employees, multiple subwaves of sampling, and sample adjustments. In addition, when variation in the weights is large, the weights are trimmed to reduce the variation. To maximize comparability of O*NET estimates to estimates from other federal sources, the final sample weights are also ratio adjusted to occupation estimates obtained from the Occupational Employment Statistics survey.

Sampling errors are computed. The analysis weights used in the sampling error computations, as noted, have been adjusted for nonresponse and are consistent with the complex sampling design.

Calculation of Descriptor Values and Reliability

For each occupation, the sample size, mean, standard deviation, and standard error of the ratings for each descriptor are calculated, together with the 95% confidence interval around the mean. Estimates with questionable precision are flagged and recommended for suppression if any of the following conditions is true:

The sample size (i.e., number of respondents who answered the question) is less than 10.
The variance is zero and the sample size is less than 15.
The relative standard error (RSE) is greater than 50%.^²¹
On average, approximately 2.7% of the estimates are flagged or suppressed.²²

Interrater Reliability and Agreement

Interrater reliability and agreement are assessed with three different analyses. The first two measure reliability in terms of the covariation among ratings, and the third reflects rater agreement. The intraclass correlation coefficients (ICCs) for each questionnaire item across all occupations are computed. These results allow one to compare respondent rating variance within an occupation with respondent rating variance across occupations. In addition to the ICCs, the mean interrater correlations (Pearson’s r) are calculated for all pairs of raters within each occupation. Finally, to assess the absolute difference among ratings of each item within each occupation, an average deviation index is calculated for each item within each occupation. For any given item and occupation, the average deviation index measures the average extent to which each individual rating deviates from the item mean. Some differences in ratings within occupations are expected because O*NET-SOC occupations comprise a variety of different jobs in most cases.

Each of the reliability analyses conducted (rater, standard errors) is influenced by the number of respondents. The O*NET data collection methods include a sufficient number of respondents in each occupation to ensure reliability (Peterson, Mumford, Levin, Green, & Waksberg, 1997).

Creation of the Occupation Database

The O*NET database is scheduled to be updated annually. Each update will include data for those occupations collected and analyzed during the previous 12-month period. Consequently, a database update includes occupations from multiple data collection waves, depending on the number of prior waves for which analysis was completed that year. For each occupation collected, the newly calculated means data replace existing analyst-based or incumbent-based means data in the database. Users are provided with metadata regarding when the data were collected and any other pertinent information that will assist the users in interpreting the data.

The O*NET database is designated with a version number denoting each update (e.g., from O*NET 20.0 to O*NET 21.0). The database is developed and administered with the MySQL database management system. Once the MySQL database is updated, it is used to generate the database for public release as a series of flat text files. It is accessible to the public on the O*NET Program Web site at https://www.onetcenter.org/.

If seeking approval not to display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.

The expiration date will be displayed on the cover of the survey questionnaires.

Explain each exception to the topics of the certification statement identified in “Certification for Paperwork Reduction Act Submissions.”

There are no exceptions.

References

Aedo, C., Hentschel, J., Javier, L., & Moreno, L. (2013). From occupations to embedded skills: A cross-country comparison. Washington, DC. The World Bank. Retrieved from http://siteresources.worldbank.org/EXTNWDR2013/Resources/8258024-1320950747192/8260293-1320956712276/8261091-1348683883703/WDR2013_bp_From_Occupations_to_Embedded_Skills_Aedo.pdf

Anseel, F., Lievens, F., Schollaert, E., & Choragwicka, B. (2010). Response rates in organizational science, 1995-2008: A meta-analytic review and guidelines for survey researchers. Journal of Business Psychology, 25, 335-349.

Balasubramanian, I. (2014). Local skill concentrations and district employment growth: A spatial simultaneous equation approach for India. The Annals of Regional Science, Working Paper No. 2014-033.

Baruch, Y., & Holtom, B. C. (2008). Survey response rate levels and trends in organizational research. Human Relations, 61, 1139-1160.

Biemer, P., Ellis, C., Pitt, A., & Robbins, K. (2006). Effects on response rates and costs of a monetary incentive for the point of contact in an establishment survey. Paper presented at the Proceedings of the American Association of Public Opinion Research, Montreal, Canada.

Bureau of Labor Statistics, U.S. Department of Labor (2014, March). Employer costs for employee compensation summary. Available from Databases, tables and calculators by subject). Retrieved from http://data.bls.gov/timeseries/CIS1010000000000Q

Cole, M. S., Bedeian, A. G., & Feild, H. S. (2006). The measurement equivalence of Web-based and paper-and-pencil measures of transformational leadership. Organizational Research Methods, 9, 339-368.

Couper, M. (2008). Designing effective Web surveys. New York, NY: Cambridge University Press.

Cycyota, C. S., & Harrison, D. A. (2006). What (not) to expect when surveying executives: A meta-analysis of top manager response rates and techniques over time. Organizational Research Methods, 9(2), 133-160. doi:10.1177/1094428105280770

Deming, D. (2015). The growing importance of social skills in the labor market. National Bureau of Economic Research Working Paper Series, NBER Working Paper 21473.

Employment and Training Administration, U. S. D. o. L., ,. (2009). O*NET OnLine in transition assistance programs at Offutt Air Force Base. Lincoln, NE: Employment and Training Administration, U.S. Department of Labor Retrieved from https://www.doleta.gov/Programs/onet/ne-offutt.cfm

Fischer, A., & Neubert, J. C. (2015). The multiple faces of complex problems: A model of problem solving competency and its implications for training and assessment. Journal of Dynamic Decision Making, 1, 1-13.

Fleishman, E. A., & Mumford, M. D. (1988). The ability requirements scales. In S. Gael (Ed.), The job analysis handbook for business, industry, and government. New York: Wiley.

Foster, L., & Gloss, A. (2015). Frontiers in international skills development: Innovative efforts in China’s transportation sector and three dimensions of work information. Industrial and Organizational Psychologist Newsletter, 52(3), 67-74.

Guion, R. M. (2011). Assessment, measurement, and prediction for personnel decisions (2nd ed.). New York, NY: Routledge.

Hanser, L. M., Campbell, J., Pearlman, K., Petho, F., Plewes, T., & Spenner, K. (2008). Final report of the panel on the Department of Defense human capital strategy. Report prepared for the Office of the Secretary of Defense by the RAND Corporation. Retrieved from http://www.rand.org/content/dam/rand/pubs/technical_reports/2008/RAND_TR610.pdf

Hirsch, B. T., Husain, M. M., & Winters, J. V. (2016). Multiple job holding, local labor markets, and the business cycle. IZA Journal of Labor Economics, 5(1), 1-29.

Igou, F. (2012). O*NET Online as a valuable tool for academic advising. Mentor.

Klein, R. J., Proctor, S. E., Boudreault, M. A., & Tuczyn, K. M. (2002). Healthy People 2010 criteria for data suppression. Hyattsville, MD: National Center for Health Statistics.

Koys, D. J. (2016). Using the Department of Labor’s “My Next Move” to improve career preparedness. Journal of Management Education, 41(1), 94-117.

Kuo, J. C. L., & Raley, R. K. (2016). Is it all about money? Work characteristics and women’s and men’s marriage formation in early adulthood. Journal of Family Issues, 37(8), 1046-1073.

Navarro, A., King, K. E., & Starsinic, M. (2011, September). Comparison of the American Community Survey voluntary versus mandatory estimates. Final report. Washington, DC. U.S. Census Bureau, American Community Survey Research and Evaluation Program. Retrieved from http://www.census.gov/acs/www/Downloads/library/2011/2011_Navarro_01.pdf

Paxon, M. C., Dillman, D. A., & Tarnai, J. (1995). Improving response to business mail survey. In B. G. Cox (Ed.), Business survey methods. New York: Wiley.

Peterson, N. G., Mumford, M. D., Borman, W. C., Jeanneret, P. R., & Fleishman, E. A. (1995). Development of prototype Occupational Information Network (O*NET) content model. Salt Lake City: Utah Department of Workforce Service through a contract with the American Institutes of Research.

Peterson, N. G., Mumford, M. D., Borman, W. C., Jeanneret, P. R., Fleishman, E. A., & Levin, K. Y. (Eds.). (1997). O*NET final technical report. Salt Lake City, UT: Utah Department of Workforce Services.

Peterson, N. G., Mumford, M. D., Borman, W. C., Jeanneret, P. R., Fleishman, E. A., Levin, K. Y., Campion, M. A., Mayfield, M. S., Morgeson, F. P., Pearlman, K., Gowing, M. K., Lancaster, A. R., Silver, M. B., & Dye, D. M. (2001). Understanding work using the Occupational Information Network (O*NET): Implications for practice and research. Personnel Psychology, 54(2), 451–492. doi:10.1111/j.1744-6570.2001.tb00100.x

Peterson, N. G., Mumford, M. D., Levin, K. Y., Green, J., & Waksberg, J. (1997). Research method: Development and field testing of the content model. In N. G. Peterson, M. D. Mumford, W. C. Borman, P. R. Jeanneret & K. Y. Levin (Eds.), O*NET final technical report. Salt Lake City: Utah Department of Workforce Services, through a contract with the American Institutes for Research.

Peterson, N. G., Mumford, M. D., Levin, K. Y., Green, J., & Waksberg, J. (1999). Research method: Development and field testing of the content model. In N. G. Peterson, M. D. Mumford, W. C. Borman, P. R. Jeanneret & E. A. Fleishman (Eds.), An occupational information system for the 21st century: The development of O*NET (pp. 31–47). Washington, DC: American Psychological Association.

Peterson, N. G., Owens-Kurtz, C., Hoffman, R. G., Arabian, J. M., & Whetzel, D. C. (1990). Army synthetic validation project. Alexandria, VA: U.S. Army Research Institute for the Behavioral Sciences.

Petroni, R., Sigman, R., Willimack, D., Cohen, S., & Tucker, C. (2004). Response rates and nonresponse in BLS and Census Bureau establishment surveys. In Proceedings of the Survey Research Methods Section (pp. 4159–4166). Alexandria, VA: American Statistical Association.

Renski, H., & Wallace, R. (2012). Workforce skills and the changing knowledge economy in Massachusetts. MassBenchmarks, 14(1).

Rothwell, J. (2013, June). The hidden STEM economy. Washington, DC: The Brookings Institution. Retrieved from http://www.brookings.edu/research/reports/2013/06/10-stem-economy-rothwell

RTI International. (2013). SUDAAN® Version 11.0.1 for Windows. Research Triangle Park, NC: Author.

Schumacher, S., Kleinmann, M., & Konig, C. J. (2012). Does item decomposition and the use of less complex items make the ratings of both groups more accurate? Journal of Personnel Psychology, 11(2), 69-76.

Slane, R. (2013). How O*NET classification helps us match jobs and skills [Web log post]. Retrieved from https://www.economicmodelling.co.uk/2013/06/11/how-onet-classification-helps-us-match-jobs-and-skills/

Statistics Research Division, Research Triangle Institute. (2000, October). O*NET report: Results of statistical analysis of pretest. Research Triangle Park, NC. Retrieved from

Tarnai, J., & Paxon, M. C. (2004). Survey mode preferences of business respondents. Paper presented at the Proceedings of the American Association of Public Opinion Research, Phoenix, AZ.

Tijdens, K., De Ruijter, E., & De Ruijter, J. (2013). Comparing tasks of 160 occupations across eight European countries. Employee Relations Law Journal, 36(2), 110-127. doi:10.1108/ER-05-2013-0046

Tijdens, K. D. R., E., & De Ruijter, J. (2013). Comparing tasks of 160 occupations across eight European countries. Employee Relations Law Journal, 36(2), 110-127. doi:10.1108/ER-05-2013-0046

Treasury Board of Canada Secretariat. (2009, May). Public servants on the public service of Canada: Summary of the results of the 2008 Public Service Employee Survey. Retrieved from http://www.tbs-sct.gc.ca/pses-saff/2008/report-rapport-eng.asp#appa RESOURCEID = 16339 Retrieved from http://www.tbs-sct.gc.ca/pses-saff/2008/report-rapport-eng.asp#appa RESOURCEID = 16339

Tsacoumis, S., & Van Iddekinge, C. H. (2006). A comparison of incumbent and analyst ratings of O*NET skills. Alexandria, VA: Human Resources Research Organization.

Tulp, D. R., Jr., Hoy, E., Kusch, G., & Cole, S. (1991). Nonresponse under mandatory vs. voluntary reporting in the 1989 survey of pollution abatement costs and experiments (PACE). In Proceedings of the American Statistical Association, Section on Survey Research Methods (pp. 272–277). Washington, DC: American Statistical Association.

U.S. Department of Labor, Bureau of Labor Statistics, and U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health. (2003). Respirator usage in private sector firms). Retrieved from http://www.cdc.gov/niosh/docs/respsurv/

U.S. Department of Labor, Employment and Training Administration. (2015). O*NET® Data Collection Program, Office of Management and Budget clearance package supporting statement. Washington, DC: Author.

U.S. Office of Personnel Management. (2016). Federal Employee Viewpoint Survey: Governmentwide management report. Washington, DC: Author. Retrieved from https://www.fedview.opm.gov/2016FILES/2016_FEVS_Gwide_Final_Report.PDF

Willenborg, L., & De Waal, T. (1996). Statistical disclosure control in practice series: Lecture notes in statistics (Vol. 111). New York, NY: Springer.

Worden, G., & Hoy, E. (1992). Summary of nonresponse studies conducted by industry division, 1989–1991. Washington, DC: U.S. Bureau of the Census.

1 For details on the uses of the O*NET Program, see Section A.2. More information about the O*NET Data Collection Program can be found at the National O*NET Program’s public Web site, https://www.onetcenter.org/ (the O*NET portal page that links to several O*NET-related Web sites), and at http://www.doleta.gov/programs/
onet (Web site of the Employment and Training Administration at the U.S. Department of Labor).

2 For detailed information on these methods, see Sections B.2 and B.3.

3 For a discussion of the preferred data source, see Section A.1.3.

4 See Revision of O*NET data collection instruments, available at https://www.onetcenter.org/reports/Data_append.html.

5 The goal of the study was to compare the psychometric quality of incumbents’ Skills ratings with that of analysts’ Skills ratings across a large sample of O*NET-SOCs. Although some mean differences between incumbents’ and analysts’ ratings were observed, the results yielded only minimal differences between the two systems of obtaining Skills information.

6 For a description of the pretest, see Section B.5.

7 For a discussion of current and future efforts to improve response rates, see Section B.4.

8For additional information about sampling with the Establishment Method, see Section B.2.1.

9 For additional information about sampling with the OE Method, see Section B.2.2.

10 See Exhibit A-2 for the list of questionnaires, number of items and scales, and data sources.

11 See Section A.16.1 for a description of data cleaning and the identification and evaluation of anomalous cases.

12 For more information on the calculation of weights and variance estimates, see Section B.2.1. For information on the suppression of estimates, see Section A.16.1.

13 See Appendix D for the nonresponse analysis conducted for recent analysis cycles.

14 For a description of analysis processes, see Sections A.16.1 and A.16.2.

15 Bright Outlook occupations are those expected to grow rapidly in the next several years or those expected to have large numbers of job openings.

16 For a more detailed discussion of the Biemer et al. (2006) results, see Section B.5.

17 See Section A.1.5 for a discussion of O*NET’s response rate experience.

18 OMB Supporting Statement Part B includes a detailed description of the sampling methodology used to select establishments. For each set of occupations being studied, the establishment sample is released periodically over time as subwaves. The schedule of subwaves can vary depending on the difficulty of the occupations of interest, which causes fluctuations in the burden estimates for each year of the burden period.

19 OMB Supporting Statement Part B includes a detailed description of the sampling methodology used to select establishments. For each set of occupations being studied, the establishment sample is released periodically over time as subwaves. The schedule of subwaves can vary depending on the difficulty of completing data collection for the occupations of interest.

20 For details about the weighting and estimation procedures, see Section B.2.1.

21 RSE = the standard error of the mean divided by the mean.

22 O*NET suppression criteria are based on substantive expert recommendations (Peterson, Mumford, Borman, et al., 1997), best practices (Willenborg & De Waal, 1996), and other large government surveys (see Klein, Proctor, Boudreault, & Tuczyn, 2002).

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Loretta Bohn
File Modified	0000-00-00
File Created	2021-01-21

1205-0421 Supporting Statement Part A_FINAL 9.25.18