NIST MEP Information Reporting - Memorandum for Panel Evaluations 3rd & 8th Years

January 16, 2018

MEMORANDUM TO: File

Through: Carroll Thomas,

Director, MEP

From: David Cranmer

Deputy Director, MEP

Subject: Process for Conducting Panel Evaluations of Centers in their
Third & Eighth Year of Operations

The MEP Statute, as amended by the American Innovation and Competitiveness Act (AICA, P.L. 114-329), at 15 U.S.C. § 278k(g)(1), requires the Secretary of Commerce to ensure that each Center is evaluated during its third & eighth year of operation by an evaluation panel appointed by the Secretary (Panel Evaluation) to determine whether the Center may continue to receive financial assistance from NIST for years 4-5 (9-10), of a Center’s cooperative agreement. Should the Panel Evaluation of the Center be positive, MEP will provide additional funding for the last two years of the cooperative agreement, which will be reviewed by the MEP Grants Officer for any pending financial issues or other matters of material non-compliance identified outside of the MEP evaluation processes that may adversely impact the funding. Should the evaluation of the Center be other than positive, the Center will be placed on probation in accordance with 15 U.S.C. § 278k(g)(5).

The Panel Evaluation of an MEP Center is carried out by a panel of private experts and Federal officials, is chaired by a NIST official. The evaluation is based on a recipient’s performance in administering its MEP Center cooperative agreement as measured against the evaluation criterion contained in 15 U.S.C. § 278k(g)(3). The MEP Director has established a process, described below, for conducting Panel Evaluations of MEP Centers, consistent with the relevant portions of the MEP statute, which at 15 U.S.C. § 278k(g) provides:

(g) Evaluations

(1) Third and eighth year evaluations by panel

(A) In general

The Secretary shall ensure that each Center is evaluated during its third and eighth years of operation by an evaluation panel appointed by the Secretary.

(B) Composition The Secretary shall ensure that each evaluation panel appointed under subparagraph (A) is composed of—

(i) private experts, none of whom are connected with the Center evaluated by the panel; and

(ii) Federal officials.

(C) Chairperson

For each evaluation panel appointed under subparagraph (B), the Secretary shall appoint a chairperson who is an official of the Institute.

(3) Performance measurement

In evaluating a Center … the Secretary … shall measure the performance of the Center against-

(A) the objective specified in subsection (c);

(B) the performance metrics under subsection (f)(2)(C); and

(C) such other criterion as considered appropriate by the Secretary.

The statute’s reference to subsection (c) is where the objective of the MEP Program is described, to wit:

(c) Objective

The objective of the Program shall be to enhance competitiveness, productivity, and technological performance in United States manufacturing through-

(1) the transfer of manufacturing technology and techniques developed at the Institute to Centers and, through them, to manufacturing companies throughout the United States;

(2) the participation of individuals from industry, institutions of higher education, State governments, other Federal agencies, and, when appropriate, the Institute in cooperative technology transfer activities;

(3) efforts to make new manufacturing technology and processes usable by United States-based small and medium-sized companies;

(4) the active dissemination of scientific, engineering, technical, and management information about manufacturing to industrial firms, including small and medium-sized manufacturing companies;

(5) the utilization, when appropriate, of the expertise and capability that exists in Federal agencies, other than the Institute, and federally-sponsored laboratories;

(6) the provision to community colleges and area career and technical education schools of information about the job skills needed in manufacturing companies, including small and medium-sized manufacturing businesses in the regions they serve;

(7) the promotion and expansion of certification systems offered through industry, associations, and local colleges when appropriate, including efforts such as facilitating training, supporting new or existing apprenticeships, and providing access to information and experts, to address workforce needs and skills gaps in order to assist small- and medium-sized manufacturing businesses; and

(8) the growth in employment and wages at United States-based small and medium-sized companies.

Subsection (f)(2)(C) of the MEP statute refers to the establishment of performance metrics for the MEP Center program, which are currently embodied in the Improving Manufacturing Productivity and Competitiveness Tracker (IMPACT). Additionally, there is currently no additional evaluation criteria identified by the Secretary or by the MEP Director pursuant to subsection (g)(3)(C) of the MEP statute.

Process:

In order to ensure sufficient time to process the new funding for the cooperative agreement, background preparation for the Panel evaluation process begins during the third quarter of a Center’s third & eighth year of operations and the evaluation should be completed no later than early in the fourth quarter of the Center’s third or eighth year of operations.

Panelists shall be chosen from among the 51 Center Directors of the MEP National Network. Senior staff and management of NIST MEP shall serve as Panel Chairs. The Evaluation Panel will typically consist of the Panel Chair and 3 Panelists, and may include other individuals as observers. Panelists and Panel Chairs shall receive training on conducting the evaluation from NIST MEP’s Panel Review Manager or designee.

NIST MEP is responsible for all scheduling associated with the Evaluation Panel.

Panelists shall evaluate the Center’s performance consistent with evaluation criterion set forth in 15 U.S.C. § 278k(g)(3) and reproduced above.

The documentation to be reviewed consists of the following:

1. The Center Performance and Profile Report.

2. Annual review reports for the Center’s 1^st and 2^nd (6^th and 7^th) years of operations. These reviews, conducted by the Regional Manager, Federal Program Officer and Grants Specialist, are interim reviews to document progress and identify issues regarding Center performance.

3. Responses to the Panel’s questions provided in advance of the review.

4. The Center’s Panel Review presentation, used the day of the review.

5. Such other information and documentation deemed relevant by the Evaluation Panel in conducting an evaluation of the MEP Center against the performance measurements identified in 15 U.S.C. § 278k(g)(3).

The evaluation itself will consist of the following steps:

1. The documentation is independently reviewed by the Panelists.

2. The Evaluation Panel will meet, typically via conference call, to evaluate the Center’s performance based on the evaluation criterion set forth in 15 U.S.C. § 278k(g)(3) (see also below crosswalk), which may include discussions focused upon the Center’s strategy, market understanding, business model, performance measurements, management, key personnel and organizational structure, financial sustainability, delivery of services and performance results. The discussion will include areas of observation, recommendation or deficiency.

3. As necessary, the Evaluation Panel will consult with the Center, Regional Management Team to get clarifications.

4. Following the review, the Panelists/Chair shall prepare and submit questions to the Center, requesting additional information or discussion of items noted during the review.

5. The Center shall provide responses to the questions and a presentation for use during the day of the review.

6. Following the evaluation, the Panelists shall provide their findings and observations to the Panel Chair, using a format/template provided by NIST MEP.

7. The Panel Chair will prepare a file memorandum memorializing the findings and recommendations resulting from the Panel Review, which will document commendations, observations, findings, programmatic recommendations and/or deficiencies. The MEP Director will review the file memorandum and make any necessary edits prior to finalizing the memorandum.

8. Letters/Notices

   1. If the overall evaluation is positive, the Panel Chair will draft an evaluation letter with commendations, observations and recommendations for transmittal to the Center, along with a notification that the Center is being recommended for continued funding. Such continued funding is subject to a pre-award compliance review by the NIST Grants Officer.

   2. If the overall evaluation is other than positive, the Panel Chair will draft a notice of probation, along with the noted deficiencies and required corrective actions, for transmittal to the Center, which will also include a notice to the Center that continued funding will be recommended during the period of probation, subject to a pre-award compliance review by the MEP Grants Officer.

   3. The MEP Center will be required to provide a Performance Improvement Plan identify steps to remediate the noted deficiencies.

9. For any of the items in 5 above, the MEP Director will review the draft and make edits or concur. In cases where a Center is being placed on probation, the MEP Director or delegate will consult with the NIST Grants Officer prior to finalizing the notice to the Center.

10. Upon concurrence, the final documents will be transmitted to the Center, Regional Management Team and Grants Management Division to take appropriate action. A copy will also be retained in the MEP file and provided to the Grants Management Division for inclusion in the Center’s official award file.

Crosswalk between Evaluation Criteria in 15 U.S.C. 278k(c) and Metrics in 278k(f)(2)(C)

For purposes of this process, the metrics identified in 15 U.S.C. 278k(f)(2)(c) are those included in NIST MEP’s IMPACT report. As of the date of this process document, those metrics include:

1. New Sales

2. Retained Sales

3. Jobs Created and Retained

4. New Investment

5. Cost Savings

6. Percent Improving Competitiveness

7. Survey Response Rate

8. Net Promotor Score

9. Manufacturing Clients/$M Federal

10. New Manufacturing Clients/$M Federal

The first eight are determined from the MEP Client Survey responses. The last two are based on Client/Project Submissions by each Center. A Center is awarded 10 points for each metric that meets or exceeds a defined threshold. If the result does not meet or exceed the threshold, no points are awarded for that metric.

It is not possible to provide a direct crosswalk from the individual criteria in 15 U.S.C. 278k(c) to specific Center results. Assuming a well-written project description, consistent from project to project and Center to Center, it might be possible to conduct an analysis that addresses k(c)(1) and k(c)(5). Such project descriptions, however, do not exist, nor would they address the remaining criteria in k(c). Determination of whether and how a Center is meeting each of the criteria comes from a reading of the documents provided for the review. Examples of how the criteria are met for each individual criterion, based on project work and/or language in various documents are shown below.

1. the transfer of manufacturing technology and techniques developed at the Institute to Centers and, through them, to manufacturing companies throughout the United States;

This criterion is met when the Center has demonstrated that it is delivering a variety of projects under the various service suites identified in MEIS¹. For purposes of this criterion, projects in the lean, quality, sustainability, technology services/product development, engineering services/plant layout, and information technology suites would fulfill this criterion.

2. the participation of individuals from industry, institutions of higher education, State governments, other Federal agencies, and, when appropriate, the Institute in cooperative technology transfer activities;

This criterion is met when the Center has demonstrated, through individual project work or partnerships involving one or more of the organizations. For project work, it is met when the Center has demonstrated that it is delivering a variety of projects under the various service suites identified in MEIS. For purposes of this criterion, projects in the sustainability, technology services/product development, engineering services/plant layout, and information technology suite, in conjunction with language in the Center’s CPPR, annual reports, responses to pre-Panel questions and/or the Center’s review day presentation, noting work with a particular company, university, college, state government agency or program, or other Federal agency would fulfill this criterion.

3. efforts to make new manufacturing technology and processes usable by United States-based small and medium-sized companies;

This criterion is met when the Center has demonstrated that it is delivering a variety of projects under the various service suites identified in MEIS. For purposes of this criterion, projects in the sustainability, technology services/product development, engineering services/plant layout, and information technology suites would fulfill this criterion. Language in the CPPR, annual reports, responses to pre-Panel questions and/or the Center’s review day presentation, noting work on such technologies or processes would fulfill this criterion.

4. the active dissemination of scientific, engineering, technical, and management information about manufacturing to industrial firms, including small and medium-sized manufacturing companies;

This criterion is met when the Center has demonstrated that it is delivering a variety of projects under the various service suites identified in MEIS. For purposes of this criterion, projects, seminars and/or workshops in lean, quality, growth, sustainability, strategic/business management/planning, technology services/product development, financial analysis/assistance services, sales/marketing/business development, engineering services/plant layout, information technology, and workforce suites would fulfill this criterion.

5. the utilization, when appropriate, of the expertise and capability that exists in Federal agencies, other than the Institute, and federally-sponsored laboratories;

This criterion is met when the Center has demonstrated that it has established relationships with non-NIST agencies and laboratories for their expertise. For purposes of this criterion, projects in the sustainability, technology services/product development, engineering services/plant layout, and information technology suites, in conjunction with language in the Center’s CPPR, annual reports, responses to pre-Panel questions and/or the Center’s review day presentation, noting work with other Federal agencies and federally-sponsored laboratories would fulfill this criterion.

6. the provision to community colleges and area career and technical education schools of information about the job skills needed in manufacturing companies, including small and medium-sized manufacturing businesses in the regions they serve;

This criterion is met when the Center has demonstrated that it is engaged in workforce development programs in concert with the State Governor’s office or agency, community colleges, or area career and technical education schools. Such language should be present in the Center’s CPPR, annual reports, responses to pre-Panel questions and/or the Center’s review day presentation.

7. the promotion and expansion of certification systems offered through industry, associations, and local colleges when appropriate, including efforts such as facilitating training, supporting new or existing apprenticeships, and providing access to information and experts, to address workforce needs and skills gaps in order to assist small- and medium-sized manufacturing businesses; and

This criterion is met when the Center has demonstrated that it is engaged in apprenticeship and/or training/education certification programs. Such language should be present in the Center’s CPPR, annual reports, responses to pre-Panel questions and/or the Center’s review day presentation.

8. the growth in employment and wages at United States-based small and medium-sized companies

This criterion is met for growth in employment when the Center meets or exceeds the IMPACT threshold for Jobs Created and Retained.

1 Current suites as of the date of this policy are described in MEP’s Reporting Guidelines, 1 January 2016. The current suites include lean, quality, growth, sustainability, strategic/business management/planning, technology services/product development, financial analysis/assistance services, sales/marketing/business development, engineering services/plant layout, information technology, and workforce.