SA 108 Interim Procedure

Office of Nuclear Material Safety and Safeguards

Procedure Approval

Reviewing the Non-Common Performance Indicator, Sealed Source and Device Evaluation Program, Interim State Agreements (SA) Procedure

SA-108

Issue Date:

Review Date:

Michael C. Layton

Director, NMSS/MSST

Date:

Paul Michalak

Branch Chief, NMSS/MSST/SALB

Date:

Stephen Poy

Procedure Contact, NMSS/MSST/SALB

Date:

Terry Derstine

Organization of Agreement States, Chair

Date:

MLxxx

NOTE

Any changes to the procedure will be the responsibility of the NMSS Procedure Contact. Copies of NMSS procedures will be available through the NRC Web site at https://scp.nrc.gov.

1. INTRODUCTION

This document describes the procedure for conducting reviews of Agreement State and U.S. Nuclear Regulatory Commission (NRC) sealed source and device (SS&D) evaluation activities using the Non-Common Performance Indicator, Sealed Source Device Evaluation Program [NRC Management Directive (MD) 5.6, Integrated Materials Performance Evaluation Program (IMPEP).] Agreement States have the option of maintaining their own SS&D program. This option has been listed as a line item in the most recently signed Agreements.

2. OBJECTIVES

To verify the adequate implementation of the three sub-elements under this indicator - (a) Technical Staffing and Training, (b) Technical Quality of the Product Evaluation Program, and (c) Evaluation of Defects and Incidents Regarding SS&Ds.

III. BACKGROUND

Adequate technical evaluations of SS&D designs are essential to ensure that SS&Ds will maintain their integrity and that the design is adequate to protect public health and safety. *NUREG-1556, Volume 3, Consolidated Guidance About Materials Licenses: Applications for Sealed Source and Device Evaluation and Registration, provides information on conducting SS&D reviews and establishes useful guidance for review teams. Three sub elements, noted above, will be evaluated to determine if the SS&D program is satisfactory. Agreement States with authority for SS&D evaluation programs who are not performing SS&D reviews are required to commit in writing to having an SS&D evaluation program in place before performing evaluations.

4. ROLES AND RESPONSIBILITIES

1. Team Leader

1. In coordination with the IMPEP Program Manager, the Team Leader determines which team member is assigned lead review responsibility for this performance indicator.

2. Communicates the team’s findings to Program management and ensures that the team’s findings are in alignment with MD 5.6.

* When performing a review, use the latest version of this and all guidance material.

2. SS&D Reviewer

1. Selects documents for review for each of the three sub-elements(e.g., training records, SS&D evaluations, event reports); reviews relevant documentation; conducts staff discussions, and maintains a summary of the review for this indicator.

2. Coordinates the review of the indicator with other reviewers, if needed.

3. Informs the Team Leader of the team’s findings throughout the onsite review.

4. Presents the team’s findings to the Program at the staff exit meeting.

5. Completes their portion of the IMPEP report for the Sealed Source and Device Evaluation Program performance indicator.

6. Attends the Management Review Board meeting for the review and is prepared to discuss their findings (this can be done either in person or via teleconference).

4. GUIDANCE

1. Scope

This guidance applies to the three sub-elements to be reviewed under this indicator.

1. Evaluation of SS&D staffing and training should be conducted in a manner similar to, but not necessarily a part of, the Common Performance Indicator: Technical Staffing and Training, but focused on the training and experience necessary to conduct SS&D activities. The minimum qualifying criteria for SS&D staff authorized to sign registration certificates should be specified by the program and should be used in the review.

2. Review for adequacy, accuracy, completeness, clarity, specificity, and consistency of the technical quality of completed SS&D evaluations issued by the Agreement State or NRC.

3. Reviews of SS&D incidents should be conducted in a manner similar to, but not necessarily a part of, the Common Performance Indicator: Technical Quality of Incident and Allegation Activities, to detect possible manufacturing defects and the root causes of these incidents. The incidents should be evaluated to determine if other products may be affected by similar problems. Actions and notifications to Agreement States, NRC, and others should be conducted as specified in the Office of Nuclear Material Safety and Safeguards (NMSS) State Agreements (SA) Procedure SA-300, Reporting Material Events.

4. This guidance specifically excludes SS&D evaluations of non-Atomic Energy Act materials (e.g., naturally occurring radioactive material (NORM)).

B. Evaluation Procedures

1. The principle reviewer should refer to MD 5.6, Part II, Performance Indicators, and Part III, Evaluation Criteria, Non-Common Performance Indicator: Sealed Source and Device Evaluation Program, for the SS&D evaluation program criteria. These criteria should apply to program data for the entire review period. A finding of “satisfactory” is appropriate when a review demonstrates the presence of the following conditions:

   1. The SS&D program meets the criteria for a “satisfactory” finding for the performance indicator, Technical Staffing and Training, as described in Section III.B.1 of the MD 5.6 Directive Handbook.

   2. Procedures compatible with NMSS Procedure SA-108, “Reviewing the Non-Common Performance Indicator, Sealed Source and Device Evaluation Program,” are implemented and followed.

   3. Concurrence review of the technical reviewer's evaluation is performed by management or staff having proper qualifications and training.

   4. Product evaluations address health and safety issues; are thorough, complete, consistent, and of acceptable technical quality; and adequately address the integrity of the products under normal conditions of use and likely accident conditions.

   5. Registrations clearly summarize the product evaluation and provide license reviewers with adequate information in order to license possession and use of the product.

   6. Deficiency letters clearly state regulatory positions and are used at the proper time.

   7. Completed registration certificates, and the status of obsolete registration certificates, are clear and are promptly transmitted to the Agreement States, NRC, and others, as appropriate.

   8. The SS&D reviewers ensure that registrants have developed and implemented adequate quality assurance and control programs.

   9. There is a means for enforcing commitments made by registrants in their applications and referenced by the program in the registration certificates.

   10. There are no potentially significant health and safety issues identified from the review, that were linked to a specific product evaluation.

   11. The SS&D reviewers routinely evaluate the root causes of defects and incidents involving the devices subject to the SS&D program and take appropriate actions, including modifications of the SS&D sheets and notifications to the Agreement States, NRC, and others, as appropriate.

2. The minimum training and qualification requirements for reviewers should be documented and be in compliance with MD 5.6, Part II, Non-Common Performance Indicator: Technical Staffing and Training. The reviewer should determine whether the training and experience of all SS&D personnel meet these or equivalent requirements.

1. For NRC, SS&D training and qualification requirements are documented in NRC Inspection Manual Chapter (IMC) 1248, Formal Qualification Programs in the Nuclear Material Safety and Safeguards Program Area.

2. Agreement States should have established, documented training and qualification requirements that are either equivalent to NRC IMC 1248 or have implemented Appendix A of NMSS Procedure SA-103, Reviewing the Common Performance Indicator, Technical Staffing and Training.

3. All SS&D evaluations completed since the last IMPEP review are candidates for review.

3. The reviewer should select a representative sample based on the number and the type of evaluations performed during the review period. The selected sample should represent a cross-section of the Agreement State’s or NRC’s evaluations completed and include as many different reviewers and categories (e.g., new registrations, amendments, inactivations, or reactivations) as practical.

3. The reviewer should include any work performed on behalf of the program under review by others, (i.e., an Agreement State, NRC, or a contractor), to ensure the technical quality of the work. The reviewer should also ensure that any individuals performing work on a program’s behalf meet the program’s training and qualification requirements.

NOTE: Because the work is being performed at the discretion of the program under review, any weaknesses or deficiencies that the review team identifies will affect the appropriate sub-element rating(s) and could ultimately affect the overall indicator rating for the program under review.

3. If the initial review indicates an apparent weakness on the part of a reviewer(s), or problems with respect to one or more type(s) of SS&D or event evaluations, additional samples should be reviewed to determine the extent of the problem or to identify a systematic weakness. The findings, if any, should be documented in the report. If previous reviews indicated a programmatic weakness in a particular area, additional casework in that area should be evaluated to assure that the weakness has been addressed.

3. The reviewer should determine whether or not a backlog exists, based on the criteria established by the program, and if the backlog has any impact on health and safety.

3. The review of incidents involving SS&Ds should be conducted in accordance with the guidance provided in Section V of NMSS Procedure SA-105, Reviewing the Common Performance Indicator, Technical Quality of Incident and Allegation Activities.

3. For Agreement States, the reviewer should also determine if the program has received notification from the NRC about potential generic SS&D issues discovered during trend analysis of the Nuclear Material Events Database (NMED) events and identified in accordance with NRC in Policy and Procedure Letter 1.57, NMSS Generic Assessment Process. The reviewer would determine if such notifications had been received under this process; the effectiveness of the States response to these notifications; the adequacy of the response when compared to the actions that would be reasonably expected to be taken by other evaluation programs within the national program; Policy and Procedure Letter 1.57; and, the programs effort to notify Agreement States and NRC of the corrective actions by the issuance of a revised certificate.

3. In cases where an Agreement State may have SS&D evaluation authority but is not performing SS&D reviews, the reviewer should verify that the program has committed in writing to having an evaluation program, as described in Section (C)(2) of Part II, MD 5.6, in place before performing evaluations.

3. Review Guidelines

1. The response to questions relevant to this indicator in the IMPEP questionnaire should be used to focus the review.

2. The reviewer should be familiar with the latest revision of NUREG 1556, Vol 3, which provides guidance for SS&D evaluations.

3. Any issues identified in the last IMPEP review should be resolved in accordance with Section V.H.4, NMSS Procedure SA-100, Implementation of the Integrated Materials Performance Evaluation Program (IMPEP).

4. For SS&D evaluations, the reviewer should evaluate the following:

1. Technical correctness with regard to all aspects of evaluations. The checklist in the latest revision of NUREG 1556, Vol. 3, or equivalent document, may be used to verify the full range of considerations;

2. Completeness of applications and proper signature by an authorized official;

3. Records to document significant errors, omissions, deficiencies or missing information (e.g., documents, letters, file notes, and telephone conversation records). The decision making process, including any significant deficiencies related to health and safety is noted during the evaluation, and adequately documented in the records;

d. The adequacy of the limitations and/or other considerations of use;

5. The conduct of the concurrence review, as defined in the Glossary, MD 5.6;

5. Acceptance of variances or exceptions to industry standards in accordance with NUREG-1556, Vol. 3, or equivalent guidance.

5. Guidance, checklists, regulations, and policy memoranda to ensure consistency with current accepted practice, standards and guidance;

5. Appropriate use of signature authority for the registration certificates.

1. Thorough technical evaluations of the SS&D designs are essential to ensure that the SS&Ds will maintain their integrity and that the design is adequate to protect public health and safety. NUREG-1556, Volume 3, “Consolidated Guidance about Materials Licenses: Applications for Sealed Source and Device Evaluation and Registration” provides information on conducting the SS&D reviews and establishes useful guidance for IMPEP teams. Under this guidance, three sub-elements: Technical Staffing and Training, Technical Quality of the Product Evaluation Program, and Evaluation of Defects and Incidents Regarding SS&Ds, are evaluated to determine if the SS&D program is satisfactory. Agreement States with authority for SS&D evaluation programs that are not performing SS&D reviews are required to commit in writing to having an SS&D evaluation program in place before performing evaluations. The following sub-elements will be considered when determining if the SS&D evaluation program is adequate:

(i) Technical Staffing and Training

(1) Evaluation of the SS&D program staffing and training should be conducted in the same manner as the evaluation conducted with respect to Common Performance Indicator 1 (refer to Section II.B.1 of this handbook).

2. The SS&D program evaluation by the IMPEP review team will focus on training and experience commensurate with the conduct of the SS&D reviews as described in IMC 1248 or compatible Agreement State procedure.

   2. Technical Quality of the Product Evaluation Program

Adequate technical evaluations of the SS&D designs are essential to ensure that the SS&Ds used by both licensees and persons exempt from licensing will maintain their integrity and that the design features are adequate to protect public health and safety. The technical quality of the product evaluation program should be assessed by the IMPEP review team on the basis of an in-depth review of a representative cross-section of evaluations performed on various types of products and actions. To the extent possible, the review team should capture a representative cross-section of completed actions by each of the Agreement State or NRC SS&D reviewers.

2. Evaluation of Defects and Incidents Regarding SS&Ds

Reviews of the SS&D incidents should be conducted in the same manner as the evaluation conducted by the IMPEP review team with respect to Common Performance Indicator 5 (refer to Section II.B.5 of this handbook) to detect possible manufacturing defects and the root causes for these incidents. The incidents should be evaluated to determine if other products may be affected by similar problems. Appropriate action should be taken and notifications made to the Agreement States, NRC, and others, as appropriate, in a timely manner.

4. Review Information Summary

The summary maintained by the reviewer for preparation of the final report will include, at a minimum:

1. The applicants name;

2. The registration certificate number;

3. The type of action, e.g., new registration, amendment, inactivation, or reactivation;

4. The date of issuance;

5. SS&D Type;

6. Narrative of the comments if any.

The summary of review information does not appear in the final report. However, it is a good practice for the reviewer to maintain this information to support the reviewer’s presentation to the MRB.

5. Discussion of Findings with the Agreement States Radiation Control Programs or NRC

1. The IMPEP team should follow the guidance in SA-100, Implementation of the Integrated Materials Performance Evaluation Program (IMPEP), for discussions of technical findings with inspectors, supervisors, and management. If performance issues are identified by the reviewer(s) that lead to programmatic weaknesses, the reviewer(s) should seek to identify the root cause(s) of the issues which can be used as the basis for developing recommendations for corrective actions. As noted in Section II.A.3, SA-100 contains criteria regarding the development of recommendations by the IMPEP team.

1. In terms of general guidance for the IMPEP review team, a finding of "satisfactory" should be considered when none or only a few or small number of the cases or areas reviewed involve performance issues/deficiencies (e.g., inspection, licensing, staffing, etc.) ; an "unsatisfactory" finding should be considered when a majority or a large number of cases or areas reviewed involve performance issues/deficiencies, especially if they are chronic, programmatic, and/or of high-risk significance; and a finding of "satisfactory, but needs improvement" should be considered when more than a few or a small number of the cases or areas reviewed involve performance issues/deficiencies in high-risk-significant regulatory areas, but not to such an extent that the finding would be considered unsatisfactory.

4. APPENDICES

Appendix A – Examples of Less than Satisfactory Programs

4. REFERENCES

1. NMSS Procedure SA-100, Implementation of the Integrated Materials Performance Evaluation Program (IMPEP).

2. NMSS Procedure SA-103, Reviewing the Common Performance Indicator, Technical Staffing and Training.

3. NMSS Procedure SA-104, Reviewing the Common Performance Indicator, Technical Quality of Licensing Actions.

4. NMSS Procedure SA-105, Reviewing the Common Performance Indicator, Technical Quality of Incident and Allegation Activities.

5. NMSS Procedure SA-300, Reporting Material Events.

6. NRC Management Directive 5.6, Integrated Materials Performance Evaluation Program.

7. NUREG 1556 Volume 3, Rev. 1, Consolidated Guidance About Materials Licenses: Applications for Sealed Source and Device Evaluation and Registration.

8. Policy and Procedure Letter 1.57, NMSS Generic Assessment Process.

4. AGENCYWIDE DOCUMENTS ACCESS AND MANAGEMENT SYSTEM (ADAMS) REFERENCE DOCUMENTS

For knowledge management purposes, all previous revisions of this procedure, as well as associated correspondence with stakeholders, that have been entered into ADAMS are listed below.

No.	Date	Document Title/Description	Accession Number
1	2/27/04	STP-04-011, Opportunity to Comment on Draft STP Procedure SA-108	ML061640162
2	6/20/05	STP Procedure SA-108, Reviewing the Non-Common Performance Indicator, Sealed Source and Device Evaluation Program, Redline/Strikeout Version	ML061640169
3	6/20/05	Summary of Comments on SA-108	ML061640173
4	6/20/05	STP Procedure SA-108, Reviewing the Non-Common Performance Indicator, Sealed Source and Device Evaluation Program	ML040620291
5	6/30/05	STP-05-049, Final STP Procedure SA-108	ML051810473
6	7/14/09	FSME-09-051, Opportunity to Comment on Draft Revision of FSME Procedures SA-108 and SA-109	ML091330602
7	7/14/09	FSME Procedure SA-108, Draft Revision with tracked changes	ML091330103
8	1/22/10	Final FSME Procedure SA-108	ML092740005
9	1/22/19	FSME Procedure SA-108, Resolution of Comments	ML092740069
10	1/22/19	FSME Procedure SA-108, Draft Revision with tracked changes	ML092740014

Appendix A

EXAMPLES OF LESS THAN SATISFACTORY PROGRAMS

NOTES:

The effectiveness of a program is assessed through the evaluation of the criteria listed in Section III, Evaluation Criteria, of MD 5.6. These criteria are NOT intended to be exhaustive but provide a starting point for the IMPEP review team to evaluate this indicator. The review team should also take into consideration other relevant mitigating factors that may have an impact on the program’s performance under this performance indicator. The review team should consider a less than satisfactory finding when the identified performance issue(s) is/are programmatic in nature, and not isolated to one aspect, case, individual, etc. as applicable.

This list is not all inclusive and will be maintained and updated in the IMPEP Toolbox on the state communications portal website.

The following are examples of potential review findings that could result in a determination of a program being found “Satisfactory, but needs improvement” for this indicator.

TECHNICAL STAFFING AND TRAINING

1. The team found that the program did not have sufficient qualified staff to complete the SS&D reviews in a timely manner. The program had only one reviewer qualified to conduct the sealed source and device evaluations, and a qualified manager to conduct the concurrence reviews. The one qualified reviewer was also responsible for other activities and only had a limited amount of time to spend on the reviews. As a consequence, the reviews were not processed in a timely manner, and were rushed when they were performed. Even with these challenges, no health and safety issues were identified with the reviews. A second reviewer in training has not been fully qualified.

2. The team found that management was not effective in identifying and taking corrective actions to address with the shortage of SS&D review staff and ensuring that new staff were properly trained in performing the SS&D reviews.

3. During the review period, the number of qualified SS&D reviewers has decreased from x down to y. The program currently has enough qualified reviewers to handle the typical SS&D volume. If the program continues the trend of losing more review staff than they add, the SS&D review program may be adversely affected.

4. The program’s SS&D training program meets most of the criteria IMC 1248 and NMSS procedure SA-103 for SS&D reviewers. The training program was deficient/did not fully address [insert training areas] to meet the criteria of IMC 1248.

5. The program review staff had ## staff working on meeting the training and qualification program for SS&D reviewers during the review period. Out of the ## staff in the training and qualification program, ## did not complete the training according to the program’s agreed to timelines with the reviewer in training. Missing the stated timelines has caused deficiencies in the SS&D review program. The program stated that the cause for missing the timelines were [insert and describe].

6. During the review period, the program hired ## new technical review staff. The review team found that ## out of ## of the new staff did not have the scientific or technical backgrounds that would equip them to receive technical training related to the review of SS&D.

7. During the review period, the team found that ## out of the ## new staff trained for SS&D reviews, the program’s training and qualification standards did not meet personnel needs of the staff. Specifically, [insert and describe].

SEALED SOURCE DEVICE PROGRAM

8. The team found that ## out of the ## SS&D reviewers did not follow the review criteria established in the program’s procedures. The review criteria routinely missed includes [insert and describe]

9. The team reviewed ## SS&D evaluations during the review. The team found ## cases where the SS&D evaluations reviewed did not have a concurrence review performed by another reviewer/manager that is/was qualified to perform the concurrence reviews.

10. The team reviewed ## SS&D evaluations during the review. The team found ## cases where the SS&D evaluations reviewed did not address the integrity of the products and important health and safety concerns with respect to thoroughness, completeness, consistency, clarity, technical quality, adherence to existing guidance in product evaluations. Specifically, ## evaluations did not fully address [insert and describe]. Another ## evaluations did not fully address […]

11. The team found in ## out of the ## of evaluations reviewed, did not summarize the product evaluation and provide license reviewers with adequate information in order to license possession and use of the product.

12. The team found in ## out of the ## of evaluations reviewed, the deficiency letters did not state regulatory positions and are not always used at the proper time. Specifically, [insert and describe].

13. The team found ## out of the ## completed registration certificates, were not clear and promptly transmitted to the Agreement States, NRC, and others, as appropriate.

14. The team found that in ## out of the ## of evaluations reviewed, the product evaluations did not include an evaluation of proposed quality assurance and control program.

15. The team found that in ## out of the ## of evaluations reviewed, that the commitments made by registrants in their applications, and referenced in the registration certificates, cannot be enforced by the program.

16. The team found that in ## out of the ## of evaluations reviewed identified potentially significant health and safety issues linked to a specific product evaluation. Specifically, [identify and describe]

17. The program had ## events of defects and incidents of devices subject to the SS&D program. The program did not fully evaluate the root causes of all defects and incidents involving devices subject to the SS&D program. Specifically, [insert and describe].

18. The program had ## events of defects and incidents of devices subject to the SS&D program. The program did not take appropriate actions, including notifications to the Agreement States, NRC, and others, as appropriate, in ## of these events.

NOTE: This list is not all inclusive and will be maintained and updated in the IMPEP Toolbox on the state communications portal website.

The following are examples of potential review findings that could result in a determination of a program being found “Unsatisfactory” for this indicator.

TECHNICAL STAFFING AND TRAINING

1. The team found that the program did not have qualified staff to complete the SS&D reviews in a timely manner. The program had no qualified reviewers to conduct the sealed source and device evaluations. As a consequence, the reviews were not adequately performed or processed in a timely manner.

2. The team found that management did not attempt any corrective actions to address the lack of qualified reviewers.

3. During the review period, the number of qualified SS&D reviewers has decreased from x down to y. The program currently does not have enough has enough qualified reviewers to handle the typical SS&D volume. The SS&D review program has been adversely affected.

4. The program’s SS&D training program does not meet most of the criteria IMC 1248 and NMSS procedure SA-103 for SS&D reviewers. The training program was deficient/did not fully address [insert training areas] to meet the criteria of IMC 1248.

5. The program review staff had ## staff working on meeting the training and qualification program for SS&D reviewers during the review period. Out of the ## staff in the training and qualification program, ## did not complete the training according to the program’s agreed to timelines with the reviewer in training. Missing the stated timelines has caused deficiencies in the SS&D review program. The program stated that the cause for missing the timelines were [insert and describe].

6. During the review period, the program hired ## new technical review staff. The review team found that ## out of ## of the new staff did not have the scientific or technical backgrounds that would equip them to receive technical training related to the review of SS&D evaluations.

7. During the review period, the team found that ## out of the ## new staff trained for SS&D reviews, the program’s training and qualification standards did not meet personnel needs of the staff. Specifically, [insert and describe].

SEALED SOURCE DEVICE PROGRAM

8. The team found that ## out of the ## SS&D reviewers did not follow the review criteria established in the program’s procedures. The review criteria routinely missed includes [insert and describe]

9. The team reviewed ## SS&D evaluations during the review. The team found ## cases where the SS&D evaluations reviewed did not have a concurrence review performed by another reviewer/manager that is/was qualified to perform the concurrence reviews.

10. The team reviewed ## SS&D evaluations during the review. The team found ## cases where the SS&D evaluations reviewed did not address the integrity of the products and important health and safety concerns with respect to thoroughness, completeness, consistency, clarity, technical quality, adherence to existing guidance in product evaluations. Specifically, ## evaluations did not fully address [insert and describe]. Another ## evaluations did not fully address […]

11. The team found in ## out of the ## of evaluations reviewed, did not summarize the product evaluation and provide license reviewers with adequate information in order to license possession and use of the product.

12. The team found in ## out of the ## of evaluations reviewed, the deficiency letters did not state regulatory positions and are not always used at the proper time. Specifically, [insert and describe].

13. The team found ## out of the ## completed registration certificates, were not promptly transmitted to the NRC for posting on the National Sealed Source and Device Registry, as appropriate.

14. The team found that in ## out of the ## of evaluations reviewed identified potentially significant health and safety issues linked to a specific product evaluation. Specifically, [identify and describe]

15. The program had ## events of defects and incidents of devices subject to the SS&D program. The program did not fully evaluate the root causes of all defects and incidents involving devices subject to the SS&D program. Specifically, [insert and describe].

16. The program had ## events of defects and incidents of devices subject to the SS&D program. The program did not take appropriate actions, including notifications to the Agreement States, NRC, and others, as appropriate, in ## of these events.