OMB Approval Memo

FDA DOCUMENTATION FOR THE GENERIC CLEARANCE

OF REQUEST FOR DATA TO SUPPORT SOCIAL AND BEHAVIORAL RESEARCH (0910-0847)

TITLE OF INFORMATION COLLECTION: Collection of Data to Support “Training Decay for Medical Products: Insulin Pump Usability Study”

(referred to as “Insulin Pump Usability Study” in the Informed Consent Form for participants)

DESCRIPTION OF THIS SPECIFIC COLLECTION

1. Statement of Need:

The Food and Drug Administration (FDA) requires that medical devices undergo usability validation testing to assess and reduce risks associated with device use. Intended users undergo a training decay period prior to validation testing to simulate use scenarios and better identify critical use errors. Training decay refers to loss of skill or information over time, which may increase the occurrence of use error.

The research aims to improve and streamline critical use error identification when performing usability testing by identifying reliable and replicable training decay behavior. Identifying generalizable training decay curves could standardize the methods for conducting usability testing for medical products and ultimately improve use error identification, while avoiding an undue toll on manufacturer resources and delays in providing life-improving, innovative products to patients.

The research questions of interest are:

• Can detailed training decay curves be identified?

• Does the difficulty of a task influence memory retention?

• What are the effects of accelerated training decay?

• Do different types of tasks have different training decay curve profiles?

• What are the differences in decay between task types?

FDA has contracted with UserWise, Inc. to conduct training decay research. They will conduct usability validation testing for medical products and supports FDA submissions of medical products on behalf of manufacturers. This research will evaluate the importance of training decay length in evaluating the use-safety of medical devices.

2. Intended Use of Information:

The contractor will analyze data from the participants to assess use errors, close calls, and difficulties with use of the device (insulin pump). They will use changes in error rates to calculate training decay curves by task types, including psychomotor, task memory, and cognitive tasks.

3. Description of Respondents:

Participants will be recruited through social media, phone, email, and in-person correspondence and will be included if they meet the following inclusion criteria:

• 18 years and older

• Is not pregnant and does not expect to be pregnant soon

• Can read and comprehend written and spoken English

• Currently lives in the United States

• Has not been a licensed healthcare provider or received formal medical training beyond CPR/first aid

• Does not have a severe phobia of needles. (Needles will be present and used during the study sessions. The presence of needles and potential risks are disclosed in the Informed Consent Form and in the screener.) 

• Does not have experience injecting insulin into themselves or others

• Has not used or previously interacted with an insulin pump

• Does not have any broken bone or other physical impairment that would prevent them from safely using a medical device

• Does not suffer from a severe mental, physical, or psychological disability that would present a significant barrier to using a medical device; examples are paralysis, Down’s syndrome, or a severe learning disability - to avoid bias caused by testing users with unrepresentative conditions

Participants will have minimal prior clinical training and no prior experience with the insulin pump being assessed. The contractor will recruit up to 150 participants for the study. Participants in the study may be grouped for analysis, based on their demographics. For each group, they will ensure a reasonable degree of geographic and demographic diversity, including literacy, race and ethnicity, and age.

There will be several groupings of participants, reference the table below for a list of all study participants. One group of participants, Cohorts 2-5, will participate in a training session for 60 minutes and may be invited to participate in another 60-minute study session. The other group of participants, Cohort 1, will participate in a 90-minute study session (without a training session). For the first group, eligibility to participate in the second study session will be disclosed upfront and will not be dependent on performance. For those participating in the training and follow up study session, they estimate the total expected duration of participation as up to approximately 2 hours (e.g., two 60-minute sessions), with a break in between that may vary from 1 hour to 1 week.

Table 1. Description of Study Participants by Cohort

Cohort	Training Decay	Minimum Number of Participants per Cohort	IRB Approval Conditions
1	No Training and no decay	N = 25	IRB Approval Condition 3
2	No Training Decay (0m + 10m/-0m)	N = 25	IRB Approval Condition 1
3	Training Decay = 1 hour (1h +9m/-0m)	N = 25	IRB Approval Condition 2
4	Training Decay = 1 day (24h +/- 3.6h)	N = 25	IRB Approval Condition 2
5	Training Decay = 7 days (168h +/- 25.2h)	N = 25	IRB Approval Condition 2
Minimum Total Number of Participants:		N = 125 (+20% = 150)

4. Date(s) to Be Conducted:

The Contractor will conduct a full study between December 1, 2021 and February 25, 2022.

5. How the Information Is Being Collected:

The study will include trained and untrained participants. Untrained participants will participate in one usability study session in which the contractor will observe and record their performance. Trained participants will undergo a single training session, after which a training decay period will begin (Attachment A, Initial Training Script). Training decay periods include 0-hour, 1 hour, 1 day, and 7 days. After the training decay period, trained participants will return for a usability study session in which the contractor team will observe and record their performance.

During each observed performance session, up to three study personnel will be present. Typically, a moderator walks the participant through the session and focuses on asking follow-up questions to better understand the root cause of use-related issues. An observer observes the interaction and records all data related to behaviors, follow-up questions from the participants, and participant-reported root causes in the moderator’s script (Attachment B, Moderator’s Script). Additionally, the participants will fill in an end of study questionnaire for collection of additional information on demographics (Attachment C, End of Study Questionnaire). The purpose of questions 1 through 7 in the end of session questionnaire is to use a validated tool to assess the health literacy of each participant. Please note that participants are told the study is about “usability” rather than about “training decay” to prevent participant response bias.

6. Confidentiality of Respondents:

The contractor will record personal details of each participant (i.e., name and address) solely for the purpose of providing informed consent, payment, and collecting initial screening information. Once the testing is initiated, they will use a numbering system to protect the identity of each participant and to remove the link between personal details and test data.

At no stage during the recruitment process will the recruiters seek to gain access to any medical records for any participant. The study team will keep personal details and will not circulate them to any third party.

All study personnel interacting with human subjects as part of this research will have completed training on the protection of human subjects.

The contractor will require that all participants sign a consent form prior to starting the study. The consent form is designed to inform the potential participant about the purpose of the study, how the study will be conducted, any risks or benefits that may impact them as a participant, and their rights (Attachment D, Informed Consent Form). In addition, the consent form documents that the participant understands the contents of the consent form and voluntarily agrees to participate in the study.

7. Amount and Justification for Any Proposed Incentive:

Participants will receive a token of appreciation for their opinions. The amounts were calculated using current industry best practice and are kept as low as reasonably possible to avoid undue pressure to participate. Participants will receive a token of appreciation for participating in the full study. The total token of appreciation for participating in the full study will be up to $80, comprised of:

• $40 for the training session (first 60-minute session);

• $40 for the usability session for trained participants (60-minute session); and

• $75 for the usability session for untrained participants (90-minute session).

This research study requires participants to return after a period of time to assess the level of training decay that may have occurred. The contractor and FDA determined the value of the token of appreciation for this group by considering that participants must return for the usability session so that the contractor may assess how much their skills have decreased over time. For example, if the token of appreciation is not adequate, participants may elect to participate in the training session and not return for the usability session. Low participation may result in inadequate data collection or loss of government funds associated with moderator and observer time. Additionally, low participation can cause delays in launching the research study, both of which may lead to increased cost.

8. Questions of a Sensitive Nature:

The contractor will not ask questions that are sensitive in nature. Participants will be allowed to abstain from answering if they are uncomfortable with any questions asked during the usability study. Additionally, participants will be notified at the beginning of the study session that they are welcome to end the session and leave at any time for any reason.

9. Description of Statistical Methods:

Use errors and difficulties will be compared between each participant’s baseline session and the second session to help quantify the level of retained knowledge and allow for comparison between lengths of training decay and participant performance. In addition, use errors and difficulties will be assessed and compared between task types to inform the methodology for the summative usability study.

Each participant will have two observations: one for their baseline assessment and one for their follow-up assessment. Timepoint will be entered as binary variables (baseline versus follow-up) as well as decay length (e.g., one hour versus one day) and task type as categorical variables.

If deemed appropriate, the contractor will also include in the analysis other collected data, such as demographics or baseline cognitive measures.

Task success can be quantified in two ways. One will be the percentage of task success rate for a given cohort (e.g., 4/10 participants successfully located the correct injection site). The contractor will analyze the data more categorically by grading each task completed as a “success,” or “failure,” or “difficulty.” This method is more in line with the usual reporting of usability studies and will allow application of different levels of rigor when quantifying decay curves.

The contractor will quantify the unique training curve for each task type by performing the same pairwise comparisons between timepoint groups and each task. The contractor will assess resulting data to determine if there are correlations between tasks that are similar in nature (e.g., Fact Memory).  For these comparisons, the expectation is that there will be no significant difference in the baseline assessments, suggesting that the task types are equally difficult at first.

One crucial additional analysis is examining the relationship between each task’s difficulty and its training decay curve.

Table 2. BURDEN HOUR COMPUTATION: (Number of responses (X) estimated response or participation time in minutes (/60) = annual burden hours):

Type/Category of Respondent	No. of Respondents	Participation Time (minutes)	Burden (hours)
Training and Usability Study Session After Training Decay	120	120 minutes	240
Usability Study Only Cohort	30	90 minutes	45
Total	150	120 or 90 minutes depending on group	285

REQUESTED APPROVAL DATE: November 2021

NAME OF PRA ANALYST & PROGRAM CONTACT:

Ila S. Mizrachi

Paperwork Reduction Act Staff

[email protected]

301-796-7726

George Neyarapally

Office of Surveillance and Epidemiology

[email protected]

7