0990 Kidney X SSB_comments removed

Evaluation of the Kidney Innovation Accelerator (KidneyX)

Supporting Statement – Section B

Submitted – May 31, 2019

Program Officer/Project Officer

Sandeep Patel, Ph.D. | Open Innovation Manager

U.S. Department of Health & Human Services (HHS)

Office of the Chief Technology Officer (CTO)

Address:   200 Independence Avenue S.W. | Washington, D.C. 20201 | Rm. 624D.2

Telephone: 202-205-0355

Fax: 202-690-6772

Email: [email protected]

Part B: Collections of Information Employing Statistical Methods

1. Respondent Universe and Sample Selection Methods

1. Interview Respondents and Sampling

For the interviews conducted as part of this information collection, we will solicit views from a small number of individuals selected from KidneyX prizewinners and non-prizewinner applicants, and people knowledgeable about KidneyX. Due to resource constraints, each set of interviewees will be a convenience sample. We plan to ensure that each sample includes respondents with diverse technical backgrounds, proposed innovations, organizational affiliations, professional experiences, and perspectives on KidneyX. The selection of respondents is designed to elicit a range of opinions and experiences that provide a rich dataset for our qualitative analysis.

Applicant Interviews: We will interview up to 12 KidneyX competition applicants in each year. In years with more than one competition, respondents will be drawn from each competition held.

Non-awardee Interviews: We will interview up to six non-awardee applicants each year. These respondents will be identified after the public announcement of awards, based on information provided during the application process.

Awardee Interviews: We will interview up to six awardees each year. These respondents will be identified after the public announcement of awards, based on information provided during the application process.

Other Stakeholder Interviews: We will interview up to six non-applicant stakeholders each year, including some non-government partners such as Kidney Health Initiative (KHI) leadership, American Society of Nephrology (ASN) executives, and patient advocates. We will work with HHS to identify appropriate stakeholders to participate in these interviews. As feasible, we will interview at least three of the same stakeholder representatives during each annual interview cycles to gain their observations on changes to KidneyX over time, while the other three respondents will be selected based on the stakeholders engaged in KidneyX during that year.

The table below provides an overview of the annual number of respondents for each interview type.

Respondent Type	Annual Number of Respondents
Applicants	12
Awardees	6
Non-awardees	6
Other Stakeholders	6
Total	30

2. Survey Respondents and Sampling

We will deploy two surveys that coincide with each competition held by KidneyX. At present, KidneyX is planning to conduct one or two competitions per year. Due to the specialized nature of the KidneyX competitions, we plan to conduct census surveys of each cohort of applicants as we expect there will be fewer than 500 applicants for any given prize competition, based on rough estimates for the size of the community of innovators involved in kidney technology.

Pre-award Survey: We will survey the entire population of KidneyX applicants upon the closure of the application deadlines for prize competitions. We will target all participants because of the high risks associated with obtaining a small, unrepresentative sample that could be caused by a small pool of applicants, high rates of non-response, or a combination of the two. The table below provides more detailed figures of the expected number of respondents to the Pre-award Survey, using the number of submissions to the first KidneyX competition as a guide.

Respondent Type	Number of Respondents
Applicants	300
Total	300

Post-award Survey: We will survey the entire population of winners and non-awardees 6 months after the prize competition announcements have been made. We will target all participants because of the high risks associated with obtaining a small, unrepresentative sample that could be caused by a small pool of applicants, high rates of non-response, or a combination of the two. The table below provides more detailed figures of the expected number of respondents to the Post-award Survey, using the number of submissions to the first KidneyX competition as a guide.

Respondent Type	Number of Respondents
Non-awardee Applicants	280
Awardees	20
Total	300

2. Procedures for Collection of Information

Our evaluation integrates a mixed-methods approach including surveys and interviews for data collection.

1. Interview Procedures

We will conduct 45-minute telephone interviews with a subset of competition applicants, awardees, and non-awardees and with other stakeholders in each year of KidneyX competitions. Potential interview respondents will receive an initial introductory e-mail from HHS notifying them that they have been selected to take part in an interview followed by a request from the evaluation team seeking their availability to schedule the 45-minute interview. Scheduled interviewees will receive a reminder e-mail 2 days before their interview and a thank you e-mail sent within 2 days of the completed interview.

The semistructured telephone interview instruments will include open-ended questions (e.g., “In what ways (beyond funding) has becoming a KidneyX innovator been beneficial to your organization?”) and will be tailored for respondent type (see Attachments C, D, E, and F).

2. Survey Procedures

We will administer an online pre-award survey to all applicants of each KidneyX prize competition upon reaching the application deadline for the competition. Applicants will receive an e-mail describing the nature, intent of use, security, and confidentiality of survey data collection. Applicants will be able to follow the link to respond to the survey online.

We will administer an online post-award survey to winners and non-awardees of each KidneyX prize competition 6 months after the prize announcements have been made for the competition. Winning and non-winning applicants will receive an e-mail describing the nature, intent of use, security, and confidentiality of survey data collection. Participants will be able to follow the link to respond to the survey online.

Both quantitative and qualitative analyses will be performed. Quantitative analyses will involve descriptive statistics, cross tabulations, and potentially correlations and regression techniques. The appropriate method may depend on the evaluation question being investigated and, ultimately, the amount and type of data available. Qualitative thematic analyses will be conducted on open-ended survey questions and interview data. Staff trained in the appropriate qualitative and/or quantitative research methods will conduct all analyses. Information will be stored on a secure shared drive with access limited to project team members.

3. Methods to maximize response rates and to deal with nonresponse

Applicants to KidneyX prize competitions are not mandated to participate in data collection for the purposes of this evaluation. As such, we have implemented the following approaches to maximize response rates and deal with nonresponse.

Telephone interviews are being employed to collect qualitative data without the costs and respondent burden associated with face-to-face interviews. Further, interview guides were developed to keep discussions at 45 minutes to minimize burden and encourage participation. We will use information gathered during the prize competition submission process and consultations with HHS to select appropriate interviewees, and document each interviewee’s consent to participate in each interview.

The Web-based mode of survey was selected to minimize burden. The surveys will take the respondent approximately 20 minutes to complete, and the surveys will be accessible across multiple browsers (e.g., Internet Explorer, Google Chrome, Mozilla Firefox) and devices (e.g., smartphones, tablets, laptops). The Post-award Survey will be conducted 6 months after the prizes are announced for a given competition, perhaps providing less motivation to participate in this second survey than the first, especially for non-awardees. Therefore, we plan to survey the entire population of applicants in both the Pre- and Post-award Surveys to maximize our sample size.

To maximize response rates, procedures will be put in place to coordinate closely with HHS and to communicate timely, appropriate reminders emphasizing the importance of applicants’ participation in the surveys. For participants who do not complete the Web-based survey within 1 week of the initial invitation, we will send an e-mail reminder 1 week after the initial invitation. Similarly, for participants who have not completed the survey within 2 weeks of the initial invitation, we will send an e-mail reminder 2 weeks after the initial invitation. A final e-mail reminder will be sent 3 weeks after the initial invitation to those that have not yet completed the survey. E-mails will come from a recognizable e-mail address such as [email protected] so that respondents are more likely to open the e-mail. The surveys will be closed upon reaching 4 weeks after the initial invitation. These procedures will help maximize the response rate.

For the Pre-Award Survey, we anticipate that interest in the survey will be high as it will be conducted shortly after the applicants submitted their entries. We will attempt to reach a response rate of at least 50 percent for each cohort. The survey invitations will encourage respondents to participate in order to improve the KidneyX initiative and any future competitions (including those in which the respondents may subsequently compete). We will send additional reminders via e-mail to non-respondents at two-week intervals from the launch of the survey until just before the survey close date. If the response rate falls substantially short of our goal, we will send personalized invitations to selected non-respondents to encourage participation, focusing on those from groups under-represented in the pool of respondents based on team structure, type of affiliated organization, technology area, and other salient characteristics. We will use data from the applicants’ submissions to identify any systematic patterns among non-respondents that might introduce bias into our results.

For the Post-Award Survey, we will attempt to achieve a response rate of at least 90 percent of the prize winners. Since each prizewinner will be surveyed shortly after receiving their awards, we anticipate that interest in responding will be high. In contrast, non-awardees may feel less compelled to respond to this survey due to the negative result from their entries. We will seek to achieve a 25 percent response rate from non-awardees in each cohort. The survey invitations will encourage respondents to participate in order to improve the KidneyX initiative and any future competitions (including those in which the respondents may subsequently compete). We will send additional reminders via e-mail to non-respondents at two-week intervals from the launch of the survey until just before the survey close date. If the response rate falls substantially short of our goal, we will send personalized invitations to selected non-respondents to encourage participation, focusing on those from groups under-represented in the pool of respondents based on team structure, type of affiliated organization, technology area, and other salient characteristics. We will use data from the applicants’ submissions to identify any systematic patterns among non-respondents that might introduce bias into our results.

For surveys where we detect a systematic pattern among non-respondents that could introduce bias, we will calculate appropriate weights to apply to specific items that are designed to mitigate that bias.

4. Tests of procedures or methods to be undertaken

All instruments and protocols for this study are similar to those used without difficulty in prior evaluation work by the study team. These prior evaluations include studies to evaluate federal programs to encourage and accelerate innovation in specific domains of biomedical science and technology. For example, members of the RTI team conducted interviews and surveys for the Combatting Anti-microbial Resistant Bacteria Accelerator (CARB-X) program sponsored in part by the Biomedical Advanced Research Project Authority (BARDA) within HHS. The team members also deployed similar instruments and protocols for the monitoring and evaluation of the NIH Research Evaluation and Commercialization Hub (REACH) program.

We will consult with members of the KidneyX Steering Committee, including experts in kidney technology, to confirm that the survey instrument items and interview questions are appropriate and are likely to be interpreted consistently by respondents. The interview guides and survey questions have also been informed by preliminary discussions with a few KidneyX stakeholders, and informal input from some winners of the Redesign Dialysis Phase 1 competition.

Prior to launching each survey, we will conduct thorough testing of the programmed survey to ensure accuracy and fidelity, including inclusion/exclusion logic; and to ensure that instructions, questions, and response options display correctly and that skip patterns function. We do not anticipate that the results of the testing will require any substantive changes to the instruments.

5. Consultants on statistical aspects of the design and people who will collect and analyze the information

In September 2018, HHS awarded contract HHSP233201500039I to RTI International to design and conduct an evaluation of KidneyX. The study team includes Jeffrey Alexander, Katherine Treiman, Cindy Soloe, Ben Anderson, and Valerie Etta of RTI International. The study team designed the evaluation in conjunction with the HHS KidneyX advisory committee. The evaluation will be conducted by RTI International. Statistical analysis will be conducted by team members with graduate degrees in public health, economics, and management, all of whom have received substantial formal training in research methods and statistics. The final study results will also be reviewed by internal RTI experts in statistical methods prior to submission. Drafts of all study results will be reviewed by appropriate experts within HHS and the KidneyX initiative prior to final publication.