Non-Sub Memo

DEPARTMENT OF HEALTH & HUMAN SERVICES
National Institutes of Health

DATE:

April 11, 2016

TO:

Stephanie Mok
Office of Management and Budget (OMB)
Reports Clearance Officer, DHHS

FROM:

Daniel Hobbs, MS
Management Analyst
Epidemiology & Biostatistics Section
Rehabilitation Medicine Department
NIH Clinical Research Center

THROUGH:

Elizabeth K. Rasch, PhD, PT
Staff Scientist and Chief, Epidemiology & Biostatistics
Rehabilitation Medicine Department
NIH Clinical Research Center
Mikia Currie
Office of Policy for Extramural Research Administration (OPERA)
Project Clearance Branch
National Institutes of Health

SUBJECT:

Non-Substantive Change Request to Currently Approved Study
(OMB # 0925-0704; Expiration Date 10/31/2017)
Request for Approval of Phases 2 & 3

The National Institutes of Health’s Department of Rehabilitation Medicine (NIH-RMD), in conjunction with Boston
University Health and Disability Research Institute (BU-HDR), are working together to develop a new testing method
that has the potential to precisely measure outcomes across the full continuum of human functioning within the SSA
disability system. Through an ongoing collaboration with the SSA, the NIH developed “Functional Assessment
Batteries” (FAB), which utilize Item Response Theory- Computer Adapting Testing (IRT- CAT) methodology. The
demonstrated advantages of IRT-CAT instruments compared to more traditional testing methods are: reduced
respondent burden, increased score precision, elimination of ceiling and floor effects, client-specific confidence intervals,
monitoring of data in real time and lower data collection costs. These are likely important advantages in high volume
assessment scenarios like in an SSA disability determination context. Difficult choices are faced between meeting the
need for an assessment that is comprehensive in its breadth, precise in its estimates and reflects the complex
dimensionality of functional assessment while meeting legitimate concerns over assessment length, administration time
and respondent burden. To solve this dilemma, this project is exploring the utility of applying IRT scale technology to
initiate a transformation in the approach to assessing function as part of the SSA disability determination process.
Per OMB instruction provided in the Notice of Action issued for this project, NIH is requesting a non-substantive
change to conduct phases 2 and 3 of this iterative research study. While initially planned as separate, sequential study
activities, conducting both phases simultaneously will substantially reduce overall burden to respondents. Additionally,
by conducting both phases concurrently, cost to the federal government and taxpayer will be reduced.
As outlined in the approved application titled: “The Social Security Administration (SSA)- National Institutes of Health
(NIH) Collaboration to Improve the Disability Determination Process: Calibration II and Predictive Validity Testing of
Item Response Theory- Computer Adaptive Testing Tools (IRT-CAT),” phases 2 and 3 consist of measuring
performance of these tools against existing gold-standard legacy measures (validation testing), and evaluating the
consistency of scoring across two administrations (reliability testing).
Initial estimates projected the need to contact 1,000 volunteer participants (500 normative sample), each subject to 1.5
hours of burden for phase 2, and 400 volunteer participants, contacted twice, each subject to .5 hours of burden (total)

for phase 3 of this research. Initial estimates for both phases conducted separately projected contact with 1,400
participants (with 400 contacted twice), with a total burden of 1,700 hours. By combining both phases, statistically
sufficient data can be captured to measure both validity and reliability of the instruments from contact with only 1,005
participants, with only 750 of these participants contacted twice, for a total burden of 626.25 hours (~63% reduction in
burden by combining both phases). Recruitment numbers and burden estimates for this consolidated approach can be
found in the attached burden table (Attachment 1).
These combined phases of study and this research approach received Institutional Review Board approval and (pending
OMB approval) will be conduced by the research survey firm YouGov Polimetrix, who maintains a pool of 1.5 million
volunteer survey respondents and uses sample matching algorithms they pioneered to accurately compile a normative
sample of the US population for comparison and calibration purposes. This approach was utilized in the OMBapproved phase 1 of this iterative research.
This approach to conducting phases 2 and 3 simultaneously involves three samples of 335 respondents. The first will
serve as a normative sample for basis of comparison with disabled populations completing the IRT-CAT instruments.
The second and third sample will each consist of 335 self-identified work disabled participants (one sample alleging
primary physical disability, the other alleging primary mental disability). Each of these samples will complete either the
“Learning & Applying Knowledge” or “Daily Activities” IRT-CAT instruments. A complete list of potential items
(questions) is included with this request (Attachment 2). It is important to note, as a key feature of IRT-CAT, each
participant will only respond to a small number of these items; Item Response Theory algorithms minimize the need to
ask every question, instead only selecting the most relevant and necessary questions to accurately assess function in these
domains. It is estimated that burden during first contact is 15 minutes (0.25 hours) for each respondent. YouGov will
administer the IRT-CAT instruments via their survey collection website. OMB control number and required
information will be posted on the initial welcome screen. This request includes an example screenshot from a previously
approved IRT-CAT study conducted during initial development of different modules (Attachment 3). This
demonstrates how the welcome screen will appear to respondents, once the instruments are uploaded and programed at
YouGov (following OMB approval).
Accounting for attrition, 250 individuals in each sample will be contacted a second time, and administered the same IRTCAT instrument. Once 250 participants are contacted for follow up, recruitment will cease. This second contact will
serve to test the consistency in scoring precision of the IRT-CAT instruments. During this second contact, the
respondents will also complete questions from the following functional assessment legacy instruments: PM-PAC
(Participation Measure for Post-Acute Care), Independent Living Skills Survey Subscales, VR PF-10 (Veterans RAND
Physical Functioning 10 question instrument), La Trobe Communication Questionnaire, BASIS 24 (Behavior and
Symptom Identification Scale; omitting 4 questions), and the AM-PAC (Activity Measure for Post-Acute Care, Applied
Cognition Short Form only). Administration of these legacy instruments will enable the research team and stakeholders
to better assess the utility of IRT-CAT instruments in measuring function. It is estimated each respondent will incur 30
minutes of burden (0.5 hours) for this follow-up contact. A complete list of questions from these legacy instruments is
provided in this request (Attachment 4). These items will also be administered through the YouGov website to their
volunteer survey responders.
While burden is not calculated for the consent document or invitation email (neither requiring response nor significant
time to review), we have included them for OMB review in this request (Attachments 5 and 6).
Funding for all phases of this project has already been awarded via a non-severable contract with the Boston University
Health and Disability Research Institute; no additional funding is required to support these phases of research. The
contract period of performance end date is August 16th, 2016, and all study related activities, including manuscript
development, must be completed by this time.
We remain sincerely appreciative of OMB’s ongoing support of this work, and have welcomed the opportunities to
discuss this collaboration with members of your organization on several occasions. Should your reviewers have any
questions or concerns about these phases of the research, please direct them to Daniel Hobbs ([email protected];
301-496-3817).

Attachments:
1.

Burden Table for Phases 2 & 3

2.

Complete List of Potential IRT-CAT Instrument Questions

3.

Screenshot of Survey Format, Containing OMB Control Number and Information

4.

Complete List of Legacy Validation Questions

5.

Study Consent Document

6.

Participation Invitation Email from YouGov