ROCIS Ready Supporting Statement

OMB SUPPORTING STATEMENT
for

Targeted Evaluations of State
Vocational Rehabilitation Agency Practices

U. S. Department of Education
Office of Special Education and Rehabilitative Services (OSERS)
Rehabilitation Services Administration (RSA)

Revised
November 12, 2008

Prepared by:
RTI International

Table of Contents
Note for Reviewers ......................................................................................................................... 1
A. Justification ............................................................................................................................... 2
1. Importance of the Information ......................................................................................... 2
2. Purposes and Uses of the Data......................................................................................... 8
3. Information Technology .................................................................................................. 8
4. Efforts to Identify Duplication......................................................................................... 8
5. Methods Used to Minimize Burden on Small Entities .................................................... 8
6. Consequences of Not Collecting the Information............................................................ 8
7. Special Circumstances ..................................................................................................... 8
8. Consultation Outside the Agency .................................................................................... 9
9. Payments or Gifts to Respondents ................................................................................... 9
10. Assurances of Confidentiality.......................................................................................... 9
11. Sensitive Questions.......................................................................................................... 9
12. Estimate of Response Burden .......................................................................................... 9
13. Estimate of Cost Burden of Collecting Information ........................................................ 9
14. Estimate of Annualized Cost to the Federal Government ............................................. 10
15. Change in Burden .......................................................................................................... 10
16. Tabulation and Analysis Plan and Schedule.................................................................. 10
17. Display Expiration Date for OMB Approval................................................................. 10
18. Exceptions to Certification Statement ........................................................................... 10
B. Statistical Methods .................................................................................................................. 10
1. Respondent Universe ..................................................................................................... 10
2. Sample Design ............................................................................................................... 10
3. Methods for Maximizing Response Rate....................................................................... 10
4. Test of Procedures and Methods.................................................................................... 11
5. Consultation on Statistical Aspects of the Design ......................................................... 11
Appendix A Paperwork Burden Statement................................................................................. A-1
Appendix B Survey of State Vocational Rehabilitation Agencies ............................................. B-1

Targeted Evaluations of State Vocational Rehabilitation Agency Practices—OMB Package

iii

A. Justification
1. Importance of the Information
The collection of information for which we are requesting approval is a survey of 80 state
vocational rehabilitation (VR) agencies.1 The survey is one component of a study entitled
Targeted Evaluations of State VR Agency Practices, which is being conducted by RTI
International and its subcontractor, InfoUse, on behalf of the Department of Education’s (ED’s)
Rehabilitation Services Administration (RSA), which administers the state/federal VR program.
The overarching purpose of the study is to strengthen the partnership through which RSA and
state VR agencies improve employment outcomes for individuals with disabilities. In addition to
the survey, study activities include site visits to 9 state VR agencies, selected on the basis of
survey results, and analysis of RSA datasets.
In this study, RSA is focusing on several aspects of VR agencies’ organizational
structures and management practices that have not been closely examined at the national level.
As shown in Exhibit 1, evaluation questions address:
1. Third-party cooperative arrangements through which state VR agencies enter
partnerships involving funds transfers whose intent is to deliver specialized
rehabilitation services to target groups, such as transitional youth or individuals with
severe and persistent mental illness.
2. Quality assurance (QA) procedures used by state VR agencies to ensure that they
deliver quality services in accordance with established policies and procedures.
Quality assurance may include case file reviews; strategic planning and performance
measurement; case management systems; program evaluation and special studies;
personnel assignments, training, and performance evaluation; management and
monitoring of vendor contracts; input from consumers and other customers; and a
variety of other procedures.
The proposed collection of information will allow analysts to examine the effects of
specific practices on VR program outcomes and individuals served. Survey data can be used in
conjunction with existing administrative records (the RSA-911 database) to compare the
outcomes achieved by consumers in cooperative programs with those achieved by consumers as
a whole. The survey will also provide information to assist RSA in helping state VR agencies
ensure effective and efficient delivery of VR services, and will identify promising practices that
could be adopted by other state VR agencies.
In this section of the supporting statement, we describe the need for this collection
request. We first briefly describe issues in each of the two topic areas. We then describe the need
for the survey, and how the information it provides will assist RSA and state agencies in
improving the performance of the VR program.
1

Because some states have separate agencies that serve only persons who are blind or visually impaired, the number
of agencies exceeds the number of states. The term “state” refers to the 50 states, District of Columbia, Puerto Rico,
and territories of the United States.

Targeted Evaluations of State Vocational Rehabilitation Agency Practices—OMB Package

2

Exhibit 1. Evaluation Questions
Third-party cooperative arrangements
1. How widespread is the use of cooperative arrangements?
2. What proportion of the states’ matching funds is derived from 3rd party cooperative arrangements?
3. What are the effects of cooperative arrangements upon: (1) state matching funds, (2) numbers and
types of consumers served, (3) types of services provided, (4) order of selection, (5) programmatic
outcomes?
4. What are the effects of cooperative arrangements upon the state services in comparison to other
states?
Quality assurance
1. To what extent do agencies perform QA and other functions to ensure that services are delivered
in accordance with statutes and regulations and in keeping with agencies’ policies and practices?
2. How well are QA functions performed?
3. What effects or outcomes are attributable to QA and/or similar program functions?
4. To the extent it is possible to determine from this evaluation, what practices are effective?

Issues in the Topic Areas
The survey instrument will collect information to address the majority of the evaluation
questions presented above. (Additional information will be collected through site visits to a
limited number of state VR agencies and analysis of RSA datasets.) Its design is based on our
understanding of issues in each topic area, as described below.
Cooperative Arrangements
The Rehabilitation Act of 1973, as amended (Section 361.28), allows state VR agencies
to enter into cooperative arrangements for delivery of VR services to targeted groups of VReligible individuals. However, several conditions must be met:
•

Services delivered through the agreement are not the services customarily provided
by the cooperating agency but “. . .are new services that have a vocational
rehabilitation focus or existing services that have been modified, adapted, expanded,
or reconfigured to have a vocational rehabilitation focus.”

•

Consumers receiving services must be applicants to or recipients of VR services from
the state VR agency.

•

The VR agency must supervise the services and providers of those services.

Targeted Evaluations of State Vocational Rehabilitation Agency Practices—OMB Package

3

•

Service arrangements must be consistent with order of selection2 and other state plan
requirements of the VR agency.

•

If services are not statewide, the VR agency must obtain a waiver of statewideness.

Under these arrangements, state VR agencies may partner with educational organizations
and institutions, agencies serving persons with mental illness or developmental disabilities, and
other entities. States may use funds provided by other agencies under these agreements to meet
federal requirements for matching funds.
Coordination and partnerships with other agencies can be beneficial in terms of effective
and efficient delivery of VR services. As indicated in an earlier study conducted by RTI,3 state
VR agencies may implement such arrangements in order to expand their consumer base, increase
consumer choice, improve services to hard-to-serve groups, or improve agency performance.
However, use of these arrangements to provide matching funds can also potentially have harmful
effects, including:
•

Skewing the distribution of disability types served by the agency. With a large
number of cooperative arrangements involving local education agencies or mental
health agencies, a state VR agency may be serving a higher proportion of transitional
youth or individuals with mental illness, or both, than characterizes the distribution of
those types of disabilities in the state. Hence by implication individuals with other
types of disabilities may not be benefiting fully from VR services, since the result
may be a reduction in available resources for persons with other disabilities.

•

Quality of services delivered under cooperative arrangements. In states with large
numbers of such agreements for the same types of programs, a particular program
design may be replicated in many localities without attention to factors that may
impinge on its effectiveness, such as local economic conditions or service resources.

•

Cost effectiveness. In some instances, programs operated under cooperative
arrangements may be more costly than the same or comparable services operated by
other vendors. These differences may result from increased overhead, administrative
oversight requirements, and staff costs associated with the cooperative programs.

•

Conflicts with other policies and regulations. The requirement for programs
operated under cooperative arrangements to serve specific target groups may be
hindered by order of selection or other competing requirements, hence reducing the
ability of such programs to operate at capacity and thereby reducing their cost
efficiency.

2

Under order of selection, a state VR agency without sufficient resources to serve all eligible individuals must first
serve those with “most significant” disabilities.
3
Tashjian, M.D., Hayward, B.J., Elliott, B.G., Levine, D., and Schmidt-Davis, J. 2004. Study of variables related
to state vocational rehabilitation agency performance. Revised draft final report. Research Triangle Park, NC:
RTI International.

Targeted Evaluations of State Vocational Rehabilitation Agency Practices—OMB Package

4

Quality Assurance
In the past, RSA’s monitoring of state VR agencies focused primarily on compliance
issues, including reviews of a sample of consumer case files in each agency. Responding to
RSA’s emphasis, VR agencies also focused their accountability activities on case reviews,
seldom going beyond this step to look at agency performance in other areas.
Following reorganization and refocusing at RSA, monitoring and other activities are now
encouraging agencies to look at performance rather than principally focusing on compliance. The
agency’s 2007 monitoring activities addressed the issue of quality assurance among other topics,
with the intent to help agencies look at their data in a different way – to understand what the data
mean broadly for agency operations, outcomes, and effectiveness. These monitoring activities
indicated that many VR agencies need assistance in understanding and implementing quality
assurance across all aspects of their operations.
Need for the Survey
In this section, we document the need for the information that the survey will collect on
the two topics.
Cooperative Arrangements
An important purpose of the survey is to collect current information about state VR
agencies’ use of cooperative arrangements for match purposes. The most recent year for which
this information is currently available for all agencies is FY 2003: during that year, 9 of 80 state
VR agencies used funding from cooperative arrangements for 10 percent or more of their match.
Partial data for FYs 2004 and 2005 indicate that, while some agencies reduced their reliance on
cooperative arrangements over the 3-year period, others began using this practice. Without
updated information, RSA officials cannot assess the extent to which reliance on cooperative
arrangements for match purposes may be affecting program performance.
RSA officials knowledgeable about VR agencies’ use of cooperative arrangements
believe that these arrangements may have observable effects on agency operations and
outcomes if they exceed a certain “critical” point. Because these effects (positive or
negative) are most likely to be observable in agencies that obtain relatively high
percentages of matching funds from cooperative arrangements, study activities (including
not only the survey but also supplemental analyses of RSA datasets and comprehensive site
visits to a small number of selected agencies) will focus on such agencies. Consequently, an
initial “screener” question in the survey instrument is designed to determine the percentage of
match attributable to cooperative arrangements.4 Agencies that obtain at least 10 percent of
4

This screener question was originally based on the percentage of funds from all public sources, including
political subdivisions, transfers from other state agencies, and cooperative arrangements. However, after
reviewing data for Fiscal Year 2003, we have determined that it is possible for an agency to meet that
criterion without receiving any funds at all from cooperative arrangements (i.e., the agency might obtain at
least 10 percent of its matching funds from political subdivisions and transfers from other state agencies).
Since the focus of the study, and that of all Section A questions, is on cooperative arrangements, we have
revised the screener question to refer only to funds from cooperative arrangements.

Targeted Evaluations of State Vocational Rehabilitation Agency Practices—OMB Package

5

their matching funds from cooperative arrangements will be asked to complete the balance
of the survey’s Section A, which collects details about:
1. Availability of state matching funds; i.e., whether additional state funding might
reduce the agency’s reliance on cooperative arrangements as a source of matching
funds in the future.
2. The number and types of partner agencies participating in cooperative
arrangements, amount of funding provided by each type of agency, and whether the
arrangements operate under waivers of statewideness.
3. Methods for administering cooperative programs, including performance
requirements, oversight activities, criteria for assessing performance, and corrective
actions for programs that do not meet requirements.
4. Agencies’ purposes and perspectives regarding cooperative arrangements, including
their advantages and disadvantages.
5. Characteristics of consumers served in cooperative programs (i.e., disability type
and significance, age, number receiving SSI/SSDI, race/ethnicity, and educational
attainment). This information will allow researchers to investigate issues concerning
possible skewing of service delivery as a result of reliance on cooperative
arrangements for matching funds.
6. Specific services provided to consumers served in cooperative programs, for
comparison to services provided to consumers as a whole.
7. Effects of cooperative programs on order of selection; i.e., whether reliance on
cooperative arrangements as a source of match may affect the state VR agency’s
ability to serve all eligible consumers.
8. Effects of order of selection on cooperative programs; i.e., whether order of
selection may affect the ability of cooperative programs to operate at full capacity.
9. Outcomes achieved by consumers in cooperative programs.5
We do not believe that asking agencies that derive less than 10 percent of their matching
funds from cooperative arrangements to provide all of the information listed above would
justify the burden involved for those agencies, nor would analysis of these data be a costeffective use of the finite amount of funds available to the contractor. However, we do
agree with OMB’s suggestion of October 16, 2008 that it may be appropriate to request a
limited amount of information from agencies that derive less than 10 percent (but at least
some) of their match from cooperative arrangements. Consequently, we have revised the
survey instrument’s skip pattern so that agencies obtaining less than 10 percent (but at
least some) of their match from cooperative arrangements will answer questions about
5

A description of purposes and uses of Quality Assurance data is contained in Section 2.

Targeted Evaluations of State Vocational Rehabilitation Agency Practices—OMB Package

6

topics 1, 4, 7, and 8 above, as well as indicating whether information on the statewide
distribution of the population by disability type and significance is available (Question 28).
Agencies that do not obtain any matching funds from cooperative arrangements will
complete only the screener question (Question 1) and then skip to Section B.
Quality Assurance
Federal officials require details about the QA systems currently used by state VR
agencies in order to determine how they can best help agencies improve program performance.
In executive interviews conducted by RTI, RSA staff members who oversee state VR agency
operations indicated that a comprehensive QA system would cover all aspects of agency
operations, including casework, vendor performance, consumer outcomes, fiscal and contract
management, personnel policies, employer relations, and consumer choice and satisfaction.
Further, an effective system should start with analysis of operations and problems, followed by
setting measurable goals for both processes and outcomes, monitoring of effects (including
possible negative impacts), and changes in policy and operations as indicated by evaluation.
Based on these premises, the survey instrument for which clearance is requested collects
information about QA practices in the following areas:
•

Case file reviews--whether reviews focus strictly on compliance issues or contribute
to program improvement, and areas for improvement suggested by this activity.

•

Strategic planning and performance measurement, including measures used to
monitor agency performance, standards for timely delivery of certain services or the
length of time a case may remain in particular statuses; use of performance-based
budgeting systems in which agency budgets are linked to performance goals; and
areas for improvement suggested by these activities.

•

Case management systems--ways in which they facilitate QA activities and methods
for verifying the accuracy of reported data.

•

Program evaluation and special studies--specific aspects or components of the VR
program studied and areas for improvement identified on the basis of formal
evaluations or studies.

•

Personnel assignments, training, and performance evaluation-- including whether
the agency employs staff members whose time is devoted primarily to QA activities,
percentage of counselors meeting Comprehensive System of Personnel Development
(CSPD)6 requirements, and use of outcome measures in counselor performance
reviews.

6

Federal legislation requires state VR agencies to describe a comprehensive system of personnel development in
State Plans for providing vocational rehabilitation services.

Targeted Evaluations of State Vocational Rehabilitation Agency Practices—OMB Package

7

•

Management and monitoring of vendor contracts, including contracting and
monitoring procedures and methods for obtaining consumer feedback on vendor
services.

•

Input from consumers and other customers in evaluation of agency performance and
areas for improvement identified through analysis of consumer satisfaction data.

The survey will also collect examples of QA materials (e.g., written policies and procedures,
manuals, and case review protocols), which other agencies seeking to improve their performance
may adopt or adapt for their own purposes if they choose to do so.
2. Purposes and Uses of the Data
RSA will use survey data on cooperative arrangements to examine the effects of these
arrangements on (1) state matching funds; (2) numbers and types of consumers served; (3) types
of services provided; (4) order of selection; and (5) programmatic outcomes. Information on
state VR agencies’ quality assurance practices will enable federal officials to determine how they
can best help agencies improve program performance.
3. Information Technology
RTI will use electronic distribution lists to distribute the survey to agencies as a Word or
PDF document. Agencies may, at their option, complete the survey in electronic form and return
it to RTI by e-mail or print the document, complete it by hand, and return it by FAX or postal
service.
4. Efforts to Identify Duplication
No similar information is currently available.
5. Methods Used to Minimize Burden on Small Entities
This information collection will not have an economic impact on small entities.
6. Consequences of Not Collecting the Information
Without the information collected by this survey, RSA will not have details about the
quality assurance practices currently used by state VR agencies and the level of knowledge of
state VR agency staff responsible for this function. As a result, RSA will be unable to determine
how best to target its technical assistance activities and to assist state VR agencies to the fullest
extent possible. RSA will also be unable to determine whether use of cooperative arrangements
for match purposes has a deleterious effect on program performance.
7. Special Circumstances
No special circumstances are required.

Targeted Evaluations of State Vocational Rehabilitation Agency Practices—OMB Package

8

8. Consultation Outside the Agency
RTI researchers conducted a number of executive interviews to obtain background
information relevant to the two topics and identify issues that should be considered in developing
the study’s data collection and analysis plans. Individuals interviewed include RSA officials who
are responsible for administering the VR program and who serve as liaisons to state VR
agencies. Others consulted included officers of the Council of State Administrators of Vocational
Rehabilitation and the National Council of State Agencies for the Blind, two professional
associations that represent the state VR agencies.
9. Payments or Gifts to Respondents
No payments or gifts are to be provided to respondents.
10. Assurances of Confidentiality
No assurances of confidentiality to agencies are necessary. The only individually
identifying information that will be collected is the name of the staff member completing the
survey. The survey instrument will include a statement that the individual’s name will be used
only for RTI’s internal purposes, in case follow-up questions are necessary. Published reports
will not contain any information that permits identification of individual human subjects.
11. Sensitive Questions
Information to be collected is not of a sensitive nature.
12. Estimate of Response Burden
The proposed survey requires agencies to obtain information from existing databases,
case management systems, or manual records. There are no direct costs to agencies; however,
we may calculate the costs in terms of time spent in responding to the survey.
Based on our experience with similar surveys, we have estimated the time required to
respond at 90 minutes. At an estimated average hourly salary of $30.00, the total burden for all
80 agencies is $3,600. The rate of $30.00 per hour represents the average, fully-loaded wage rage
(i.e., includes pre-tax wages, fringe benefits, and overhead costs) for four different classes of
labor ranging from clerical to managerial labor. The average wage rate accounts for the amount
of time different types of agency personnel (i.e., clerical, technical, professional, and managerial)
are expected to expend in responding to the survey.
13. Estimate of Cost Burden of Collecting Information
There are no capital costs nor is there any equipment purchase necessary for this
collection.

Targeted Evaluations of State Vocational Rehabilitation Agency Practices—OMB Package

9

14. Estimate of Annualized Cost to the Federal Government
This data collection is to be conducted under an existing contract between RSA and RTI
International of Research Triangle Park, NC. Costs to the federal government for the study’s
data collection task, including the survey, site visits to state VR agencies, and several other
activities are $258,240.
15. Change in Burden
This is a new collection; there are no changes or adjustments in burden.
16. Tabulation and Analysis Plan and Schedule
The survey for which clearance is requested is one component of a study entitled
Targeted Evaluations of State VR Agency Practices, which includes a number of other activities.
The study began on October 1, 2007 and is scheduled to end on September 30, 2009. The survey
of state VR agencies will take place in November and December 2008, followed by site visits
to a small number of agencies in early 2009 to explore the two topics in greater detail. Evaluation
reports will be submitted to RSA in June 2009.
17.

Display Expiration Date for OMB Approval
RSA will display the expiration date for OMB approval of the information collection.

18. Exceptions to Certification Statement
There are no exceptions to the certification statement.

B. Statistical Methods
1. Respondent Universe
No statistical methods will be used for this data collection. At present, there are 80 state
VR agencies. RTI will ask all of these agencies to respond to the survey.
2. Sample Design
None; the data collection will include the universe of agencies.
3. Methods for Maximizing Response Rate
Although agencies are not required as a condition of federal funding to respond to the
survey, RSA will inform them about the study, encourage their participation, and explain the
importance of collecting valid and reliable data. Should the survey fail to achieve a response rate

Targeted Evaluations of State Vocational Rehabilitation Agency Practices—OMB Package

10

of at least 80 percent, RTI will undertake prompting efforts, including e-mail messages and
telephone calls, to increase the rate.
4. Test of Procedures and Methods
No tests of procedures and methods were needed.
5. Consultation on Statistical Aspects of the Design
No consultation was needed.

Targeted Evaluations of State Vocational Rehabilitation Agency Practices—OMB Package

11

Appendix A
Paperwork Burden Statement

Targeted Evaluations of State
Vocational Rehabilitation (VR) Agency Practices
Survey of State Vocational Rehabilitation Agencies

Paperwork Burden Statement
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a
collection of information unless such collection displays a valid OMB control number. The valid
OMB control number for this information is XXXX--XXXX. The time required to complete this
information collection is estimated to average 90 minutes per response, including the time to
review instructions, search existing data resources, gather the data needed, and complete and
review the information collection. If you have any comments concerning the accuracy of the time
estimate or suggestions for improving this form, please write to U.S. Department of Education,
Washington DC, 20202. If you have any comments or concerns regarding the status of your
individual submission of this form, write directly to Steven Zwillinger, Rehabilitation Services
Administration, U.S. Department of Education, Potomac Center Plaza, 550 12th Street, SW, Room
5066, Washington, DC, 20202.

Targeted Evaluations of State Vocational Rehabilitation Agencies—OMB Package

A-1