Analysis Plan

Vocational Rehabilitation and Employment (VR&E) is one of five “business lines” provided by the Veterans Benefit Administration. VR&E provides comprehensive services and assistance to enable veterans with service-connected disabilities (SCD) and related employment handicaps to obtain stable and suitable employment. There are two types of handicaps: employment handicap and serious employment handicap. When disabilities are too severe for useful/suitable employment, VR&E assists veterans in attaining independent living (IL) to the maximum extent possible.

VR&E provides vocational and educational counseling services to veterans, active duty service members, and dependents who apply for benefits and are determined to be eligible. The services are designed to help eligible individuals choose a vocational direction and goals and to determine a course of action needed to achieve those goals. Assistance can include interest and aptitude testing, occupational exploration, career counseling, vocational goal setting, and investigating educational and training facilities that might assist in achieving occupational goals.

VA has posed five key objectives for this evaluation study of the VR&E program:

Research Question 1: Provide descriptive analyses of VR&E participants and non-participants
Research Question 2: Provide descriptive analysis of the VR&E program
Research Question 3: Compile a state-by-state inventory of other programs and compare them to the VR&E program
Research Question 4: Perform a broad-based return on investment analysis
Research Question 5: Project future program participation and caseload

To address these questions, the Study Team will organize the analyses according to the following topics:

Profile of participant and non-participant veterans with disabilities (Research Question 1)
Participation rates, both historical and projected, and reasons for participation and non-participation (Research Questions 1 and 5)
VR&E services and processes including outreach, case management, staffing, and coordination with other programs (VHA, C&P including IU, DOL VETS, State Vocational Rehabilitation, VSOs) (Research Question 2)
Employment and cost benefit analysis (Research Question 4)
Other outcomes which are not financial outcomes such as measures relating to independence in living and quality of life (Research Questions 2 and 4).
Comparison to other programs (Research Question 3).

Research Question 4 in the Statement of Work points to performing a “broad-based” return on Investment analysis (ROI) that is not limited to post-rehabilitation income. Broad-based is interpreted to mean that a traditional ROI analysis will be conducted that focuses on the financial benefits of employment outcomes, plus perform analysis of non-financial benefits as well.

The Study Team will analyze VR&E services and processes for each of VR&E’s five tracks:

Independent Living
Self-employment
Rapid employment
Reemployment
Longer-term education and training.

Chapter II summarizes the numerous data sources to be used in the study and relates them to the study objectives. Chapter III describes our approach to developing profiles of participants and non-participants. Chapter IV presents our plan for analyzing participation and non-participation rates, both actual and projected, and reasons for participation or non-participation. Chapter V is the analysis plan for VR&E services and processes. Chapter VI focuses on cost-benefit analysis with financial outcomes while Chapter VII considers non-financial outcomes. Finally, Chapter VIII addresses comparison to other programs including state level programs and VA’s Compensated Work Therapy Program.

Key Data Sources

The Study Team will gather information and data from numerous sources for use in the evaluation including interviews and meetings with VA staff, site visits, VA administrative files, survey data collection, secondary data files, previous studies, and the literature at large. Table II‑1 identifies the study objectives that each data source will support:

Table II‑1. Study objectives and data sources

Data Sources	Study Objectives
Data Sources	Profile of Participants and Non-Participants	Participation Rates	VR&E Services and Processes	Cost Benefit Outcomes	Other Outcomes	Comparison to Other Programs
Meetings with VR&E Personnel						
Site Visits						
Literature Review						
C&P Master Record				
Chapter 31 Benefits Delivery Network				
VBA CWINRS Database				
VHA Patient Surveys					
SSA Earnings Records &Disability Benefit Records					
Review VR&E Folders					
DoD Defense Manpower Data Center Data		
Previous VR&E Customer Satisfaction Surveys						
2001 National Survey of Veterans				
2007 Veterans Employability Research Survey				
2007 Survey of Disabled Veterans			
New Survey of Participant/Nonparticipants				
New Survey of VR&E Staff and Contractors						
Data Collection from VETS Organizations						
Interviews with VSO Staff						

Below the data sources to be used in this study are briefly described:

Meetings with VR&E personnel. The Study Team will hold several meetings with key VR&E personnel in the central office. These meetings will aid in gathering information, collecting data, and obtaining feedback.

Site visits. In order to gain familiarity with VR&E services and processes, the Study Team will visit selected VA Regional Offices and Medical Centers. During these site visits, the Study Team will hold discussions with Vocational Rehabilitation Counselors (VRCs), Counseling Psychologists (CPs), Employment Coordinators (ECs), and other VBA staff and with VAMC staff that interact with VR&E staff, serve on Vocational Rehabilitation Panels, and manage the Compensated Work Therapy Program. The intent of the site visits is to assess the level of interactions between VR&E and medical staff, to compare VR&E with CWT, and to identify best practices in program operations. The number of sites and the site selection will depend on agreement between VA and the Study Team on reimbursement of travel expenses. The contract travel budget is for $5,000.

Literature review. The Study Team will conduct a literature review to gather information and research related to the study questions and vocational rehabilitation in general. The literature review will include a brief legislative history of the program to provide an overview, identify legislated outcomes, and to review the Veterans Education Act. In addition, the literature review will review program outcomes that include program performance measures and outcome measures used in the vocational rehabilitation field for client evaluations and rehabilitation outcomes.

C&P Master Record. The Compensation and Payment (C&P) Master Record is a database of veterans receiving compensation from VA, which includes primary diagnoses, disability levels, military service information, as well as demographic data. These records will be used in conjunction with CWINRS to derive the sample population for the survey of VR&E participants and nonparticipants. Additionally, the database will also be used to supplement VR&E’s CWINRS database and Chapter 31 BDN data.

Chapter 31 Benefits Delivery Network (BDN) data. The VA maintains information on all beneficiaries and pensioners in its Benefit Delivery Network. The Study Team will use this data for Chapter 31 veterans to supplement some data fields, since the BDN data may be more up–to-date than information in CWINRS or C&P. The Chapter 31 BDN data is the only source for the receipt of subsistence allowance payments provided to the veteran while enrolled in VR&E.

VBA CWINRS database. CWINRS is a nationwide case management and information system created in 2001 for VA’s VR&E Program, which contains records for all VR&E applicants since 2001 as well as those active in the program as of September 2001. Thus, inactive applicants or participants who completed their programs prior to September 2001 are not in CWINRS. The exception would be anyone who had applied prior to September 2001, was inactive as of that date, and subsequently became active again.

CWINRS data have some limitations, however, and the Study Team will make use of the C&P and Chapter 31 BDN data to supplement the CWINRS data on several levels. First, there may be missing information in CWINRS for some individuals with respect to their disability rating, primary diagnosis, and length of military service. C&P Master Record data includes this information and will be referenced for populating these fields in cases where they were missing in CWINRS. In addition, the CWINRS system only updates certain fields when the VR&E counselor accesses and refreshes the data for the individual record. This data refresh is not necessary for the counselor to do his/her day-to-day job and may not occur each and every time an individual’s case file is accessed. Furthermore, before CWINRS was implemented in 2001, the individual VR&E offices had systems that were not linked at the national level, and as a result were unable to update a central data source to keep track of individuals in VR&E. Chapter 31 BDN data will be used for cohorts who applied for VR&E prior to the creation of CWINRS. Finally, while CWINRS tracks the readjustment benefits such as tuition and fee payments provided to the veteran while enrolled in VR&E, there is no recording of the individual subsistence allowance payments provided to the veteran. Such information is recorded in the Chapter 31 BDN data.

VHA patient survey data on VR-12/VR-36 health status. The VR-12/VR-36 health status survey data that VHA collects for many cases offers a measure of health-related quality of life outcomes other than employment-related ones. Health status might also serve as a control variable affecting other outcomes. The Study Team will request aggregate data on the results of the VR-12 and VR-36 surveys in order to compare the results of VR&E participants, non-participants, and CWT participants.

Social Security Administration earnings data. This study will analyze Social Security Administration earnings data to assess the change in earnings before and after participation in VR&E programs and the financial attractiveness of the programs.

Social Security Administration disability benefit records. The study can use SSA records to determine the extent and amount of receipt of Supplemental Security Income (SSI) or Social Security Disability Insurance (SSDI) benefit payments by eligible veterans with disabilities. A veteran could receive SSI or SSDI or both at the same time.

Department of Education RSA-911 data. The Rehabilitation Services Administration (RSA), housed in the federal Department of Education, oversees the provision of vocational rehabilitation services to eligible persons with disabilities. This federal-state partnership (i.e., “public-sector VR”) consists of 80 “general disabilities,” blind, and “combined” state agencies which provide a wide range of employment-related services to persons with physical, mental, and/or emotional impairments. The RSA maintains case load statistics for all “closures” (i.e., cases terminated) reported by all 80 state agencies in a given federal fiscal year through the RSA-911 Case Service Report. There are 43 data elements pertaining to each closed case reported on the RSA-911. There is a state agency code that designates which state or territory the case was closed from and whether the agency was a general disability vs. a blind or a “combined” agency. There are several variables that help to identify veteran status of the closed case. There are many other variables that will allow comparisons of veterans receiving public-sector VR with those veterans participating in the VR&E Program.

DoD Defense Manpower Data Center data. The DoD Defense Manpower Data Center (DMDC) maintains records on personnel and manpower. Information from this database such as education level, years of military service, rank at discharge, medical discharge rating and diagnosis, military occupational series, and Armed Forces Qualification Test (AFQT) and ASVAB scores can be used to supplement both the CWINRS and C&P Master Record databases for veterans released from active duty after the year 1980 and these variables can be used in the analysis. These variables are used to control for differences in human capital that affect earnings comparisons.

Previous VR&E Customer Satisfaction Surveys. Previous VR&E Customer Satisfaction Surveys conducted in 1999, 2000, and 2001-2002 will be used to analyze trends in customer satisfaction and identify potential factors that influence participation and nonparticipation.

2007 Veterans Employability Research Survey. This study is very recent and contains survey results for customer satisfaction, reasons for participation, non-participation, dropout, and several other topics of interest.

2001 National Survey of Veterans (NSV). Data from the 2001 NSV will be used to compare and contrast characteristics of veterans. NSV data includes questions on reasons for participation in each of the VBA programs. Note that the NSV data is self-reported, and it is possible that some of the self-reported responses may result in findings that do not match exactly with VA administrative data.

2007 Survey of Disabled Veterans. The 2007 Veterans’ Disability Benefits Commission survey of disabled veterans collected information regarding the self-assessed health status of 21,221 service-connected veterans from all ratings, special monthly compensation, and body systems to evaluate quality of life. The database from the 2007 Survey of Disabled Veteran can be used as needed to supplement information needed to provide study findings. These data could be matched with CWINRS to compare participants and non-participants. However, it should be pointed out that approximately two-thirds of the survey respondents were released from service prior to 1980.

New survey of veteran participants and non-participants. The survey of veterans for this study will be designed to collect information for both employment-related outcomes and other outcomes, outreach, track-specific activities, case management, income, and reasons for dropping out and participation in other federal VR programs. This survey will be explained in detail in the study’s OMB Clearance Package.

New survey of VR&E staff —approximately 700 staff members, plus 1,500-2,000 contractors. In order to gain information pertinent to many study objectives, the Study Team will administer two surveys to staff members and contractors of VR&E. Two separate but similar surveys will be created to complete this task. There are roughly 700 staff members within VR&E and another 1,500 contract counselors. The survey will be Internet-based and administered to all staff; therefore, no sampling plan will be required. The survey encompasses the Vocational Rehabilitation Counselors (VRC), Employment Coordinators, Counseling Psychologists, and the 1,500 contract counselors. It will also include the 57 VR&E Officers (Division Chiefs).

In order to create the survey, the Study Team will conduct interviews with stakeholders to assess the information which should be collected from the survey. The Study Team plans to make site visits to Regional Offices and Medical Centers. During the visits, interviews will be requested with VRC staff and the Division Chiefs. The insights gained from these meetings will allow the Study Team to formulate a final survey document.

The Study Team will break the survey into sections based on different question groupings such as demographics, job skills, departmental/regional office processes, outreach efforts, and communication. Stakeholder and site visit interviews may lead to the expansion of survey question groupings. Once the survey questions have been finalized, they will be administered using the Study Team’s web-based survey instrument. The software will send emails to all relevant staff with a personal link to the survey. In order to complete this task, the Study Team will work with VR&E staff members to obtain email addresses for all applicable survey participants including contractors.

The survey of contract counselors and VRCs will ask a range of questions in order to ascertain information relevant to the various objectives of the overall study including Research Question 2 which requires analysis of the VR&E program and processes. Preliminary meetings with VR&E officials gave the Study Team insight into some of the pertinent topics which should be addressed in the survey. The first topic is a demographic profile of VRC and contracting staff. This section will include questions regarding age, sex, salary, tenure, experience, and educational background. The next topic the survey should focus on is the processes in place at VR&E for each regional office to analyze for any significant regional variance. Caseload management is another topic that can be addressed within the survey. The Study Team will attempt to gain information regarding VRC caseload levels, how cases are processed, and whether caseloads are viewed as manageable by counselors and managers. Other possible areas might include collaboration with VHA CWT, VETS, identification of other programs which are important to the veterans VR from the counselor’s vantage point.

The Study Team will build in additional or separate questions for Division Chiefs In particular, division chiefs will answer questions that relate toward their management styles and how their regional office is run. For instance, the Study Team may ask how much training should counselors receive and whether or not they have been receiving the required amount.

Data collection from DOL VETS. The Study Team will be meeting with senior officials in the Washington, DC area to obtain their insights on interaction with the VR&E program in general and outreach activities. The COTR will review and approve questions in advance.

Interviews with VSO staff. The Study Team will interview VSO staff at the national level (in person) and at the field level (by phone). These meetings will allow the Study Team to gather information pertaining to the level of coordination and interaction between VSOs and VR&E offices.

Profile of Participant and Non-Participant Veterans

A key study objective is to provide profiles of VR&E participants and non-participants.

The Study Team will analyze the CWINRS database, Chapter 31 BDN data and C&P Master Record data to generate descriptive statistics and create profiles of both participants and non-participants of the VR&E program along with a description of the typical participating and nonparticipating veteran. The Study Team will supplement these primary administrative data sources with information from the Defense Manpower Data Center and earnings data from the Social Security Administration.

The Study Team will tabulate characteristics related to participants and non-participants such as demographics, earnings level, education level, type and severity of SCD, and those related to military service. Demographic characteristics that will be included in the profile are gender, age, and ethnicity. Education level at entry on duty and at discharge, and, if available, Armed Forces Qualification Test (AFQT) scores will be included. Additionally, the type of primary impairment and disability rating levels — as well as characteristics related to military service background such as branch of service, length of service, military occupational series, and rank at discharge— will be tabulated in the profile. Separate analyses will be conducted for new enrollees entering the VR&E program in recent years (2006 through 2008). Data are only available on which of the five tracks participants are in from 2006 and later. The profile analysis is intended to be descriptive only, not addressing the major factors predicting participation and non-participation. The latter is addressed in the next chapter.

Table III‑1 provides an example of the type and format of tables that will be included in the profile.

Table III‑1. Sample Profile Table: Age and Rehabilitation Rate

Age at Application	Eligible/Entitled to VR&E		Developed a VR&E Plan		Rehabilitated		Rehabilitation Rate (Overall)
Age at Application	Number	Percent	Number	Percent	Number	Percent	Rehabilitation Rate (Overall)
< 25
25-34
35-49
50-65
> 65
Total

Source: CWINRS April 2008 data extract

Participation Rates and Analysis

Key study questions or objectives pertaining to participation in VR&E include:

What are the participation rates overall and in the five different tracks?
What are the reasons that veterans give for participating in the program?
What are the reasons for not participating, for interrupting, and for withdrawing?
How do different factors affect participation rates?
What is the projected future program participation and caseload?
What is the potential impact of the new Chapter 33 Veterans Educational Assistance Program on participation in VR&E?

To address these questions, the Study Team will:

Analyze participation rates based on VR&E administrative files
Analyze veteran survey responses regarding reasons for participating or not participating, for interruption, or for withdrawal
Conduct statistical analysis of between participation rates and plausible explanatory variables including multivariate regression analysis
Develop a projection model to estimate future program participation and caseload.

The Study Team will analyze participation and non-participation rates back to the 1992 cohort based on data from VA administrative files including the CWINRS database, Chapter 31 BDN data (for cohorts prior to 2001), and C&P Master Record data as well as other data sources. The team will also analyze the formula currently used by VR&E to calculate participation rate to assess if an improved formula could be used.

To examine reasons for participation and nonparticipation as well as those for interruption, or withdrawal, the Study Team will conduct a survey of VR&E participants and non-participants. The Study Team will also assess whether these are related to reasons that may have affected trends or differences in participation rates over time or across cohorts. These survey results will be compared with the 2007 survey results.

Analysis will be conducted that will include statistical tests to explain why some veterans are more likely to participate in the vocational rehabilitation program than others. The Study Team will apply multivariate techniques such as logit analysis to analyze the probability of participation as a function of several variables. Factors that VA has already identified in a recent preliminary study include reliable transportation, adequate housing, family care burdens (where veteran is caretaker), and lack of basic financial means/resources. These possible reasons will be included in the study survey for respondents to identify although quantification of terms such as “adequate,” “burden,” and “inadequate” may be somewhat of a challenge. Other reasons might pertain to: veteran felt that VR&E services were not needed; veteran received support from other sources; or outreach efforts influenced the decision. Another analysis the Study Team may conduct is CHAD analysis with a decision tree.

In addition to cross-sectional analysis of individual participation versus non-participation, the Study Team will analyze trends in participate rates at a macro or aggregate level. The Study Team plans to perform trend analysis, starting with the year 1990 to the present, subject to the availability of data. Trend lines will be graphed out and correlated with plausible explanatory variables. Increases in participation depend on many different factors including expanded outreach efforts on the part of VR&E, increases in C&P claims, the changing nature of impairments incurred in active duty, and economic factors. Other possible factors affecting participation rate trends include prospects for future military conflicts and changes in the process of determining service-connected disabilities. Future military conflicts, of course, are difficult to predict, but the Study Team can make certain assumptions about trends in casualty rates arising from the ongoing Iraq and Afghanistan conflicts. These factors and others that the Study Team believed are relevant will be included as part of the analyses.

The Study Team will work with VR&E officials to create a participant and caseload projection model. The model will use inputs from past and present participation and predict the number of participants expected in the upcoming years. In order to complete this task, the Study Team requires access to the current projection methods used by VR&E. The Study Team will consider the projection models built by C&P as well. The Study Team will examine the statistical distribution of new enrollees for VA disability compensation (post 9/11) in terms of diagnostic condition, rating level, and age. This analysis will then help us to formulate reasonable assumptions about future trends. The Study Team believes that these methods and data will assist in creating a projection model for VR&E, which will help VR&E plan for staffing needs and changes in participation before they occur.

A major factor affecting participation in VR&E programs is likely to be the perceived financial payoffs the veteran would receive from completing the program. The Study Team will examine the relative financial attractiveness of VR&E and other education programs to determine which programs are the most financially advantageous for whom.

In particular, DoD is deciding whether to allow service members to allocate their post 9/11 education benefits to their dependents. This would have a major impact on decision-making by the veteran. The Study Team knows that among veterans in VR&E CWINRS who have completed a rehabilitation plan, 44 percent have no dependents, 21 percent have one, 14 percent have two, 13 percent have three, and 9 percent have four or more. Veterans in Independent Living, however, have a very different profile: 32 percent have no dependents; 46 percent have one, 11 percent have two, 7 percent have three, and 4% have four or more. Transfer of post-9/11 education benefits provides no incentive to those without dependents. In addition, some Army personnel were allowed to transfer their GI Bill benefits to their spouses and children. Depending on data availability, this could be incorporated into the analysis as well.

Both factors (levels of compensation and number of dependents) would likely to weigh heavily on the veteran’s decision to use post-9/11 education benefits or VR&E. The Study Team will ask questions on this issue in the survey of VR&E participants and non-participants to gauge intent.

If VA proceeds with a transition benefit that pays family living expenses for veteran participants in VR&E, consideration could be given to this as well. For example, approximately 58 percent of veterans in CWINRS who have completed a plan have disability ratings of 50 percent or less. They would likely receive more in a transition benefit than they currently receive in combined disability compensation and VR&E subsistence. Thus, they would have a stronger incentive to use VR&E if they received a transition benefit.

This projection model will also consider the possible impact of the Veterans Employment Act.

VR&E Services and Processes

Study objectives for analyzing VR&E services and processes include:

Describe VR&E services
Assess VR&E outreach
Assess VR&E case management and follow-up activities
Assess coordination with other organizations including VHA, C&P, DoD, DOL VETS, State Vocational Rehabilitation, and VSOs including barriers to coordination
Assess coordination between medical rehabilitation and vocational rehabilitation

Description of VR&E Services

A key study objective is to describe the services and processes of VR&E. To address this, the Study Team will:

Provide a descriptive analysis of the services provided by VR&E for the five program tracks
Describe services according to the continuum of service provision known as the rehabilitation process.
Develop flow charts to aid in understanding the rehabilitation process.
Assess differences in VR&E processes across different offices.

Interviews with VR&E management and site visits at selected VR&E regional offices will yield information required to complete the description. VR&E and VMAC principals will be asked by the Study Team to describe VR&E operations and certain aspects of them. In addition, the interviews and site visits will help the Study Team to understand the roles and functions of VR&E counselors and what is involved in the management of cases and caseloads.

Analysis of CWINRS data will enable assessment of the utilization of different services and the duration of the veteran’s participation in VR&E within the limits of these data. For example, approximately one-fifth of veterans in CWINRS do not have a track assignment as this field is not mandatory. These findings will be compared and contrasted with the veteran’s perspective using data from the veteran survey. Finally, survey data from the VR&E counselors will deepen the Study Team’s understanding of the services that are provided and the processes that are followed. All three sources of data will enable adjustments in the “flow chart” and narrative descriptions of the rehabilitation process to ensure completeness and accuracy. At this point to the Study Team will identify specific benchmarks by which progress can be measured as veterans move through the rehabilitation process. Key aspects of the rehabilitation process that will receive special attention include the application, evaluation, rehabilitation plan development, vocational training/education, and job placement.

The Study Team’s intent in scrutinizing the rehabilitation process is to develop highly detailed and descriptive flow charts that accurately depict how it is being currently executed. Only then can differences between different VBA facilities and programs become visible, and only then can different offices learn from each other. As evaluators, our job is to 1) look at the way things are actually being done; 2) by careful analysis, to identify processes that are not noticed internally, and 3) to shine a light onto these processes to develop findings and recommend affirmation/replication, abandonment, or modification.

Additional process-related matters require close examination. For example, how are veterans assigned to “tracks” within the VR&E program? How frequently do veterans transition among the tracks? What is the level of veteran involvement in this decision and how are disagreements reconciled? The CWINRS database does not contain all of this information; therefore, the Study Team will have to rely on other sources including on-site interviews of VR&E staff and web-based surveys of VR&E staff.

Veteran Evaluation

Evaluation of the veteran by VHA, Compensation and Pension (C&P), and VR&E involves a series of procedures which are highly individualized and take place at several points in time. Any and every life area of the veteran may be subject to evaluation: physical, sensory, cognitive, neurological, psychological, educational, vocational, motivational, and more. Once the key life areas are identified, potential savings of time and cost can be realized by using extant resources of evaluative information. It can also be noted that VA disability ratings are based on medical impairment while VR&E evaluation is based more on functional capacity. While VA may have impairment rating (vs. residual functional capacity), impairment rating is irrelevant to employment. Acquisition of employment (outcome) is about residual functional capacity and accommodation of impairment. The Study Team will examine through interviews and a review of VR&E folders how evaluations help to establish the rehabilitation goals and the objectives pursuant to that goal.

Rehabilitation Plan Development

What process is used in plan development? Is it comprehensive? Are the elements involved necessary? Are they balanced? Are they redundant? Is the process evenly applied across counselors and regions? Can additional steps be added to planning that would enhance rates of retention and program completion? Examination of a random sample of veteran files will enable us to address these questions.

Vocational Training

VR&E makes significant investment in training and education. Are veterans being trained for occupations in which growth is anticipated, especially in the new economy? To what extent are veterans evaluated for “transferable skills” from their former job (this may mean the military occupation, but not in all cases) and efforts made to achieve job placement as quickly as possible? This may obviate the protracted periods of unfocused vocational training during which lifelong illness behaviors are frequently learned. Certain of the VR&E five tracks, rapid access to employment and re-employment, focus on this approach.

Training may be recommended in many cases because certain military skills do not translate to the civilian workforce. For example, the Air Force tends to have more technically trained personnel who can more easily find work. The Army tends to have less technically trained personnel. There is a large Army contract called ACAP which hires contract counselors to help transitioning military personnel write resumes, learn job hunting skills, and so forth. ACAP counselors are present at TAP/DTAP sessions. Also, it has long been recognized that military occupations can be difficult to translate into civilian occupations. Of course, the Study Team will consider that VR&E participants have a disability or Serious Employment Handicap and often cannot return to previous work.

All transferable skills from previous civilian and military work are never completely eliminated by a disability short of coma. Using transferable skills analysis, those skills can be brought forward (or accommodated) and others if necessary can be trained.

The Study Team will delve into this issue during interviews with VR&E staff and site visits and confer with other subject matter experts.

Job Placement

As civilian unemployment is at 7 percent and likely to increase further, what is being done to create employment opportunities specifically for veterans? (This relates to Research Question 2: describe VR&E services.) Do private and public partnership arrangements exist, and if these involve national organizations are they being replicated in all regions? Are there demographic differences (see above) that exist between the groups of veterans who are participating in training/education vs. rapid employment vs. re-employment vs. self-employment? These questions will be addressed as information is gathered or analyzed from various sources such as interviews, survey data, and administrative data.

Outreach

One may naturally assume that application to the program is the first step in the rehabilitation process. However, the process may be thought of as beginning with outreach activity on the part of VBA to inform veterans about the existence, resources, and availability of the VR&E program. The 2004 Task Force report suggested that VBA could be doing more with respect to outreach. Hence, outreach as a front-end activity will be addressed in the analysis including the specific mechanisms that this involves and also as a response to the contractual requirement. Research question 2-m in the Statement of Work is: What does VR&E do in terms of outreach to disabled veterans. This may lead to the discovery of “best outreach practices” that would explain participation rates more than other factors considered. Conversely, practices may be identified that have a deterrent effect on applications.

Study questions pertaining to VR&E’s outreach efforts include:

How do veterans find out about VR&E benefits?
What does VR&E do in terms of outreach to veterans with disabilities? Are outreach materials developed and targeted for maximum effectiveness?
Are there underserved veteran groups for whom there is a need for increased emphasis on outreach? Examples might include veterans with visual impairment, National Guard/Reserve personnel, or select minority groups. Organizationally speaking, EconSys needs to determine who is responsible for outreach related to VR&E, whether it is programmatic, embodied in the person of one staff member or assigned to all personnel as a regularly evaluated job responsibility.
Does outreach include marketing to employers?

To address these questions, the Study Team will:

Evaluate the effectiveness of materials/media and staff performing outreach.
Assess whether the outreach efforts appropriately address groups such as the blind, hearing impaired, and those with cognitive deficiencies for whom accessibility to public information is compromised.
Assess the outreach efforts to employers, National Guard/Reserve personnel, and all racial and ethnic groups.

Building upon the analysis of participation and application rates outlined above, the Study Team will examine the impact and effectiveness of the current level of outreach provided for by the VR&E program. This includes outreach efforts aimed at educating eligible veterans prior to their application, and efforts aimed at educating participants about the programs they are enrolled in or eligible for. The team will attempt to determine to what degree current outreach levels affect the likelihood that a veteran applies for VR&E benefits, for example, and whether there is a need to increase outreach efforts in order to ensure that all potential applicants are informed of their eligibility.

The Study Team will utilize several sources of information to assess outreach, including VR&E meetings and site visits, veteran survey data, survey of VR&E counselors, and documents and materials pertaining to VR&E’s outreach efforts. The Air Force offers TAP seminars every quarter. Family Support Center informs VA representative of the seminar dates. The Study Team will attempt to confer with the VA representative.

Track-Specific Analysis

In its 2004 report on the state of the Vocational Rehabilitation program, the VA’s VR&E Task Force recommended that a five-track “employment-driven service delivery system” be implemented in the program. The Task Force concluded that such a system would allow for the “rapid assessment of veteran needs so as to quickly direct the veteran into specialized services.”¹ As a result of these recommendations, the VR&E program now directs entitled veterans into one or more of five tracks: rapid access to employment, reemployment, long-term training and employment, self-employment, and independent living.

As noted above, however, the CWINRS database contains only information on which track each veteran is currently assigned to, with no retention of information on previous tracks. Many veterans may not be able to identify their specific track in a survey. The Study Team is therefore investigating alternative ways to identify veterans by track, such as by correlating certain types of training with specific tracks. Challenges in this regard will be most significant for the three employment-tracks—reemployment, rapid access to employment, and self employment. Participants may receive services associated with one track while in another track. Each track may have discrete services and there may be one primary track, but this does not preclude accessing services “rostered” under a different track. For example, the VR&E participant can receive IL services even though he is not in the IL track. Also, the Study Team will consider that the five track program is not entirely new in that the same services have always been offered to one extent or other.

Furthermore, because these recommendations were implemented so recently, comprehensive data on the successes of the track program may not be available. For example, many veterans who entered into secondary education at a college or university through the long-term training track may not have completed their education program yet, while those who have may not have significant work histories since completion. Therefore, the Study Team will use CWINRS data and staff and veteran surveys to the maximum extent possible to analyze the tracks while acknowledging the limitations of the data. The survey of staff may help us assess the limitations of the data.

Rapid Access to Employment

The rapid access to employment track applies to veterans whose disabilities do not significantly impair their employability and who have marketable skills, but who may need some vocational assistance or training. This may include assistance with resume-writing, networking, interview and application skills, job-search training, and post-employment follow-up.

Identifying veterans in this track will pose particular challenges, especially because veterans may be transferred to this track upon completing another. For example, upon completing a university program through the long-term training track, a veteran may be recorded as entering the rapid access track to receive assistance in applying their new skills toward obtaining employment. The Study Team will work with VR&E staff experts to identify ways in which veterans who participated in this track can be identified and then analyze its effectiveness.

Key study objectives or questions include:

How does VR&E assist veterans with disabilities gain rapid access to employment?
How frequently do participants in other tracks transfer into this track to obtain employment?
Do other tracks offer similar services when veterans complete lengthier training programs?
If other tracks offer similar services, what sorts of specialized services differentiates the Rapid Access track?
What are the specific ways in which VR&E assists veterans with disabilities in gaining rapid access to employment?
Is there a consistent understanding of the concept and execution of “transferable job skills” analysis?
Does VR&E have a system in place that provides for transferable skills analysis from military to civilian occupation?
Has VR&E examined transferable skills technology that exists in the private sector (e.g., DOL’s “Military Skills Translator” program?
Does VR&E fully utilize the many incentive programs available to stimulate the hiring of veterans or otherwise extend veterans preference in hiring considerations (e.g., DOL’s “Hire Vets First” program)?

Reemployment with a Previous Employer

The reemployment track applies primarily to National Guard and Reserve veterans with disabilities. The Uniformed Services Employment and Reemployment Rights Act of 2004 (USERRA) requires that these veterans be allowed to return to their previous employer upon completing their military service. In this track, VR&E coordinates with the employer to return the veteran to their previous position if possible, identify a new position if the previous is unavailable, and provides training and counseling to prepare the veteran for employment.

Key study objectives or questions include:

How does VR&E assist veterans in this track?
To what extent do National Guard and Reserve veterans participate in this track versus the other tracks?
Are National Guard and Reserve veterans sufficiently informed by VA about their legal rights for re-employment, such as those guaranteed by USERRA?
Are veterans with disabilities sufficiently informed about state and federal laws that protect them from workplace discrimination as members of a protected class?
- The Americans with Disabilities Act
- The Rehabilitation Act
- The Workforce Investment Act
- The Vietnam Era Veteran’s Readjustment Assistance Act
- The Civil Service Reforms Act
- State Fair Employment Practices Act
Are veterans with disabilities sufficiently informed about state and federal laws that provide for medical and disability-related leave?
- Family and Medical Leave Act
- Americans with Disabilities Act
- State and federal worker compensation statutes
Are veterans with disabilities sufficiently informed about federal laws and executive orders that go beyond non-discrimination and extend affirmative action protections for persons with disabilities among federal contractors? These laws are particularly important because of the anticipated growth in industries affected by federal stimulus initiatives (e.g., construction related to infrastructure). While it may be premature to discuss federal stimulus initiatives at the moment, our study will require another year and a half to complete and the stimulus package was signed on February 17, 2009.
- Employment Discrimination and Equal Opportunity in Supply and Service Contracts (Executive Order 11246)
- Employment Discrimination in Construction Contracts
  (Executive Order 11246)
- Section 503 of the Rehabilitation Act of 1973
- Vietnam Era Veterans’ Readjustment Assistance Act of 1974 (VEVRAA)
- Veterans Employment Opportunities Act of 1998 (VEOA)
- Veterans Benefits and Health Care Improvement Act of 2000 (VBHCUA)
- Jobs for Veterans Act of 2002
Would more extensive outreach efforts aimed at increasing awareness of those rights increase participation in this track?

Long-Term Training and Employment

This track provides assistance to veterans who require more extensive or specialized training in order to return to the civilian workforce. This may be because the veteran’s disabilities require advanced adjustment training and counseling, or because the veteran presently lacks certain skills, abilities, or certifications to enter the workforce.

As its name implies, this track focuses on training which requires an extended time period. Examples of the services provided in this track include on-the-job training, internships or apprenticeships, vocational education or certification, work monitoring, work-study, and continuing formal education, such as at a college or university. In recent years, VR&E has faced criticism because of the high proportion of veterans who pursue degrees of higher education through the program. Critics argue that such an inordinate focus on higher education may hinder the program’s intended emphasis on employment.

This emphasis on higher education is not unique to the VR&E program; it is evident that cultural and social developments over the past several decades have placed increasing significance on the need to obtain a college degree to pursue high-level employment. As a result, it is only natural that these trends carry over into the VR&E program, causing veterans to prefer to pursue a university education rather than a vocational program. The Study Team will investigate the rates at which this track is used to provide higher education versus vocational training, and whether targeted informational and outreach campaigns could reduce the imbalance between the two. As this track has a focus on higher education, comparison will be made to the Post 9/11 Veterans Educational Assistance Act as well, particularly in making projections of utilization of the long-term training track.

Key study objectives or questions include:

Have veterans been fully appraised of all five tracks before training/employment are recommended/requested as the main thrust of the rehabilitation plan?
Are VR&E programs mutually exclusive? For example, could a veteran use transferable skills to find a job immediately and then participate in training/education to build a career?
Does training include “work adjustment training,” i.e., remediation of those skills not directly related to job performance, but equally important in getting and keeping a job? Examples include response to civilian authority, getting along with coworkers, punctuality/hygiene/grooming, learning and use of occupational jargon and social norms, and workplace communication skills.
Using transferable skills, can training be recommended that builds upon existing knowledge and skills from military service and adds values, rather than downplay or repudiate those skills?
Are military traits such as leadership, decision making, giving/taking direction, teamwork, and leadership emphasized in creating the veteran’s job seeking portfolio?
Are caseloads prohibitively large that high quality vocational counseling is precluded? Example, are veterans given a professionally assessed picture of their vocational self to include interests, values, aptitudes, abilities, work adaptive abilities, motivation, and learning style?
There exist 19,000 unique occupations in the U.S. labor market. Are veterans exposed to sufficient amount of occupational information prior to making a commitment to a single targeted occupation?

Training for Self-Employment

This track provides aid and assistance to veterans who are found to be best suited for self-employment. This may be because the veteran has limited access to traditional employment, or requires a more flexible work schedule or environment due to disability. VR&E works with these veterans to analyze business proposals, create and edit business plans, provide training in business-related skills, and provide financial assistance for business startups in some instances. To help with these tasks, VR&E obtains support from other agencies, including the Small Business Administration (SBA), and has access to such programs as:

Veteran Business Outreach Centers (VBO Centers). VBO Centers provide business training, counseling and mentoring, and referrals. Their staff can assist you in the development of market research and business plans, and give you training assistance on how to become an entrepreneur.
Veteran Business Development Officers (VBD Officers). Assigned to local SBA servicing offices, VBD officers provide general business development assistance. In addition, they have crucial knowledge of local markets and businesses.
Small Business Development Centers (SBDCs).Business development centers operate in each state, the District of Columbia, Puerto Rico, and the Virgin Islands. They provide consulting and education services to small business owners on topics ranging from management practices to technical skills. For example, they provide training on conflict management and Oracle databases at the same center.
Service Corps of Retired Executives (SCORE). A nonprofit organization, SCORE is made up of retired business owners and executives with decades of experience starting and operating businesses of all types. SCORE provides counseling and training to help the veteran write a business plan, apply for a loan, hone management skills, or become a more confident small business owner.
Small Business Training Network (SBTN). SBTN provides online courses in a variety of business and management areas, online counseling, access to the full SBA library of resources, and connections to other educational and training opportunities.

Key study objectives or questions include:

What training does VR&E provide to veterans who want to go into business for themselves and are these programs adequate preparation?
Which of the aforementioned entrepreneurial programs are utilized and to what degree?
How are veterans screened as being appropriate candidates for small business ownership?
Is self-employment presented early on as an option or is it considered a placement alternative of last resort?
What are the patterns of success and challenges shown by the data so that findings can be understood at a micro or individual level rather than just at the institutional level?

To address these questions, the Study Team will:

Assess the extent to which the VA partnership programs (as listed above) are included in VR&E rehabilitation plans.
Assess the comparative utilization of entrepreneurial programs.
Assess the level of consideration given to the entrepreneurial program in the rehabilitation planning process.

The Study Team will assess the training provided in the self-employment track for veterans who desire to go into business for themselves. The nature of the training currently provided will be reviewed to assess its effectiveness. The Small Business Administration and other experts in the field will be consulted as part of the assessment. The extent to which entrepreneurial skills are included and the extent to which the non-VA rehabilitation programs include training in this area will be assessed. The Study Team will provide a matrix showing comparative program features across different programs.

Independent Living

IL was statutorily authorized as a pilot program in 1980 with a cap of 500 new cases each year.² The cap was increased to 2,500 cases each year by P.L. 107-103.³ The cap was further increased to 2,600 by P.L. 110-389.⁴ Issues include the impact of the annual cap and how services would be affected by an increase in IL (Independent Living) participation. The various goals that veterans in IL have established will be analyzed to determine the extent to which employment is planned or possible. The time period during the fiscal year that the application is submitted and the amount of time between application and actual entry into IL will be analyzed to assess the extent of any delay caused by the cap and other problems the cap may cause.

The Study Team will also analyze the past participation of veterans with mild, moderate, or severe traumatic brain injury (TBI), and estimate the future volume of such individuals and the concurrent impact on VR&E workload. The Study Team would use both the C&P Master File and other data to identify veterans with TBI as a primary disability to assist in estimating the increase that can be expected. Information from the Department of Defense on the estimated number of TBI injuries will be analyzed since it is understood that a large portion of TBI has gone undiagnosed—particularly the less severe injuries. The Study Team will assess the nature of the training and services provided in IL to ensure that it meets the needs of veterans, many of whom may need more supervision than assistance with activities of daily living.

Study questions on the independent living (IL) track include:

What portion of veterans enrolled in IL has rehabilitation plans that include eventual employment as a goal? What percentage achieves that goal?
Enrollment in the IL track is currently capped at 2,600. Is this cap limiting VR&E’s ability to provide IL services to those in need? What is the need or purpose for a cap at all? How many individuals in need of IL are delayed or not allowed in IL due to the cap, and how is VA addressing their needs? How would VR&E services be affected by a significant increase or elimination of the cap in IL participation?

To address these questions, the Study Team will:

Analyze the various goals that veterans in IL have established to determine the extent to which employment is planned or deemed feasible.
Analyze the amount of time between application and actual entry into IL comparing applications during the early portions of the fiscal year to applications late in the fiscal year to assess the impact of the cap and related issues.
Analyze the past VR&E participation of veterans with mild, moderate, or severe traumatic brain injury (TBI), and project the future volume of such individuals and the impact on VR&E workload
Assess the nature of the training and other services provided in IL to ensure that it meets the needs of veterans.
Assess the extent to which IL services are resulting in true IL outcomes; i.e., independent vs. institutional living, and assess the appropriateness of current rehabilitative goals and how they are set and measured.

The Study Team proposes to analyze the 2006-2008 applicant cohort for much of the research on the different tracks and processes, since the five-track approach which was deployed nationally in November 2005. However, the earlier cohorts—VR&E applicants in 1992, 1997 and 2002—will be much more relevant for looking at IL outcomes, because this data could then be contrasted with the preliminary experience of the 2006-2008 VR&E applicant cohort.

Previous analysis indicated that 38 percent of participants in Independent Living have a primary service-connected diagnosis of PTSD. The Study Team will analyze the IL services provided for this portion of the population and assess coordination with VHA. .

Case Management

An integral part of the operation of the VR&E program is case management, in which the staff and contract counselors work directly with the veteran participants to assess the veteran’s employment or independent living needs and capabilities, design a program to assist these veterans, develop a written rehabilitation plan, monitor their progress, and assist in obtaining employment or achieving maximum independent living. Each counselor has an assigned group of veterans and this constitutes the counselor’s caseload.

The VR&E Task Force and the Veterans’ Disability Benefits Commission both questioned the adequacy of the number of counselors and their resultant caseload. The Commission cited caseload goals for other programs such as the DOL Disabled Veterans’ Outreach Program (DVOP) (50 veterans/counselor) and the VR&E program’s performance goal (125 cases/ counselor; actual caseload was 146 as of August 2006.)⁵ The Commission recommended increased staffing to improve case management.⁶

Study questions pertaining to VR&E’s case management efforts include:

What is the general process for managing a VR&E case?
What is the appropriate versus actual caseload size and caseload composition for each counselor/case manager?
How do actual caseload sizes vary among Regional Offices?
How does these numbers compare with those of state-federal vocational rehabilitation program?
How are veterans assigned to tracks within the program?
What is the process by which a veteran changes tracks? How often does this occur?
How are differences in rehabilitation planning between veterans and their counselors resolved?
What is considered a successful closure?
What follow-up or follow-on services are provided to veterans after closure?
How are closed cases re-activated? How often does this occur and why?
How can VR&E serve larger numbers of veterans with Traumatic Brain Injury, the so-called “signature” condition of the current conflict?
What are best practices in case management? Which sites, areas, or locations are using best practices?
What training is provided to VR&E counselors regarding case management? How does this training comport with standards of the Case Management Society of America?
How do the Employment Coordinators engage with employers? What services are employers expected to provide, and how are employers supported?
What areas are there for potential improvement?

To address these questions, the Study Team will:

Assess the level of follow up activities and the case management techniques that are currently employed in the VR&E program.
Assess the extent to which program interruptions and dropouts could be reduced through improved case management.
Assess the level of follow up activities and the case management techniques that are currently employed in the VR&E program.
Assess the extent to which program interruptions and dropouts could be reduced through improved case management, particularly with veterans with cognitive difficulties such as TBI.

Review the results of previous work by the VA Office of Inspector General on follow-up with TBI to assess how case management techniques can be applied within VR&E.⁷

The Study Team realizes that case management was once considered one of many roles and functions of the rehabilitation counselor, but over the past 20 years it has emerged as a freestanding profession complete with certification standards, examinations, and a unique body of knowledge and skills. Accordingly, a greater amount of training in case management is required today to achieve proficiency. For this topic, the Study Team will rely on sources of information such as literature review, analysis of administrative data, meetings with relevant VR&E staff, site visits at select VR&E offices, and the survey of VR&E staff.

A literature review will be conducted to provide insight into caseload issues and the optimum follow-up period for veterans with different types and severity of disability.

A survey of counselors and employment coordinators will be conducted to provide information on their credentials, experience, training, and workload. In particular, the survey will provide information on the opinions of staff, contract counselors, and employment coordinators on the level of effort required, training, best practices, and the proper follow-up period for veterans in each of the five tracks.

Interviews with staff at VA Central Office and in the field will provide information on the current process of managing a VR&E case. The Study Team will flow chart the process, identify variations that occur, and identify best practices that are observed. Analysis of the coordination between the VR&E program and VHA will include assessment of the processes used and the follow up that is appropriate for veterans with various types and severity of conditions, especially cognitive conditions such as TBI and mental health conditions. The nature and scope of case management training will be examined as well as the utilization of various training methods.

Interviews with DOL officials will provide information with which to compare the VR&E and DVOP programs. Analysis of statistical information on staffing levels nationally and at individual Regional Offices in context of the number of veterans in the five tracks and the mix of caseloads will be conducted.

Staffing

Study questions pertaining to VR&E’s staffing include:

What are the demographic characteristics of VR&E staff? What training, professional credentials and proficiency characteristics do they have?
What differences exist between federal employees and contract counseling staff in qualifications, training, and caseload?
What differences exist between federal employees and contract counseling staff with respect to areas of emphasis in the rehabilitation process (e.g., training versus job placement?
For pre-service graduate education, are counselors being trained by institutions per distance learning or in-vivo? Are the universities selected strong in vocational rehabilitation, transferable skills, and case management (preferred), or are they primarily counseling and mental health-oriented (less desirable)?
Are the counselors more professionally oriented toward personal adjustment counseling (less desirable) or employment counseling (preferred)?
Are the counselors/employment coordinators themselves engaged in employer development and demand side job placement?

To address these questions, the Study Team will analyze the profile characteristics of VR&E staff obtained from the study survey of VR&E staff. The nature and scope of case management training will be examined as well as the utilization of various training methods.

VR&E has invested heavily in training for its counselors since 1999 to comply with Comprehensive System of Personnel Requirements (CSPD) (i.e., all counselors must have MS degrees in Rehabilitation Counseling or a closely related field). So they obtain training at one of 90 programs. Depending on which school they attended, they obtain varying degrees of emphasis on the three areas critical to VR&E current and future success, vocational rehabilitation, transferable skills, and case management. Questions will be added to the counselor survey to address these points:

Graduate program attended
Masters completed?
Discipline?
State licensure? (Licensed Professional Counselor? Certified Rehab Counselor? National Certified Counselor? Certified Case Manager?)
Program was:
1. In person
2. Online
3. Combination of both
Rate quality of instruction in these three specific areas:
1. Vocational Rehabilitation
2. Transferable skills
3. Case management

Coordination with Other Organizations

Other organizations that VR&E coordinates with include VHA, C&P for IU, DOL VETS, state vocational rehabilitation, and VSOs. Study questions pertaining to VR&E’s coordination efforts include:

How does VR&E interact with each partner?
How does information on veterans with disabilities flow between VR&E and each partner? What type of information is shared between VR&E and its partners?
What processes are in place with each partner to facilitate seamless referrals that will not cause delays or breakdown in rehabilitation services?
How can coordination with each partner be enhanced? Should there be enhanced coordination or interaction between C&P’s Individual Unemployability and VR&E? Should VR&E assess each IU applicant’s employability?

To address these questions, the Study Team will utilize multiple sources of information including meetings with VR&E and partners’ management, site visits, and the survey of VR&E staff. Specific activities are as follows:

Describe the interactions between VR&E and each partner.

Identify and describe patterns of referral and any barriers to eligibility or seamless transition of the veteran between partners.
Review the processes used by VBA and VHA for coordination. Describe the barriers that individuals may face when moving between programs.
Assess the extent to which vocational rehabilitation plan takes into account medical rehabilitation requirements.
Assess the extent to which VR&E interruptions and withdrawals are due to medical rehabilitation requirements.

Assess whether referral information from partner to partner is timely, relevant, and thorough in a way that minimizes duplication of effort (especially evaluation). Does the information have sufficient robustness? Is it individualized, i.e., does it address the specifics of the individual’s impairments, functional limitations, disabilities, and social and physical environmental factors.
Assess whether referrals are accompanied by specific referral questions and recommendations for how the next level of service will help the veteran reach long-term goals.
Differentiate the key features of veterans who may have SCD associated with chronic illness from those who experienced acute and life altering injuries in combat. These may be different populations with respect to their VR&E needs and responses.
Consider alternative rehabilitation models and recognize that today’s veterans may have multiple diagnoses and require evaluations that emphasize both medical and functional components.
Summarize recommendations for enhanced collaboration.

Coordination between VHA and VR&E

The Study Team will assess program coordination between VHA and VR&E for those who have been medically discharged from the military and those needing both medical rehabilitation and vocational rehabilitation. The Study Team will assess the coordination between medical and vocational rehabilitation. This process will include interviews of VHA and VR&E managers and staff managing or involved with coordination activities, examination of administrative data and review of other studies that have examined this issue.

The Study Team will interview managers from both partners in order to:

Determine how many veterans eligible for VR&E enrolled, dropped the program prior to completion, or completed the program and how many from each group are employed.
Assess key variables in VR&E and CWT (including demographics, impairments, disabilities and health status such as VR-12 or VR-36 or equivalent if these measures are available).
Determine congruence between VR&E assessment/plan and rehabilitation interventions.

Collaboration with VSO’s

Does the PVA program at the Richmond VAMC represent a model with respect to VSO collaboration? The Study Team will conduct interviews with VR&E and VSO staff at Regional Offices and VA Medical Centers including questions on the subject of collaboration with VR&E. The Study Team will meet with the PVA and visit the PVA service office at the Richmond VAMC. The Study Team will review the effectiveness of the PVA centers which are understood to have assisted over 100 veterans by July 2008, 20 of whom were placed into employment.⁸ The Study Team would expect to identify best practices that could be replicated nationally.

Coordination with DOL VETS

The Study Team will interview VR&E management involved in coordination efforts with Department of Labor VETS program, beginning with meetings with DOL program managers in Washington, DC. In compiling state-level inventory of programs supported by VETS, the Study Team will seek out evidence of collaboration between VR&E and VETS. In addition, this will be a topic addressed in our site visits at selected VA Regional Offices and VA Medical Centers and in the survey of VR&E counselors.

Employment and Cost-Benefit Outcomes

A key study objective in the Statement of Work is to perform a broad-based return on investment analysis (ROI) that is not limited to post-rehabilitation income (Research Question 4). This section in the Analysis Plan focuses on post-rehabilitation income or financial outcomes while the next section addresses other outcomes. The ROI analysis will not duplicate prior cost-benefit analyses performed for the VR&E program. Instead, the Study Team will develop a more comprehensive methodology. The Study Team will identify and quantify, to the extent possible, the benefits to veterans and to the government of program outcomes such as independent living, volunteer work, and participation in community activities.

Study objectives or questions for employment and cost-benefit outcomes include:

Does VR&E improve outcomes (in terms of jobs and earnings)?
What is the economic return on investment (ROI) in training and employment services for the veteran? For the federal government?
Are there specific characteristics of veterans that affect the likelihood of successful rehabilitation? Do rehabilitation rates vary by type of disability (e.g., mental versus physical)? By combined degree of disability rating? By time to completion (by interrupt status, number of months of benefits, and total time elapsed)? By how veterans find out about VR&E benefits? What proportion of successful completers finds employment in the public sector? What proportion of successful completers finds employment in the private sector?
Do outcomes vary by VR&E program track? How do costs vary by track?
How do VR&E costs and outcomes compare with those of other public and private sector programs providing similar services to veterans (e.g., state and other federal VR programs?

To address these questions, the Study Team will:

Estimate the impact of different factors on VR&E earnings for three cohorts of Chapter 31 applicants, those who applied in 1992, 1997, and 2002. Factors analyzed will include measures of VR&E-funded training hours received, military experience, severity of disability, and annual employment.
Utilizing study survey data, analyze the reasons for dropping out by comparison group, and investigate using alternative “internal” comparison groups drawn from VR&E program withdrawals and “screen-outs” (e.g., veterans with 0 to 20% SCD ratings).
Develop a comparison group drawn from veterans applying for and receiving benefits from the C&P Service but who do not apply for Chapter 31 services.
Investigate feasibility of obtaining comparison groups of veterans applying for VETS or public-sector VR, which requires merging the RSA-911 veteran files with the VBA BIRLS files. RSA data have an indicator for veteran status.
Conduct specification tests for the different comparison groups using available pre-program employment and other data as available to determine their comparability to the treatment group of Chapter 31 recipients.
Link Chapter 31 BDN data files with Master Beneficiary Records and Supplemental Security Record to see if receiving VR&E services reduces dependence on Social Security Disability Insurance (SSDI) and Supplemental Security Income (SSI).
Analyze how VR&E costs and post- rehabilitation earnings vary depending on the characteristics of veterans or track of service.

Evaluation Design

Using the data and considerations described above, our evaluation of the VR&E Chapter 31 Services will have three distinct components.

First, the Study Team will summarize basic information about the characteristics and outcomes of beneficiaries of the program. The Study Team will analyze how rehabilitation rates vary by type of disability (e.g., mental versus physical), by the combined degree of disability rating, and by the time to completion. The Study Team will analyze what portion of successful completers are employed by federal or other government agencies, and will analyze what portion of successful completers are employed with the private sector as well. The Study Team will focus on how outcomes vary by track and how costs vary across tracks and will compare VR&E costs and outcomes with those of other public and private sector programs providing similar services to veterans.

Second, the Study Team will evaluate the impact of the different VR&E service tracks on labor market outcomes.

Third, the Study Team will estimate the economic return on investment (ROI) in training and employment services for veterans and for the government using a traditional analysis, but will also develop an non-economic ROI which will be subsequently described in Other Incomes of the VR&E Program.

The first set of questions is descriptive and can be answered directly from the data. Impact and ROI evaluations inherently are more complicated and ambiguous. The data alone cannot answers these types of questions. Researchers must combine data with assumptions that, in many cases, can be controversial. Our solution is to conduct the evaluation using alternative assumptions or scenarios. This not only provides a range of possible outcomes, but also lets us assess whether particular findings are robust with respect to scenarios tested. Below, the basic methodological approaches to be used to conduct the impact and ROI evaluations are described.

Outcome Analysis

The Study Team will use the updated SSA earnings, military experience, and VR&E records to estimate the impact of VR&E services on labor market outcomes. The Study Team will use 15 years of labor market data for the 1992 applicant cohort, 10 years for the 1997 cohort, and 5 years for the 2002 cohort. In addition to work-related outcomes, the Study Team will analyze other outcomes such as independent living, volunteer work, and participation in community activities. Quality of life measures might also be relevant.

The basic idea of an impact evaluation is simple and appealing. Program outcomes—employment and wages—are measured and compared to the outcomes that would have resulted in the absence of the program. In practice, however, it is difficult to design a credible evaluation in which such a comparison can be made. The fundamental difficulty is that the outcomes in the absence of the program are counterfactual and not observable. What would have happened to recipients had they not received Chapter 31 benefits? This fundamental methodological problem⁹ requires that the evaluation design provide some basis for constructing a credible estimate of the outcomes for the unobserved conditions (i.e., identically-situated veterans who did not receive VR program services). Consequently, the estimation of Chapter 31 treatment outcomes will require qualifying assumptions or the identification of a “comparable” group of individuals. The data alone cannot resolve this outcome problem.

Establishing credible estimates of what the outcomes would have been without the program is the most critical and demanding part of impact evaluation. When those estimates are convincing, the effects found in the evaluation can be attributed to the program rather than to any of the many other possible influences. Otherwise, the evaluation can be misleading. For example, a simple comparison of the employment outcomes of Chapter 31 recipients to non-recipients is unlikely to reveal the true impact of the Chapter 31 program. Any differences in labor market outcomes could be due to preexisting differences between the groups (e.g., unobserved health or disability status). The job of a good impact evaluation design is to neutralize or rule out such alternative explanations for observed outcome differences.

The counterfactual outcomes problem can be resolved if the employment data are combined with sufficiently defensible assumptions. Because there is no established solution to the counterfactual outcomes problem that is valid in all settings, the Study Team plans to use a variety of different methods to assess the robustness of the findings using different or competing models. The Study Team will consider three broad design approaches: quasi-experimental designs, structural/parametric designs, and non-parametric designs (see Appendix A for a technical discussion of the different design approaches).

Quasi-experimental designs compare the outcomes among VR&E recipients to carefully chosen comparison groups that are thought to be similar except for the fact that they did not receive treatment. For example, the ROI analysis provided in “An Outcome-Based Assessment of the Chapter 31 program” used an “internal” comparison group of veterans drawn from non-participating applicants for VR&E services. These designs are appealing and commonly used, but often controversial. Does the comparison group credibly reveal the counterfactual outcome (i.e., how the person would have fared in the absence of Chapter 31 services)?

Structural designs formally combine prior information about the institutional details of the VR&E program with models of behavior. For example, the SCD ratings used to assign VR&E treatments might provide a natural way to evaluate the impact of the employment services by comparing veterans with scores just above and below the cutoff. This institutional feature of the program might then be coupled with a model of how individuals decide whether to work, and how schooling impacts employment and wages. Once estimated, structural models allow researchers to formally characterize how treatments are assigned, how employment outcomes are determined, and the how the program impacts schooling and employment outcomes. These models have been developed by James Heckman, the Nobel Prize winning economist, and others, including Steven Stern, a member of our research team. In the case of the Chapter 31 program, a structural model would allow an examination of the various employment tracks and estimate the various employment outcomes resulting from assignment to a given track.

Rather than impose the rigid modeling assumptions inherent in structural designs, nonparametric designs begin by asking what the data alone reveal about the Chapter 31 program and then add on stronger assumptions as appropriate. Without restrictive assumptions, the data generally do not reveal the impact of the Chapter 31 program but can provide informative bounds around employment outcomes. Then, by imposing stronger assumptions, the bounds narrow. While it is tempting to impose assumptions strong enough to yield a definitive finding about employment outcomes, the problem is that strong assumptions may be inaccurate and yield flawed conclusions (Manski, Newman and Pepper, 2002).¹⁰ Evaluators must be concerned about the credibility of findings to policymakers and the public, a potentially diverse group, some of whose members may not share the evaluator's beliefs about what are and are not plausible assumptions. Thus, nonparametric designs allow evaluators to explicitly reveal how inferences about employment outcomes vary with different assumptions about the Chapter 31 program.

Return on Investment Evaluation

Using the results of the outcome analysis, the Study Team will then evaluate the Return on Investment (ROI). Here, the Study Team is interested in long run ROI for both the veteran and the government. The Study Team will use the updated SSA earnings, military experience, and VR&E records to calculate a 10-year ROI for the 1997 VR&E applicant cohort, which would then be compared to the revised results for the 1992 applicant cohort. Additionally, the Study Team will undertake a five-year ROI calculation using the 2002 applicant cohort and compare this to 5-year ROI results for the 1992 and 1997 cohorts. In calculating the ROI, EconSys will consider the yearly cost of living adjustment to the overall earnings.

In addition to work-related outcomes, the Study Team will analyze other outcomes such as the extent of independent living, extent of involvement in volunteer work, and extent of participation in other community activities as discussed in the next chapter of this document.

The Study Team will also consider making longer run lifetime ROI estimates. The ROI analysis provided in “An Outcome-Based Assessment of the Chapter 31 program” (Dean and Schmidt, 2005) argued that the 10-year ROI is too conservative. Earnings gains incurred many years after the program are not incorporated into the 10-year ROI estimates. Of course, the problem with conducting a longer run ROI analysis is that we do not observe the outcome of interest—lifetime earnings.

Thus, to undertake the lifetime analysis, we face the problem of trying to use auxiliary outcomes (i.e., earnings over the 10 or 15 years post treatment) to draw conclusions about lifetime labor market experiences. To resolve this auxiliary outcome problem we will combine historical measures of lifetime earnings profiles (using an external data source such as the CPS) with assumptions linking the observed patterns in the historical data to the auxiliary outcomes.¹¹

Once the Study Team establishes this link between historical and auxiliary data, we can then undertake the ROI analysis. Our approach to estimate the Return on Investment (ROI) to the VR&E program will be patterned after a framework elaborated in Thornton et al.¹² Thornton views the benefits and costs of any social program from three distinct perspectives – the government’s, the individual’s, and society’s. Table VI‑1 lists a series of tangible and intangible benefits as well as tangible costs.

Two separate sets of benefits are listed for the Agency’s perspective. The first set includes benefits to the government resulting from revenue (tax) increases and expenditure (public assistance) reductions. The second set includes benefits to the individuals participating in the program, specifically increased income offset by increased taxes and reduced public assistance receipts. Agency costs include both purchased services as well as general operating costs.

From the individual’s perspective, program benefits include those same earnings gains (net of taxes on those earnings) and reductions of public assistance payments as well as intangible benefits for self and family due to improved functioning and quality of life. Of course, the individual does not incur the costs of the program but often foregoes income while participating in VR.

Table VI‑1. Identifying Potential Benefits (+) and Costs (-) for Use in Calculating the Return on Investment in VR&E from Alternative Perspectives

Tangible Benefits (– indicates a benefit offset)
	Perspective
Agency's			Individual's	Society's
to Government		to Individual	Individual's	Society's
		(+) Increased Future Earnings	(+) Increased Future Earnings	(+) Increased Future Earnings
(+) Increased Tax Receipts (FICA, Fed, State)		(–) Increased Tax Payments (FICA, Fed, State)	(–) Increased Tax Payments (FICA, Fed, State)	(+) Increased Tax Receipts (FICA, Fed, State)
(+) Reduced Public Assistance Payments		(–) Reduced Public Assistance Receipts	(–) Reduced Public Assistance Receipts	(+) Reduced Public Assistance Payments
Intangible Benefits
		(+) Spillover effects to family members	(+) Spillover effects to family members	(+) Spillover effects to family members
				(+) Other spillover effects to society
Tangible Costs
(–) Purchased Services		(–) Purchased Services		(–) Purchased Services
(–) General operating costs		(–) General operating costs		(–) General operating costs
			(–) Foregone earnings while in program	(–) Foregone earnings while in program

In addition to increased future post-rehabilitation earnings from linking to Social Security, our proposed ROI will include estimating the tax payments and reduction in public assistance receipts. The former will be obtained using tax formulas applicable to each state. The latter will be obtained by linking the veteran’s files with the SSA disability benefit payment records. These include the Master Beneficiary Record, which contains Social Security Disability Insurance (SSDI) benefit payment information and the Supplemental Security Record, which contains disability payments through the Supplemental Security Income (SSI) program.¹³

Other Outcomes of the VR&E Program

A key study objective in the Statement of Work is to perform a broad-based return on investment analysis that is not limited to post-rehabilitation income (Research Question 4). This section in the Analysis Plan addresses other outcomes. Closely related to this objective is to analyze alternative definitions of success (Research Question 2).

To address these objectives, the Study Team will:

Review intended or statutory outcomes with VA
Conduct a literature review of potential measures of success for VR outcomes
Compare alternative definitions of success
Analyze the results of the VR 12 and VR 36 Health‐Related Quality of Life
Analyze the results of the study’s survey of veterans that pertain to alternative outcomes

The initial activity will be a review of the existing statutory or operational definitions of success that VA currently uses in its VR&E program. Two approaches will be utilized: (1) statutes and publications (technical literature, lay publications and websites) review and (2) interview of VA officials to obtain program and individual client evaluation templates/forms that are routinely used to evaluate success. If the term “success” is not currently used, attempts will be made to identify what is used to determine benefit or positive outcome of the program. Effort will be made to determine whether the denotation or connotation of the term “success” used by the VA is similarly used in the vocational rehabilitation literature.

In addition, the Study Team will compare indicators of success (both individual and program) for each of the VR&E five tracks (employment through long-term services, independent living, reemployment, rapid access to employment, and self-employment services). This will determine whether there is uniformity among these tracks and whether each track is considered independently. Also, it will be important to determine if eligibility for one track enables the use of the other tracks’ resources.

A literature review will be conducted to identify models and potential indicators for vocational success, including review of state vocational rehabilitation programs (civilian) and international rehabilitation programs (veteran and/or civilian) and eligibility restrictions. Selected vocational counselors/program directors of state vocational rehabilitation programs will be interviewed for program details. The literature review will also include assessment of what is known about the impact of vocational rehabilitation on health status, quality of life, and well being in order to identify key parameters that have been considered valuable in terms of quality of life or health status outcomes (e.g., social behavioral studies, visits to the emergency room, life expectancy, substance abuse).

The Study Team will explore the possibility of obtaining additional data from three different sources. First, an interview guide will be used in meetings with state vocational counselors in order to evaluate the measurements of success (programs/individuals). The Study Team will explore the possibility of obtaining quality of life survey data (VR-12 and VR-36) from VHA that can be matched to VR&E participant data for the four cohorts selected for this study. Since 59 percent of SCD veterans were patients of VHA in FY 2006 and the usage rate increases as severity of disability worsens so that 83 of SCD rated 50% or higher were patients,¹⁴ a significant proportion of SCD veterans, both participants and non-participants, should have survey results available. EconSys expects that many participants will have results from pre-vocational rehabilitation and post-rehabilitation periods for comparison of the change in quality of life due to VR&E. Finally, the study’s primary survey of VR&E participants and non-participants will obtain data on outcomes that reflect alternative definitions of success. Based on the literature review and assessment, the Study Team will generate candidate questions for the study survey of veterans designed to assess the views and outcomes of VR&E participants and non-participants. The Study Team will conduct statistical analyses of responses to questions pertaining to health status, quality of life, well being, and other measures of program success. This information will be used to develop rationale for (against) developing outcome measures of vocational rehabilitation based on health status, quality of life and well-being.

Comparison to other Programs

The study questions related to comparison to other programs include:

State-level programs
- What programs at the state level provide services to veterans with disabilities similar to VA?
- What state agencies receive Department of Labor (DoL) Veterans Employment and Training Service (VETS) grants?
- How do the state vocational rehabilitation programs compare to the VR&E program in terms of eligibility criteria (i.e., how do the elements of eligibility vary across the states and compared to VR&E)?
- How do the state programs market (i.e., outreach) their services to eligible veterans and other participants?
- How do the state programs provide or deliver services to program participants? Do the service delivery models at the state level differ from VR&E?
- How do the rehabilitation rates and post-rehabilitation earnings compare across the state programs and to VR&E?
Other programs
- What other programs (outside of state level programs) provide vocational rehabilitation support to veterans with disabilities?
- How are these programs funded?
- How many veterans make use of these programs and why do they use these programs versus state level programs and/or VR&E?
Best practices
- What programs could serve as national models of best practices?
- What are the best practices exemplified by these programs?
- How replicable are these best practices across different circumstances (e.g., across the spectrum of disabilities)?
Compensated Work Therapy Program
- What are the features of the Veterans Health Administration’s (VHA) Compensated Work Therapy (CWT) Program?
- How do the features of CWT compare to the features of VR&E?
Veterans Use of Other Programs
- To what extent do veterans with disabilities participate in other programs in addition to or instead of VR&E?
- Are eligibility criteria related to the rate of participation?

State Level and Other Programs

To address the study questions, the Study Team will employ multiple data collection methods as shown in Table VIII‑1.

Table VIII‑1. Research Questions and Data Sources and Methods

Research Question / Topic Area	Data Source / Method
State-level programs	Interviews with stakeholders Interviews with state-level disability program coordinators Secondary analysis of extant data sources, including state rehabilitation program Web sites and other available literature Rehabilitation Services Administration (RHA)-911 data files (Department of Education)
Other non state-level programs	Secondary analysis of extant data sources, including local (e.g., municipal) Web sites, and other federal government Web sites (e.g., Department of Labor, Department of Education)
Best practices	Interviews with stakeholders Interviews with state-level disability program coordinators
Compensated Work Therapy Programs (WTP)	Interviews VHA management staff for CWT Review available documents that describe CWT and its participants Conduct site visits to the Salem and Richmond VA Medical Centers to gather data on interaction with VR&E
Veterans Use of Other Programs	Secondary analysis of extant data sources, including state rehabilitation program Web sites and other available literature Interviews with state-level disability program coordinators RSA-911 data files (Department of Education) Survey of VR&E participants and non-participants

The specific data collection methods and the analysis of the resultant data are described in detail in the following sections. The analysis plan is summarized in the Appendix.

State Level Programs

Table VIII‑2 restates the data sources and methods associated with this research question.

Table VIII‑2. State Level Program Data Sources and Methods

Research Question / Topic Area	Data Source / Method
State -level programs	Interviews with stakeholders Interviews with state-level disability program coordinators Secondary analysis of extant data sources, including state rehabilitation program Web sites and other available literature RSA-911 data files (Department of Education)

Research Question /

Topic Area

Data Source / Method

State -level programs

Interviews with stakeholders

Interviews with state-level disability program coordinators

Secondary analysis of extant data sources, including state rehabilitation program Web sites and other available literature

RSA-911 data files (Department of Education)

The Study Team will employ a multi-pronged approach in obtaining state-level program information, which will include:

Stakeholder interviews. The Team will identify the persons who have knowledge of and/or a vested interest in state-level programs offering vocational rehabilitation services, including those programs that have partnerships with federal agencies, such as VR&E. The Study Team will conduct 10-15 interviews with various stakeholders.
Interviews with state-level disability program coordinators. The Study Team will conduct interviews with 2-3 disability program coordinators in each state (as feasible).
Secondary analysis of state vocational rehabilitation Web sites and other documentation. The Study Team will analyze the available information from Web sites.
Data from the Rehabilitation Services Administration (RSA)-911 tracking system. The Study Team will request data from the RSA-911 system covering the past five fiscal years, which provides data on VR services across the states. The Study Team will analyze the data available from the RSA-911 (case services report) system.

Interviews. Table VIII‑3 presents the various sources from which to draw potential interview candidates.

Table VIII‑3. Potential Sources of Interview Candidates

Type	Potential Source
Stakeholders	VR&E Staff DOL VETS Staff Local Veterans Employment Representative (LVER) Staff VA Compensated Work Therapy (CWT) Program Staff
State Level Officials	State Vocational Rehabilitation Agencies (SRVAs) State Directors of Veterans Affairs State Deaf and Blind Vocational Programs State Independent Living Councils (SILCs)

Type

Potential Source

Stakeholders

VR&E Staff

DOL VETS Staff

Local Veterans Employment Representative (LVER) Staff

VA Compensated Work Therapy (CWT) Program Staff

State Level Officials

State Vocational Rehabilitation Agencies (SRVAs)

State Directors of Veterans Affairs

State Deaf and Blind Vocational Programs

State Independent Living Councils (SILCs)

Prior to conducting these interviews, or all interviews with stakeholders, the Team will develop specific interview protocols to guide and structure the interviews. Interview protocols and operational procedures will be reviewed and approved by the COTR in advance. The stakeholder interviews will be used to identify additional sources of information, which will then be added to overall set of activities.

State-by-State Reference Directory. From the interviews, from secondary data sources (e.g., program documentation, Web sites), and from the analysis of the RSA-911 data, the Team will construct a reference directory of state-level programs as a self-sustaining document with a cover and/or binder, introduction, and table of contents, directory with other feature such as tabs to be added if needed for navigational aids. At a minimum, the directory will contain the following for each state:

A list of all state and local agencies that provide vocational rehabilitation programs for which veterans are eligible and the contact information for these programs.
A brief description of the purpose of each program and eligibility criteria employed. Eligibility restrictions for State programs will be identified.
An indication on the extent to which a state-level program is similar to VR&E, using a three-valued labeling scheme: 1) Non-duplicative; 2) Overlapping; and, 3) Duplicative. Criteria for making these assessments will be provided to the COTR for review prior to execution. For categories #2 and #3, a brief justification on the findings will be provided.
An indication of the number of participants in each program (per year or in a recent calendar year), including the number of veterans and the number of non-veterans.
An indication of the whether a state-level program is an example of a “best practice” that can be replicated in other programs (if available).
An indication of whether a state level program can be used as a model of an “exemplary partnership” among state and federal agencies (if available).

The information in the directory will be aggregated to obtain a “collective snapshot” of state-level VR programs. The elements that are common across states and programs, such as funding sources, will be identified and highlighted.

Other Programs

Table VIII‑4 restates the data sources and methods associated with addressing this research question. There are a number of VR programs that may be offered at the local level, such as a municipality. There also are programs operated by not-for-profit organizations, such as Goodwill Industries, AmVets, and others. The not-for-profit organizations may have a geographically, local (e.g., within a city or county), regional (e.g., within a state or several states), or a national presence (e.g. a number of states from coast to coast).

Table VIII‑4. Other Programs Data Sources and Methods

Research Question / Topic Areas	Data Source / Method
Other non state-level programs	Secondary analysis of extant data sources, including local (e.g., municipal) Web sites, and other federal government Web sites (e.g., Department of Labor, Department of Education)

Research Question /

Topic Areas

Data Source / Method

Other non state-level programs

Secondary analysis of extant data sources, including local (e.g., municipal) Web sites, and other federal government Web sites (e.g., Department of Labor, Department of Education)

Among the extant data sources that the Study Team will use to identify other sources (non-state) for VR programs are:

Guidestar.org. This is a Web site that currently stores information on approximately 1.7 million not-for-profit organizations. The Team will use this as a starting point to obtain preliminary information on those organizations offering VR services.
Federal government Web sites. The Study Team will use Web sites belonging to the Department of Labor, and to the Department of Education; both of these fund VR programs. The Study Team will take care in avoiding duplicating the programs identified here with those identified in the inventory of state-level programs (using RSA-911 data) previously described.

The team will develop a directory similar in nature to the directory of state-level programs. Some of the information obtained about the programs will include:

Sources and amount of funding and budget (discretionary versus mandatory)
Number of cases handled annually (e.g., number of VR closures or “graduates”)
Types of disabilities specialized in
Average duration of services offered
Degree of follow-up after successful completion of the program.

The analysis will create a compendium of such programs alongside the state-level program inventory.

Best Practices

Table VIII‑5 summarizes the data sources and methods that The Study Team will use in ascertaining best practices among VR programs that can be exported and replicated in other programs.

Table VIII‑5. Best Practices Data Sources and Methods

Research Question / Topic Area	Data Source / Method
Best practices	Interviews with Stakeholders Interviews with state-level disability program coordinators

Research Question /

Topic Area

Data Source / Method

Best practices

Interviews with Stakeholders

Interviews with state-level disability program coordinators

As discussed above, the Study Team will use the following data collection methods to develop best practices:

Stakeholder interviews
- The Team will identify the persons who have knowledge of and/or a vested interest in state-level programs offering vocational rehabilitation services, including those programs that have partnerships with federal agencies, such as VR&E. The Study Team will conduct 10-15 interviews with various stakeholders.
Interviews with state-level disability program coordinators
- The Study Team will conduct interviews with 2-3 disability program coordinators in each state (as feasible).

Best practices will emerge during the stakeholder interviews and interviews with the state disability program coordinators, which the Study Team will document. Where appropriate, the team will also identify the best ways to achieve “best practice transfer” among programs and will note any unique conditions or circumstances that mitigate the successful deployment of specific best practices.

Veterans Use of Other Programs

Table VIII‑6 recapitulates the data sources used to address this set of research questions.

Table VIII‑6. Data Sources and Methods for Veterans Use of Other Programs

Research Question / Topic Area	Data Source / Method
Veterans Use of Other Programs	Secondary analysis of extant data sources, including state rehabilitation program Web sites and other available literature Interviews with state-level disability program coordinators RSA-911 data files (Department of Education) Survey of VR&E participants and non-participants

Research Question

/ Topic Area

Data Source / Method

Veterans Use of Other Programs

Secondary analysis of extant data sources, including state rehabilitation program Web sites and other available literature

Interviews with state-level disability program coordinators

RSA-911 data files (Department of Education)

Survey of VR&E participants and non-participants

The data for this question will come from the following sources:

Extant data sources. Data on the use of these programs by veterans will come from Web sites and any publications issued by agencies.
State coordinator interviews. The state coordinators will be asked about participation in other programs by veterans, including any criteria for eligibility, duration of services offered, follow-up services offered, and any special services targeting veterans in particular.
Survey of VR&E participants and non-participants. As noted elsewhere in this analysis plan, the Study Team will ask veterans about non-VR&E programs that they are using, either in addition to, or in place of the VR&E services and the reasons why in a questionnaire that will be distributed to approximately 48,000 veterans.

The Study Team is fully prepared to deal with any discrepancies between estimates of veteran participation among the different sources, in order to develop an accurate count and proportion of veterans using other VR programs as compared to those using VR&E programs.

Apart from the counts and proportions, the analysis will focus on the reasons that veterans turn to other VR programs for assistance. The reasons may be manifold, including eligibility criteria, degree of magnitude of assistance provided, convenience to home or work location, the ability of integrate family supports, and other factors.

Summary of Approaches

Table VIII‑7 summarizes the research questions, approach, and analytical techniques for state level program analysis:

Table VIII‑7. Summary of research questions, approaches, and analytical techniques

Research Questions	Approach		Analytical Techniques
State-level programs
What programs at the state level provide services to veterans with disabilities similar to VA? What state agencies receive Department of Labor (DoL) Veterans Employment and Training Service (VETS) grants? How do the state vocational rehabilitation programs compare to the VR&E program in terms of eligibility criteria (i.e., how do the elements of eligibility vary across the states and compared to VR&E)? How do the state programs market (i.e., outreach) their services to eligible veterans and other participants? How do the state programs provide or deliver services to program participants? Do the service delivery models at the state level differ from VR&E? How do the rehabilitation rates and post-rehabilitation earnings compare across the state programs and to VR&E?	Interviews with stakeholders Interviews with state-level disability program coordinators Secondary analysis of extant data sources, including state rehabilitation program Web sites and other available literature Analysis of Rehabilitation Services Administration (RHA)-911 data files (Department of Education)	Content data analysis of interview data, and Web site data Development of Program Inventory schema for enumerating all identified programs Quantitative analysis of RSA-911 data: Frequency distributions, cross-tabulations of dependent variables by demographic and other variables of interest; statistical measures of association, such as correlations and chi-square.
Other Programs
What other programs (outside of state level programs) provide vocational rehabilitation support to veterans with disabilities? How are these programs funded? How many veterans make use of these programs and why do they use these programs versus state level programs and/or VR&E?	Secondary analysis of extant data sources, including local (e.g., municipal) Web sites, and other federal government Web sites (e.g., Department of Labor, Department of Education)	Content analysis of qualitative data Synthesis of secondary quantitative data Budget/financial analysis

Table VIII‑7, Summary of research questions, approaches, and analytical techniques (Continued)

Research Questions	Approach		Analytical Techniques
Best Practices
What programs could serve as national models of best practices? What are the best practices exemplified by these programs? How replicable are these best practices across different circumstances?	Interviews with stakeholders Interviews with state-level disability program coordinators	Content analysis of qualitative data
Compensated Work Therapy Program
What are the features of the Veterans Health Administration’s (VHA) Compensated Work Therapy (CWT) Program? How do the features of CWT compare to the features of VR&E?	Interviews VHA management staff for CWT Review available documents that describe CWT and its participants Conduct a site visit to Salem VA Medical Center to gather data on interaction with VR&E	Content analysis of qualitative data
Veterans Use of Other Programs
To what extent do veterans with disabilities participate in other programs in addition to or instead of VR&E? Are eligibility criteria related to the rate of participation?	Secondary analysis of extant data sources, including state rehabilitation program Web sites and other available literature Interviews with state-level disability program coordinators RSA-911 data files (Department of Education) Survey of VR&E participants and non-participants	Content analysis of qualitative data Summary and synthesis of secondary quantitative data Quantitative analysis of RSA-911 and questionnaire data: Frequency distributions, cross-tabulations of dependent variables by demographic and other variables of interest; statistical measures of association, such as correlations and chi-square.

VA’s Compensated Work Therapy Program

In order to gain an understanding of the CWT program and access to information, the Study Team will interview VHA management staff responsible for managing the CWT program. The Study Team will also review available documents and data that describe the program, its participants, processes, and outcomes. In addition, visits to selected VA Medical Centers will provide an understanding of the CWT’s interaction with the VARO’s VR&E program.

Table VIII‑8 summarizes the date sources and methods to address this research objective.

Table VIII‑8. Compernsated Work Therapy Program Data Sources and Methods

Research Question / Topic Area	Data Source / Method
Compensated Work Therapy Programs (WTP)	Interviews VHA management staff for CWT Review available documents that describe CWT and its participants Conduct site visits to Salem and Richmond VA Medical Centers to gather data on interaction with VR&E

Research Question /

Topic Area

Data Source / Method

Compensated Work Therapy Programs (WTP)

Interviews VHA management staff for CWT

Review available documents that describe CWT and its participants

Conduct site visits to Salem and Richmond VA Medical Centers to gather data on interaction with VR&E

The Compensated Work Therapy (CWT) Program provides a number rehabilitative of services available to veterans to meet a range of veteran goals:

Vocational rehabilitation services
Job matching and employment supports
Vocational case management
Work site and job analysis
Consultation regarding assistive technology
Reasonable accommodation
Mental health counseling
Guidance in addressing ADA regulations compliance.

The CWT goal is not permanent placement but rather a component of therapy provided at the VA medical center.

CWT, also known as “Veterans Industries” in some contexts, works with employers as well as veterans in providing training and employment services. In order to gain an understanding of the CWT program and access to information, the Study Team will:

Interview VHA management staff (5-10 staff) responsible for managing the CWT program.
Review available documents and data that describe the program, its participants, processes, and outcomes.
Conduct site visits to selected VA Medical Centers.

As part of the analysis, the Study Team will develop a family of matrices that tabularizes the information and makes qualitative comparisons with programs and services provided by VR&E. This will aid in comparing and contrasting the two programs.

Participation Rates in Other Programs

The Study Team will analyze the extent of participation of veterans with disabilities in other programs in addition to or instead of VR&E. Part of our approach will be to examine veterans who apply for public-sector VR services. The Study Team will utilize the RSA-911 reporting system for the analysis. The RSA-911 is a national data base of all persons “closed” from public-sector VR in a given fiscal year. Information is reported by the separate state agencies so it is possible to examine all data at the state level.

There are roughly 700,000 applications for public-sector VR services nationwide in a given year, with about half ultimately being accepted for services. In Federal Fiscal Year 2006 more than 350,000 persons were terminated from the 80 state VR agencies, with over 200,000 of these individuals having an employment outcome and the balance determined to be “not rehabilitated”. The annual cost of purchasing the requisite VR services and administering this program exceeds $3 billion.¹⁵

There are several variables that help to identify what the veteran status is of the closed case. There is a binary variable “to indicate if the individual had served in the active military, naval or air service, and was discharged or released under conditions other than dishonorable.”¹⁶ There is a data element for “type of public support” received by the individual at application and closure which includes a designation for veteran’s disability benefits which “are payments made by the Department of Veterans’ Affairs for partial or total disability.”¹⁷ There is also a designation for the monthly amount of public support received at application and closure although it is not VA-specific in that the payment is categorized as “all other public support”.

There are many other variables that will allow comparisons of veterans receiving public-sector VR versus those veterans participating in the VR&E Program. In addition to the standard demographic and socioeconomic variables (e.g., gender, age, race, education level, number of dependents) there are several variables detailing the nature and severity of the individual’s disabling condition. Both an individual’s primary and secondary disability are designated by a four-digit code that is a combination of 19 impairment codes (e.g., sensory, physical, mental) and 37 codes for the causes and sources of the impairment (e.g., amputations, TBI, SCI). There is also a binary designation whether the person is classified as severely disabled or not.

There is information collected about the individual’s employment status, weekly earnings, and hours worked reported in the week prior to application for VR services. If the individual is successfully rehabilitated, the same information is obtained for the week after being employed for 90 days. Additionally, a four-digit Dictionary of Occupational Titles code is recorded for the type of employment at the time of successful completion of VR.

This employment information is necessarily limited to the time frame when the individual is enrolled in VR. However, since the RSA-911 records the person’s social security number it is possible to obtain the individual’s longitudinal earnings profile from the Social Security Administration (SSA). A Memorandum of Agreement between the SSA and the RSA was signed in 2008 allowing for the merging of the RSA-911 Case Service Report file with various files from the SSA. These files include the Master Earnings File (MEF) and the Supplemental Security Record (SSR) and Master Beneficiary Record (MBR). The MEF contains annual earnings from 1954 through two years prior to the current calendar year; the SSR and MBR contain monthly benefit payments for Supplemental Security Income (SSI) and Social Security Disability Insurance (SSDI) since 1954 and 1974, respectively, through the latest available month. The RSA-911 files for all VR program closures in a given federal fiscal year are available since FFY 1997. The current holdings include 11 years of closures through FY 2007.

The Study Team will create an extract of all persons self-identified as veterans closed from public-sector VR for the 11-year period spanning 1997-2007. The Study Team will then develop a profile of these veterans that provides detailed information about their demographic, disability, and socioeconomic attributes, the type and amount of VR services provided, the administrative outcome of their VR service program, and their employment information gleaned from the earnings “cross-match” with SSA as well as what is reported in the RSA-911. The Study Team will also conduct a cross-state analysis of application, enrollment/dropout rates, and success/non-success rates for public-sector VR enrollees and examine differentials in types and amount of purchased VR services provided among the states. If the requisite agreements can be obtained by VBA/VR&E and RSA, the Study Team will also seek to have a linkage between the RSA-911/SSA records and the VBA’s BIRLS file to verify the individual’s veteran status and ascertain what VBA benefits the veteran has received.

Appendix A: Description of Alternative Design Approaches for Comparison Groups

Quasi-Experimental Designs

In practice, the most common assumption used to evaluate the impact of VR&E programs is that participation in the program is unrelated or independent of the employment outcome, as they would be in a classical randomized experiment that includes both control and experimental (treatment) groups. Generally speaking, these designs are approximations to randomized experiments that compare selected cases receiving the treatment with “similar” cases not receiving it.

The basic idea is to carefully specify control and comparison groups that are thought to be similar to treatment groups except for the fact that they did not receive treatment. The appeal of this design is obvious, and there is an active methodological research agenda aimed at developing new approaches that can be used to justify the application of this assumption. The better versions of these designs attempt to statistically account for observed extraneous influences in order to argue that there are no omitted confounding variables. Matching methods developed by Rosenbaum and Rubin¹⁸ and others provide formal statistical tools for identifying relevant comparison groups that can be tested statistically to ensure the development of valid data findings on differences.

Observing the same individuals (e.g., veterans) over many years affords us a range of flexible and effective approaches for accounting for extraneous influences. In particular, panel data designs, such as fixed effects and difference-in-difference models, will allow us to effectively account for certain types of unobserved but stable confounding variables, such differing work ethic and/or personal motivation, undiagnosed disabilities, and other factors that are difficult or impractical to measure in this kind of study. For example, one might be concerned that there are fixed but unobserved individual characteristics associated with VR treatment decisions and labor market outcomes (e.g., severity of the disability). With panel data, one can simply net out these individual factors that are fixed over time. Thus, in this case, the basic fixed effects model can be used to resolve the counterfactual outcomes problem.

The Study Team will use different internal and external control groups. The ROI analysis provided in “An Outcome-Based Assessment of the Chapter 31 program”¹⁹ used an “internal” comparison group of veterans drawn from non-participating applicants for VR&E services. The comparison group included those veterans who had applied for Chapter 31 benefits some time during 1992 but dropped out of the program after being determined eligible and entitled but prior to receiving a subsistence allowance payment. Excluded from the comparison group were those VR&E applicants administratively "screened out" as being ineligible or not entitled to Chapter 31 benefits and program withdrawals prior to an eligibility determination. The Study Team will consider a similar internal comparison group for the 1997 and 2002 applicant cohorts. In addition, the Study Team will investigate using alternative “internal” comparison groups drawn from VR&E program withdrawals and “screen-outs” (e.g., veterans with 0-20% SCD ratings).

The Study Team will also use an "external" comparison group drawn from C&P recipients with release from active duty dates from 1980 through 2007 who have not applied for VR&E. The Study Team will use those veterans with an SCD who are temporally matched on the customary socio-demographic attributes along with a combination of their "release from active duty" date and/or their date of initial notification of eligibility for compensation benefits (which is also the same date they receive an application for Chapter 31 benefits). VA C&P data files will the source of identifying comparable non-participant veterans. Likewise, the Study Team will investigate the feasibility of obtaining comparison groups of veterans applying for public-sector VR, which requires a merging of the Department of Education’s Rehabilitation Services Administration RSA-911 client files with file from the VBA Beneficiary Identification Record Locator Subsystem (BIRLS)²⁰ to identify veteran’s military status and types of VBA benefits received files.

This RSA-911 Case Service Record will be used to develop several possible external comparison groups for determining programmatic impacts of participating in the VR&E program. The annual RSA-911 closure files include the gamut of all possible closure statuses including: 1) those persons who withdraw prior to an eligibility determination; those persons not accepted for services, 3) persons who were accepted for VR services and dropped out prior to an Individualized Employment Plan was developed; 4) persons who were not rehabilitated after receiving significant service provision; as well as 5) persons “successfully” rehabilitated. These closure statuses are virtually identical to the statuses used in the VR&E Chapter 31 BDN data and allow for several types of external comparison groups to be examined for comparability to participants in the Chapter 31 program.

Given that the date of application for VR is a data element on the RSA-911 record and that the average duration in public-sector VR is less than two years, it is possible to construct an applicant cohort for all cases applying in 1997. That is, the 1997 applicant cohort will be generated by combining all cases who apply in that year but who are closed in 1997, 1998, 1999, etc. Similarly, the Study Team will also be able to construct a nearly complete 2002 applicant cohort consisting of closures in 2002 and beyond. The caveat is that there may be a few applicant cases in 2002 that have not yet been closed by 2007, the last closure year currently available from RSA.

These public-sector VR applicant cohorts are directly comparable to the VR&E applicant cohorts for 1992, 1997, and 2002 to be used in the return on investment analysis. There will be ten years of post-application earnings for the 1997 cohorts and five years for the 2002 cohort.

Finally, the Study Team will consider the feasibility of using external data drawn from data from national surveys of the non-institutionalized population. These data can be limited in that there may be few veterans in the sampling frame, and it may be difficult to identify much about whether the respondent received employment related services. However, all comparison groups have problems, and these external data sources have much to offer including large samples, detailed covariates (including information on disabilities, training/schooling and, in some cases, veterans status and payments/education assistance), and a long time series. With the proper care, these data can be used to help address the counterfactual outcomes problem and extrapolate beyond the ten or fifteen year window of labor market outcomes observed in the VA data.

Structural/Parametric Designs

Despite the different quasi-experimental approaches that can be used to support an exogenous selection assumption, in general, this assumption might not be credible. VR services are not likely to be randomly assigned, and any imaginable control comparison group is likely to differ in ways that will lead to spurious correlations in the observed data.

An alternative route to address this counterfactual outcomes problem is to assume that treatment effects are constant across the population and that there exists some covariate, termed an instrumental variable (IV), that is independent of outcomes but not of treatments.²¹

There are a number of possible instrumental variables that might be feasible. Two sources of variation may be useful. The first is the SCD rating used to assign treatment. Using the scoring cutoff points that determine treatment assignment rules, the Study Team might be able to implement a regression discontinuity design which effectively identifies the impact of treatment by comparing those with scores just before and after the cutoff.²² The basic idea behind the RD regression discontinuity design is that assignment to the treatment is determined by the value of a covariate being on either side of a fixed and observed threshold. This predictor may itself be associated with the potential outcomes, but this association is assumed to be smooth. Thus, so any discontinuity of the conditional expectation of the VR&E employment outcome as a function at the SCD rating cutoff value is interpreted as evidence of a causal effect of the Chapter 31 treatment. A second variable that might be useful is the time to initial notification. This duration may impact when services are provided but may not be directly related to future labor market outcomes.

Another alternative the Study Team will explore is to use a parametric latent variable model jointly describing how treatments are selected and outcomes determined. Heckman's binary treatment selection model, for example, can be extended to examine VR&E programs where there are many potential treatments.²³ In particular, the Study Team can allow for treatments to be assigned endogenously, using a variant of the polychotomous choice models developed and applied by multiple researchers.²⁴

Semi- and Nonparametric Methods

Concern with the validity of the strong assumptions needed to identify treatment effects has led to the recent development of a body of literature imposing weaker assumptions. In some cases, these weaker models yield only bounds on the counterfactual mean outcome. There is a large body of literature on semi-parametric alternatives. More recently, attention has focused on nonparametric models. One set of results consider the implications of a threshold -crossing model (like the SCD rating choice model above) but without the distributional restrictions.²⁵ A second design illuminates the identifying power of instrumental variable assumptions. When Chapter 31 is imposed alone, treatment effects vary across the population.²⁶ A third set of results shows the identifying power of various assumptions about the treatment Chapter 31 selection process when nothing is known about the process determining VR&E employment outcomes. For example, one may assume that each member of the population was assigned a VR&E the service track yielding the better employment outcome for that individual. A fourth set of results shows the identifying power of assumptions determining employment outcomes when nothing is known about the Chapter 31 selection process. All of these approaches fit into our problem in a relatively straightforward way.

1 Report to the Secretary of Veterans Affairs: The Vocational Rehabilitation and Employment Program for the 21^stCentury Veteran, VA Vocational Rehabilitation and Employment Task Force, March 2004, p. 7.

2 Statement of Richard Daley, Associate Legislative Director, Paralyzed Veterans of America, before the House Committee on Veterans’ Affairs, July 10, 2008.

3 Statement of Ruth Fanning, Director, VR&E Service, before the House Committee on Veterans’ Affairs, July 10, 2008.

4 P.L. 110-389, §332 amends §3120(e) of Title 38.

5 Veterans’ Disability Benefits Commission, p. 267.

6 Ibid, p. 270

7 Health Status of and Services for Operation Enduring Freedom/Operation Iraqi Freedom Veterans after Traumatic Brain Injury Rehabilitation, VA Office of Inspector General 05-01818-165, July 12, 2006. and Follow-up VA’s Role in Ensuring Services for Operation Enduring Freedom/Operation Iraqi Freedom Veterans after Traumatic Brain Injury Rehabilitation, VA Office of Inspector General 08-01023-119, May 1, 2008.

8 Testimony of Richard Daley, Associate Legislative Director, PVA, before the House Committee on Veterans’ Affairs, July 10, 2008.

9 Called the counterfactual outcomes or selection problem.

10 “Using Performance Standards to Evaluate Social Programs with Incomplete Outcome Data:

General Issues and Application to a Higher Education Block Grant Program,” Evaluation Review, 26(4), August 2002, 355-381.

11 In particular, we will consider the equal conditional means assumptions that the historical average lifetime earnings profile conditional on a 10- or 15-year window of observed earnings equals the lifetime earnings profile VR&E recipients will realize after the 10-year window. See.Manski, C. and Pepper, J. (2000). Monotone Instrumental Variables with An Application to the Returns to Schooling. Econometrica,68(4), July 2000, 997-1010.

12 Thornton, C., R. Agodini, and V. Jethwani, 2000, “Design for Evaluating the Net Outcomes for the State Partnership Initiative,” Social Security Administration.

13 SSI is based on General Tax Revenues and thus amount can change from year to year. Also receipt of SSI can change in matter of one month because the financial means test is based on total Household income. SSDI is not necessarily a lifetime benefit. While SSI is funded from general revenues and not from the Trust Fund (as SSDI is funded), the amount of the benefit is not tied to the amount of general revenues but rather set by Congress.

14 Veterans’ Disability Benefits Commission, p. 243.

15 U.S. Department of Education. (2008). Quarterly Cumulative Caseload Report. RAS Annual Review Report. Retrieved January, 2009 from http://rsamis.ed.gov

16 U.S. Department of Education. (2004). RSA Policy Directive PD-04-04, RSA 911 Case Service Report, September 2004. p. 44

17 U.S. Department of Education. (2004). RSA Policy Directive PD-04-04, RSA 911 Case Service Report, September 2004, pp. 19 & 38

18 Rosendbaum, P. & Rubin, D. (1983). The Central Role of the Propensity Score in Observational Studies for Causal Effects, Biometrika, 70, pp. 41-55.

19 Dean,D. & Schmidt, R. (2005). “An Outcome-Based Assessment of the Chapter 31 Program,” Final Report, U.S. Department of Veterans Affairs, Contract No. 101-Y27247.

20 U.S. Department of Veterans Affairs, Veterans Information Resource Center. (2008). Beneficiary Identification Record Locator Subsystem – Death File. Retrieved January 13, 2009, from http://www.virec.research.va.gov/DataSourcesName/BIRLS/BIRLS.htm

21 Bjorklund, A. & Moffitt, R. (1987). Estimation of Wage Gains and Welfare Gains in Self-Selection Models. Review of Economics and Statistics, 69, pp. 42-49.
Friedlander, D., Greenberg, D.H., & Robins, P.K. (1997). Evaluating Government Training Programs for the Economically Disadvantaged. The Journal of Economic Literature. XXV(4), pp. 1809-1855.

Heckman, J. & Honore, B. (1990). The Empirical Content of the Roy Model. Econometrica, 58, pp. 1121-1149.

Heckman, J. & Hotz, J. (1989). Choosing Among Alternative Nonexperimental Methods for Estimating the Impact of Social Programs: The Case of Manpower Training. Journal of the American Statistical Association, 84, pp. 862-874.

Heckman, J. & Robb, J. (1985). Alternative Methods for Evaluating the Impact of Interventions. Longitudinal Analysis of Labor Market Data, Cambridge: Cambridge University Press.

Maddala, G.S. (1983). Limited-Dependent and Qualitative Variables in Econometrics. Cambridge: Cambridge University Press.

Manski, C. (1995), Identification Problems in the Social Sciences, Cambridge, Mass.: Harvard University Press.

22 Hahn, J., Todd, P., & Van der Klaauw, W. (2001). Identification and Estimation of Treatment Effects with a Regression Discontinuity Design, Econometrica 69, pp. 201-209.

23 Heckman, J. (1979). Sample Selection Bias as a Specification Error, Econometrica, vol. 47, pp. 153-161.

24 Duncan, G. (1980). Formulation and Statistical Analysis of the Mixed, Continuous Distrete Variable Model in Classical Production Theory. Econometrica, 48(4), pp. 839-852.

Dubin, J.A. & McFadden, D.L. (1984). An Econometric Analysis of Residential Electric Appliance Holdings and Consumption. Econometrica, vol. 52, 345-362.

Lee, L.F. (1983). Generalized Econometric Models with Selectivity. Econometrica, vol. 51, pp. 507-512.

Dahl, G. B. (2002). Mobility and the Returns to Education: Testing a Roy Model with Multiple Markets. Econometrica, vol. 70, pp. 2367-2420.

25 Shaikh & Vytlacil. (2005). Threshold Crossing Models and Bounds on Treatment Effects: A Nonparametric Analysis. Working paper.

26 Balke, A. & Pearl, J. (1997). Bounds on Treatment Effects from Studies With Imperfect Compliance. Journal of the American Statistical Association, 92, pp. 1171-1177.

Manski, C. and Pepper, J. (2000). Monotone Instrumental Variables with An Application to the Returns to Schooling. Econometrica,68(4), July 2000, 997-1010

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Chapter I
Author	George Kettner
File Modified	0000-00-00
File Created	2021-02-04

Analysis Plan

Final VRE Evaluation Study Analysis Plan - April 27 09.docx

Evaluation of VAs Vocational Rehabilitation and Employment (VR&E) Program

Analysis Plan

OMB: 2900-0735

Deliverable 2.2: Analysis Plan (Final)

Department Of Veterans Affairs

Evaluation of VA’s Vocational Rehabilitation and Employment (VR&E) Program

Introduction

Key Data Sources

Profile of Participant and Non-Participant Veterans

Participation Rates and Analysis

VR&E Services and Processes

Description of VR&E Services

Veteran Evaluation

Rehabilitation Plan Development

Vocational Training

Job Placement

Outreach

Track-Specific Analysis

Rapid Access to Employment

Reemployment with a Previous Employer

Long-Term Training and Employment

Training for Self-Employment

Independent Living

Case Management

Staffing

Coordination with Other Organizations

Coordination between VHA and VR&E

Collaboration with VSO’s

Coordination with DOL VETS

Employment and Cost-Benefit Outcomes

Evaluation Design

Outcome Analysis

Return on Investment Evaluation

Other Outcomes of the VR&E Program

Comparison to other Programs

State Level and Other Programs

State Level Programs

Other Programs

Best Practices

Veterans Use of Other Programs

Summary of Approaches

VA’s Compensated Work Therapy Program

Participation Rates in Other Programs

Appendix A: Description of Alternative Design Approaches for Comparison Groups

Quasi-Experimental Designs

Structural/Parametric Designs

Semi- and Nonparametric Methods

Evaluation of VAs Vocational Rehabilitation and Employment (VR&E) Program