Veterans Employability Survey
OMB Clearance Package
November 2005
June 2006
Susan Krumhaus, Co-Principal Investigator
Office of Policy
Department of Veterans Affairs
810 Vermont Avenue, NW
Washington, DC 20420
Telephone: 202-273-5108
E-Mail: [email protected]
Abstract
A. JustificationAbstract ii
A. Justification
1. Circumstances Making the Collection of Information Necessary 1
2. Purpose and Use of the Information Collected 3
4. Efforts to Identify Duplication and Use of Similar Information 34
5. Impact on Small Businesses or Other Small Entities 34
6. Consequences of Collecting the Information Less Frequently 34
7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 45
8. Comments in Response to the Federal Register Notice and Efforts to
9. Explanation of Any Payment or Gift to Respondents 5
10. Assurance of Confidentiality Provided to Respondents 5
11. Justification for Sensitive Questions 67
12. Estimation of Hour Burden Including Annualized Hourly Costs 67
13. Estimate of Other Total Annual Cost Burden to Respondents or Recordkeepers 78
14. Annualized Cost to the Federal Government 78
15. Explanation for Program Changes or Adjustments 78
16. Plans for Tabulation and Publication and Project Time Schedule 78
17. Reason(s) Display of OMB Expiration Date is Inappropriate 910
18. Exceptions to Certification for Paperwork Reduction Act Submissions 910
B. Collection of Information Employing Statistical Methods
1. Respondent Universe and Sampling Methods 1011
2. Procedures for the Collection of Information 1012
3. Methods to Maximize Response Rates and Deal with Non-response 1315
4. Tests of Procedures or Methods to be Undertaken 1316
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or
Analyzing Data 1316
Attachment 1. Authorizing Legislation 1517
Attachment 2. 60-Day Federal Register Notice 1719
Attachment 3. Draft 30-Day Federal Register Notice 2022
Attachment 4. Advance Letters 2325
Attachment 5. Telephone Script 26 28
Attachment 56 Survey Questions by Cohort and Research Issues With Justification 2830
6.Attachment 67. Survey Instruments 28602
Abstract
The goal of the Department of Veterans Affairs’ Vocational Rehabilitation and Employment (VR&E) Program is to enable veterans to live independently, achieve the highest quality of life possible and, given advances in medical science and technology, to secure gainful employment. AIn 2004, an independent task force assessed the VR&E program and concluded that approximately one-third of veterans enrolled in the program did not proceed directly throughersist and complete the programto completion. Instead, they had their program of rehabilitation interrupted or they completely discontinued; indeed about twenty to twenty-five percent of the new applicants were in fact veterans who had previously discontinued the program. Further, in 2003, 12 percent of the veterans in the program had to interrupt their rehabilitation plans, primarily due to health problems, family and financial issues, and problems arising from their disabilities.1
The proposed study, called the Veterans Employability Survey, consists of a nationally representative survey of veterans who have applied to thediscontinued the program at one of four VR&E Program . phases, as well as a comparison group of veterans who have completed the program. An advance letter from the VA explaining the general purpose of the survey and how the data will be used along with a letter from Abt Associates will be mailed to the sampled veterans. Telephone interviews will be completed with a total of 5,000 veteran applicants to the VR&E Program. The 5,000 will comprise 1,000 interviews with veterans who discontinued the program at various points as well as aeach of four phases and 1,000 veterans who belong to the comparison group of veterans who have completed the program.
A. Justification
A.1. Circumstances Making the Collection of Information Necessary
The goal of the Department of Veterans Affairs’ Vocational Rehabilitation and Employment (VR&E) Program is to enable veterans to live independently, achieve the highest quality of life possible and, given advances in medical science and technology, to secure gainful employment. The demands being placed on the VR&E Program are exceeding the organization’s capability to effectively deliver services. It has been reported that the processing of claims has taken precedence over the provision of rehabilitative services. Similarly, the administration and oversight of service provision for the veteran under the VR&E Program hasve not been a leadership priorityies. The application of the VR&E Program likely differs in each Region.2
The VR&E Program does not have data and management information to analyze either the factors that drive demand for services or the population of veterans that are applying for those services. In March 2004, the VR&E Task Force presented its report, “The Vocational Rehabilitation Employment Program for the 21st Century Veteran,” to the Secretary of Veterans Affairs. That report was an independent assessment of the VR&E Program. Per the VR&E Task Force report, the program does not have the measurement systems to “design and implement interventions to reduce the number of veterans who drop out of the [vocational rehabilitation] program or have to interrupt their rehabilitation plans” and to “provide for long-term evaluation of program outcomes.” On average, it takes 1,095 days for a veteran to complete the program without interruption. However, for a veteran who has for some reason discontinued the program, the average number of days increases to 1,625. The VR&E Task Force report found that approximately one-third of veterans enrolled in the program did not proceed directly throughersist and complete the programto completion. Instead, they had their program of rehabilitation interrupted or they completely discontinued; indeed about twenty to twenty-five percent of the new applicants were in fact veterans who had previously discontinued the program. Further, in 2003, 12 percent of the veterans in the program had to interrupt their rehabilitation plans, primarily due to health problems, family and financial issues, and problems arising from their disabilities.
Under the 38 United States Code 527, the Secretary of Veterans Affairs has the authority to gather data for the purposes of planning and evaluating VA programs (Attachment 1). The Veterans Employability Survey will be conducted under this general authorization.
A.1.1. Need for Data Collection
The Department of Veterans Affairs (Office of Policy, Planning and Preparedness) has initiated this project to determine the factors, primary and secondary, that have an impact on veteran employability resulting from veterans discontinuneed to further study those discontinuing or interrupting the VR&E Program or having that programprogram is documented in the 2004 Ttask Fforce report to the Secretary of Veterans Affairs on the results of an independent assessment of the VR&E Program. In that report, Ttask fForce members indicated that “there has not been a concerted effort to research the interrupted , and to compare this VR&E Program population with veterans who successfully completeand discontinued case problem and then to design and implement interventions to mitigate risk of a veteran dropping out temporarily or permanently from the programand with the general veteran population. The resulting analysis of the data, then, may be used to modernize the National Survey of Veterans, conducted periodically by the VA, and to develop a program of timely interventions to improve the success rate of employment resulting from the VR&E Program.
A.1.2 Project Description
Briefly, the proposed study consists of a nationally representative survey of.” In its recommendations, the task force proposed a followup with Chapter 31 veterans who have applied to theinterrupted or discontinued the VR&E program.
Furthermore, as the United States is at war, the number of veterans with disabilities requiring the services of the VR&E Program continues to increase. As this number increases so will the number of veterans who fail to persist and attain VR&E Program goals unless the VA devises intervention strategies. The Veterans Employability Survey will allow the VA to identify the risk factors for interruption/discontinuation so that an intervention plan can be implemented that will increase the number of program completers and, in turn, improve the quality of life for disabled veterans with disabilities.
A.1.2 Project Description
Briefly, the Veterans Employability Survey consists of a nationally representative survey of veterans who have discontinued the VR&E Program as well as veterans who have successfully completed the program. An advance letter from the VA explaining the general purpose of the survey and how the data will be used will be mailed to the sampled veterans along with a letter from Abt Associates Inc., the contractor hired by the VA to conduct the study, that will explain the procedures for the survey. Telephone interviews will be completed with 54,000 veteran applicants to the VR&E Program who discontinued the program at various points as well as a comparison group of 1,000 veterans who have completed the program.
The study will address the following research issues:
(1) What effect does veteran satisfaction with the VR&E program have on persistence and attainment of program completion?
(2) What role does communication play in the ability of the veteran to persist in the program?
(3) What are the internal (VR&E program) and external (e.g., familial, financial) barriers to persistence and attainment?
(4) How does disability level affect persistence?
(5) How likely will participants find employment that matches their abilities, training and interests?
(6) How do characteristics and reasons of veterans who discontinue the program differ by phase of the program (cohort) they were in? How do characteristics of veterans who discontinue the program differ from those who complete the program?
A.2 Purpose and Use of the Information Collected
As described in Section A.1.1 above, resultsData collected in the Veterans Employability Survey will be used will be used by the VA in their efforts to modernize the National Survey of Veterans, conducted periodically by the VA, and to develop a program of timely interventions to improve the success rate of employment resulting from the VR&E Program.to identify the risk factors that have an impact on veterans not completing the Chapter 31 VR&E Program, a program designed to help service-connected veterans secure gainful employment. Results of analyses of these data will then be used to make informed recommendations on developing risk mitigation interventions that will lead to improvements in the success rate for the program.
Attachment 65 contains a crosswalk of the Veterans Employability Survey questions, which lists each question, identifies the cohorts that will be asked the question, and the research issue(s) that will be addressed.
A.3. Use of Information Technology and Burden Reduction
The computer-assisted telephone interviewing (CATI) data collection method to be used for the telephone interviewing employs computer software that presents the questionnaire on computer screens to an interviewer. The computer program guides the interviewer through the questionnaire, automatically skipping to the appropriate questions based on answers to previous questions. Interviewers enter survey responses directly into the computer, and the CATI program determines if the selected response is within an allowable range, checks it for consistency against other data collected during the interview, and saves the responses into a survey data file. This data collection technology reduces the time required for transferring, processing, and releasing data.
A.4. Efforts to Identify Duplication and Use of Similar Information
A literature review was conducted in preparation for this study, as well as a review of studies and materials provided by VA’s Veteran Benefit Administration Customer Satisfaction Survey staff and VA’s Vocational Rehabilitation and Employment staff. Although data collected in the past from VA customer satisfaction surveys have suggested risk factors contributing to discontinuation in the VR&E Program, samples drawn for these customer satisfaction surveys did not focus specifically on those who did not persist in the VR&E Program. There are no recent national studies of veterans designed to specifically study veterans who have discontinued the VR&E Program. Program.
A.5. Impact on Small Businesses or Other Small Entities
While not expected to account for a large proportion of the respondents, it is anticipated that some of the respondents might own a small business. In an effort to minimize respondent burden, the questionnaire has been kept as brief as possible, averaging 20 minutes to complete.
A.6. Consequences of Collecting the Information Less Frequently
As the United States is at war, the number of veterans with disabilities and thus requiring the services of the VR&E Program continues to increase. has increased and is expected to increase. Because data regarding veterans are limited, the VA will use Veterans Employability Survey data to develop plans that will accommodate veterans’ needs so that they can complete the program. Without these data and the intervention strategies that eventually will be developed based on the data, the number of veterans who discontinue the VR&E program will continue to increase.
whoas baseline data.
Respondents will be asked to complete only one questionnaire during the course of the Veterans Employability Survey. While the interview concludes with locating questions that would facilitate additional rounds of data collection, the VA has no current plans to follow up this group of respondents with additional rounds of data collection.
A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
There are no special circumstances for this information collection. The Veterans Employability Survey is in full compliance with the guidelines of 5 CFR 1320.5.
A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agencies
In accordance with the Paperwork Reduction Act of 1995, the VA published a notice in the Federal Register announcing the intention to request an OMB review of Veterans Employability Survey data collection activities. The notice was published on September 27, 2005, Volume 70, Number 186, and provided a 60-day period for public comment (Attachment 2). The VA has received no public comments in response to this notice. The draft 3030-Day Federal Register Notice is displayed as Attachment 3.
Staff from Abt Associates Inc. contributed to the design of the survey.
A.9 Explanation of Any Payment or Gift to Respondents
This survey does not include any payments or gifts for respondents.
A.10. Assurance of Confidentiality Provided to Respondents
The contractor has established standards and procedures for maintaining confidentiality and these procedures will be used for the Veterans Employability Survey. Name, preferred mailing address, telephone number, and, where available, e-mail address, will be recorded and maintained by the contractor for mailing and telephone interviewing purposes only. Cases will be tracked using a survey identification number. Once the surveys have been completed, identifying information will be purged from the data files prior to delivery to the VA. (While an updated sample file will bPrior to data collection, in keeping with the basic tenet of human subjects research as noted in the Belmont Report in 1979, all respondents will be assured of the confidentiality of their responses and clearly and completely informed about their choice to participate. Respondents will be informed with:
A description of the research,
A clear explanation of the research purpose,
Identification of risks and benefits of the research,
Assurance that confidentiality will be respected except as otherwise required by law, and
Reassurance that participation is voluntary.
e provided to the VA, it will not be possible to link the sample file records with the data in any way.) Prior to data collection, in keeping with the basic tenet of human subjects research as noted in the Belmont Report in 1979, all respondents will be assured of the confidentiality of their responses and clearly and completely informed about their choice to participate. Respondents will be informed with: 1) a description of the research, 2) a clear explanation of the research purpose, 3) identification of risks and benefits of the research, 4) assurance that confidentiality will be respected except as otherwise required by law, and 5) reassurance that participation is voluntary.
Veterans will receive assurance and information about confidentiality in the initial survey contact through the U. S. Mail and over the telephone. The advance letters (Attachment 4) will include descriptive information about the study and an assurance of confidentiality. The telephone interviewers will explain the study to potential participants and assure them of confidentiality during the initial and any follow-up calls. (Attachment 5).
The contractor has established standards and procedures for maintaining confidentiality and these procedures will be used for the Veterans Employability Survey. The contractor will use contacting information (name, address, and telephone number) for mailing and interviewing purposes only. A unique survey identification number will identify respondents. Once the surveys have been completed, identifying information will be purged from the data files prior to delivery to the VA. While an updated sample file will be provided to the VA, it will not be possible to link the sample file records with the data.
The study contractor also has routine measures to safeguard safeguards confidentiality during the collection, storage, and disposition of survey data. All staff who have access to confidential information are instructed on the requirement to protect confidentiality and are required to sign a pledge to maintain confidentiality. Only authorized personnel are allowed access to confidential records, and only when their work requires it. Hard copies of confidential information not in use are stored in locked filing cabinets in the contractor’s secured data preparation area.
In all contractor offices, access to the data processing areas is controlled, with only authorized personnel allowed in the telephone and data preparation centers, computer rooms, and tape libraries. In addition, individual data banks and files are protected by passwords and other techniques that prohibit access by nonapproved staff. At all contractor sites, building security forces are on duty 24 hours, seven days a week.
Data security also encompasses back-up procedures and other file management techniques to ensure that files are not inadvertently lost or damaged. All important data files are routinely backed up to tape, using fast dump/restore software. File protection is additionally provided by in-place procedures to prevent unauthorized changes or access to data files.
The contract specifies that the contractor shall protect the privacy of survey respondents as well as information about prospective respondents. No information which may allow the identification of a respondent will be provided VA. Abt Associates will strip such information from the data to be provided prior to delivery.
Furthermore, information obtained in the coarse of this project will be protected under the Privacy Act and section 5701 of Title 38 U.S. Code. The VA maintains a System of Records entitled “Veterans, Dependents, of Veterans, and VA Beneficiary Survey Records (43VA008).
A.11. Justification for Sensitive Questions
The Veterans Employability Survey does not include questions regarded as personally sensitive. The questionnaires are included as Attachment 76. The contractor’s procedures to safeguard documents and files containing confidential information have been described above in Section A.10.
A.12. Estimation of Hour Burden Including Annualized Hourly Costs
The hour burden estimate associated with the Veterans Employability Survey appears in Table A.12-1. We estimate that 5,000 veterans will respond to the survey, and thatPrior to the main data collection effort, the average administration time for the survey is twenty minutes for allcontractor will conduct a pretest of 150 respondents (30 from each cohort), currently estimated to last 20 minutes on average. We estimate that 5,000 veterans will respond to the survey. This implies a total hour burden of 1,667717 hours. The annualized cost to respondents is shown in Table A.12-2. If we estimate the value of a veteran’s time at $17.7518.09 per hour3, the annualized hourly cost of the survey is $29,583.33approximately $31,061.
A.12-1 Estimates of Hour Burden
Type of Respondents |
# of Respondents |
Frequency of Response |
Average Time per Response |
Annual Hour Burden |
Applicants to the VR&E Program Pretest |
5,000150 |
1 |
0.3320/60 |
1,66750 |
Main Study |
5,000 |
1 |
20/60 |
1,667 |
A.12-2 Annualized Cost to Respondents
Type of Respondents |
# of Respondents |
Frequency of Response |
Hourly Wage Rate |
Respondent Cost |
Applicants to the VR&E Program Pretest |
5,000150 |
1 |
$17.75$18.09 |
$29,583.33 $905 |
Main Study |
5,000 |
1 |
$18.09 |
$30,156 |
A.13. Estimate of Other Total Annual Cost Burden to Respondents or Recordkeepers
There are no capital costs, operating costs, or maintenance costs to report.
A.14. Annualized Cost to the Federal Government
The eighteen-month contract cost to the Federal government for the design, implementation, and analysis for the Veterans Employability Survey is $836,218. Thus, the average annual cost is $557,479.
A.15. Explanation for Program Changes or Adjustments
This is a new collection of information.
A.16. Plans for Tabulation and Publication and Project Time Schedule
The database from which all analyses will be tabulated will include, for each respondent, selected variables from the database from which the veterans were selected, updated information on locating efforts and response status, and Veterans Employability Research Survey responses. No individual respondent will be separately identified in any analyses or reports and all personal identifiers will be removed from the analysis files.
A.16.1 Analysis Plan
After data collection is complete, Abt Associates will analyze the survey data. This analysis will describe, overall and by cohort, overall response ratesparticipant and item nonresponse rates, and describe the pattern of responses.
Descriptive analyses. The initial analyses will consist of basic frequency distributions on all survey items. This will provide information to enable the VA to form an overall picture of the results and of the survey characteristics of the responding veterans to guide the more complex analyses.
For dichotomous variables (e.g. the proportion of., veterans that have been tested in the VR&E Program), frequency distributions will be constructed, along with proportions and 95% confidence intervals. For non-dichotomous variables (e.g. income), summary statistics will be calculated, including means, medians, minimums, maximums, along withand 95% confidence intervals.
Comparing respondents to non-respondents. It is important to assess whether responding veterans are different from those that did not respond. The sampling frame will provide the necessary data to conduct this analysis. This analysis will be used to help form classes for development of sampling weights.
Cross-tabulations. A selected subset of the numerous possible cross-tabulations of the veteran characteristics (“independent” or “predictor” variables) with the survey response (“dependent”) variables will be selected and produced, based on study hypotheses (e.g., to test the hypothesis that availability of transportationoverall general health affects program completion, these two variables would be cross-tabulated and the appropriate statistical test, for example a chi-squared or an exact test, would be applied).
Multivariate analysis. Multivariate analyses offer a powerful way of identifying veteran characteristics correlated with a successful employment outcome. An appropriate model will be developed using multivariate methods such as logistic regression, to identify the existence and strength of a relationship between dependent and independent variables and to control, as appropriate, for confounding variables.
A.16.2 Project and Publication Schedule
The schedule for the Veterans Employability Survey is shown in Table A.16‑1. Interim data tabulations will begin during the data collection period, with formal analysis to occur immediately after data collection activities are closed. While the schedule below shows the final report to the VA being prepared in the month following data analysis, many components of this report will be developed throughout the life of the project and will be in place long before the last month.
Table A.16-1. Veterans Employability Survey Schedule
Activity |
Dates |
Months after OMB Approval |
|
Interviewer Training |
March 6-7, 2006 |
1 month |
|
Data Collection |
March 8-June 2, 2006 |
1-4 months |
|
Quality Control of Data Collection |
March 8-June 2, 2006 |
1-4 months |
|
Interim Tabulations |
March 15-June 2, 2006 |
1-4 months |
|
Data Analysis |
June 16-August 4, 2006 |
4-6 months |
|
Final Report |
September 9, 2006 |
7 months |
|
Activity |
Months after OMB Approval |
|
|
Interviewer Training |
1 month |
|
|
Data Collection |
1-4 months |
|
|
Quality Control of Data Collection |
1-4 months |
|
|
Interim Tabulations |
1-4 months |
|
|
Data Analysis |
4-6 months |
|
|
Final Report |
7 months |
|
|
A.17. Reason(s) Display of OMB Expiration Date is Inappropriate
All instruments will display the expiration date for OMB approval.
A.18 Exceptions to Certification for Paperwork Reduction Act Submission
No exceptions to the Certification for Paperwork Reduction Act (5 CFR 1320.9) are requested.
B. Collection of Information Employing Statistical Methods
B1. Respondent Universe and Sampling Methods
The target population for the Veterans Employability Survey consists ofcomprises five cohort groups of veterans. The current status of the veteran will be defined in terms of the sampling frame at the time the sample is drawn. The cohort groups are:
Veterans who applied to the VR&E Program, were found to be eligible, but have a current status of “did not show up for an initial appointment; ;”
Veterans who had to temporarily interrupt the evaluation and planning phase of the VR&E Program , and have a current status of “dropped out rather than returning to the program; ;”
Veterans who continued into the evaluation and planning phase of the VR&E Program, and have a current status of “dropped out before a plan was developed; ;”
Veterans who completed the evaluation and planning phase of the VR&E Program, began a plan of rehabilitation, and then have a current status of “dropped out or were otherwise discontinued from the program;;” and
A control group of veterans who successfully completedhave a current status of “successful completion” of the VR&E Program.
The sampling frame will be limited to veterans who had a disposition of completed or, interrupted/discontinued, or actively participating in the VR&E Program. Bbetween January 2004 and November 2005,. tThe population size of each cohort group is shown below. The time period of the sampling frame will be updated prior to the conduct of the study.
Cohort 1 |
27,490 |
Cohort 2 |
11,714 |
Cohort 3 |
14,754 |
Cohort 4 |
9,730 |
Cohort 5 |
15,898 |
The objective of the proposed sampling design is to select a sample from each of these five cohort groups so as to achieve 1,000 completed interviews in each. A response rate of 80% is expected for the Veterans Employability Survey. Based on the expected response rate alone, it will be necessary to sample around 6,250 veterans in order to complete 5,000 interviews.
B.2. Procedures for the Collection of Information
Sample Selection. The sampling frame, which is essential for the selection of a probability sample of veterans from each cohort group, will be providedabstracted from the Chapter 31 Master Record, a database maintained by the VR&E Program. The VA will provide Abt Associates with the sampling frame in the form of computerized lists of the five cohort groups. The lists will contain names, addresses, telephone numbers, and other pertinent contact information. Abt Associates will work with the VA to ensure that correct veteran contact information is available for the national survey, and will provide the VA with a database of the names and contact information for the sample once it is finalized..
To As noted above, to achieve a sample of 1,000 veterans from completed interviews for each of the five groups, we need to will select a larger sample,. In addition to accountan allowance for nonresponse. Also, the, this larger sample will have to be increased to accountallow for the possibility that the contact information for some of the sampled veterans might not be correct. The sampled veterans on the contact lists supplied by the VA may not have correct information. The list of veterans selected in the sample will be matched with the National Change of Address (NCOA) service and names with undeliverable addresses will be removed. WeAbt Associates will work with the VA to ensure that the number of names deleted from the frame is kept to a minimum. In addition, some it might not be possible to locate all of the NCOA-matched cases may subsequently be unable to be located.. Given these factors, a larger than needed initial sample size of 3,000 veterans will be drawn from each cohort group (15,000 in total). We will group the initial sample of 3,000 veterans per cohort group into replicates (or random subsamples) of 50 cases each and release replicates selectivelyin a controlled fashion to obtain close to the required 1,000 interviews per cohort., while working towards achieving the target response rate of 80%.
In stratified sampling (such The sampling design for the selection of the sample is relatively straightforward, since separate computerized lists for the five cohorts will be provided. We plan to sort the population for each cohort group by Census Region to ensure geographic representation of the sample. Within each Census Region, the sample will be sorted by Service-Connected Disability rating (<50%, >=50%, and unknown). For cohorts 3 to 5, within each Service-Connected Disability rating, the population will then be sorted by Serious Employment Handicap (yes versus no). Because each cohort sample will produce 1,000 completed interviews, the VA will be able to analyze data on such characteristics as age, race, ethnicity, and sex.
The required sample from each cohort group will be selected using systematic random sampling. The sorting described above followed by systematic sampling will ensure proportional representation of key subgroups.
In stratified sampling, such as that to be used for the Veterans Employability Survey), we may sometimes find that some units were either misclassified at the time of selection and were therefore selected in the wrong stratum or changed strata after sample selection. This should not cause problems in obtaining unbiased estimates. If we find that after selection, some selected units (i.e., veterans) are in a different stratum (i.e., cohort group) than at the time of selection, we will classify the unitveteran in the correct stratum but retain the sampling weights computed on the basis of their original probabilities of selection. An estimate for that stratum is obtained using the corresponding sampling weights. The units in the same stratum may have different weights, but the estimates are will be unbiased. The variance will slightly increase due to different differential weights.
The sampling design for the selection of the sample is straightforward, since separate lists for the five cohorts are to be provided. We plan to stratify the population in each list by Census region to ensure geographic representation of the sample. Based on their address, veterans will be classified into one of four U.S. regions. We propose to select an independent sample in each of the four regions within each cohort. The overall sample for each cohort will be allocated to each Census region in proportion to the number of veterans estimated in the region.
The required sample from each Census region will be selected using the Systematic Sampling Procedure. The list will be sorted by relevant information available prior to sample selection. For example, if age and gender data are available, the list will be sorted by these two variables. Sorting with systematic sampling will ensure proportional representation to various subgroups. We will consider all the available variables for such sorting (for example, we might also sort by state within a region), prior to beginning the sample selection process. stratum movement is small.
Estimation Procedure. For obtaining population-based estimates of totals, means, etc., we will assign a sampling weight to each responding veteran in each of the lists.cohort groups. This combines the base sampling weight, which depends on the probability of selection of the respondentveteran, and an adjustment for unit nonresponse. The sampling frame contains variables that can be used to rake the sample to known population control totals. These final weights will be used for all statistical analyses.
Degree of Accuracy. With a Veteran’s Employability Survey sample of 1,000 persons from each cohort, we will be able to estimate population percentages of characteristics of interest in each cohort group with a margin of error of plus or minus 3.1 percentage points at the 95% confidence level. The sample design for the Veterans Employability Survey recognizes that the five cohort groups have very different levels of experience with the VR&E Program and that it is important to understand how their exposure to the program relates to their current status and prospects in the labor force. The sample design calls for separate estimates, with a reasonable level of precision, for each cohort group. We selected a target sample size of 1,000 interviews because this sample size will provide estimates of population percentages with a 95% confidence interval half-width no larger than about 3.3 percentage points, assuming a design effect from weighting of around 1.1. A target of 1,000 interviews per cohort group will also provide reliable subgroup estimates within cohorts. Examples of important subgroup estimates include estimates by degree of Service-Connected Disability and Serious Employment Handicap status, which apply only to cohort groups 3 to 5. Table B.2-1 shows the expected 95% confidence interval half-widths, for a percentage equal to 50%, for Service-Connected Disability categories within each cohort group. Table B.2-2 shows the expected 95% confidence interval half-widths, for a percentage equal to 50%, for Serious Employment handicap categories within cohorts 3 to 5.
Table B.2-1: Expected 95% confidence interval half-widths for a percentage equal to 50% for Service-Connected Disability categories within each cohort group
Service-Connected Disability rating |
Cohort 1 |
Cohort 2 |
Cohort 3 |
Cohort 4 |
Cohort 5 |
<50% |
4.4 |
4.7 |
4.8 |
4.6 |
4.3 |
>=50% |
5.2 |
4.8 |
4.6 |
4.9 |
5.5 |
Unknown |
13.1 |
12.8 |
15.1 |
13.4 |
11.5 |
Table B.2-2: Expected 95% confidence interval half-widths for a percentage equal to
50% for Serious Employment Handicap categories within cohort groups 3 to 5
Serious Employment Handicap |
Cohort 3 |
Cohort 4 |
Cohort 5 |
Yes |
4.9 |
4.8 |
5.3 |
No |
4.3 |
4.4 |
4.1 |
An equally important aspect of the design is to allow for comparisons between cohort groups. A sample size of 1,000 completed interviews in each cohort group will make it possible to detect a difference in percentages, at the .05 significance level for a 2-tailed z-test, of around 5.7 percentage points between any two cohort groups, with 80% power. We expect some of the differences to be of this order of magnitude or somewhat larger.
Unusual Problems Requiring Specialized Sampling Procedures. There are no unusual problems associated with these sampling procedures.
Use of Periodic Data Collection Cycles to Reduce Burden. This clearance request is for a one-time survey.
Data Collection Procedures. The contractor is responsible for drawing the veteran sample and performing the necessary mailing, and interviewing operations related to the survey. Specifications The VA will have oversight responsibility for all activities conducted by Abt Associates Inc. These activities include: specifications for data collection procedures, sample design, specific content of the questionnaire, instructions for the administration of the interview, and procedures to measure quality control are provided by the VA.. The contractor coordinatesis responsible for drawing the veteran sample, performing the advance letternecessary mailing and telephone-interviewing operations related to the sampled veterans. The VA monitors the interviewing activities through remote monitoring and ongoing communications with the contractor duringsurvey, and montoring quality throughout the data collection phase.
Quality Control Procedures. Telephone data collection for the Veterans Employability Survey will be subject to strict quality control procedures. Five percent of all telephone interviewing work will be monitored by supervisory staff to ensure that introductory material is properly read, the item wording and sequence of the questionnaire are followed, respondent questions are properly answered, responses are properly probed, and interviewers’ recording of responses into the CATI system is accurate. Interviewers will be given active coaching and feedback on their performance. All data requiring data entry will be 100% verified.
B.3. Methods to Maximize Response Rates and Deal with Non-response
A variety of methods will be used to maximize response rates, such as limiting the time required to complete the questionnaire to 20 minutes, quality control of telephone interviews, and careful monitoring of production rates on a weekly basis.
Survey Non-response. As a result of the survey topic and the ease of participation, a high response rate is expected. Nonetheless, careful monitoring to ensure that the data are collected in an accurate and timely manner and that non-response is minimized will be maintained. Efforts to ensure high response rates will include efficient scheduling of callbacks through the CATI call scheduler, training interviewers in refusal aversion techniques to prevent initial refusals, and training refusal converters to overcome resistance to the survey.
Interview data will be weighted to describerepresent the population from which the sample was drawn. The weight combines a basic weight reflecting the probability of selection and an adjustment for veterans not responding to the survey. For non-response adjustment, suitable unit nonresponse weighting classes will be created after evaluating non-response bias in the sample. The resulting weighted data yields estimates for the target population. The following nonresponse bias evaluation is planned. Because we are using a list sample for this survey, there are background variables available for both nonrespondents and respondents:
Census Region
Date of birth
Gender
Severe employment disability indicator
Degree of disability
Once the data collection has been completed, respondents within each cohort group will be compared with the population on these five characteristics. Any major differences will be addressed with poststratification weighting adjustments.
B.4. Tests of Procedures or Methods to be Undertaken
A 150-case pretest (30 interviews with veterans in each of the five sample strata) to refine data collection methods is planned after OMB approval is obtained.
B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
Michael P. Battaglia
Abt Associates Inc.
55 Wheeler Road
Cambridge, MA 02138
David C. Hoaglin, Ph.D.
Abt Associates Inc.
55 Wheeler Road
Cambridge, MA 02138
Susan Krumhaus
Office of Policy
Department of Veterans Affairs
810 Vermont Ave., N.W.
Washington, DC 20005
Michael P. Battaglia Abt Associates Inc. 55 Wheeler Road Cambridge, MA 02138
|
David C. Hoaglin, Ph.D. Abt Associates Inc. 55 Wheeler Road Cambridge, MA 02138
|
Christine Elnitsky Office of Policy Department of Veterans Affairs 810 Vermont Ave., N.W. Washington, DC 20005
|
Abt Associates Inc. will complete the data collection and will assistconduct the VAanalyses in consultation with analysisthe VA.
1 VA Vocational Rehabilitation & Employment Task Force. (2004). Vocational Rehabilitation and Employment Program for the 21st Century Veteran. Washington, DC.
2 Glazier, Raymond E. and Jeffrey W. Anderson. (2005). Veterans Employability Literature Review and Analysis Report, Report delivered to Department of Veterans Affairs Office of Policy under contract #101-J47306.
3 Based on data for all occupations, July 20032004, from the National Compensation Survey, Bureau of Labor Statistics, U.S. Department of Labor.
| File Type | application/msword |
| File Title | DEPARTMENT OF HEALTH AND HUMAN SERVICES |
| Author | OlsonL |
| Last Modified By | Denise McLamb |
| File Modified | 2006-09-08 |
| File Created | 2006-09-08 |