B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
The target population for the pilot C&P Enrollment questionnaire will include individuals who have received a decision on a compensation or pension benefit claim within 30 days prior to the fielding period. The sample will be stratified as follows: (1) type of benefit (i.e., Compensation, Pension) (2) claimants who were found eligible (3) claimants who were found ineligible and are not appealing their claim. The target population for the pilot Compensation Servicing questionnaire will include individuals who have been receiving compensation benefits for at least 6 months or individuals who received a decision on a compensation claim 6-18 months prior to the field period. The sample will be stratified as follows: (1) individuals who were granted a decision, are receiving benefits and not appealing their benefits, (2) individuals who were granted a decision, are receiving benefits and are appealing their benefist, (3) individuals who were denied benefits and are appealing (4) individuals who were denied benefits and are not appealing. The target population for the pilot Pension Servicing questionnaire will include individuals who have been receiving pension benefits for at least 6 months or individuals who received a decision on a pension claim 6-18 months prior to the field period. The sample will be stratified as follows: (1) individuals who were granted a decision, are receiving benefits and not appealing their benefits, (2) individuals who were granted a decision, are receiving benefits and are appealing for additional special benefits (i.e., Aid and Attendance, Housebound), (3) individuals who were denied benefits and are appealing.
The number and types of questions answered will vary slightly based on the experiences of the target population (e.g., whether or not the respondent has had contact or not with the VA regarding their claim).
This section describes the universe of respondents for the survey, and the sampling plan that will be employed to achieve a representative sample for the survey.
1) Compensation Enrollment
Individuals who have received a decision on a compensation benefit claim within 30 days prior to the fielding period. The sample will be stratified as follows: (1) type of benefit (i.e., Compensation, Pension) (2) claimants who were found eligible (3) claimants who were found ineligible and are not appealing their claim.
TABLE 1: Individuals who have received a decision on a compensation benefit claim EXPECTED RESPONSE RATE AND SURVEY YIELD
|
||
Number of Claimants |
Expected Response Rate |
Completed Surveys Expected |
10,000 |
30% |
3,000 |
2) Pension Enrollment
Individuals who have received a decision on a pension benefit claim within 30 days prior to the fielding period. The sample will be stratified as follows: (1) type of benefit (i.e., Compensation, Pension) (2) claimants who were found eligible (3) claimants who were found ineligible and are not appealing their claim.
Table 2 displays the sampling frame and the expected response rates and the expected yield.
TABLE 2: Individuals who have received a decision on a pension benefit claim EXPECTED RESPONSE RATE AND SURVEY YIELD
|
||
Number of Veterans |
Expected Response Rate |
Completed Surveys Expected |
10,000 |
30% |
3,000 |
3) Compensation Servicing
Individuals who have been receiving compensation benefits for at least 6 months or individuals who received a decision on a compensation claim 6-18 months prior to the field period. The sample will be stratified as follows: (1) individuals who were granted a decision, are receiving benefits and not appealing their benefit, (2) individuals who were granted a decision, are receiving benefits and are appealing their benefit, (3) individuals who were denied benefits and are appealing (4) individuals who were denied benefits and are not appealing.
Table 3 displays the sampling frame and the expected response rates and the expected yield.
TABLE 3: Individuals who have been receiving compensation benefits for at least 6 months or individuals who received a decision on a compensation claim 6-18 months prior EXPECTED RESPONSE RATE AND SURVEY YIELD
|
||
Number of Claimants |
Expected Response Rate |
Completed Surveys Expected |
10,000 |
30% |
3,000 |
3) Pension Servicing
Individuals who have been receiving pension benefits for at least 6 months or individuals who received a decision on a pension claim 6-18 months prior to the field period. The sample will be stratified as follows: (1) individuals who were granted a decision, are receiving benefits and not appealing their benefit, (2) individuals who were granted a decision, are receiving benefits and are appealing for additional special benefits (i.e., Aid and Attendance, Housebound), (3) individuals who were denied benefits and are appealing.
Table 4 displays the sampling frame and the expected response rates and the expected yield.
TABLE 4: Individuals who have been receiving pension benefits for at least 6 months or individuals who received a decision on a pension claim 6-18 months prior EXPECTED RESPONSE RATE AND SURVEY YIELD
|
||
Number of Claimants |
Expected Response Rate |
Completed Surveys Expected |
10,000 |
30% |
3,000 |
The target population for the pilot Education Enrollment questionnaire will include individuals who have received a decision on their education benefit application within 90 days (i.e., the original end-product has been cleared within the past 90 days) prior to the fielding period. The sample will be stratified as follows: (1) accepted and enrolled, (2) accepted and not enrolled, (3) denied. The target population for the pilot Education Servicing questionnaire will include beneficiaries who have been enrolled and receiving education benefit payments for at least 2 consecutive school terms prior to the fielding period.
The number and types of questions answered will vary slightly based on the experiences of the target population (e.g., whether or not the respondent has had contact or not with the VA regarding their claim).
This section describes the universe of respondents for the survey, and the sampling plan that will be employed to achieve a representative sample for the survey.
1) Education Enrollment
Individuals who have received a decision on their education benefit application within 90 days (i.e., the original end-product has been cleared within the past 90 days) prior to the fielding period. The sample will be stratified as follows: (1) accepted and enrolled, (2) accepted and not enrolled, (3) denied.
TABLE 5: Individuals who have received a decision on an education benefit application within the past 90 days EXPECTED RESPONSE RATE AND SURVEY YIELD
|
||
Number of Claimants |
Expected Response Rate |
Completed Surveys Expected |
10,000 |
30% |
3,000 |
2) Education Servicing
Beneficiaries who have been enrolled and receiving education benefit payments for at least 2 consecutive school terms prior to the fielding period.
TABLE 6: Beneficiaries who have been enrolled and receiving education benefit payments for at least 2 consecutive school terms EXPECTED RESPONSE RATE AND SURVEY YIELD
|
||
Number of Beneficiaries |
Expected Response Rate |
Completed Surveys Expected |
10,000 |
30% |
3,000 |
The target population for the pilot Loan Guaranty Enrollment questionnaire will include individuals who closed a VA home loan in the 30 days prior to the fielding period. The sample will be stratified as follows: (1) those who closed on purchase loans, (2) those who received loans for interest rate reductions, and (2) those who obtained cash out or other refinancing. The target population for the pilot Specially Adapted Housing Servicing questionnaire will include individuals who are eligible for a specially adapted housing grant in FY 2009. The sample will be stratified as follows: (1) those who have not yet applied, (2) those who have applied but have not yet received a decision, (3) those who have received an approval on their grant and are currently somewhere in post-approval, (4) those who have had all their funds disbursed and final accounting is not yet complete, and (5) those who have had all of their funds disbursed and final accounting is complete.
The number and types of questions answered will vary slightly based on the experiences of the target population (e.g., whether or not the respondent has had contact or not with the VA regarding their claim).
This section describes the universe of respondents for the survey, and the sampling plan that will be employed to achieve a representative sample for the survey.
1) Loan Guaranty Enrollment
Individuals who closed a VA home loan in the 30 days prior to the fielding period. The sample will be stratified as follows: (1) those who closed on purchase loans, (2) those who received loans for interest rate reductions, and (2) those who obtained cash out or other refinancing.
TABLE 7: Individuals who closed a VA home loan in the past 30 days EXPECTED RESPONSE RATE AND SURVEY YIELD
|
||
Number of Beneficiaries |
Expected Response Rate |
Completed Surveys Expected |
10,000 |
30% |
3,000 |
2) Specially Adapted Housing Servicing
Individuals who are eligible for a specially adapted housing grant in FY 2009. The sample will be stratified as follows: (1) those who have not yet applied, (2) those who have applied but have not yet received a decision, (3) those who have received an approval on their grant and are currently somewhere in post-approval, (4) those who have had all their funds disbursed and final accounting is not yet complete, and (5) those who have had all of their funds disbursed and final accounting is complete.
TABLE 8: Individuals who were eligible for a SAH grant in FY 2009 EXPECTED RESPONSE RATE AND SURVEY YIELD
|
||
Number of Beneficiaries |
Expected Response Rate |
Completed Surveys Expected |
5,000 |
30% |
1,500 |
The target population for the pilot Vocational Rehabilitation and Employment Enrollment questionnaire will include individuals who had an initial meeting with their VR&E counselor and were granted a decision regarding their entitlement in the past 60 days prior to the fielding period. The sample will be stratified as follows: (1) those who applied, showed up for an initial appointment, were found entitled to and decided to pursue the program, (2) those who applied, showed up for an initial appointment, were found entitled to and decided not to pursue the program, (3) those who applied, showed up for an initial appointment and were not found entitled to the program. The target population for the pilot Vocational Rehabilitation and Employment Servicing questionnaire will include individuals who have entered and been enrolled in one of the five tracks for at least 60 days prior to the fielding period. The sample will be stratified as follows: (1) Veterans who are currently participating, (2) Veterans who have been rehabilitated, (3) Veterans who did not fully complete program (negative closures), and (4) Veterans who have reached maximum rehabilitation gain and could not proceed in program.
The target population for the pilot VR&E Escaped Beneficiary questionnaire will include individuals who dropped out of the program prior to completing a rehabilitation plan. The sample will be stratified as follows: (1) applicants who never attended the initial meeting with a counselor, (2) applicants who were determined to be entitled and did not complete a rehabilitation plan, and (3) applicants who started, but did not complete rehabilitation (i.e., negative closures)).
The number and types of questions answered will vary slightly based on the experiences of the target population (e.g., whether or not the respondent has had contact or not with the VA regarding their claim).
This section describes the universe of respondents for the survey, and the sampling plan that will be employed to achieve a representative sample for the survey.
1) Vocational Rehabilitation and Employment Enrollment
Individuals who had an initial meeting with their VR&E counselor and were granted a decision regarding their entitlement in the past 60 days prior to the fielding period. The sample will be stratified as follows: (1) those who applied, showed up for an initial appointment, were found entitled to and decided to pursue the program, (2) those who applied, showed up for an initial appointment, were found entitled to and decided not to pursue the program, (3) those who applied, showed up for an initial appointment and were not found entitled to the program.
TABLE 9: Individuals who had an initial meeting with their VR&E counselor and were granted a decision regarding their entitlement in the past 60 days EXPECTED RESPONSE RATE AND SURVEY YIELD
|
||
Number of Claimants |
Expected Response Rate |
Completed Surveys Expected |
10,000 |
30% |
3,000 |
1) Vocational Rehabilitation and Employment Servicing
Individuals who have entered and been enrolled in one of the five tracks for at least 60 days prior to the fielding period. The sample will be stratified as follows: (1) Veterans who are currently participating, (2) Veterans who have been rehabilitated, (3) Veterans who did not fully complete program (negative closures), and (4) Veterans who have reached maximum rehabilitation gain and could not proceed in program.
TABLE 10: Individuals who had an initial meeting with their VR&E counselor and were granted a decision regarding their entitlement in the past 60 days EXPECTED RESPONSE RATE AND SURVEY YIELD
|
||
Number of Beneficiaries |
Expected Response Rate |
Completed Surveys Expected |
10,000 |
30% |
3,000 |
1) Vocational Rehabilitation and Employment Escaped Beneficiary
Individuals who dropped out of the program prior to completing a rehabilitation plan. The sample will be stratified as follows: (1) applicants who never attended the initial meeting with a counselor, (2) applicants who were determined to be entitled and did not complete a rehabilitation plan, and (3) applicants who started, but did not complete rehabilitation (i.e., negative closures).
TABLE 12: Individuals who dropped out of the program prior to completing a rehabilitation plan EXPECTED RESPONSE RATE AND SURVEY YIELD
|
||
Number of Escaped Beneficiaries |
Expected Response Rate |
Completed Surveys Expected |
5,000 |
30% |
1,500 |
2. Describe the procedures for the collection of information, including: Statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose in the proposed justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.
The Voice of the Veteran Compensation and Pension surveys will entail stratified random sampling of 10,000 veterans for each of the four population segments outlined in Section 1. VA is using a 95% confidence interval for categorical variables for all surveys. There are no unusual procedures that will be required to draw a representative sample meeting these criteria.
The Voice of the Veteran Education surveys will entail stratified random sampling of 10,000 veterans for the each of the population segments outlined in Section 1. VA is using a 95% confidence interval for categorical variables for all surveys. There are no unusual procedures that will be required to draw a representative sample meeting these criteria.
The Voice of the Veteran Loan Guaranty Enrollment survey will entail stratified random sampling of 10,000 veterans and the Specially Adapted Housing Servicing survey will entail stratified random sampling of 5,000 veterans for the population segment outlined in Section 1. VA is using a 95% confidence interval for categorical variables for all surveys. There are no unusual procedures that will be required to draw a representative sample meeting these criteria.
The Voice of the Veteran Vocational Rehabilitation and Employment Enrollment and Servicing surveys will entail stratified random sampling of 10,000 veterans for the population segment outlined in Section 1. The Voice of the Veteran Vocational Rehabilitation and Employment Escaped Beneficiary survey will entail stratified random sampling of 5,000 veterans for the population segment outlined in Section 1. VA is using a 95% confidence interval for categorical variables for all surveys. There are no unusual procedures that will be required to draw a representative sample meeting these criteria.
VBA has obtained the services of a contractor to develop, administer, and analyze this set of surveys.
Strategies to Maximize Response Rates
VBA will employ methods to minimize respondent burden and to maximize survey response rates. This section identified the strategies to be employed to reach these objectives. Each strategy is outlined below.
Strategy # 1 to Maximize Response Rates: Using Web Technologies for Ease of Response
Veterans will have the option to complete the Voice of the Veteran surveys on paper or via the web-based form. The web address that the surveys will be posted on will be included in all of the mailing notifications as indicated below. This initial notification will include a cover letter and a URL and password. VBA expects most surveys will be administered online, which will maximize the timeliness, efficiency, and response rate of data collection.
Both the paper and web-based surveys will be developed with the end user in mind; a user-friendly form will help maximize response rates.
The online survey technology will incorporate several features to maximize response rates and respondent usability. These include a password system, which prevents any one person from completing more than one survey and allows respondents to begin the survey then come back at a later point to finish it.
Strategy # 2 to Maximize Response Rates: Using Advance and Follow-Up Mailings to Publicize the Surveys and Encourage Response
VBA will use a 4-step survey and follow-up process to administer the surveys (see Table 13 below). An increase in the overall response rate is the major advantage of using this process. The use of a letter as a follow-up tends to increase the response rate by between 5 and 8 percentage points.
Mailing |
Table 13: Mailing Material |
#1
|
Notification/cover letter w/ URL & password and paper survey
|
#2
|
Reminder notification/cover letter w/ URL & password and paper survey
|
#3
|
Second notification/cover letter w/ URL & password and paper survey
|
#4
|
Second reminder notification/cover letter w/ URL & password and paper survey
|
Strategy # 3 to Maximize Response Rates: Conduct Cognitive Labs/Pre-testing of Surveys
The contractor will conduct cognitive labs with three test users for each survey to determine whether respondents understand the survey questions and answer choices, as intended. VBA will provide the contractor with lists of potential test users. The contractor shall be responsible for securing the participation of test users, from this list. Prior to user testing, the contractor shall provide VBA staff with a list of the selected test-users.
VBA will conduct cognitive lab sessions aimed at identifying needed additions or refinements to the questionnaire. Cognitive labs are one-on-one or group sessions with potential survey participants, in which respondents are asked to complete the questionnaire while thinking aloud. The primary purpose of these sessions is to gather feedback on survey questions and answer choices to ensure they are easily understood and correctly interpreted. Outcomes of cognitive labs include, but are not limited to: addition or omission of specific questions, changes to wording of questions, clarification of question response options, addition of response options, and changes to ordering of questions.
The contractor will prepare a summary report of the cognitive testing session for the Voice of the Veteran surveys. The results of the cognitive labs will be taken into account when revising and finalizing the survey questionnaires.
Strategy # 4 to Maximize Response Rates: Maintaining a Toll-Free Survey Hotline
During the period that the survey is in the field, the contractor will provide and maintain a toll-free telephone line and dedicated e-mail address to answer any questions respondents may have about the survey (e.g., how to interpret questions and response items, the purpose of the survey, how to get another survey if their copy has been lost/damaged). Project staff will be available to answer telephone calls or emails during regular business hours (8:30 a.m.-6 p.m. ET). A voice messaging system will be available to receive phone messages after regular business hours so after-hours calls can be responded to within 24 hours. Respondents who utilize the e-mail option will be sent an initial automatically generated response email informing them that their e-mail was received and that they will receive a response within 24 hours.
Strategy # 5 to Maximize Response Rates: Excluding Questions of a “Sensitive” Nature
None of the questions included in the surveys are sensitive, or private in nature, which will encourage compliance.
Strategy # 6 to Maximize Response Rates: Assuring and Maintaining Confidentiality
Survey respondents for all surveys will be assured that their personal anonymity will be maintained. All hard copy questionnaires will be scannable, and consist of approximately eight printed pages, printed back to back with a numeric Litho-Code on the front and back cover. Veterans will be provided unique passwords that will allow the contractor to identify when a respondent has completed the survey and exclude them from further reminder letter or postcards.
Strategy # 7 to Maximize Response Rates: Secure Networks and Systems
The contractor will have a secure network infrastructure that will protect the integrity of the databases, the survey application, and all associated server resources. The servers must be protected by a strong firewall system and the operations center must be in a secure temperature-controlled environment with video surveillance, where network services are continually monitored by automated real-time programs to ensure the integrity and availability of all critical components. All key servers will be supported by a backup power supply that can continue to run the systems in the event of a power outage. Additionally, the contractors must be immediately alerted if critical monitor thresholds are exceeded, so that they can proactively respond before outages occur.
Approach to Examine Non-Response Bias
Non-response bias refers to the error expected in estimating a population characteristic based on a sample of survey data that under-represents certain types of respondents. Stated more technically, non-response bias is the difference between a survey estimate and the actual population value. Non-response bias associated with an estimate consists of two components – the amount of non-response and the difference in the estimate between the respondents and non-respondents. While high response rates are always desirable in surveys, they do not guarantee low response bias in cases where the respondents and non-respondents are very different. Two types of non-response can affect the interpretation and generalization of survey data: item non-response and unit non-response. Item non-response occurs when one or more survey items are left blank in an otherwise completed, returned questionnaire. Unit non-response is non-participation by an individual that was intended to be included in the survey sample. Unit non-response – the failure to return a questionnaire – is what is generally recognized as survey non-response bias.
There are two approaches to tackling the effects of non-response. One is to minimize the chances of non-response at the data collection stage. This may involve introducing measures, which aim to maximize the response rate. The other approach is to make statistical adjustments at a survey follow-up stage when all the data is collected. Both approaches are described in the next paragraphs of this section.
Since it is not always possible to measure the actual bias due to unit non-response, there are strategies for reducing non-response bias by maximizing response rates across all types of respondents. In the face of a long-standing trend of declining response rates in survey research (Steeh, 1981; Smith, 1995; Bradburn, 1992; De Leeuw & Heer, 2002; Curtin & Presser, 2005), these strategies include:
Use of notification letters, duplicate survey mailings, reminder letters and postcards.
Use of novelty in correspondence such as reminder postcards designed in eye-catching colors.
Use of an extended survey field-period to afford opportunities to respond for subgroups having a propensity to respond late (e.g., males, young, full-time employed).
Use of well-designed questionnaires and the promise of confidentiality.
Providing a contact name and telephone number for inquiries.
Employing these strategies to the administration of these surveys will be crucial for maximizing high response rates across all respondent types (see section on maximizing response rates above).
Non-response follow-up analyses can help identify potential sources of bias and can help reassure data users, as well as the agency collecting and releasing the data, of the quality of the data collected. The approach to examining the presence of non-response bias will be conducted as follows:
Compare the Demographics of Respondents from the Voice of the Veteran surveys to the Demographics of Non-Respondents from the Voice of the Veteran surveys. To examine the presence of non-response bias, VA will compare the demographics of responders (i.e., those who responded to each Voice of the Veteran Survey) to the non-responders (i.e., those who did not respond to each Voice of the Veteran Survey).
The comparison between responders and non-responders will be made on the following variables for this survey:
Region – it is possible that participants from a certain part of the country (i.e., region) may respond to the survey at a higher rate than those who are from another part of the country.
Age – it is possible that participants from a certain age group (i.e., over 30 years old) may respond at a higher rate than those who are members of another age group.
Gender – it is possible that participants from a certain gender (i.e., male) may respond at a higher rate than their counterpart.
Disability rating – it is possible that participants from a certain range of disability ratings may respond at a higher rate than another set of participants from another disability range.
Separation date - it is possible that participants who separated from the military at a later date may respond at a higher rate than participants who separated at a later date
Length of service – it is possible that participants who served in the military for longer periods of time may respond at a higher rate than participants who did not serve for as long a period
Based on the steps discussed above, VBA will identify issues with respect to non-response bias for the survey.
The contractor will conduct cognitive labs with three or more test users for the survey to determine whether respondents understand the survey questions and answer choices, as intended. Working closely with VBA, the contractor will draw a small pool of names from potential participants in each of the surveys for inclusion in the cognitive labs. Cognitive lab participants will be drawn from the same population that will be used for the main study. The contractor will submit the list of potential participants to VBA for review and approval. Once identified, the contractor will contact potential participants by telephone and ask them to participate. Cognitive lab sessions will take place in the metropolitan Washington, DC area or via conference call depending upon participant availability.
Once the participants have been selected, VBA will conduct cognitive lab sessions aimed at identifying needed additions or refinements to the questionnaire. Cognitive labs are one-on-one or group sessions with potential survey participants, in which respondents are asked to complete the questionnaire while thinking aloud. The primary purpose of these sessions is to gather feedback on survey questions and answer choices to ensure they are easily understood and correctly interpreted. Outcomes of cognitive labs include, but are not limited to: addition or omission of specific questions, changes to wording of questions, clarification of question response options, addition of response options, and changes to ordering of questions.
5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
The VBA C&P contact persons are:
Carolyn Bryant, [email protected]
Pamela Liverman, [email protected]
The VBA EDU contact persons are:
Bob Macomber, [email protected]
Pamela Liverman, [email protected]
The VBA LGY contact persons are:
Carleton Sea, [email protected]
Pamela Liverman, [email protected]
The VBA VR&E contact persons are:
Pamela Salazar, [email protected]
Pamela Liverman, [email protected]
VBA has contracted the services of JD Power & Associates (JDPA) to administer the survey. JDPA contacts are as follows:
Greg Truex, 805-418-8522
| File Type | application/msword |
| Author | Hee-Seung L. Seu |
| Last Modified By | capplive |
| File Modified | 2010-12-09 |
| File Created | 2010-12-09 |