National Institute of Food and Agriculture
U.S. Department of Agriculture
OMB No. 0524-0050
Veterinary Loan Repayment Program
SUBJECT: Supporting Statement B. Statistical methods for surveys associated with the Veterinary Loan Repayment Program (VMLRP).
Respondent Universe and Sampling Methods
(Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate during the last collection.)
Participant and Employer Feedback Surveys. Each VMLRP award cohort consists of approximately 50 veterinarians (participants) with 30 employers (~20 of 50 participants are self-employed). Results will not be used to make statements representative of all VMLRP participants and are not intended to be generalizable beyond the respondent group.
Post-Award Termination Survey. All participants will receive a post-award termination award survey at one, three and five years post-contract completion. Assuming an average of 50 participants each year, 45 participants need to respond to have a confidence level of 95% and 5% margin of error in the calculated retention rate for a particular cohort. The program is interested in the retention rate of veterinarians in shortage situations for the program as a whole, versus for each cohort, at one, three and five years post-contract; therefore, cohorts will be merged based on the number of years post-contract as there is no reason to assume the factors surrounding retention would be any different among cohorts.
Response rate will be participant driven. The National Health Services Corps (NHSC), a program for physicians that is similar to VMLRP, surveys their loan repayment participants one year post contract and achieves a 58.9% to 75.2% response rate. For VMLRP, we estimate that the response rate will be at least a 65%. Thus with the current number of participants, we do not anticipate having a large enough sample of respondents for statistical analysis until we have surveyed four cohorts at each post-contract time point. Until these numbers are reached, retention rates will not be considered generalizable to the program and all analyses will be restricted to the respondent group.
(Describe the procedures for the collection of information including:
Statistical methodology for stratification and sample selection;
Estimation procedure;
Degree of accuracy needed for the purpose described in the justification;
Unusual problems requiring specialized sampling procedures; and,
Any use of periodic (less frequent than annual) data collection cycles to reduce burden.)
No estimation or stratification is used because we will survey 100% of the VMLRP participant population and their employers. Participant and employer contact information will be obtained from the VMLRP administrative database. This database is continually updated as new participants join the program, change employers and as status or contact information changes for participants. The VMLRP also uses information from the database to generate all reports about the program. Quarterly communications via email (and phone if needed) occur between VMLRP staff, participants and participants’ employers throughout the program and continue one year after completion of a contract for tax purposes. Upon completion of a VMLRP contract the information is retained.
Participant and Employer Feedback Surveys. Participant and employer contact information will be obtained from the VMLRP database. Participants and employers will be e-mailed a link to a web-based survey instrument or, if a web-based survey instrument cannot be used, the survey will be attached to the email as a pdf-fillable form. For employers who primarily communicate with the program via fax, a copy of the survey will be faxed to them. Participants will receive the survey after they complete the first year of their contract. Employers will receive the survey during the last quarter of the participant’s contract.
We are interested in qualitative information about the program. All analyses will involve frequencies and text analysis.
Post-Award Termination Survey. Participant contact information will be obtained from the VMLRP database. The VMLRP database is not updated after one year post-contract. Therefore, as time goes on it is possible that VMLRP will lose contact with some past participants due to lack of updated contact information. Attempts will be made to obtain updated contact information (see Section 3b below). Non-response due to lack of updated contact information was assumed when estimating the anticipated response rate of 65%.
At one, three and five years post-contract, participants will be e-mailed a link to a web-based survey instrument or, if a web-based survey instrument cannot be used, the survey will be attached to the email as a pdf-fillable form. A modified version of Dillman’s Total Design Method for mailed, self-administered questionnaires will be employed.
Data from responses received in electronic format will be directly loaded into a database. Responses received in hard-copy format will be keyed, edited, entered in the database manually, and reviewed by another person for accuracy.
Retention in our analyses will be defined as a proportion of participants retained in shortage situations at one, three and five years post-contract. Reasons for retention or lack of retention will be described descriptively with no generalizations.
(Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.)
Participant and Employer Feedback Surveys. Information collected for the feedback surveys will not yield generalizable quantitative findings; it can provide useful customer input, but it will not yield data about customer opinions that can be generalized.
Post-Award Termination Survey. Most veterinarians being surveyed are relatively young and technology-oriented, so all surveys will be sent electronically. To accommodate those not wanting to respond via the Internet, the email invitation will offer the option to request a pdf-fillable form (when a web-based survey instrument is used) or to have a paper copy sent to them which can then be returned by U.S. Mail. Additionally, non-respondents will be sent a reminder along with the link or copy of the survey two, four and six weeks after the initial email inviting them to take the survey.
Rejected emails and other indications of bad email addresses will be followed up by attempting to contact the participant by phone at the number on file and/or at the last place of known employment. If all contact information on file appears to be outdated the state veterinary board in which the participant was last known to be practicing will be contacted for current contact information.
Surveys are sent to a population in which the program already has relevant demographic information. Non-respondents’ demographics will be compared to those responding to the survey and assessed for any significant differences. Any significant differences found will be adjusted for in the overall analysis.
(Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.)
All survey questions were developed based on the needs of the program and after reviewing survey questions from other loan repayment programs assessing retention. Pretesting was done with internal staff and seven current or past VMLRP participants.
(Provide the name and telephone number of individual consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.)
NIFA did not consult with any outside individual on the statistical aspects of the design of its survey. NIFA employees who will collect and/or analyze the information are:
Michelle Colby
202-401-6802
With analysis support as needed from:
Jason Boim
202-401-6188
Lisa Stephens
202-401-6438
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | THE SUPPORTING STATEMENT |
| Author | arfqrs |
| File Modified | 0000-00-00 |
| File Created | 2021-01-15 |