The Department of Defense (DoD) TRICARE Management Activity (TMA) Office of Force Health Protection and Readiness (FHP&R) is requesting OMB approval to conduct a survey of former military psychologists and psychiatrists to improve the retention of mental health providers in the military. FHP&R refers to this effort as the Retention of Behavior Health Providers Survey (RBHP). Specifically, FHP&R is seeking OMB approval for a questionnaire and related correspondence.
The RBHP is authorized by the National Defense Authorization Act of 2008, Section 1618, which requires the secretaries of DoD and the VA “to conduct joint planning for the prevention, diagnosis, mitigation, treatment, and rehabilitation of, and research on” traumatic brain injury (TBI), post-traumatic stress disorder (PTSD), and other mental health conditions among military personnel.
The survey is being conducted in response to the 2007 DoD Mental Health Task Force report, which found that “the military system does not have enough fiscal or personnel resources to adequately support the psychological health of service members and their families in peace and during conflict.” The task force itself was established by Section 723 of the National Defense Authorization Act of 2006, in which Congress required the Secretary of Defense to establish a task force to examine all matters “relating to mental health and the Armed Forces” and to produce a report that would both assess DoD mental health services and provide recommendations for those services. One of the key findings of the task force was that there are not enough active duty mental health professionals serving in the military to meet the needs of its personnel, and that the supply of these professionals would likely decline in coming years. Not surprisingly, a key recommendation of the report was “to ensure an adequate supply of military [mental health] providers.”
The proposed survey was designed in response to those findings and recommendations; its twin goals are to determine the key factors in military mental health providers’ decisions to join as well their decisions to leave military service, and to identify incentives that might have encouraged them to extend their service. These findings will inform the development of policy recommendations by FHP&R that would improve the retention of active duty psychologists and psychiatrists.
The primary purpose for this data collection is to better understand factors that contribute to uniformed mental health professionals’ decisions to enlist in the military, as well as the factors that cause them to leave military service. This information will help inform the development of effective policies so that the military health system can recruit and retain high quality mental health professionals.
Data will be collected and analyzed by Mathematica Policy Research (Mathematica) under Contract 7200004115, Task Order M0110. Mathematica is a subcontractor to Lockheed Martin Corporation, which is under contract to DoD W81XWH-08-D-0039 DO 003, titled "Retention of Behavior Health Providers Survey (RBHP).” Findings from the analysis will be presented to FHP&R in the form of a final report and briefing on the results to FHP&R.
For the RBHP, the primary mode of data collection will be a web-based instrument. It will be supplemented by a hard copy version of the instrument, which will be used to provide the respondent the option to submit responses by mail, and a follow-up telephone reminder script, which will allow a telephone interviewer to contact a sample member and, if warranted, complete the survey using the web form. The follow-up telephone call will provide the respondent with the opportunity to complete the survey as an interview over the telephone. Over the course of an eight-week field period, Mathematica will make several attempts to encourage respondents to complete the web survey (see Table 1).
TABLE 1.
PRACTICES TO ENCOURAGE RESPONDING VIA THE WEB SURVEY
Prior to field period |
Advance letter |
Week one |
Email invitation or hard copy packet if email not available |
Week two |
Reminder notice to those with email addresses |
Week three |
Second reminder notice to those with email addresses, first reminder by US mail to those without email addresses |
Week four |
Non-respondents w/o email addresses receive telephone reminder, with option to complete survey as a telephone interview |
Week five |
All non-respondents sent a hard copy packet |
Weeks six through eight |
All non-respondents receive telephone reminder, with option to complete survey as a telephone interview |
We expect that at least 50 percent of those responding will complete the web-based version of the questionnaire, and that a maximum of 50 percent of those responding will complete the hard copy version or elect to complete the survey as a phone interview. To minimize the burden on respondents, we will accept data in any of the three formats. Any possible mode effects will be tracked and analyzed.
The advance letter to all sample members will explain the survey and alert them that they will be contacted in the coming weeks. Mathematica will also set up a toll-free help desk number and an email address to answer the respondents’ questions and to help them complete the web survey.
The proposed data collection does not duplicate any other effort, and the information cannot be obtained from any other source. Some data relevant to the study, such as provider rank, end of obligation, etc., are available in existing records and therefore are not items in the survey. Only those questions for which we need to collect new, up-to-date data or to confirm existing data are in the survey instrument, which has 35 questions and we project will require approximately 15 minutes to complete.
Some of the services have conducted their own research in this area. The Navy recently (May 2009) completed and released findings from a Needs Assessment Survey developed by the Navy Psychology Steering Committee. While the survey covers many of the same topics as our proposed study, including key factors impacting retention such as deployments and working conditions, it was administered to only 111 members of the Navy Psychology listserv. We carefully considered the findings and recommendations from this survey in the construction of the current TMA survey instrument, but feel our study is not redundant in that it surveys both psychologists and psychiatrists from all three branches of the service, and will provide some common metrics and a large enough sample size by which to make comparisons across subgroups such as provider type, rank, years in service, number of deployments, etc.
The RBHP survey is unique in that it will capture critical data from providers who have ended their military careers. It is also unique in that it captures consistent data from a sample of a fairly large population of personnel in all three branches of the military, allowing us to make comparisons across services and other subgroups.
No small entities are involved in the study.
This is a one-time data collection limited to an eight-week period and to mental health professionals volunteering to participate in the study.
This data collection complies fully with 5 CFR 1320.5(d)(2).
The notice soliciting public comment on this data collection required in 5CFR 1320.8(d) was published in the Federal Register on October 30, 2009 (volume 74, number 209, pages 56184-56185). No comments were received in response to this notice.
No payment or gifts are provided to respondents for participating.
Mathematica will take several steps to assure respondents that the information they provide will be treated as confidential and used for research purposes only. Respondents will not be identified individually in any reports or in any communications to TMA. The assurances and limits of confidentiality will be made clear in advance material mailed to sample members and will be restated on the first page of the web survey and the paper instrument, and at the beginning of any telephone interview. Respondents will also be told that their participation in the survey is voluntary, and that they have the option to refuse to answer questions.
Access to the web survey. Mathematica will restrict access to the RBHP web instrument to protect the confidentiality of respondents. Each sample member will be assigned a unique ID and password, which will be included in emails and letters. The web instrument will be hosted by Mathematica on a secure (SSL) web server.
Data will be processed and stored on Mathematica’s password-protected local area network (LAN). Mathematica protects its LAN with several security mechanisms available through the network operating system. Access to confidential information stored on LAN directories is restricted to authorized project staff by means of IDs and passwords. In addition, network servers containing confidential information are kept in a locked area. Finally, all staff assigned to work on the project sign a confidentiality pledge as a term of employment. The pledge requires staff to maintain the confidentiality of all information collected.
Only project staff who have the appropriate government security clearances will have access to the completed surveys and the resulting electronic database. Once the data is entered from them, the hard copy surveys will be stored for one year in a locked cabinet in the Mathematica project director’s office. After one year, the hard copy surveys will be shredded and destroyed. For the telephone interviews, interviewers will enter the respondents’ answers directly into the web survey program as the survey is conducted. Access will be limited as explained on the previous page.
There are no questions of a sensitive nature in the RBHP.
A pilot study of the survey instrument was conducted with five military health providers, and the estimates of response burden are based on this study.
As shown in Table 2, the estimated time to respond to the questionnaire is 15 minutes. The estimated cost per hour for former military psychiatrists and psychologists is drawn from the Bureau of Labor Statistics’ Wage Statistics (May 2008, http://www.bls.gov/oes/2008/may/oes_nat.htm#b29-0000).
TABLE 2.
ESTIMATED TOTAL RESPONSE BURDEN FOR THE RBHP SURVEY OF FORMER PROVIDERS
Respondent Type* |
Number of Respondents |
Responses |
Average Hours per Response |
Total Hour Burden |
Cost per Hour |
Total Cost Burden |
Former military psychiatrists |
90 |
1 |
.25 |
22.5 |
$74 |
$1,665 |
Former military psychologists |
74 |
1 |
.25 |
18.50 |
$43 |
$796 |
Total burden cost |
|
|
|
|
|
$2,461 |
*The RBHP will also survey 396 current military behavior health providers. Since OMB clearance is not required for the survey of current military providers, this table reflects the burden for former providers only.
There are no capital, start-up, operations, or maintenance costs to respondents associated with this project.
The RBHP will be executed under a 23 month contract for which the cost of the data collection is approximately $160,000, of the total contract value of $334,522.
This is a new collection.
The RBHP is a survey of 566 current providers (of whom 70 percent, or 396, are expected to respond), for which OMB clearance is not required, as well as 234 former providers (of whom 70 percent, or 164, are expected to respond), for which OMB clearance is required. The survey of former providers is scheduled to start after the survey of current providers (see Table 3 for the former provider survey schedule). The sample design and instrument development process is the same for both surveys, and at the time of this application, sample and instrument design as well as instrument testing are complete.
Results of the survey will be presented for both psychologists and psychiatrists and by branch of service for branches with at least 50 respondents. A complete set of descriptive measures from the survey will be presented in table format, with accompanying text and charts highlighting the key findings. In addition to the descriptive analysis, the report will present bivariate analysis of key subgroups, with an interpretation of findings focusing on possible recommendations. Cross-tabulations may include, in addition to provider type and service branch, a comparison of reasons for termination of service with length of service, number of tours of duty, age, and rank.
An exemption for the requirement to display the expiration date is not requested.
This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.
TABLE 3.
RBHP SCHEDULE FOR SURVEY OF FORMER PROVIDERS
Task |
Deliverable |
Dates |
Project management |
|
Ongoing |
Survey development |
|
Completed |
Clearance |
TMA IRB clearance OMB clearance DMDC clearance and licensing* |
In process |
Implement survey |
Data collection |
4/25/10–6/18/10 |
Locating round 1 |
4/23/10–5/25/10 |
|
Locating round 2 |
6/1/10–6/18/10 |
|
Data processing |
6/18/2010–6/25/2010 |
|
Prepare table shells |
4/5/10–4/19/10 |
|
Analysis |
Conduct analysis |
6/18/2010–6/25/2010 |
Draft report |
6/25/2010 |
|
Report and briefing |
Draft briefing slides |
6/25/2010 |
Final report |
6/30/2010 |
|
Final briefing |
6/30/2010 |
*DMDC is the Defense Manpower Data Center
The respondent universe for the full RBHP study includes both current and former military psychiatrists and psychologists (Table 4). The current group includes those serving on active duty in any branch of the military at the time the survey is conducted. The former military psychiatrists and psychologists include those who separated from military service within the three years before the survey was administered; that is, since December 1, 2006. The sampling frame has been provided by Defense Manpower Data Center (DMDC).
The sample design for the proposed study is a stratified random sample with four strata defined by the person’s current status with the military (current or former military) and occupational specialty (psychiatrists or psychologists). Using the DMDC data file, the listing of current and former military psychiatrists and psychologists will be partitioned into these four strata. We describe here the full design, but, as noted before, we seek clearance only for the survey of former military psychiatrists and psychologists.
The target sample size of completed interviews was determined based on the desired precision for estimates within each of the four strata and for comparative analysis across the strata (such as comparing the difference in the percentages on the attitudes for current military psychologists relative to former military psychologists). As shown in Table 4, current tabulations indicate that there are only 234 former military psychiatrists and psychologists (129 former military psychiatrists and 105 former military psychologists), but approximately 1,351 current military psychiatrists and psychologists. To permit the precision desired for comparing estimates for former and current psychiatrists and psychologists, we propose the selection of all former military psychiatrists and psychologists and a sample of 566 current military psychiatrists and psychologists for a total sample size of 800 former and current military psychiatrists and psychologists.
In each of the two strata of former military psychiatrists and psychologists, we project a response rate of 70 percent and this will result in approximately 90 former military psychiatrists and 74 former military psychologists. By assuming that the propensity to respond can be treated as equivalent to sample selection, the response rate of 70 percent becomes a finite population correction.1 Under this assumption, the 129 former military psychiatrists in the sample is projected to result in 90 respondents and this sample size will allow for a 95 percent confidence interval of less than +/- 5.6 percentage points (the confidence interval half-width) around an estimated percentage near 50% (for example, the percentage of former military psychiatrists who left the military for a specific set of reasons). Similarly, the sample of 105 former military psychologists is expected to result in 74 respondents, allowing for a 95 percent confidence interval of less than +/- 6.3 percentage points. For comparison of percentages between the strata (for example, between former psychiatrists and former psychologists), a difference of 12 percentage points can be detected using a 2-sided test with an alpha level of 0.05 and statistical power of 80 percent.
Assuming that the sample allocated to the current military psychiatrists and psychologists is as presented in Table 4, we expect that for comparisons of percentages between current and former psychiatrists and psychologists, a difference of 8.6 percentage points can be detected using a 2-sided test with an alpha level of 0.05 and statistical power of 80 percent. For comparisons of percentages for current and former psychiatrists, we would expect to detect a difference of approximately 11 percentage points and 12 percentage points for current and former psychologists.
For our power calculations, we assumed that all those listed on the frame are eligible and a response rate of 70 percent. We will select all former military psychiatrists and psychologists for the proposed survey, and we expect approximately 164 completed surveys.
TABLE 4.
SAMPLING FRAME AND SAMPLE SIZES FOR RETENTION OF BEHAVIOR HEALTH PROVIDERS (RBHP) SURVEY OF CURRENT AND FORMER PSYCHIATRISTS AND PSYCHOLOGISTS
Respondent Type |
Sampling Frame |
Sample Size |
Target Number of Completed Surveys |
Current military psychiatrists |
517 |
283 |
198 |
Current military psychologists |
834 |
283 |
198 |
Former military psychiatrists |
129 |
129 |
90 |
Former military psychologists |
105 |
105 |
74 |
Total respondents |
1,585 |
800 |
560 |
Source: DMDC data file of current and former psychiatrists and psychologists and Mathematica computations.
For estimation purposes, weights will be computed for responding former military psychiatrists and psychologists. These weights will essentially be the inverse of the response rate because we propose all former military psychiatrists and psychologists will be selected with certainty.
Because we project a response rate of 70 percent, we will conduct a nonresponse bias analysis to determine what factors may be associated with nonresponse and potentially result in bias in the survey estimates. This nonresponse bias analysis will assist in identifying factors to use to form weighting classes for the computation of the weights for the former military psychiatrists and psychologists. Because we have limited information on the former military psychiatrists and psychologists, we will use bivariate and multivariate cross-tabulations to explore for differential nonresponse patterns.
In order to maximize response rates, MPR will monitor data collection throughout the eight-week field period, and will work in several ways to locate and prompt non-respondents throughout that time.
In preparation for week 1, one week before the start of data collection, MPR will send an advance letter to all of the psychologists and psychiatrists in the sample. This letter will not contain the actual invitation to take the survey with username and password, which will be included in the invitation sent at the start of data collection one week later.
Week 1. During the first week, no later than three to five days after the advance letter mailing, MPR will issue invitations:
• For those with e-mail addresses on record, MPR will send personalized email invitations, general survey information, and a hyperlink to the survey website. A hyperlink to a PDF version of the survey will also be provided for those preferring to complete the survey in hard copy. Embedded in the hyperlink will be the psychologists and psychiatrists’ unique usernames and passwords.
• For those without email addresses, MPR will mail a hardcopy of the same materials. During this week, MPR will learn whose email addresses are invalid and a hard copy of the study materials will be mailed to these psychologists and psychiatrists. All respondents who are sent a hard copy packet will be provided with pre-labeled, postage-paid, return envelopes to facilitate response.
Week 2. During the second week, MPR will send out thank you/reminder notices via email to those with email addresses who have not yet completed the survey. Those who were sent hard copy surveys will not be contacted in the second week.
Week 3. MPR will repeat this procedure in Week 3, for those who have not yet responded. MPR will also send out thank you/reminder notices via regular mail to those without email addresses who were sent a hard copy survey, who have not yet completed the survey.
Week 4. In Week 4, the psychologists and psychiatrists with valid email addresses who have still not responded will receive another e-mail reminder; while those with no email addresses will receive telephone reminders to complete the survey either via the web or on hard copy. As part of the telephone reminder call, those with no email addresses will be given the option to complete the survey during the call as a telephone interview.
Week 5. In week 5, those who have not yet responded (both those with valid email addresses and those without) will receive hard copy questionnaires and pre-labeled, postage-paid, return envelopes to facilitate response.
Weeks 6-8. In weeks 6, 7 and 8, those who have not yet responded (both those with valid email addresses and those without) will be contacted by phone and asked to participate in a telephone interview. Trained MPR staff will conduct the interviews.
MPR will devote up to 480 person hours to locating the current and the former provider respondents. The great majority of these person hours are expected to be used in locating the former provider respondents, as a smaller percentage of these are expected to have current email addresses available. Among the locating approaches that can be brought to bear are the following:
1. Professional association files
2. Lexis-Nexis Person and Business Locators (a commercial service to which MPR subscribes)
3. Meta-directory of databases for the field of psychology and psychiatry
• http://www.webgator.org/gator77.htm, which provides links to more than 70 medical directories and databases ranging from the AMA’s physician search website (for psychiatrists) to many state licensing boards and several mental health care provider meta-sites.
• Association of State Medical Board Executive Directors “Docfinder” website, http://www.docboard.org/, which lists both state medical licensure sites for psychology and psychiatry and locator sites maintained by state departments of health
4. Specialty organizations
• The American Board of Medical Specialties maintains a meta-website at http://abms.org/ that permits searches both within and across specialties, including psychiatry.
• Lexis-Nexis Asset Locator - especially for providers at or nearing retirement age, MPR will consult this source to determine whether a psychologist or psychiatrist has recently bought or sold property.
• Foreign medical websites - such as http://www.indmedica.com (for India), if information leads us to believe the individual may have returned to his or her homeland.
5. Worldwide person locator: Advanced Search page http://hotbot.lycos.com to see if any published articles or other information provides clues. This general search engine is especially useful for finding specialty and foreign medical websites by permitting complex mathematical (Boolean) searches with pre-specified parsing.
6. General Population Locators - three-tier directory assistance process:
• Checking by name, address, and phone number
• Accurint—a product that enables searching by name, address, date of birth, employment status, criminal records, professional licenses, and other choices, alone or in combination; and
• Web searching
• Written correspondence or telephone calls to potential contacts of the sample member, such as family members
With this approach, MPR expects to achieve a 70 percent response rate.
No tests of procedures were undertaken, beyond pilot testing of the survey instrument with five respondents. The pretest was used to refine question wording and to determine that 15 minutes would be required to complete the survey. The pretest participants are detailed in Table 5.
TABLE 5.
SUMMARY OF PRETEST OF QUESTIONNAIRE
Respondent Type |
Male |
Female |
Total |
Current military psychiatrists |
3 |
1 |
4 |
Current military psychologists |
1 |
0 |
1 |
|
|
|
|
Total respondents |
|
|
5 |
The following people have contributed to the study design and the design of the survey instruments, sample design, and data collection procedures:
A. TMA Force Health Protection and Readiness
Dr. Jill Carty, Health Prevention Coordinator, Psychological Health Strategic Operations
TMA Force Health Protection and Readiness
Psychological Health Strategic Operations
Falls Church, VA 22041-3258
703-845-3317
Dr. Jill Carty is the project officer for the federal government.
B. Mathematica Policy Research, Inc.
Mathematica Policy Research, the subcontractor, will have overall responsibility for conducting the project and for preparing all tables.
Patty Cloud, Project Director
Mathematica Policy Research
111 E. Wacker Drive Suite 920
Chicago, Illinois 60601
(312) 994-1006
Frank Potter, Statistical Consultant
Senior Fellow and Associate Director of Statistical Services
Mathematica Policy Research, Inc.
P.O. Box 2393
Princeton, NJ 08543-2393
(609) 936-2799
Mail: [email protected]
1 In sampling theory, the finite population correction is an adjustment to the sampling variance to account for the relative size of the sample to the sampling frame. The finite population correction is 1 – n / N, where n is the sample size and N is the number of units in the sampling frame. For an estimated proportion p, the variance estimate is (1 – n / N) * var(p).
DRAFT
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | RBHP (6551) OMB Supporting documentation |
| Subject | RBHP OMB supporting documenation |
| Author | Patty Cloud |
| File Modified | 0000-00-00 |
| File Created | 2021-02-02 |