Terms of clearance memo

Date: February 1, 2007

To: OMB Desk Officer

Subject: Terms of Clearance, OMB No. 0915-0146

From: HRSA Desk Officer

This memorandum addresses the issues identified under the Terms of Clearance for OMB Number 0915-0146, the National Health Service Corps (NHSC) Scholarship Application. The Terms of Clearance on the Notice of Action dated 11/21/2003 required that HRSA evaluate the information collection with regard to its ability to improve retention of NHSC scholars. Considerable concern was expressed regarding the necessity and effectiveness of the personal questions, particularly those in the personal interview. In addition, the terms required that the application be made online for respondents by the next submission.

An evaluation of the NHSC Scholarship Award Process was completed in March, 2006. The primary purpose of the project was to evaluate and validate the instruments used to select NHSC scholarship recipients. After careful consideration of the recent assessment of the application and personal interview, HRSA has determined that the personal interview questions do not provide a valid instrument for purposes of NHSC retention. As a result, the personal interview has been discontinued and is not included in this clearance request.

Since the early 1990's, the process for applying for the NHSC scholarship program involved two stages. A written application form was submitted and based on the application information, a personal interview was conducted. Individuals were selected for awards based on the information provided in both the application and the interview. The personal interview contained questions on the applicant’s personal life, family background and stability, family conflicts and stress, and other questions of a personal nature. These questions were developed to attempt to identify applicants with personal characteristics that would assist in determining a commitment to working with the underserved.

The mission of the NHSC is the improvement of the delivery of health care in health professional shortage areas by the appropriate placement of health professionals. The scholarship program has been used as an incentive to attract individuals committed to serving the underserved. One of the selection criteria required by Federal statute and regulation specifies that priority will be given to an “individual who has characteristics that increase the probability of continuing to serve in a Health Professional Shortage Area (HPSA) after the period of obligated service is completed.”

In developing the process for applying to the program, Public Health Service staff and consultants were asked to review the job requirements, personal qualities, and interview and application materials with respect to the ability to identify the personal characteristics referenced above for making scholarship selections. The selection process that was developed involved the application form and the personal interview; a semi-structured interview consisting of completely open-ended responses to personal questions. The personal interview has experienced difficulty in the clearance process due to its subjective nature, its personal questions, and the lack of coded response categories for such items. As a result of the OMB clearance process in both 2000 and 2003, the program was given ‘Terms of Clearance’ requiring an assessment of the requirement and utility of the personal interview.

Although the assessment included an examination of the utility of the application form and interview in predicting retention, measuring retention was not possible for all scholars since the current selection process for the NHSC scholarship program has not been in place long enough to capture these data. The typical medical scholar does not serve until up to 7 to 9 years after submitting an application, and does not complete NHSC service until an additional 4 years have passed. Thus, retention data will not be available for most of the 1998 medical scholars receiving a scholarship award in their first year of training until approximately 2009. Most of the non-medical scholars awarded scholarships in 1998 have retention data available unless an approved deferral occurred during the training program or NHSC service obligation. The analysis of the selection process and eventual service and retention of health care professionals in HPSAs was conducted using non-medical scholars, since they were the only group with available outcome data.

The evaluation also examined the extent to which service obligations were met from those applicants awarded scholarships. (No data were available on applicants who were not selected.) The percentage of 1998 non-medical scholars completing their service obligations was 96%, and the percent of 1998 non-medical scholars retained in a medically underserved area following their service obligation was 72%. It is possible that self selection plays a significant role in this process, as those individuals applying to a program of this nature might already be personally predisposed to serving in underserved areas.

The evaluation determined that the selection process was operating effectively for non-medical scholars, evidenced by the high completion and retention rates of non-medical scholars selected in 1998. Results did not support the predictive validity of the selection instruments regarding completion and retention for the 1998 medical cohort, due to the lack of maturity of the outcome data.

The scholarship selection system was developed using a content validation approach to ensure that the application and interview were measuring behaviors and personal preferences consistent with those needed to be successful in the scholarship program; however, predictive validity analyses showed no relation between the composite score for these instruments and service obligation completion or retention. This lack of predictive validity for the composite score can be potentially explained by data-related artifacts that minimize the probability of obtaining strong predictive validity findings for these types of data. The findings are not surprising given that the selection process was attempting to predict career decisions many years after the instruments were administered. Despite alternative explanations, the program determined that the personal interview should be discontinued, and information from the application would be used for selection without any interview questions.

In conclusion, terms of clearance were issued by OMB for this activity in 1997, 2000, and 2003 due to considerable concerns with questions in the interview about the applicant’s personal life and the utility of such questions. The evaluation results failed to provide evidence that the combined application and interview instruments could successfully predict retention. Given this finding and the concerns about the utility of the personal interview questions, the interview component has been discontinued in the selection process.

Regarding electronic capability: when the application was submitted to OMB for approval in 2003, the instrument was still in a paper form. Since that time, the instrument has been converted to an online application and is fully electronic.

2