Memo to OMB Desk Officer

Health Resources and Services Administration

National Health Service Corps Scholarship Application

(OMB NO. 0915-0146)

This memorandum addresses the issues identified under the Terms of Clearance for OMB Number 0915-0146, the National Health Service Corps (NHSC) Scholarship Application and responds to comments from the ICR submitted in February, 2007. The Terms of Clearance on the Notice of Action dated 11/21/2003 required that HRSA evaluate the information collection with regard to its ability to improve retention of NHSC scholars. Considerable concern was expressed regarding the necessity and effectiveness of the personal questions in the application process, particularly those in the personal interview. In addition, the terms required that the application be made online for respondents by the next submission. The issues raised in the 2003 Terms of Clearance were not new for the application process; when the application and interview were revised in 1996, OMB expressed concerns in the Terms at that time about the “utility of questions about the applicant’s personal life in selecting recipients.”

Following internal review of the application and interview, and an assessment of the practical use of the personal questions in the application process, it was decided that the interview did not contribute information that was useful in predicting completion of service obligation or retention. In February, 2007, HRSA submitted the NHSC application without the interview component to OMB to extend approval. The application form; however, still retained several sections with personal items that did not have clear relevance or utility for selection of scholars. The OMB desk officer provided comments and questions on the application that specifically asked about the utility of these items.

In late spring of 2007, HRSA established a new Bureau, the Bureau of Clinician Recruitment and Retention, and the National Health Service corps program was moved into this Bureau and is under new oversight. The new Bureau Administrator determined that a thorough review of the application process was necessary, and began internal review of the application materials, recent comments from OMB, and historical Terms of Clearance to revise and improve the application.

Following this review, and after careful consideration of the comments and questions from OMB, the program determined that the application form required substantial revision to give the information obtained from the form greater utility for making award determinations. HRSA withdrew the ICR from review to begin the process for revising the application and bulletin. The NHSC National Advisory Council provided consultation and expertise on the revisions to the application.

Sections of the application that did not have clear relevance to award determinations have been removed from the application. The sections on “Personal Preferences” and the “Activities Inventory Items” have been omitted from the application. New materials that applicants must now include are academic transcripts to provide information on academic performance and one academic reference (as outlined in 42 CFR 62.6). In addition, applicants will respond to five essay questions about career goals and provide a brief narrative response which will be reviewed by an Advisory Panel.

When HRSA submitted this ICR to OMB in 2007, OMB provided questions on the submission, and responses are provided below. As a result of the substantial revisions; however, some of the sections to which the questions refer are no longer in the application. This is noted in the response where applicable.

Questions from OMB on the ICR 0915-0146 submitted 02/20/2007:

Can HRSA please elaborate on what was involved in the assessment of the personal interview and the application? In particular, if retention data is not available for most of the 1998 medical scholars, how did HRSA determine that the personal interview did not provide enough utility?

Response: The personal interview component has had substantial questions from OMB reviewers over the years regarding its practical use, and the only support for the interview had been anecdotal information from interviewers. The evaluation that was conducted attempted to assess the predictive validity of the application process (application form and interview) for completion of obligated service and, where possible, retention. The variable retention was found to have severe limitations due to very small numbers of medical scholars that had applied following the last revision (1998), completed their education, completed their service obligation, and were now in service. Only the non-medical scholars had sufficient numbers for the analysis.

An important focus of the assessment was the interview component of the process. Regression equations were computed to predict completion of service obligation and retention for the 1998 non-medical scholars only, again, due to very restricted numbers of 1998 medical scholars. Retention data for the 1998 Medical Scholars receiving a scholarship award in their first year of training will not be available until approximately 2009 since the typical Medical Scholar does not service until up to 7-9 years after application and does not complete NHSC service until an additional 4 years have passed. The results indicated that the interview was not positively related to either outcome; indeed, the relationship between the interview score and either completion of service obligation or retention was nonexistent.

When the evaluation found that there was no predictive validity for the interview regarding completion or retention of the non-medical cohort, it provided solid evidence of the limitations of the interview component. There was no evidence to indicate that the interview component would have significant predictive validity for the medical cohort. The personal interview consisted of a series of completely open-ended personal questions and no method of scoring or coding responses was ever provided to the Reports Clearance Officer. Interviews had been conducted and scored by a contractor, not by program staff, and following these evaluation results, the program determined that the interview could not be justified.

The OMB Terms of Clearance for the interview component and personal questions of the scholarship application in 1997, 2000, and 2003 expressed considerable concern regarding the interview questions, and since the evaluation did not support the utility of the interview, it was discontinued. Instead, other information is now requested from the applicant, such as, faculty recommendations, academic transcripts, and a brief personal essay.

In evaluating the utility of the application instrument, what did the assessment show as far as the utility and predictive validity of the application? Why were revisions not made to the application?

Response: The evaluation that was conducted did not include an assessment of the utility and predictive validity of the overall application separately from the personal interview. The assessment looked at the correlation between the application scores and the interview scores as a composite, and at the interview as a predictor. Given the history of concerns expressed by OMB regarding the personal interview, this component of the application process received greater consideration regarding revisions to the application overall.

The evaluation did include a review of the personal questions in Sections E (Personal Preferences) and F (Activities Inventory Items) in the application for their relationship to completion of service obligation. These sections had been highlighted as a concern by OMB desk officers in previous reviews due to the personal nature of the questions and whether or not the items were relevant for the selection process.

Section E contained 39 pairs of statements that represented contrasting attitudes towards various issues and situations. Applicants selected one statement from each pair that best describes their beliefs and feelings. Section F contained a number of items concerning activities, to which the applicant would respond in terms of how well each item describes them. The evaluation results did not support the contribution of these sections in predicting the completion of the scholar’s service obligation; therefore, these sections have been deleted from the application form.

Is there some reason why the application does not require the reporting of information regarding the applicant’s work experience, faculty recommendation, and academic performance, as detailed in the CFR? In the context of this scholarship, it would seem particularly useful to have the applicant demonstrate their interest and commitment to working with underserved populations through the preparation of a short essay or activities list.

Response: HRSA agrees with the suggestions for the information listed above, and the revised application requires the reporting of information that is detailed in the CFR. Scholarship applicants are required to complete the application form and to provide supplemental materials to support their application. Faculty recommendations and academic transcripts are now required materials for a complete application to the NHSC Scholarship program.

In our opinion, the questions on the application with the most predictive validity seem to be questions 4d and 4e, which assess the applicant’s dedication to working with underserved populations. And yet these questions are optional. Is there a reason why?

Response: Applicants are required to complete these questions on the application form.

What were the characteristics predicted to be relevant to service retention, as developed by the Advisory Panel?

Response: The characteristics were developed by the Advisory Panel in the 1990s, and little documentation remains regarding the process and final determinations of the characteristics. Three Suitability scales were developed in the early years of the program: Suitability for Rural Assignment, Suitability for Urban Assignment, and General Suitability. Responses to the personal interview and Sections E and F were reviewed for these characteristics and interviewers rated applicants on these scales.

How will applicants know what the “community code” is for the areas in which they have lived? For example, how would they know that Chevy Chase, MD is a small city, while Los Altos, CA is a medium city?

Response: The community code is no longer required on the application.

For some questions, it is not entirely clear to us why the questions are relevant (e.g why does HRSA want to know whether the applicant finds “the neighborliness of a small town” desirable? Who would honestly prefer to “improvise or do without the most modern equipment”?). For other questions, the questions themselves seem relevant but the “right” answer seems to be too obvious (i.e. applicants would probably not disclose that they do NOT plan to work in a HPSA after fulfilling the service obligation, even if that was true). How will HRSA gain useful information from either of these types of questions? Are sections D-F optional?

Response: We agree, and the revised application addresses these issues and concerns. The issues referenced above resulted from the statements in sections D-F from the old application in which respondents indicated which statement best described their goals, their likes, and future plans. Certain sections have been completely removed from the application, and others have been revised, as outlined below.

The old application sections D-F were as follows:

Section D: Career Goals

Section E: Personal Preferences

Section F: Activities Inventory Items

Sections E and F have been completely removed from the application and are no longer asked.

Sections D-F in the revised application are as follows:

Section D: Degree Information

Section E: Background

Section F: Career Goals.

In Section F, the response categories “not very well, slightly, etc.” seem to be at odds with the question. To the statement “I am the leader in my group,” for example, the response would probably be “strongly agree, agree, neither agree nor disagree, etc.”

Response: This section, the Activities Inventory Items, has been removed from the new application form. It has been replaced with a new section F, “Career Goals”, consisting of essay responses to five questions that ask the applicant to describe their goals related towards serving underserved populations and medically underserved communities, as well as future professional plans.

The instructions appear to be relevant for paper-based applications only. Is there a paper-based version of this application? And are there instructions for the web-based application?

Response: The NHSC Scholarship Application and the Scholarship Bulletin are web-based and available online. At the time of the previous submission of this ICR to OMB, the application form had just been converted to an online application, and the Scholarship Bulletin that was on the web still had references to the form as though it were on paper. The Bulletin has been revised and now reflects the online submission process.

5