Pra-2126new.nrcme.ss.070606.ost

Information Collection Clearance

The Supporting Statement

Survey of Medical Examiners Who Certify the Physical Qualifications of Commercial Motor Vehicle Drivers

This is to request approval from the Office of Management and Budget (OMB) for a new survey on the role of medical examiners who certify the physical qualifications of commercial motor vehicle (CMV) drivers.

1. Circumstances that make collection of information necessary.

The Secretary of Transportation is required to establish and maintain a current national registry of medical examiners who are qualified to perform examinations and issue medical certificates that verify whether a CMV driver’s health meets the Federal Motor Carrier Safety Administration (FMCSA) standards. See 49 U.S.C. § 31149(d) (Attachment A) added by section 4116(a) of the Safe, Accountable, Flexible, Efficient Transportation Equity Act: A Legacy for Users (SAFETEA-LU) (Attachment B). In addition, section 4116 (b) of SAFETEA-LU amends 49 U.S.C. § 31136(a)(3) (Attachment C) to require that the physical examinations of CMV operators are performed by medical examiners who have received training in physical and medical examinations standards, and, after the National Registry is established, are listed on the Registry.

FMCSA is developing the National Registry of Certified Medical Examiners (NRCME) program as one initiative to improve highway safety by producing trained, certified medical examiners who can effectively determine if a CMV driver’s health meets FMCSA standards. FMCSA’s goal is to improve safety and reduce fatalities on our Nation’s highways by 41 percent (%) by 2008. FMCSA has determined that focusing on driver factors, including medical issues, is one component of reaching this goal.

Once the NRCME program is implemented, FMCSA would accept physical examinations performed only by certified medical examiners listed on the NRCME, as required by 49 U.S.C. § 31149(d)(3). The NRCME program would require medical examiners to complete training developed from standardized curriculum specifications and pass a national certification test. The procedures used to develop and maintain the quality of the program are expected to be in accordance with national accreditation standards for certification programs established by the National Commission of Certifying Agencies (NCCA), the accreditation arm of the National Organization for Competency Assurance (NOCA).

The Role Delineation Study identifies the knowledge, skills and abilities required to perform the critical tasks that medical examiners use to determine if a CMV driver meets Federal physical qualification standards. The Role Delineation Study is the primary process for developing both the specifications for the standardized training curriculum and the certification test for FMCSA medical examiners. The Medical Examiner Survey is a component of a Role Delineation Study that analyzes, defines, and makes available both to the population of medical examiner candidates and to program stakeholders the domains and tasks that will be assessed by the certification test. The survey is one method for gathering data from FMCSA medical examiners and addresses components 2 through 4 of the following components of the Role Delineation Study:

(1) Develop a task list through a variety of techniques. In particular, section 1607.14, Technical Standards for Validity Studies, of 29 CFR part 1607 Uniform Guidelines on Employee Selection Procedures (Attachment D) indicates certification test content should be based on “work behavior(s) and the tasks associated with them.”

(2) Measure (a) the extent to which each task in the list is performed by a sample of medical examiners, and (b) how important respondents in the sample find each task. Section 1607.14 of title 29, CFR, indicates these studies should focus on task importance for success. When important tasks are also extensively performed among practitioners, confidence grows in the legal and professional soundness of study results.

(3) Disqualify tasks lacking sufficiently strong extent and importance measurements.

(4) Remove disqualified tasks to identify critical tasks.

(5) Create specifications for certification test items linked to critical tasks.

(6) Identify knowledge, skills, and abilities that support performance of critical tasks.

(7) Identify scope, sequence and priority of tasks supporting development of training curriculum and guidelines.

(8) Provide a baseline description of the medical examiner role that can be compared to future role delineation studies as one indicator of the impact of the NRCME program.

This information collection supports the DOT Strategic Goal of Safety (i.e., reducing the highway fatality rate). The quality of the medical certification for CMV drivers has been an issue for a long time. Based on post-accident examination of medical certifications, too often drivers have been certified by medical examiners who say these drivers were qualified when basic physical qualifications required by FMCSA regulations were not met. The Agency wants to be sure that drivers are carrying valid medical certificates and that those certificates accurately reflect that the driver has met all FMCSA medical standards at the time of certification. FMCSA currently has insufficient confidence that all CMV drivers have been medically certified based on consistent application of the FMCSA medical standards and guidelines. The Agency has a responsibility to ensure that physically unqualified drivers are not on the roadway with the motoring public, and the goal of reducing crashes, injuries and fatalities involving large trucks and buses is met.

2. How, by whom, and for what purpose the information will be used.

Purpose for collecting the information:

FMCSA will use the information from the survey to support the creation and defense of a certification test ensuring that medical examiner candidates demonstrate a minimum level of competency such that only “qualified” medical examiners are listed on the National Registry. The Role Delineation Study will be used as the primary evidence for establishing that 1) the certification test was developed using accepted psychometric principles, 2) the test content was developed and validated by the candidate population’s peers and 3) provide evidence that the certification program tests job-related medical examiner competence.

FMCSA has modeled the activities for the Role Delineation Study and Medical Examiner Survey similar to those typical of healthcare certification programs that have achieved ⁱNCCA accreditation. ⁱⁱHenderson (1996) identifies two phases in performing a job analysis (role delineation study). The NRCME Role Delineation Study project has progressed through this first phase, task identification, and has described the FMCSA medical examiner role via the survey task list. The second phase, validation, requires that the task list be endorsed by job incumbents which will be accomplished through the survey.

Task validation is based on two responses to the task list: performance of the task and importance of the task. A task is identified as critical when it meets both the minimum performance and importance thresholds. A certification test developed without the validation phase is at risk for including content that is not critical to minimum competency, which could become a legal issue difficult to defend. Supplementing the task list with survey data showing endorsement of critical tasks leads to development of a certification test with highly-defensible and candidate-respected content.

Raymond (2001) reports that 90% of credentialing agencies use some kind of questionnaire for job analysis studies task validation. Task list validation is the primary purpose of the survey.

How the information will be used:

The Working Integrated Product Team (WIPT) of medical examiners and FMCSA regulations expert advisors who identified and organized the task list into the survey will be using the results of the survey to support the development of the certification test blueprint and detailed content outline. Both the members of the WIPT and the survey respondents are medical professionals (Advanced Practice Nurses, Doctors of Chiropractic, Doctors of Osteopathy, Medical Doctors and Physician Assistants, (see 49 CFR § 390.5 (Attachment E)) who are authorized under current FMCSA regulations to perform physical examinations of CMV drivers and are peers of the certification test candidate population.

First, the WIPT will use the resulting snapshot of current medical examiner performance to validate the completeness of the task list. The recipients are specifically asked if the task list is complete; if not complete, they are asked to indicate what is missing.

Second, the results will be used to determine the potential subset of critical tasks which will describe job behaviors for which certification candidates can be tested. Critical tasks survive all exclusion rules and will be organized into a detailed content outline. An exclusion rule requires that a task 1) must be performed by at least a majority of examiners and 2) have an importance rating equal to or above a specific threshold. Limiting test content to these most extensively performed and important tasks ensures the minimum competency determination is based on tasks that are critical to competent performance.

The ultimate goal is to link test content to the medical examiner role, including an accurate delineation of work tasks as well as the knowledge, skills and other abilities required to perform these tasks competently. The outline structure of the detailed content outline will be used by FMCSA to develop a test specifications table that dictates item selection by each content domain, including the total number and complexity level of items for each certification test form. This ensures all test forms are equivalent in content scope, domain distribution, and difficulty.

The resulting test specification will be used at least annually by item writing and item review WIPTs to develop new test items. The test blueprint and linkage to the task list is critical for test maintenance to ensure that outdated items are replaced with appropriate new items.

In addition, a third use of the survey results by FMCSA will be to support development of core curriculum content and guidelines for medical examiner training mandated by SAFETEA-LU. Content for training is not limited to tasks that survive exclusion rules. However, the survey results will assist FMCSA subject matter experts developing the training guidelines to identify, prioritize, sequence and chunk training content to provide the necessary background and content coverage. This will support candidate passage of the certification test and competent performance of the FMCSA medical examiner role.

It is anticipated that the changes to FMCSA regulations and guidelines combined with the changes in medical practice will require a new Role Delineation Study at least every five to seven years to ensure the linkage between the certification test and the role of the medical examiner remains current with advances in the science and technology impacting this role. However, role delineation studies may be required at shorter intervals if FMCSA changes in medical regulations and/or guidelines impact the validity of the test blueprint.

Dissemination:

Survey results will be included in the Role Delineation Study Report. The Executive Summary of the report will be posted in the training and certification section of the NRCME Web site. Should FMCSA seek accreditation for the certification program, the entire report will be available upon request to support that accreditation.

The detailed content outline of critical tasks and test specifications will be disseminated to the public or used to support publicly disseminated information, including candidate handbooks and curriculum guidelines so that all candidates and training organizations will have a clear understanding of the scope of the certification test.

As explained in the preceding paragraphs, the information gathered has utility. The Physical Qualifications Division of FMCSA will retain control over the information and safeguard it from improper access, modification, and destruction, consistent with FMCSA standards for confidentiality, privacy, and electronic information. See response #10 of this Supporting Statement for more information on confidentiality and privacy. The information collection is designed to yield data that meet all applicable information quality guidelines. Prior to dissemination, the information will be subjected to quality control measures and pre-dissemination review pursuant to guidelines for the Data Quality Act, section 515 of Public Law 106-554 (Attachment F).

3. Extent of automated information collection.

FMCSA and Axiom Resource Management, Inc. (FMCSA’s contractor for developing the NRCME program) propose to mail a cover letter and survey (Attachment G) to approximately 5,000 medical examiners who currently perform the FMCSA physical examination for CMV drivers. Survey responses will be digitized for analysis and summarization after optical scanning.

It was the unanimous consensus of the WIPT that a paper-based survey was more user-friendly than an electronic survey for the targeted clinician respondent. One factor identified by the WIPT was that an examiner in the typical clinical setting could more easily carry a paper survey with him or her and use the short periods of time between examinations to respond to some statements, thus completing the survey over the course of the day. Rarely does a medical examiner have either a block of time or dedicated access to a computer that supports online completion of a survey of this length. Given the length and related nature of the tasks, if examiners want to flip back and forth between sections of the survey, it is easier to do so with a paper version than a long electronic document. Axiom’s subcontractor, Applied Measurement Professionals, Inc. (AMP) confirmed that they have observed non-response rates increase with survey length for electronic surveys. Therefore, for this research effort, selecting a paper format is a factor that should encourage a higher response rate.

4. Efforts to identify duplication.

No duplication of data was discovered by a literature review for the Role Delineation Study. This is the first formal research focused specifically on defining the role of the medical examiner in performing FMCSA CMV driver physical examinations and is limited to tasks specific to examination conduct. It does not duplicate other research efforts and similar data are not available. Existing related data are primarily the result of collection of CMV crash data. These crash investigation data reflect whether a driver medical problem was identified during the crash investigation and the outcome of the driver physical examination. These data are not specific to tasks required by the medical examiner in making the determination to certify the driver.

5. Efforts to minimize the burden on small businesses.

It is anticipated that this survey will not affect small businesses since the data is being collected from individuals, not establishments.

6. Impact of less frequent collection of information.

If the Medical Examiner Survey is not conducted, then there will be substantially less evidence supporting the content of the training, testing and certification components of the FMCSA Medical Examiner Certification Program. Confidence in program outcomes would be diminished. Specifically, decisions to certify certain persons and deny certification to other persons would be less well supported. Persons who are unable to demonstrate sufficient mastery of program content to earn certification may be motivated to seek a legal remedy. There would be less evidence to refute the claims of a plaintiff in such a circumstance.

If FMCSA applies for accreditation of the NRCME Medical Examiner Certification Program, it must be able to document that the certification test assesses the tasks performed by the certification population and that identification of the critical tasks was based on psychometrically sound methodology.

Changes in the Federal Motor Carrier Safety Regulations (FMCSRs) that affect the CMV driver physical examination and the medical guidelines for various medical conditions will potentially require FMCSA to conduct either a partial or entirely new Role Delineation Study (including a survey) periodically. The FMCSA Medical Review Board (MRB) will be reviewing and revising current FMCSA medical standards and guidelines, as well as developing new ones for medical conditions that are not currently addressed, every six to 18 months. Revisions that are implemented from this process will result in a need to conduct either a partial or new Role Delineation Study every year while these changes are being made. The Role Delineation Study will need to be repeated every five to seven years to ensure continued content relevance of the NRCME program and maintenance of the certification program national accreditation.

7. Special circumstances.

There are no special circumstances related to this information collection.

8. Compliance with 5 CFR § 1320.8.

On September 29, 2005, FMCSA published a notice in the Federal Register (70 FR 56964) advising the public of its intent to seek OMB approval to collect this information (see Attachment H). Two comments were received in response to this notice but did not specifically address the information collection. The comments and FMCSA responses are found in Attachments I and J. FMCSA has published another notice in the Federal Register with a 30-day comment period to announce when this package is being sent to OMB for review and approval (see Attachment K).

9. Payments or gifts to respondents.

No payments or gifts will be provided to the respondents. Survey responses will be provided by volunteers who practice within the five medical professions authorized to perform FMCSA CMV driver physical examinations.

10. Assurance of confidentiality.

Surveys will be distributed to specific individuals; however, the survey will not prompt respondents to identify themselves, so responses will be anonymous. Survey response data will be used to identify critical medical examiner tasks for training curriculum and certification test content. These data will be housed in an FMCSA Medical Examiner Certification Program secure database.

FMCSA through its contracted services, will ensure that storage and access to these data, both paper and electronic, is restricted only to those persons authorized by FMCSA. Raw data will not be available to the public. After data collection we will discard all links between direct identifiers, including zip code, from the data so that all responses are anonymous.

11. Justification for collection of sensitive information.

There are no sensitive items included in the questionnaire.

12. Estimate of total annual burden hours for respondents.

FMCSA estimate that 4,000 of the 5,000 medical examiners who currently perform FMCSA CMV driver physical examinations will provide responses to the survey. The estimated annual total burden hours for the collection of the survey data is 4,000 hours [4,000 returned surveys x 1 hour per response = 4,000 hours]. FMCSA and Axiom Resource Management, Inc. will send the survey to medical examiners who are currently authorized to perform FMCSA CMV driver physical examinations.

Estimated Annual Burden to Respondents: 4,000 hours.

FMCSA further estimate that the cost for the medical examiner respondents for the initial Medical Examiner Survey is $216,000 [4,000 respondents x 1 hour per survey x $54.00 per hour = $216,000]. $54.00 is the weighted hourly average wage for the five medical examiner professions authorized to perform FMCSA CMV driver physical examinations.

13. Estimate of total annual costs to respondents.

There are no costs to respondents except those identified above in Item 12.

14. Estimate of cost to the Federal government.

FMCSA estimates that the total cost for collecting and analyzing the initial Medical Examiner Survey data is $37,500. These costs will be absorbed in a current contract agreement between FMCSA and Axiom Resource Management, Inc.

Estimated Total Annual Cost to Federal Government: $25,000. This cost is figured on the need to conduct a new Role Delineation Study (including a survey) at the cost of $37,500 every 18 months for the next six years while the MRB is reviewing and revising all of the FMCSA medical standards and guidelines. This would be four role delineation studies in 6 years, so the annual cost would be $25,000 [$37,500 x 4 years = $150,000/6 years = $25,000].

15. Explanation of program changes or adjustments.

This is a new information collection that will result in a one-time program change of 4,000 estimated annual burden hours.

16. Publication of results of data collection.

The listing of included tasks will be used to develop a certification program, detailed content outline and test specifications.

The time schedule for survey conduct and results release is as follows:

Month Task

1 Distribute surveys to potential respondents

3 Collect, digitize and summarize survey responses

4 Convene group to review survey results, select task

exclusion criteria, develop detailed content outline, and test specifications

5 Convene group to review survey results and develop training

core curriculum content and guidelines

6 Draft Role Delineation Study results and submit for review

7 Produce final Role Delineation Study report

Tables summarizing results will be included in an overall report defining the critical tasks required to perform competently a CMV driver physical examination. An Executive Summary of results of the Role Delineation Study, including survey results, will be posted in the training and certification section of the NRCME Web site (http://www.nrcme.fmcsa.dot.gov). The entire study report will be available for purposes of accreditation and legal defense or as determined on a case-by- case basis if the distribution includes materials that could compromise the integrity of testing materials and processes used for certifying medical examiners in a fair and standardized manner.

17. Approval for not displaying the expiration date for OMB approval.

No approval is being requested.

18. Exceptions to certification statement.

None.

B. Collections of Information Employing Statistical Methods

1. Universe and Respondent Selection:

Background

Medical examiners who qualify CMV drivers are not conveniently available for use in sample surveys. They are included in unknown subsets of five different types of medical professionals based on their decision to perform this type of activity in their practice. They may be listed to various degrees in the rosters of a number of organizations but not necessarily identified as medical examiners, much less as to whether they examine CMV operators. Defined in this manner, medical examiners who perform this service constitute a hidden population relative to obtaining accurate information about their practice through a sample survey. Since there is no sample frame for hidden populations, attempts to study them using standard sampling and estimation techniques have often produced misleading results. When the NRCME program is fully implemented, the Registry itself will act as a sampling frame because it will contain the entire population of medical examiners who provide this service. Before that can be achieved, reasonable unbiased information is needed to validate the tasks performed by medical examiners which will permit the development of the National Registry. Given this, it is necessary to consider other approaches to obtaining useful information about medical examiners to allow this development to continue.

An approach which has gained much attention and produced promising results utilizes a collection of methods generally described under the term network analysis. The networks that are the focus of attention here are those social structures made up of family and friends, work partners, professional associates, acquaintances, and the organizations in which we participate. Anthropologists have studied the composition of these relationships, or social networks, in villages, towns and urban areas across the world. Social scientists have generally approached the work for two purposes. One is to identify the members of and the patterns of interactions among collections of individuals of various types such as kinship, work and friendship groups. The other purpose involves how to select respondents or participants for a research study by identifying individuals who know others or know about them. This latter activity is generally called network sampling or sometimes referred to as snowball sampling (Shensul et al. 1999).

More recently, however, the social network approach has been applied to more sophisticated problems; research on hidden populations is one of these areas. Here, the work has been directed at estimating the size of hidden populations and, of more interest to the present project, discovering the knowledge, beliefs and attitudes held by networks. In studying these areas, the effort has focused on how to estimate the parameters that describe social and professional networks and how these parameters behave under different sampling approaches and levels of missing data. The parameters of a network allow researchers to uncover information about members of the network even if they are not included in the sample. This reach into a network from a respondent is largely based on the concept of homophily. Homophily is the tendency for people to associate and develop friendships, marriages, work relationships, etc. with people similar to themselves. Various demographic characteristics such as age, gender, socio-economic class, ethnicity, education and occupation can provide boundaries around relationships. An individual's beliefs and values (e.g., religious, political) can also be influential. Individuals in homophilic relationships share common characteristics (e.g., beliefs, values, education) that make communication and relationship formation easier.

Parameters of most relevance to this study concern measures of centrality. Measures of centrality describe actors’ (those asked about the network) positions in a network relative to others and in relation to the complete network. Centrality measures assess such network factors as degree, betweenness, closeness, radiality and integration. While these measures have technical definitions, their names are reasonably self explanatory except for degree. This measure relates to the number of individuals in the network known to an actor in the data collection process. This measure can be expressed as an individual’s degree, the total degree of the network, or the network’s average degree. It is further defined as being manifested in two ways, i.e., in-degree and out-degree. In-degree is measured as the number of other individuals who identify the actor and out-degree is the number of others who the actor identifies. To obtain a measure of in-degree, there is a need to collect in-depth information from all of the network members in the sample. Out-degree can be measured easily by asking respondents to report the number of individuals they know within the identified network boundary. Out-degree is the measure of greatest interest here because it is the type of information that can be obtained in the data collection processes to be used in this study. For example, one can ask a physician how many other physicians they know who are medical examiners who examine CMV operators to qualify them for an interstate commercial driver’s license.

A number of researchers have studied the behavior of the degree measure in different circumstances. In this work, the main types of sampling strategies examined were network sampling and general survey sampling. Among this research, Salganik and Heckathorn (2004) showed that asymptotically unbiased estimates of degree can be obtained using a type of snowball sample. They found that estimated percentages of the population with specific traits were also asymptotically unbiased and that these estimates had this quality no matter how the initial members (seeds) of the network were selected.

Other researchers examined the effect of missing data. In his area, a study was performed that investigated stability of centrality measures when networks were sampled with various levels of missing data (Costenbader and Valente, 2003). The conditions investigated involved amounts of missing data ranging from 20% to 90%. Analyses of stability were performed by correlating the estimates taken from the full data set with those estimates based on the varying levels of missing data. The analyses were performed on 11 different sets of data and out-degree was found to have the most stability across the levels. The correlations were around .90 at 50% missing data and were still at .70 with 90% missing data. This is not surprising because the data provide an in-depth description of the network and are labor intensive to obtain. In-degree did not perform as well, but still had significant correlations (.60) with 50% missing data. This is encouraging because, as was mentioned, out-degree data can easily be obtained from respondents in a survey.

This indicates that under some circumstances researchers may still be able to use network data for which some data are missing to study network properties or create network-based interventions such as developing training and certification examinations. In other words, researchers who do not interview all members of a community or network may still be able to take advantage of some aspects of network theory and techniques to obtain useful information when the target group initially has the characteristics of a hidden population.

Sampling Strategy

In this study, we plan to obtain two types of samples. One will be obtained using a snowball type of approach while the other will be a random sample of medical examiners from organizations that are expected to have these individuals in their membership. Two samples are used so that more information will be available on this hidden population, and the results can be examined in different contexts. For example, we are confident that we can obtain unbiased results using the snowball sample but are not entirely sure of what an organizational survey may produce. If the results are similar, then there can be confidence that the estimation of the degree distribution is valid and will produce accurate estimates of the traits that are the focus of the task validation (see Estimation section). Should the results differ significantly, then there will be a need to investigate, explain and reconcile this discrepancy. One approach to doing this will involve asking the actors in each sample how many medical examiners they know who are members of the organizations involved and how many they know who are not. The degree distributions of the various subsets will be examined to determine if membership (or lack of membership) is a confounding factor. This approach is suggested through the concept of “critical multiplism” which is promoted by researchers and evaluators who argue that increased external validity of findings can be achieved by using multiple data sources even if some are flawed (General Accounting Office (GAO), 1992).

In 1992, the GAO issued a report that introduced a collection of approaches for generating coherent scientific evidence called Cross Design Synthesis. This report was critical of the flagship of medical scientific investigations, clinical trials, saying that they often had poor external validity. This was the case, the report said, because it was frequently difficult to apply findings to a broader patient population based on the methods of selecting the subjects into the trials. While internal validity of the approach was usually excellent, the weakness in relation to external validity was a serious flaw. The report argued for the broader use of non-experimental data to enhance the generalizability of findings. This meant, of course, there is a need to employ a range of methods to handle observational data so that findings can be based on a preponderance of evidence. The spirit of this approach focused on obtaining coherence from a complex program environment.

Underlying this view is the concept of “critical multiplism”. Here, its proponents point out that scientific results are often weakened because all aspects of the scientific approach are flawed and open to bias (Cook, 1985). It is argued that one can best approximate truth through a strategy of multiple approaches to demonstration of results in an attempt to control bias. In the course of doing this, one must adopt a critical focus relative to each approach. While this is not different from the concept of triangularization to achieve acceptable results, it is more of an elaboration. We feel it will be useful in this circumstance because medical examiners have not been systematically studied before and, as has been pointed out, they have some characteristics of a hidden population.

Estimation

The purpose of this survey is to determine the prevalence of tasks performed by medical examiners so as to validate earlier observational findings. As has been pointed out, the approach to this is to estimate the percentage of occurrence of a task as it is reported by the medical examiners. For example, the exclusion rule requires that a task 1) must be performed by a majority of examiners and 2) have an importance rating equal to or above a specific value to be used for testing for minimum competency. To provide data which allows a valid assessment of the tasks, it is necessary to obtain as unbiased estimates as possible of the percentages of task occurrence as reported by the medical examiners. The best approach to this is to use the data describing network degree as an aid in making the estimates. Given the expected quality of the estimates of degree, the use of these with the data describing the occurrence of tasks will help adjust the prevalence estimates towards what actually occurs in the networks.

Guidance in conducting this type of analysis is provided by some recent research. Using survey data and measures of degree, researchers were able to develop models that combined the measures of interest to obtain estimates of the prevalence of traits in the network (Zheng et al., 2006). The researchers modeled network degree as count data in a non-homogenous Poisson framework. They used degree as a dependent variable in a Poisson regression with the occurrence of the traits as independent variables. The occurrence of the traits was estimated as the coefficients of the independent variables. We shall use the same approach with the occurrence of tasks as independent variables.

Sample Size and Response Rates. There are five critical subgroups in the population defined by a background as (1) an Advanced Practice Nurse, (2) a Doctor of Chiropractic, (3) a Doctor of Osteopathy, (4) a Medical Doctor, or (5) a Physician Assistant. We intend to identify 1,000 respondents for each of these five subgroups during an opt-in phase of the study. Surveys will be sent to 1,000 people from each subgroup who know about the proposed NRCME and the role of the survey in the project. We intend to follow up with sample members multiple times with a goal to achieve at least an 80% response in each subgroup. We feel that we will be able to realistically achieve this rate based on the saliency of the topic and the follow-up methods we will be using.

Only practitioners who have functioned as medical examiners will become constituents of the survey sample. Information released before the survey will inform constituents future medical examinations will only be performed by medical examiners listed on the NRCME . We expect that by informing sample members of the study purpose before they opt-in, the commitment to give responses will be higher than normal. Most importantly we intend to follow-up with weekly communications among those we know who have not responded.

Figure 1

Erroneous deviations from population responses for a sample are inversely related to the square root of sample size, which Figure 1 displays. Error in observed survey responses would reach a practical minimum when approximately 400 responses are received. Our target response of 800 per subgroup should yield results in which survey response error is minimized for each subgroup.

We will calculate a standard error of the mean for each task so WIPT members can know when they risk making a false negative exclusion error. Task exclusion methodology will permit committee members to retain tasks within the margin of error for certification program content. Such a method will ensure only clearly peripheral tasks are excluded. Conversely, confidence will be high that surviving tasks over which certificants will be tested are truly critical.

2. Procedures for Collecting Information:

As indicated, the sample will be equally stratified among the five groups of professionals who are authorized to perform FMCSA CMV driver physical examinations. Within some of these smaller cohorts (e.g., Doctors of Chiropractic), the stratified sample may be the population of professionals. For larger cohorts (e.g., Medical Doctors) random sampling would be ideal. As discussed, one challenge in targeting survey respondents is that FMCSA does not currently know how many medical examiners are performing the FMCSA CMV driver physical examinations, the breakdown per medical profession or where these professionals are located.

Each cohort includes specialty practice areas where practitioners are more likely to perform the FMCSA CMV driver physical examination as part of their practice (e.g., within the Medical Doctor cohort, there are those who practice in occupational health settings). The proposed sampling will target a national distribution from these practice areas. However, the sampling will be completed from the general population of the professional group if there is an insufficient number of practitioners within the specialty areas. The goal is to increase the degree of accuracy and relevance of the responses. Only the initial survey will depend on targeting specialty areas. Subsequent surveys will be distributed to a sampling of the certified medical examiner population listed on the National Registry.

3. Methods to Maximize Response:

We will ensure that the survey sample of 5,000 includes those medical examiner sub-populations that perform the CMV physical, including 1,000 respondents from each of the five subgroups. Methods used to distribute the request to participate in the NRMCE survey include:

1. Word of mouth by medical examiners already contributing subject matter expert services to the NRCME program,

2. Professional association distribution through print and electronic publications,

3. Information prominently displayed on the NRCME Web site and other professional Web sites, and

4. Direct mailings to random samples of professionals in the high-probability specialty sub-populations to pre-screen that all participants perform the CMV driver physical examinations.

As of July 3, 2006, more than 2,400 survey participants have been identified.

The study will include an opt-in phase to ensure we start with 5,000 people who have completed physical examinations for CMV drivers and are willing to participate in the Medical Examiner Survey. We are more confident an opt-in sample will include persons who have at least some understanding of the study and are committed to following through by completing a survey. We will send postcards to mailing lists we expect include medical examiners. These individuals will be instructed to opt-in to receive a survey by registering their contact information at a Web site. These individuals will be added to the approximately 2,400 individuals who have already signaled their interest in receiving a survey until we have identified 1,000 willing recipients for each of the five subgroups.

Just prior to distribution of the survey, participants will be sent a postcard informing them when the survey will be sent, who to contact if they do not receive it, and encouraging them to complete the survey. Participants will also receive an electronic version of the message by e-mail.

Each survey will be sent in an envelope that is personally addressed to the medical examiner. This envelope will prominently display the NRCME banner and FMCSA logo to identify clearly the Federal source of its contents. The cover letter accompanying the survey will describe the proposed NRCME program and encourage each respondent to complete the survey. This cover letter will also be signed by a senior FMCSA Agency official. Survey introductory information will include a description of the typical time it should take to complete the survey, a support contact name, an email address and telephone number to answer recipients’ questions as well as a confidentiality pledge. Pre-addressed and postage-paid envelopes will be included with the survey.

After the survey is distributed, the following intensive methods will be implemented to maximize response rate:

1. Weekly e-mail reminders encouraging prompt and complete responding until we have received at least 800 completed surveys from each of the five subgroups,

2. A mailed reminder; this letter will include a request for those not completing the full survey to respond to demographic questions,

3. Regular promotion of survey completion through the same professional and contact resources (e.g., listservs) that have been used to identify volunteers for the survey.

As indicated above, in an effort to capture information on non-respondents and assess non-response bias, the post-survey distribution reminder letter described above will include the same demographic questions as the survey. If participants do not intend to return the entire survey, they will be asked to respond to each demographic question contained in the letter. This way we hope to assess and account for nonresponse bias to the extent possible.

Intra-class correlation values will also help to provide an answer to the theoretic question, “Would other samples from the population have responded similarly to survey items?” Intra-class correlation values typically exceed .90 when the expected number of respondents reacts to a survey. Values this high are associated with high probability for similar responses from a survey sample and theoretic responses of other samples from the same population.

4. Testing of Procedures:

There was no pilot test of the survey instrument since the test development professionals involved in the Role Delineation Study have a well-documented, excellent record of past performance in conducting numerous role delineation studies.

5. Contacts for Statistical Aspect of Data Collection:

Kaye Kirby, Federal Motor Carrier Safety Administration, Program Manager, 202-366-3109

Glenna Tinney, Axiom Resource Management, Inc., Onsite Project Manager, 202-366-0549

Robert C. Shaw, Jr., PhD, Applied Measurement Professionals, Inc. (AMP), 913-495-4467

ATTACHMENTS

A. 49 U.S.C. 31149

B. SAFETEA-LU, Section 4116, Public Law 109-59, 119, Stat. 1726-28 (August 10, 2005)

C. 49 U.S.C. 31136

D. 29 CFR 1607.14

E. 49 CFR 390.5

F. Section 515 of Public Law 106-554

G. Cover Letter and Survey Instrument

H. FMCSA’s 60-day notice requesting comments (Federal Register notice, dated September 29, 2005 (70 FR 56964))

I. Comment from the 60-day notice and FMCSA response thereto

J. Comment from the 60-day notice and FMCSA response thereto

K. FMCSA’s 30-day notice requesting comments

References

Cook, T. D., (1985), “Post-Positive Critical Multiplism.” In Shotland and Mark (Eds.), Social Science and Social Policy. Beverly Hills, CA, Sage.

Costenbader, E. and Valente, T. (2003), “The Stability of Centrality Measures When Networks are Sampled,” Social Networks, 25, 283-307.

Raymond M.R. (2001). Job analysis and the specification of content for licensure and certification examinations. Applied Measurement in Education. 14(4), 369-415.

Salganik, M. J. and Heckathorn, D. D. (2004) “Sampling and Estimation in Hidden Populations Using Respondent-Driven Sampling.” Sociological Methodology, 193-239.

Schensul, J., LeCompte, M., Trotter, R., Cromley, E. and Singer, M. (1999), Mapping Social Networks, Spatial Data and Hidden Populations, Beverly Hills, CA, Sage.

U.S. General Accounting Office, “Cross Design Synthesis: A New Strategy for Medical Effectiveness Research,” March 1992, GAO/PEMD-92-18.

Zheng, T., Salganik, M. J. and Gelman, A. (2006) “How many people do you know in prison?: Using overdispersion in count data to estimate social structure in networks.” Journal of the American Statistical Association, to appear.

i NCCA is the accreditation body of NOCA. Certification programs may apply and be accredited by the NCCA if they demonstrate compliance with each accreditation standard. NCCA's Standards exceed the requirements set forth by the American Psychological Association and the U.S. Equal Employment Opportunity Commission. http://www.noca.org/ncca/ncca.htm

ii Henderson, JP (1996). Job analysis, in Browning, A.H., Bugbee, Jr., A.C., and Mullins, MA (Eds.). Certification: A NOCA handbook. Washington DC: NOCA.

18