SS Part B -- Evaluation of the 2009 CHIPRA Quality Demonstration Grant Program -- Survey Data Collection 8-22-2013

SUPPORTING STATEMENT

Part B

Evaluation of the Children’s Health Insurance Program Reauthorization Act of 2009 (CHIPRA) Quality Demonstration Grant Program: Survey Data Collection

Version: July 26, 2013

Agency of Healthcare Research and Quality (AHRQ)

Table of contents

B. Collections of Information Employing Statistical Methods 2

1. Respondent universe and sampling methods 2

2. Information Collection Procedures 3

3. Methods to Maximize Response Rates 6

4. Tests of Procedures 7

5. Statistical Consultants 7

B. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The universe of respondents for this study is the group of physicians who provide primary care to children enrolled in Medicaid in selected States (Massachusetts, North Carolina, Ohio, and Pennsylvania). This group of child-serving physicians includes both pediatricians and family physicians. According to the American Medical Association (AMA) Masterfile, nationally, there are about 42,000 office-based pediatricians and about 68,000 office-based family physicians. Within the selected States, there are an estimated 5,970 office-based pediatricians and 8,608 office-based family physicians.¹ Based on available studies, the contractor estimates that 80 percent of pediatricians and 50 percent of family physicians serve at least some children enrolled in Medicaid.² The estimated size of the survey-eligible population in the selected States is therefore 9,080 (4,776 pediatricians and 4,304 family physicians).³

The sample will be selected from the AMA’s Physician Masterfile, a dataset frequently used to sample the physician workforce in the U.S. The Masterfile includes current and historical data for more than 900,000 physicians, residents, and medical students, including about 42,000 office-based pediatricians and about 68,000 office-based family physicians. Specifically, it offers information on physician demographics, professional activities (patient care versus nonpatient care), specialties (self-reported), medical education, certifications and licensures, and contact information. The AMA aims to capture in the Masterfile every physician in the U.S., including those who are not members of the AMA.

Massachusetts, North Carolina, and Pennsylvania were selected because they are implementing demonstration-funded activities, such as statewide efforts to promote use of standard quality measures that are designed to influence physicians’ beliefs and attitudes about the value of quality measurement for improving child health services. Of the remaining CHIPRA grantees, only South Carolina is implementing an activity that is likely to have statewide effects on attitudes toward quality measurement and has a provider population large enough to generate our target number of respondents. However, South Carolina already conducts an annual survey of its primary care providers. Hence, we excluded South Carolina from considerations. The remaining demonstration states are not implementing activities that are likely to have statewide effects on attitudes toward quality measurement. One comparison State not participating in the demonstration (Ohio) was selected based on similarities to the selected demonstration States in terms of geography and provider population.

Two sampling strata will be defined: pediatricians and family physicians, in order to control the release of the sample. To preserve the original distribution of these specialties, the sample will be proportionally allocated to these two strata.

In selecting the sample both explicit and implicit stratification will be used. Explicit stratification involves defining groups (strata) within which samples of a specified size are selected. Implicit stratification, by which the frame is sorted on the stratifying variables before sampling, is used when sequential sampling methods are used to help ensure that the sample is proportionately distributed on the stratifying variables. Within each specialty, implicit stratification will be used on physician characteristics to help ensure that the samples are distributed in the same way as the sampling frame.⁴ Variables available in the AMA Masterfile that will likely be used for implicit stratification include age, gender, and years since completion of residency.

2. Information Collection Procedures

a. Overview

Information for this study will be collected primarily through a pencil-and-paper survey; respondents also will be offered a web-based option and the opportunity to complete the survey by phone during follow-up calls.

We conservatively estimate 40 percent of these physicians will respond to the letter based on previous surveys conducted by the American Academy of Pediatrics and the American Academy of Family Physicians.⁵ Based on the published literature, it is likely that 20 percent of pediatricians and 50 percent of family physicians who respond to the letter will indicate that they do not serve Medicaid-enrolled children.⁵

We expect that the remainder of those who respond to the letter (that is, 80 percent of pediatricians and 50 percent of family physicians who respond to the letter) will indicate that they do serve Medicaid children and will complete the survey.

Assuming an equal number of pediatricians and family physicians in the sample, Table1 illustrates these assumptions.

Table 1. Response Pattern Assumptions

Expectations Per State	Pediatricians	Family Physicians	Total
Number of letters sent	577	577	1,154
Number responding to letter	231	231	462
Number of respondents replying to the letter who serve Medicaid children and complete the survey 1	185	115	300

¹ Assuming 80 percent of pediatricians and 50 percent of family physicians serve children enrolled in Medicaid, 231 responding pediatricians will result in 185 pediatricians (80 percent of 231) who serve children enrolled in Medicaid and 231 responding family physicians will result in 115 family physicians (50 percent of 231) who serve children enrolled in Medicaid

b. Estimating Population Characteristics

Population characteristics will be estimated from survey data using sampling weights that reflect differences in probability of selection for the physicians. We will then adjust the sampling weights to compensate for physicians who do complete the survey by computing the propensity to respond. The inverse of the response propensity will be used as the adjustment factor and the weights will be post-stratified to the frame counts. Because the expected response rate is less than 80 percent, an analysis of the potential for nonresponse bias will be conducted and the results used in making nonresponse adjustments. Below we first describe these procedures for conducting nonresponse analysis and then present the plan for weighting. Nonresponse analysis is routinely conducted as part of the weighting process, and involves identifying which measurable factors are associated with nonresponse.

The goals of nonresponse analysis are to assess the extent to which (1) nonresponse has introduced an appreciable risk of bias and (2) the weighting process has corrected for any such risk. The procedures to achieve these goals will entail (1) identifying factors associated with nonresponse, (2) seeing which of the factors identified in step 1 are also associated with key study variables (to be identified by the research team), (3) using this information in the logistic regression response propensity models for nonresponse adjustments, and (4) checking to see if the weighting process corrected imbalance on characteristics associated with both nonresponse and key study variables.

To identify variables associated with nonresponse, AHRQ will use two techniques—Chi-Square Automatic Interaction Detection (CHAID) followed by logistic regression. Potential sources of data for this analysis include characteristics of sample members available from the sampling frame and environmental variables such as length of time in the field and number of contacts.

Once characteristics associated with nonresponse are identified, logistic regression will be used to examine whether any of these characteristics are also associated with response to the interview. The inverse of the response propensity scores will be used directly or will be used to form weighting classes for the nonresponse adjustments.

The analysis weights will be used to compute estimates by and within specialty, state, and overall. Weights are needed for analysis because the contractor will use differential sampling methods (sampling pediatricians and family physicians at different rates within and across states) and because some level of nonresponse is expected. The differential sampling and nonresponse will result in samples of pediatricians and family physicians that are distributed differently than the populations from which they have been sampled. To the extent that the samples differ from their respective populations on dimensions that are related to study variables, estimates made with unweighted data will be biased. The weights will be designed to bring the weighted distribution of the sample back in line with the population distributions.

The weights will be the products of several factors designed to correct for differences in probabilities of selection and response propensities. The first factor is the sampling weight, defined as the inverse of each physician’s probability of selection. The next factor will be a nonresponse adjustment. In making the nonresponse adjustments, we will use the response propensity score or form cells based on the nonresponse analysis described above.

The product of the two factors yields the nonresponse adjusted weight (NRW):

(1)

where

P_Sji=the probability of selection of physician i in specialty j in State S

RR_Sjik is the response propensity score for the individual physician K or for the individual physician in cell K. (The formation of response rate adjustment cells is discussed below.)

The next step will be to ratio-adjust the sample in each specialty each State sums for the frame total:

(2)

The post stratified physician weight is then

₍₃₎

Weights will be examined at this point to see if trimming is required. If weights are trimmed, the sum of the trimmed weights will be readjusted to the estimated population total.⁶

_{Multiple sets of weights will be constructed as needed to facilitate the following analyses: (1) statewide estimates (or comparisons of physicians from two States), (2) estimates for all States combined, and (3) comparisons of physicians between the three demonstration states and the non-demonstration state.}

_{The sample design is a stratified random sample and the computation of survey estimates based on the sample requires the use of survey data analysis procedures. These survey data analysis procedures are available in the SAS, STATA, and SUDAAN statistical software packages. The sampling variance will be computed for nonlinear estimates (such as proportions, percentages, means, and regression coefficients) using the Taylor series linearized expansion of the survey estimator and the explicit equations for stratified random sampling. The data file of respondents will include the stratification parameters to permit the computation of correct sampling variances.}

c. Expected Confidence Intervals

Expected confidence intervals help indicate the degree of precision with which the survey estimates represent the overall population from which a sample is drawn. Table 2 presents the expected half-width of 90 and 95 percent confidence intervals for alternative values of a characteristic (percentage) for all physicians interviewed in a State. Minimum detectable differences (MDDs) are also given for percentages between two states assuming alpha of 0.05 and statistical power of 70 percent and 80 percent for alternative values of a characteristic.

Table 2 Precision Level (90 And 95 Percent Confidence Interval) For Estimated Percentages For Single States And Minimum Detectable Difference In Percentages Between Two States For Alternative Values Of An Outcome Measure For 300 Completed Questionnaires In Each State

Outcome Measured as a Percentage	Half-width of Confidence Interval for Single State			Minimum Detectable Difference (Alpha = 0.05)
	90 Percent	95 Percent		70 Percent Power	80 Percent Power
25.0	4.3	5.1		9.2	10.4
33.3	4.7	5.6		10.0	11.3
40.0	4.9	5.8		10.4	11.8
50.0	5.0	5.9		10.6	12.0

Note: The half-width of the confidence interval and minimum detectable difference are computed assuming a binomial variable. A design effect of 1.1 was assumed to account for the weight differential arising from variation in the response rates of the physicians.

d. Quality Control Procedures

A team of experienced quantitative researchers will collect and analyze the survey data described in this statement. In addition, sampling statisticians will review (1) specifications for the coding that will be used to conduct the analyses to ensure that appropriate weights are applied and (2) descriptions of the results to ensure that appropriate conclusions are drawn regarding population parameters. An independent senior programmer will review all code developed for the analyses.

3. Methods to Maximize Response Rates

Physicians can be a challenging population to contact and survey due to their workload and scheduling demands, and because of this, we generally do not expect high response rates from them. The estimated response rate for this study across all modes of survey completion is 40 percent. We will mail out 3,000 surveys for the random sample (750 per selected State) to yield 1,200 completed surveys (300 per State for each of four States).

The survey topics related to quality measures and improving the quality of pediatric patient care are highly salient to the universe of sampled pediatricians and family physicians. In addition to survey topic relevance, numerous methods and materials will be used to encourage response and reduce challenges to participation:

• Branded materials. The advance letter will be printed on AHRQ letterhead and mailed in an AHRQ envelope so that the study is legitimized and the importance is stressed. In addition, the web-based survey will include the AHRQ logo.

• Endorsement of stakeholders. The advance letter will include an endorsement letter from the American Academy of Pediatrics (AAP) and American Academy of Family Physicians (AAFP). The endorsement will highlight the importance of the study and encourage participation.

• Use of incentives. Prepaid monetary incentives are associated with increased survey response, even when small in amount. The initial survey mailing will include a $5 cash incentive to demonstrate appreciation for the respondent’s time and heighten the norm of reciprocity.

• Eligibility postcard experiment. A randomized experiment in which a randomly selected group of physicians will have a postage-paid postcard inserted into their initial correspondence. The postcard contains the two survey questions that screen for eligibility for participation. We hypothesize that ineligible physicians will be more likely to return the postcard than the full paper survey indicating their ineligibility. Return of the postcard can help differentiate ineligible physicians versus nonresponders and potentially allow for more efficient release of additional case as needed to reach the target number of responses. In addition, physicians that return their postcard after the AHRQ advance letter but before the survey mailing will be removed from our mailing database. This saves costs on mailings. This method has been used in other surveys of physicians but there has not been a rigorous assessment of its effects.

• Flexible participation. Respondents whose preferred mode of contact is not mail will be offered the opportunity to complete the survey by fax or over the telephone with an interviewer. This flexibility will allow the respondent to participate in the manner most convenient based on their own scheduling needs.

• Closed-ended questions: To facilitate completion of the survey by mail or web, the vast majority of survey items will be closed-ended with categorical response categories.

• Targeted nonresponse follow-up. Seven follow-up contact attempts will be made after the initial mailing to encourage survey response. These contact attempts will vary by mode in order to reach respondents in the manner most convenient for them and will include a postcard mailing, fax, emailing (if available), and telephone reminders. During reminder calls (up to two), trained interviewers will call respondents to (1) remind them about the survey and seek their participation; (2) gather updated contact information for the clinician, especially email addresses; and (3) offer to assist sample members over the phone to complete the survey.

• Survey support. There will be a toll-free helpline and an email address for this survey. This phone and email contact information will be made available to sample members in the advance letter, along with the paper and web survey versions, and during reminder telephone calls. This toll-free line and email box restrict access to project personnel only.

4. Tests of Procedures

The survey questionnaire for the mail-based survey was pretested with four pediatricians and one family medicine physicians in May-July 2013. The survey pretest was designed to determine whether the information being requested in the survey is reasonable, clearly stated in coherent and unambiguous language, and collected in the least burdensome way possible. Through the pretest, AHRQ learned about challenges respondents experienced in completing the survey and providing the requested information. As a result, following the pretest, recommendations for survey revisions were made.

Pretest responses to the survey questionnaires were collected by mail because that will be the primary mode of response. Staff followed up with pretest respondents by telephone to learn their reactions and determine how to improve survey language. The pretest established the average survey length was less than 15 minutes per respondent. The results of the pretest were incorporated into the OMB package prior to the posting of the 30-day notice.

5. Statistical Consultants

AHRQ has contracted with Mathematica Policy Research, Urban Institute, and Academy Health to conduct the evaluation of the CHIPRA quality demonstration grants. Table 3 identifies the individuals at these organizations who were consulted regarding the quantitative methods used in this project. Table 4 identifies members of technical expert panel that was consulted on this survey.

Table 3 Individuals Consulted Regarding Quantitative Methods of Evaluation

Name	Title	Email	Phone Number
Henry Ireys, Mathematica	Senior Fellow	[email protected]	202- 554-7536
Frank Potter, Mathematica	Senior Statistician	[email protected]	239-558-5956
John Hall, Mathematica	Senor Statistician	[email protected]	609-275-2357
Kirsten Barrett, Mathematica	Senior Survey Researcher	[email protected]	202-554-7564
Joe Zickafoose, Mathematica	Researcher	[email protected]	734-794-1123
Jenny Kenney, Urban Institute	Senior Associate	[email protected]	202-261-5825
Lisa Simpson, Academy Health	President and CEO	[email protected]	202-292-6747
Cindy Brach, AHRQ	Project Officer	[email protected]	301-427-1444

Table 4 Members of the CHIPRA Quality Demonstration Evaluation Technical Expert Panel Sub-Group on the Physician Survey

Name	Title	Affiliation
Bruce Bagley	Medical Director for Quality Improvement	American Academy of Family Physicians
Steve Blumberg	Senior Scientist and Research Survey Statistician	National Center for Health Statistics, Centers for Disease Control and Prevention
David Kelley	Chief Medical Officer	Office of Medical Assistance Programs, Pennsylvania Department of Public Welfare
Jon Klein	Associate Executive Director and Director of the Julius B. Richmond Center of Excellence	American Academy of Pediatrics
Cynthia Minkovitz	Director, Women's and Children's Health Policy Center; Professor, Department of Population, Family and Reproductive Health and Pediatrics	Johns Hopkins Bloomberg School of Public Health
Lynn Olson	Director, Department of Research	American Academy of Pediatrics
Mark Weissman	Chief, Division of General Pediatrics & Community Health; Executive Director, Children's National Health Network, DC Partnership to Improve Children's Healthcare Quality	Children’s National Medical Center

1 Smart DR. Physician Characteristics and Distribution in the U.S. Chicago: Division of Survey & Data Resources, American Medical Association; 2012.

2 Zickafoose JS, Clark SJ, Sakshaug JW, Chen LM, Hollingsworth JM. Readiness of primary care practices for medical home certification. Pediatrics. 2013;131(3): 473-482.

3 80 percent of the number of office-based pediatricians and 50 percent of the office-based family physicians.

4 We will consider the use of gender, age, other demographic characteristics, subspecialty, or whether the medical degree was received in the U.S.

5 Because the expected response rate is less than 80 percent, we will conduct a nonresponse analysis, described in the next section on “Estimating Population Characteristics.”

6 Trimming (of weights) involves reducing the value of very large (outlier) weights and redistributing the “trimmed” amount to other (non-trimmed) cases. Trimming is used mainly to reduce the potential increase in sampling error (design effect) due to unequal weights.