20475 ID_Supporting Statement Part B 120513

The impact analyses of the Regional Extension Centers (REC) program will rely on three distinct comparisons; each will include REC participants and matched non-participants. Comparison 1 will use the screener data for outcomes. Comparison 2 will use the Centers for Medicare & Medicaid Service (CMS) Medicare Electronic Health Record (EHR) Incentive Program attestation data for outcomes. Comparison 3 will use the survey data for outcomes. The comparisons are illustrated in Exhibit B1.

Comparison 1. First, we will merge the American Medical Association (AMA) Master File data and the REC program’s customer relationship management (CRM) database using the individual National Provider Indentifier (NPI). This will identify primary care physicians participating and not participating in the REC program. A sample of primary care physicians participating in the REC program and a matched sample of primary care physicians not participating in the REC program will be drawn. All sampled physicians will be invited to take the screener. Responses from the screener from REC participants will be compared to those from non-participants to examine the impact of REC program participation on EHR adoption (Research Question 1).

Comparison 2. Physicians will also be matched to the Centers for Medicare & Medicaid Service (CMS) Medicare Electronic Health Record (EHR) Incentive Program attestation data^¹ to determine each physician’s meaningful use status. These data will be used to compare REC participants to non-participants on Medicare EHR incentive program attestation (Research Question 2).

Comparison 3. Screener respondents who reported having an EHR and working in small practices or with an underserved community will be interviewed by telephone to collect information about their EHR experience. Again, REC participants will be matched to nonparticipants, which will enable us to identify the impact of the REC program on meaningful use attestation in the Medicaid Electronic Health Record (EHR) Incentive Program, difficulty in adopting EHRs, assistance received by EHR adopters, and participation in care transformation programs. (Research Questions 2-6)

The specific steps are illustrated in Exhibit B1 below.

Exhibit B1. Design of the impact analysis

Matching

As indicated in Exhibit B1, there are three points in the overall impact study where we will conduct propensity score matching to minimize differences between REC participants and nonparticipants. These matching efforts are necessary because REC participation is not randomly assigned and providers select to either participate or not participate in the program. Because of this there will be systematic differences between REC participants and nonparticipants that could bias our impact estimates. Propensity score matching is among the most rigorous non-experimental methods to minimize these potential biases.

We first conduct propensity score matching at the initial sampling stage (propensity score match 1 in Exhibit B1). After merging AMA and CRM data we will select an initial sample of 2,356 REC participants. We then use a Caliper Radius Matching algorithm (detailed in Attachment B-1) to find the most closely matched nonparticipants among the remaining sample members. The matching algorithm uses information from the AMA dataset, including but not limited to, gender, age, state, zip, MD/OD status, major professional activity, specialty, year of graduation from medical school, and medical school variables as its explanatory variables.

Once we have created a matched sample, we will administer a short screener questionnaire to collect information on whether the provider has an EHR and to collect a number of additional background variables, including practice size, type of practice, and percent of patients with Medicaid or uninsured. These data help to determine whether the REC program increased adoption of EHRs (Research question 1). Physicians will be invited to work with colleagues most familiar with EHR selection, implementation, and use to complete the screener.

We will also conduct an impact analysis focusing on EHR attestation outcomes for the Medicare and Medicaid EHR incentive program using CMS data and self reported data (Research question 2). This impact analysis will include all matched sample members responding to the screener. In addition, we can use the CMS attestation data to conduct an analysis of nonresponse bias on the screener.

Because we do not expect 100 percent of the initial matched sample to respond, we will repeat the propensity score matching effort within the screener questionnaire respondent subsample (propensity score match 2 in Exhibit B1). This is necessary to re-balance the groups and useful because:

REC participant respondents may differ from nonparticipant respondents,
We now have more background data and can do a more thorough job matching REC participants and non-participants.

However, because we start with a sample that was already matched at the outset, we expect that the propensity score matching at this stage will mostly confirm the validity and balance of our sample rather than change its composition drastically.

Using the subsample of screener respondents meeting eligibility criteria, we then administer the survey via a computer-assisted telephone interview (CATI).^² This survey collects outcome data on assistance with EHR adoption, satisfaction with the EHR, usage of the EHR in daily practice, progress toward meaningful use, and other practice management outcomes (Research questions 2 to 6). Physicians will be invited to work with colleagues most familiar with EHR selection, implementation, and use to complete the survey.

Because respondents among REC participants may again differ from respondents among REC non-participants, we will again repeat the propensity score matching effort (propensity score match 3 in Exhibit B1) and will add additional practice background data to this effort, as appropriate and feasible. (Note that we will not include any matching variables that could possibly have been affected by the REC, even indirectly, since this would introduce bias rather than removing it. One example is EHR adoption. If we matched on this factor, we would observe no differences between REC participants and non-participants on this key outcome.)

Estimating sample size

To estimate the sample sizes necessary for this approach, we used Optimal Design Software. This is a software package, developed by Stephen Raudenbush and colleagues, which helps researchers determine sample size, statistical power, and optimal allocation of resources for multi-level and longitudinal studies.^³

We specified the following parameters to estimate the number of providers per REC: alpha = 0.05; clusters = 62 RECs; power = 0.80; effect size = 0.15 to 0.30; effect size variability = 0.01 to 0.05 (the amount of variability in effect sizes as a function of REC); proportion of variance explained in the outcome by REC characteristics = 0.10; and proportion of variance explained by provider characteristics = 0.10.(Exhibit B2)

Exhibit B2. Power and sample size estimates

We will have sufficient power to detect an effect size of 0.15 with an average sample of 24 physicians per REC which creates a desired final, total sample size of 1488 physicians. We feel that the large number of RECs is the main reason why our sample sizes per REC do not have to be large to provide us with the ability to detect effects of small to moderate size. To avoid over-representing providers from places where the number of target primary care providers is small, we will allocate the sample proportionate to the size of the REC so that we sample more providers from larger RECs. To do this we will identify how many potential sample members we have in each of the 62 RECs (K_1,K_2,K_3,…K₆₂). Our total (n) will be the sum of all of the potential sample members from all the RECs together (K₁+K₂+K₃ +…K₆₂). We will then sample [1488*K_(1,2,3…62)]/n from each REC. In the end we will weight any pooled estimates by K/N, where N is the realized respondent sample and K is the population it represents.

Response rate

The expected final response rate for this study is 50%. In order to achieve the desired final, total sample size of 1488 (average of 24 providers per REC), we will oversample and invite 4712 providers to take part in the study. This oversample assumes: a) 75% response rate for the screener questionnaire; b) 30% of screener respondents will be ineligible for the full survey because they work in large practices;^⁴ c) 28% of the remaining screener respondents will be ineligible because they do not have EHRs;^⁵ and d) 83.5% response rate for the survey. (Exhibit B3)

Exhibit B3. Response rate calculation

4712 providers invited with screener

0.25 do not respond to screener (n=1178)

3534 responders to screener

0.30 ineligible because work in large practice (n=1060)

2474 responders to screener in small practices

0.28 ineligible because no EHRs (n=693)

1781 responders to screener in small practices and with EHRs

0.1645 do not respond to survey via CATI or paper (n=293)

1488 final responders to survey

final 50% response rate =

1488 final responders to survey

4712 providers invited with screener - 1060 ineligible because work in large practice - 693 ineligible because no EHRs

2. Information Collection Procedures

We will use the following procedures to collect screening and survey data.

Screener

All sampled providers (n=4,712) will receive a letter introducing the study. (Attachment B-2) Then we will mail a second letter describing the study and enclose the paper screener, a postmarked, self-addressed envelope, and a $0, $2, $5, or $10 cash incentive.^⁶ Providers will be requested to complete the screener and return it in the enclosed envelope. (Attachment B-3 and B-4)

The screener asks questions about whether the recipient’s practice uses an electronic health record (EHR) system and the type of practice/setting in which the recipient works. The screener will also request phone numbers in order to contact eligible respondents for the survey, which is administered via a computer-assisted telephone interview (CATI).

Nonrespondents to the screener will receive up to two mailed reminders. (Attachment B-5 and B-6) Providers who do not respond to the final mail reminder will receive calls from interviewers who will attempt to collect the screening data over the phone.

We will determine eligibility for the survey based on responses in the screener. Providers deemed ineligible will be thanked for their participation via mailed postcard. (Attachment B-7) Eligible providers will be invited to take the survey (described below).^⁷

Survey

Eligible providers will be invited to take the survey via phone using Computer Assisted Telephone Interviewing (CATI) software. All eligible providers will receive a letter indicating that they will receive a phone call from an interviewer; describing the survey itself; and offering a $15 up front cash incentive for participating. (Attachment B-8)^⁸

Nonresponders to the telephone survey will receive up to 9 follow-up calls to encourage participation. Non-responders will receive a mailed reminder accompanied by a paper survey and a postmarked envelope as a final attempt to collect the survey data. (Attachment B-9) The paper version of the survey will consist of 29 questions that are also included in the CATI instrument. We will request that providers complete the survey and return it in the enclosed envelope.

3. Methods to Maximize Response Rates

To maximize response rates we use several techniques including multiple non-response follow-up, incentives, and mixed mode data collection.

Nonresponse follow-up. We plan intensive follow-up with nonresponders at both the screener and survey stages. We will mail up to two additional reminders to providers who do not respond to the screener. If providers still do not respond, then we will call them on the telephone to administer the screener and the survey, if eligible. We plan to conduct up to 9 call backs if we are unable to reach our sampled respondents or if they are unavailable when we call. We will also follow-up telephone survey non-responders with up to 9 calls. If our call back attempts fail, then we will mail a shortened paper survey.

Incentives. To increase response rates we plan to administer prepaid non-conditional cash incentives. A $2, $5, or $10 prepaid non-conditional incentives will be sent by mail with the screener. The different incentive denominations will allow us to study the effect of denomination on reponse rate. A $15 prepaid non-conditional incentive will be sent by mail in advance of the phone survey. Our goal is to increase response rates at the first stage to reduce non-response follow-up and potential non-response bias.

Mixed Modes. We plan to use two modes of contact and data collection to increase response rates and minimize non-response bias. We first contact respondents through the mail to administer the prepaid incentive and collect the screener data. We follow-up with non-responders over the phone. Next we conduct the survey over the phone. We follow-up with non-responders of the survey using a mailed, paper survey. If certain providers are less likely to respond using a particular mode of administration then we maximize the likelihood of them participating if we provide different modes of administering the screener and survey.

4. Tests of Procedures

Screener and survey items underwent cognitive testing with no more than nine participants. We interviewed primary care physicians, nurses, nurse practitioners, and practice managers recruited through ONC, the American Academy of Family Physicians, and a recruitment firm. The interviews provided an in-depth exploration of concepts and language within the screener and survey instruments from the perspective of a study participant. The interviews examined what the participant was thinking or feeling when responding to the questions and how questions were interpreted.

5. Statistical Consultants

The following indivdiuals consulted on statistical aspects of the design.

Affiliation	Name	Telephone	Email
AIR	Johannes Bos	650-843-8110	[email protected]
	Steven Garfinkel	919-918-2306	[email protected]
	HarmoniJoie Noel	202-403-5779	[email protected]
	Grace Wang	650-843-8191	[email protected]
Anderson, Niebuhr & Associates, Inc.	Marsha A. Niebuhr	651-486-8712	[email protected]

Attachment B-1. Specification of Propensity Score Matching Strategy

Our primary sample of interest is providers participating in the REC program. We propose to include a comparison group comprised of similar providers in small practices who are not participating in the REC program. The selection process is described below.

Step 1. Link AMA and CRM to distinguish REC participants from non-participants.

We will identify primary care providers (i.e., general practice, general preventive medicine, internal medicine, obstetrics/gynecology, pediatrics, adolescent medicine, public health, etc) in the AMA masterfile who work in patient care.^⁹ We will link AMA and CRM administrative data using NPI in order to distinguish REC program participants and non-participants.

Step 2. Create propensity scores

The propensity score will be estimated using a logit function predicting participation in the REC program versus no participation:

Where

is the probability of being in the program for unit i.

is a vector of regression coefficients.

is a vector of explanatory variables.

Explanatory variables from the AMA masterfile and other datasets, include but are not restricted to: Gender, age, state, zip, MD/OD status, major professional activity, specialty, year of graduation from medical school, and medical school.

Step 3. Conduct matching of propensity score via Caliper Radius Matching.

We propose to use radius matching by the log odds of the propensity score. We will choose a radius to minimize the mean integrated standard error (MISE) for the comparison cases in a comparison sample weighted to correspond to the distribution of the treated sample.^¹⁰ In general, if at least 90 percent of the participants were matched, the matching proportion is considered acceptable.

The matching procedure uses the following model:

Where

is the number of comparison cases.

is the outcome variable for comparison case j.

is the mean outcome for those cases matched to j.

is the propensity score for case j, is the radius, and is a weight.

Estimates are more stable if all comparison cases that are sufficiently close to a given REC participant are considered as is done in many-to-one caliper matching with replacement. A radius is used to determine how close the propensity score of a comparison case needs to be before it can be considered a potential match. The size of the radius is an important consideration because it becomes a tradeoff between bias and the number of matches found (which affects precision).

Our proposed approach is to select the one comparison unit (Non-REC) that is closest to the treatment (REC) on the propensity score. By using a single comparison unit for each treatment unit, we ensure the smallest propensity-score distance between the treatment and comparison units and reduce bias.^¹¹ We also reduce bias by matching with replacement so that each treatment unit can be matched to the nearest comparison unit, even if a comparison unit is matched more than once, and by setting a radius to determine how close a comparison needs to be before it can be included as a potential match.

Balancing Propensity Scores. Following the matching, it is important to test how well the participant and comparison cases are balanced. Tests for statistically significant differences between variable means for the participant cases and the weighted comparison sample are performed to assure that the propensity score in fact balances the independent variables.^¹²

Linear adjustment. Regardless of how the matching is done, there will most likely still be some small differences between participant and comparison cases. A linear adjustment will be used to correct for the asymptotic bias inherent in matching estimators.^¹³ This method fits a linear model in the comparison sample and then uses those coefficients to adjust for any differences between the participant and comparison samples.

Standard errors. It is necessary to obtain an indicator of sampling error in the estimates. We will use a conditional standard error, which provides an estimate of the variation in an impact estimate, conditional on the independent variables.^¹⁴ ^¹⁵ The conditional standard error can be estimated by:

Where

K(i,j) is an indicator of whether comparison case i is matched with treated case j,

M(j) is the number of comparison cases matched with treated case j,

N1 is the number of treated cases,

N is the total number of cases,

Wi is an indicator of whether case i is treated (1) or comparison (0),

is an estimate of the variance of the dependent variable conditional on particular values of the independent variable matrix X.

Step 4. Select REC participants and matched non-participant pair.

We will stratify on REC and randomly select an average of 38 providers from each of the 62 RECs (n=2356).

Attachment B-2. Announcement letter

DEPARTMENT OF HEALTH & HUMAN SERVICES

Office of the Secretary

Office of the National Coordinator for Health Information Technology

Washington, D.C. 20201

[Date]

[name]

[address]

Dear [participant’s name]:

We are writing to invite you to participate in a new research study about electronic health records (EHRs). The goal of the study is to understand challenges with adopting and using EHRs and the help you may have received to meet those challenges.

The Office of the National Coordinator for Health Information Technology (ONC) is conducting this survey with the American Institutes for Research (AIR). We have worked with physicians and staff from the American Academy of Family Physicians (AAFP), American Academy of Pediatrics, American College of Physician (ACP), and American Congress of Obstetricians and Gynecologists (ACOG) to make sure this study is relevant to you.

By participating, you will be providing crucial information to ONC and policy makers for:

showing overall impact of federal programs on EHR adoption and achievement of meaningful use of EHRs by practices that provide primary care.
identifying challenges faced by practices providing primary care when adopting and achieving meaningful use of EHRs.
capturing opinions about EHRs.

These findings will help ONC and policy makers to improve existing programs, to create new programs that meet providers’ needs better, to understand gaps and barriers to EHR adoption and to prioritize these needs when making policy decisions.

We have randomly chosen practices that provide primary care services across the United States to participate in this study. Your participation is crucial as no one can be replaced, and each response is critical to the study’s success. In the coming days, you will receive a questionnaire and a postmarked envelope. The questionnaire will take less than 5 minutes and should be completed by the person most familiar with EHR selection, implementation, and use in your practice. This may be you, another clinician, practice manager, nurse, or other employee. Information Technology staff may also help complete some questions. Depending on your responses, you may be contacted about taking a follow-up survey.

Please contact: Dr. Grace Wang, American Institutes for Research, at [placeholder REC provider survey email] for questions about the study and your rights as a participant. Thank you for your consideration.

Sincerely,

Jacob Reider, MD

Acting National Coordinator for Health Information Technology

Attachment B-3. Cover letter for screener for people receiving an incentive

DEPARTMENT OF HEALTH & HUMAN SERVICES

Office of the Secretary

Office of the National Coordinator for Health Information Technology

Washington, D.C. 20201

[Date]

[name]

[address]

Dear [participant’s name]:

The Office of the National Coordinator for Health Information Technology (ONC) is conducting this study with the American Institutes for Research (AIR). We have worked with physicians and staff from the American Academy of Family Physicians (AAFP), American Academy of Pediatrics, American College of Physician (ACP), and American Congress of Obstetricians and Gynecologists (ACOG) on this study to make sure it is relevant to you.

By participating in the study, you will be providing crucial information to ONC and policy makers. These findings will help ONC and policy makers to improve existing programs, to create new programs that meet providers’ needs better, to understand gaps and barriers to EHR adoption and to prioritize these needs when making policy decisions.

Kindly fill out the enclosed questionnaire in the next day and return it using the enclosed postmarked envelope. The questionnaire should be completed by the person most familiar with EHR selection, implementation, and use in your practice. This may be you, another clinician, practice manager, nurse, or other employee. Information Technology staff may also help complete some questions. Enclosed is a token of our appreciation for being part of this study.

This questionnaire will take less than 5 minutes to complete. Participation is completely voluntary, and you can stop at any time. You will not lose any benefits if you decide not to participate or to discontinue in the study. We will keep your responses confidential, and we do not anticipate any risks associated with participating. Depending on your responses, you may be contacted about taking a follow-up survey.

Please contact: Dr. Grace Wang, American Institutes for Research, at [placeholder REC provider survey email] for questions about the study or your rights as a participant. Thank you for your consideration.

Sincerely,

Jacob Reider, MD

Acting National Coordinator for Health Information Technology

Attachment B-4. Cover letter for screener for people not receiving an incentive

DEPARTMENT OF HEALTH & HUMAN SERVICES

Office of the Secretary

Office of the National Coordinator for Health Information Technology

Washington, D.C. 20201

[Date]

[name]

[address]

Dear [participant’s name]:

Sincerely,

Jacob Reider, MD

Acting National Coordinator for Health Information Technology

Attachment B-5. First reminder for screener

DEPARTMENT OF HEALTH & HUMAN SERVICES

Office of the Secretary

Office of the National Coordinator for Health Information Technology

Washington, D.C. 20201

[Date]

[name]

[address]

Dear [participant’s name]:

You received an invitation last week to participate in a new research study about electronic health records (EHRs). The goal of the study is to understand challenges with adopting and using EHRs and the help you may have received to meet those challenges.

Your participation will help ONC and policy makers to improve existing programs, to create new programs that meet providers’ needs better, to understand gaps and barriers to EHR adoption and to prioritize these needs when making policy decisions.

Sincerely,

Jacob Reider, MD

Acting National Coordinator for Health Information Technology

Attachment B-6. Second reminder for screener

DEPARTMENT OF HEALTH & HUMAN SERVICES

Office of the Secretary

Office of the National Coordinator for Health Information Technology

Washington, D.C. 20201

[Date]

[name]

[address]

Dear [participant’s name]:

As you may recall, you received an invitation 2 weeks ago to participate in a new research study about electronic health records (EHRs). The goal of the study is to understand challenges with adopting and using EHRs and the help you may have received to meet those challenges.

Sincerely,

Jacob Reider, MD

Acting National Coordinator for Health Information Technology

Attachment B-7. Thank you postcard for ineligibles

DEPARTMENT OF HEALTH & HUMAN SERVICES

Office of the Secretary

Office of the National Coordinator for Health Information Technology

Washington, D.C. 20201

Dear [participant’s name]:

You or a colleague recently provided information about your practice as part of a study about electronic health records (EHRs). Thank you for responding and for completing our study. Your participation has been critical to understanding challenges with adopting and using EHRs.

Sincerely,

Jacob Reider, MD

Acting National Coordinator for Health Information Technology

Attachment B-8. Cover letter prior to telephone survey

DEPARTMENT OF HEALTH & HUMAN SERVICES

Office of the Secretary

Office of the National Coordinator for Health Information Technology

Washington, D.C. 20201

[Date]

[name]

[address]

Dear [participant’s name]:

You or a colleague recently completed a survey describing the electronic health record (EHR) system at your practice. We thank you for providing this information.

We are writing to invite your continued participation in this research study about electronic health records (EHRs). The goal of the study is to understand challenges with adopting and using EHRs and the help you may have received to meet those challenges.

The Office of the National Coordinator for Health Information Technology (ONC) is conducting this study with the American Institutes for Research (AIR). We have worked with physicians and staff from the American Academy of Family Physicians (AAFP), American Academy of Pediatrics, American College of Physician (ACP), and American Congress of Obstetricians and Gynecologists (ACOG) to make sure this study is relevant to you.

An interviewer will be calling in the coming week to ask you a few additional questions. The call will take less than 30 minutes to complete. Your participation is voluntary, and you can stop at any time. You will not lose any benefits if you decide not to participate or to discontinue in the study. We will keep your responses confidential, and we do not anticipate any risks associated with participating. We have enclosed $15 as a token of our thanks for being a part of this important interview.

Please contact: Dr. Grace Wang, American Institutes for Research, at [placeholder REC provider survey email] for questions about the study or about your rights as a participant. Thank you for your consideration.

Sincerely,

Jacob Reider, MD

Acting National Coordinator for Health Information Technology

Attachment B-9. Cover letter with paper version of survey for telephone survey non-responders

DEPARTMENT OF HEALTH & HUMAN SERVICES

Office of the Secretary

Office of the National Coordinator for Health Information Technology

Washington, D.C. 20201

[Date]

[name]

[address]

Dear [participant’s name]:

You or a colleague recently completed a survey to participate in a new research study about electronic health records (EHRs). We thank you for providing this information. We attempted to follow-up with you by calling your work number but were unsuccessful at reaching you.

The goal of the study is to understand challenges with adopting and using EHRs and the help you may have received to meet those challenges. The Office of the National Coordinator for Health Information Technology (ONC) is conducting this study with the American Institutes for Research (AIR). We have worked with physicians and staff from the American Academy of Family Physicians (AAFP), American Academy of Pediatrics, American College of Physician (ACP), and American Congress of Obstetricians and Gynecologists (ACOG) to make sure this study is relevant to you.

Kindly fill out the enclosed survey about your experiences with your EHR system and return it as soon as possible using the enclosed postmarked envelope. The survey should be completed by the person most familiar with EHR selection, implementation, and use in your practice. This may be you, another clinician, practice manager, nurse, or other employee. Information Technology staff may also help complete some questions.

This survey will take less than 10 minutes to complete. Participation is completely voluntary, and you can stop at any time. You will not lose any benefits if you decide not to participate or to discontinue in the study. We will keep your responses confidential, and we do not anticipate any risks associated with participating.

Sincerely,

Jacob Reider, MD

Acting National Coordinator for Health Information Technology

1 http://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/DataAndReports.html

2 Eligibility is defined as primary care providers (i.e., general practice, general preventive med, internal medicine, obstetrics/gynecology, pediatrics, adolescent med, public health) AND working in practices of 10 providers or less OR working in settings serving >30% Medicaid or uninsured patients. Exclusion criteria are: providers not working in direct clinical care OR providers working for hospitals or within health maintenance organizations.

3 Raudenbush, S. W., et al. (2011). Optimal Design Software for Multi-level and Longitudinal Research (Version 3.01) [Software]. Available from www.wtgrantfoundation.org.

4 http://www.cdc.gov/nchs/data/ad/ad383.pdf

5 http://www.cdc.gov/nchs/data/databriefs/db111.htm

6 Providers will be randomly assigned a denomination. This will allow us to examine the effect of denomination on response rate.

7 Providers who completed the screener over the phone with an interviewer will have the options of continuing on with the survey, scheduling an appointment to take the survey at a later time, or ending participation in the study.

8 Providers who complete both the screener and survey over the phone during the same call will receive the $15 incentive in the mail after completing the survey.

9 By relying on the AMA masterfile for provider characteristics, we will be excluding physician assistants and nurse practitioners from the analysis.

10 Galdo, Jose, Jeffry Smith, and Dan Black, “Bandwidth Selection and the Estimation of Treatment Effects with Unbalanced Data,” Unpublished, May 2008.

11 Dehejia, Rajeev H. and Sadek Wahba. “Propensity score-matching methods for

Nonexperimental causal studies.” The Review of Economics and Statistics, February 2002, 84(1): 151–161.

12 Smith, Jeffrey A. and Petra E. Todd, “Does Matching Overcome LaLonde’s Critique of

Nonexperimental Estimators?” Journal of Econometrics 125 (March-April, 2005), 305-53.

13 Abadie, Alberto and Guido W. Imbens, “Large Sample Properties of Matching Estimators for

Average Treatment Effects,” Econometrica 74:1 (January 2006), 235-267.

14 Imbens, Guido W., “Estimating Variances for Estimators of Average Treatment Effects,”

Unpublished, Harvard University (September 2008).

15 Imbens, Guido W. and Jeffrey M. Wooldridge, “Recent Developments in the Econometrics of

Program Evaluation,” Institute for Research on Poverty Discussion Paper no. 1340-08, University of Wisconsin, 2008.

File Type	application/msword
File Title	SUPPORTING STATEMENT
Author	wcarroll
Last Modified By	Funn, Sherrette (OS/ASA/OCIO/OEA)
File Modified	2013-12-18
File Created	2013-12-18