2344ss01b.rev 8.13.09

ICR SUPPORTING STATEMENT: Part B

Auto Body Compliance Assessment Pilot Project

Part B applies only to the Pilot project; methodologies described here will not apply to surveys used for anecdotal purposes in support of the CRC activities.

1. Survey Objectives, Key Variables, and Other Preliminaries

1. Survey Objectives

The purpose of the pilot study is to evaluate whether and how EPA’s specific compliance assistance activities in Region 1 helped owners or operators of auto body shops improve their operations with respect to Subpart HHHHHH National Emission Standards for Hazardous Air Pollutants: Paint Stripping and Miscellaneous Surface Coating Operations at Area Sources (NESHAP Subpart HHHHHH, hereinafter referred to as the Surface Coating Rule.). Specifically, the study has been designed to assess whether or not the assistance provided helped owners or operators understand these and other applicable environmental regulations, and whether or not the assistance helped owners or operators implement operational changes that improved environmental practices; including changes in behavior that resulted in the auto body shop either returning to compliance with regulations or taking steps toward achieving compliance.

The questions contained in the surveys are designed to obtain information on (1) environmental performance related to current hazardous waste management and training requirements under RCRA, (2) environmental performance related to air emissions control requirements associated with the recently promulgated Surface Coating Rule, (3) environmental compliance assistance received by government agencies or other entities, and (4) perceptions regarding the factors that influence shop behaviors related to environmental performance. Some of these behaviors are related to current regulatory requirements, while others will be required in the future or are purely voluntary. Survey-based measurements related to these behaviors will be referred to in this document as measures of “environmental performance.” In addition, the survey will collect information on the type of compliance assistance received, i.e., the extent to which the shop has been affected by outreach, site visits, or other types of assistance from EPA or others.

There are two objectives in collecting this information:

• The primary objective is to assess the degree to which EPA’s compliance assistance activities lead to, or are associated with, improvements in environmental performance at auto body shops.

• A secondary objective of the survey is to assess the validity of information about environmental performance data collected through telephone surveys. EPA typically uses telephone surveys to gather information about environmental performance; however, OMB has expressed concerns that such data is subject to potential self-reporting and non-response bias. The current survey will involve on-site data collection designed to verify telephone responses and evaluate potential non-response and self-reporting bias.

2. Key Variables:

The key variables in the survey involve shops’ environmental performance related to employee training, air pollutants (specifically addressing requirements of the Surface Coating Rule), RCRA generator and waste determination requirements, emergency procedures, and the type of compliance assurance received by shops.

Key variables related to employee training are as follows:

• Whether the shop has a training program focused on proper hazardous waste management procedures, and whether these training records are available.

• Whether spray technicians have been properly trained, and whether these training records are available.

Key variables related to air emissions are as follows:

• Type of spray guns used to apply coatings.

• Manufacturer of spray guns.

• Configuration of area where spray coatings are applied (e.g., whether spray coatings are applied in an enclosed, ventilated spray booth, and how long this booth has been in operation).

• Capture efficiency of booth exhaust system for area where spray coatings are applied.

• Whether particle filter is used on spray booth exhaust and condition of filter.

• Configuration of paint mixing room (e.g., whether mixing rooms are enclosed and ventilated, and how long they have been in operation).

• Configuration of prep station for spray coating (e.g., whether prep stations are enclosed and ventilated, and how long they have been in operation).

• Methods used for cleaning spray guns, and how long these methods have been in use.

• Whether the shop uses waterborne paints, and for what purposes.

• Amounts of different types of coatings applied in the shop each week.

• Primary supplier of paint coating products.

• Whether shop has MSDS and coating formulation data for solvents and coatings, and whether this documentation is available.

• Whether coatings used by the shop contain chromium, lead, cadmium, nickel, or manganese.

• Whether paint stripping products used by the shop contain methylene chloride, and whether shop has minimization plan for methylene chloride.

• Whether the respondent is aware of the Surface Coating Rule, and if so, when and how the shop first heard of these requirements.

Key variables related to RCRA requirements are as follows:

• How much hazardous waste the shop generates.

• Whether all hazardous waste streams have been identified.

• Whether hazardous wastes are properly labeled and stored in an appropriate manner.

• Whether hazardous wastes are shipped to an appropriate off-site facility.

Key variables related to emergency procedures are as follows:

• Whether the shop has implemented emergency procedures.

• Whether documentation related to emergency procedures is available.

Key variables related to the type of compliance assurance are as follows:

• Sources of information about how to comply with federal and state environmental regulations (e.g., trade associations, paint manufacturers, regulatory agencies, etc.).

• Whether shop has been visited by a government environmental official within the last six months, and type of government official that visited.

• Whether shop has made any changes in its operations in the last year, and if so, what changed and why.

3. Sampling Approach:

The survey will involve a probability sample of auto body shops. As described in detail in Section B.2, the probability sample will be taken within the context of a study designed to evaluate the impact of compliance assistance on environmental performance. The Agency chose not to conduct a census due to the prohibitive costs and the unnecessary burden that would be imposed on auto body shops. The Agency has concluded that its information needs can be met through a probability sample and that the sampling error will not compromise the objectives of the survey. Section B.2.c provides additional information about precision.

The survey is being designed and conducted with the assistance of a contractor:

Contractor	Contractor Roles
Industrial Economics, Incorporated 2067 Massachusetts Avenue Cambridge, MA 02140	Study design Questionnaire development Statistical sample design Random assignment Database design

4. Feasibility:

The Agency believes it has developed a strong study design that will meet the objectives described above. The survey and overall study design is the result of a two-year collaborative effort involving an Agency inter-office workgroup; consultants with expertise in program evaluation, survey design, and statistics; and independent peer reviewers. The Agency has allocated sufficient funds to complete the survey effort, and the data will be available within a timeframe that is acceptable to users within the Agency.

The risk of failure has been minimized through the collection of multiple types of data. As discussed in the next section, the study involves two different approaches to policy evaluation (random assignment experiment and comparison group design) and two different data collection modes (telephone and on-site surveys). Furthermore, the survey will gather data on many different measures of environmental performance.

The data collection approach has been designed to minimize the impact of non-response and self-reporting bias. In order to address these potential sources of bias in the telephone survey, the data collection effort incorporates (1) on-site follow-ups with telephone respondents and non-respondents and (2) on-site verification of self-reported environmental performance. The Agency expects minimal non-response bias in the on-site component of the survey effort, as past site visit cooperation rates have been near one hundred percent.

The Agency does foresee a potential risk related to the type and quantity of compliance assistance received by the comparison group in the long-term component of the study. As we discuss below, our approach to assessing the long-term impact of compliance assistance requires a comparison group that receives limited, if any, compliance assistance from the Agency. However, the Agency cannot control compliance assistance provided by states, local governments, private vendors, or trade associations. If these groups replace EPA compliance assistance with substantial compliance assistance efforts of their own, the long-term study may be compromised.

2. Survey Design

This section contains a detailed description of the survey design, including a description of the target population, sampling frame, sample size, stratification variables, sampling methods, precision targets, non-sampling error, and questionnaire design. The section begins with an overview of the study design, as the survey is best viewed within the overall context of the policy evaluation. The schedule for the various components of the study is depicted graphically in Appendix A.

Short-Term Experiment: Impact of Compliance Assistance Outreach and Workshops

The short-term impact of compliance assistance outreach and workshops will be evaluated through a random-assignment experiment involving auto body shops in areas of eastern Massachusetts that have elevated cancer and/or non-cancer health risks from air pollutants. The auto body shops will be randomly assigned to either a treatment group (hereafter, Group A) or a control group (hereafter, Group B) (Exhibit B-1). The shops in the treatment group will receive (1) a multimedia guidebook providing a summary of air, water, and RCRA requirements impacting auto body shops in Massachusetts, (2) a brochure summarizing the Surface Coating Rule requirements, (3) an invitation to attend workshops covering the requirements of the Surface Coating Rule, and (4) a copy of the presentation slides that will be used at the workshops.¹ EPA anticipates that a total of approximately 150-300 shops will participate in the workshops. The shops in the comparison group will not receive the mailings, and they will not be invited to the workshops.²

Shortly after the workshops have been completed, EPA personnel will assess environmental performance through a combination of telephone interviews and on-site visits with probability samples of shops from each of the two groups. The impact of compliance assistance will be assessed by comparing the estimated environmental performance for Group A to the estimated environmental performance for Group B.

The environmental performance estimates for Groups A and B will incorporate information from both the telephone interviews and the on-site visits. Although some shops may provide inaccurate information during the telephone survey or refuse to respond to the telephone survey altogether due to concerns about potential EPA enforcement actions (or other reasons), site visits at random samples of respondent and non-respondent shops will provide data required to address these issues (see next section for additional details).

Exhibit B-1: Assessing Short-Term Impact of Compliance Assistance Outreach and Workshops

Assessing the Validity of Telephone Surveys

The validity of telephone survey responses will be evaluated by using a two-phase sampling approach to assess environmental performance in the short-term experiment (Cochran, 1977, pp. 327-355) (Exhibit B-2). In the first phase, EPA will measure performance through a telephone survey with a sample of shops. In the second phase, site visits will be conducted with samples of telephone survey respondents and nonrespondents. The site visits will determine shop environmental performance through direct observation and a review of on-site records. Site visits at respondent shops will allow us to adjust for any potential self-reporting bias in the telephone survey, while site visits at non-respondent shops will allow us to adjust for any potential non-response bias. By comparing performance levels estimated using data from this two-phase approach (combining data from the telephone survey with data from follow-up site visits) with performance levels estimated using data from the telephone survey alone (excluding the site visit data), the bias resulting from relying entirely on telephone survey responses can be determined.

In addition, we anticipate that the supplemental information obtained through the telephone survey will improve the precision of our performance estimate relative to estimates derived from site visits alone. If the performance estimates from this two-phase sampling approach provide substantial precision gains relative to the performance estimates obtained from site visits alone, the Agency will consider using the two-phase approach in the 2011 component of the long-term study.

Exhibit B-2: Schematic for Assessing the Validity of Telephone Surveys

Long-Term Study: Impact of Compliance Assistance Package

The long-term impact of a more comprehensive compliance assistance package will be evaluated through a comparison group design involving auto body shops in areas of eastern Massachusetts with elevated air toxics risks (hereafter, Group C) ³ and a similar population of auto body shops in a comparison location (hereafter, Group X). By “long-term” impact, we are referring to the cumulative impacts of a package of compliance assistance activities provided over approximately two years. The post-treatment measurement for assessing long-term impacts will occur at a point in time when the Surface Coating Rule requirements are in effect.

The shops in Group C will be offered a full suite of compliance assistance activities related to RCRA and surface coating requirements, including outreach, workshops, and site visits:⁴

• Outreach: All shops will receive a multimedia guidebook providing a summary of air, water, and RCRA requirements impacting auto body shops in Massachusetts; a brochure summarizing the Surface Coating Rule requirements; and a copy of the presentation slides that will be used at the workshops.

• Workshops: All shops will be offered an opportunity to participate in compliance assistance workshops led by EPA personnel.

• Site Visits: One hundred eighty randomly selected shops will be offered compliance assistance through site visits by EPA personnel in 2009. The purpose of these 2009 site visits is two-fold: they will be used to assess environmental performance and, after the on-site interview has been completed, they will be used to deliver compliance assistance based on interviewer observations.⁵ An additional 60 shops will receive compliance assistance visits in 2010; most will be randomly selected, but some shops may request a site visit after hearing about workshops or site visits at neighboring shops.

The shops in Group X will receive limited, if any, compliance assistance from EPA or state environmental assistance programs regarding RCRA requirements or the Surface Coating Rule.

In each of the two locations, site visits by qualified personnel at random samples of shops will be used to estimate performance before and after compliance assistance has been provided.⁶ This will result in four separate estimates of performance:

1. Group C 2009 Pre-treatment

2. Group C 2011 Post-treatment

3. Group X 2009

4. Group X 2011

The impact of compliance assistance will be assessed by comparing the change in performance in Group C with the change in performance in Group X. (See Exhibit B-3) (i.e., through a “difference-in-differences” approach, see Bloom, 2005, pg. 179).

Exhibit B-3: Schematic for Assessing Long-Term Impact of Compliance Assistance Package

1. Target Population and Coverage:

The population of interest for the pilot study is auto body shops located in areas with elevated air toxics risks in eastern Massachusetts and Group X, that are subject to the recently promulgated Surface Coating Rule. Auto body shops are subject to the Surface Coating Rule if they are paid to complete at least two surface coatings each year on vehicles or pieces of mobile equipment.

Areas of elevated air toxics risks are defined as areas with elevated cancer and/or non-cancer risks from air pollution according to National Air Toxics Assessment (NATA) data⁷. The Agency has chosen to focus on areas with elevated air toxics risks for this study because we believe that the need for auto body compliance assistance may be greater in these areas, given elevated risks to residents from other sources of air pollution.

The Agency is currently considering two potential locations for Group X: Oklahoma and southeastern Virginia. The characteristics of these locations are being evaluated, and the comparison location will be selected prior to beginning the study. The evaluation criteria used in selecting Group X include:

• number of shops located in elevated-risk areas;

• ;

• state regulations related to RCRA and air emissions; and

• expected compliance assistance from EPA, state, or other entities.

At the conclusion of the study, EPA will use additional data gathered in the survey to assess the adequacy of Group X as a comparison group and to identify any compliance assistance received by Group X shops.

2. Sample Design:

2 (b) (i) Sampling Frame

The sampling frame for the survey is auto body shops located in areas with elevated air toxics risks in eastern Massachusetts and Oklahoma/Virginia. Auto body shops are defined as businesses with Standard Industrial Classification (SIC) 7532 – “Automotive Body, Paint, and Interior Repair and Maintenance” – that are listed in either Dunn & Bradstreet or Reference USA. SIC 7532 comprises establishments primarily engaged in repairing or customizing automotive vehicles, such as passenger cars, trucks, and vans, and all trailer bodies and interiors; and/or painting automotive vehicles and trailer bodies. Auto body shops located in Worcester, MA, Lawrence, MA and Boston, MA were excluded from the sampling frame because for each of these communities aggressive assistance outreach and or enforcement activities had been conducted for a number of years (Lawrence and Boston) or being planned (Worcester) at the local or federal levels.

2 (b) (ii) Sample Size

The proposed sample sizes for the study are provided in Table B-1. In the short-term experiment we will select 400 shops for the telephone survey and 180 shops for the on-site survey. In the long-term study, we will select an additional 300 shops for the on-site survey: 100 shops in 2009 and 200 shops in 2011. Sample sizes were determined by weighing expected costs against expected precision gains from larger samples (see Section B.2.c.i). Sample sizes in 2009 are greater than in 2011 in order to balance the precision needs for the short- and long-term studies.

The Agency is planning to conduct an interim telephone survey with 200 Group C shops in 2010 to gauge the progress of auto body shops toward achieving compliance with the new Surface Coating Rule requirements. This interim telephone survey will help the Agency determine whether additional outreach activities (e.g., additional mailings, workshops, or site visits) may be needed prior to the 2011 compliance date.

Although the Agency currently does not plan to conduct telephone surveys in 2011, telephone surveys will be added in 2011 if the two-phase sampling approach increases the precision of the environmental performance estimates substantially without adding undo complexity to the study. If telephone surveys are conducted in 2011, the sample sizes will be 200 in each state, for a total of 400 additional shops. The Agency does not propose to conduct a phone survey in Group X in 2009 due to potential measurement effects (i.e., the phone survey itself could act as a form of compliance assistance, which would bias results from this comparison group at the baseline measurement.)

Table B-1: Summary of Sample Sizes

Shops Sampled	Sample Size for Telephone Survey	Sample Size for On-Site Survey
Group A 2009	200	80
Group B 2009 a	200	100
Group X 2009	0	100
Group C 2010	200	0
Group C 2011	0b	100
Group X 2011	0b	100
Notes: a Group B serves two purposes: it is the control group for the short-term experiment, and it provides the pre-treatment estimate of Group C environmental performance for the long-term experiment. The long-term experiment requires larger sample sizes to improve precision; hence, the sample size for Group B is larger than that for Group A. b As discussed in the text, telephone surveys may be conducted in 2011 if the two-phase sampling approach is successful in 2009.

2 (b) (iii) Stratification Variables

Sampling for the telephone and site-visit surveys in Groups A and C will be stratified (using proportional allocation) by receipt of face-to-face compliance assistance. The face-to-face compliance assistance stratum is defined as all shops that either attended an EPA workshop or received an EPA compliance assistance site visit as part of this effort. This stratification will ensure that both types of shops (i.e., those that receive face-to-face assistance and those that do not) will be included in the sample. The size of the strata will be unknown until compliance assistance has been provided.

In addition, the two-phase sampling approach used in Groups A and B will involve stratification in the second sampling phase. That is, the shops sampled for the telephone survey will be stratified into telephone respondents and telephone non-respondents for the on-site survey. This stratification into respondents and non-respondents will ensure that shops from both groups will be sampled for the on-site survey, ensuring that the Agency will be able to assess potential self-reporting bias and potential non-response bias in the telephone survey. Once again, the Agency expects to use proportional stratification in allocating the sample among the strata. (Exhibit B-4 illustrates the implementation of both stratification approaches within Group A.)

Exhibit B-4: Stratified Sampling Approach for Group A

(Includes all Shops Invited to Attend Workshops)

2 (b) (iv) Sampling Method

The sampling method for Group X in 2009 and 2011 will be simple random sampling from the population of shops. In order to minimize potential measurement effects associated with the 2009 interviews, the 2009 sample will be excluded from the sampling frame prior to drawing the sample for 2011.

The sampling method for Group C in 2011 will be random sampling stratified by receipt of face-to-face compliance assistance, with simple random sampling used within each stratum.

The sampling method for Groups A and B in 2009 will be somewhat more complex. In Group A, the survey will be completed in two phases (see Exhibit B-4). In the first phase, a stratified random sample of 200 shops will be selected for the telephone survey. There will be two strata for the telephone survey: shops that attended a workshop and shops that did not. In the second phase, site visits will be conducted at a stratified random sample of 80 of the shops selected for the telephone survey. Thus, the second phase will involve four distinct strata, defined by workshop attendance/non-attendance and response/non-response to the telephone survey.

In Group B, the survey will also be completed in two phases. However, as none of the Group B shops will have attended workshops prior to the telephone survey, a simple random sample of 200 shops will be selected for the telephone survey in Group B. Site visits will then be conducted at a stratified random sample of 100 of the shops selected for the telephone survey, with shops stratified by response/non-response to the telephone survey.

2 (b) (v.) Multi-Stage Sampling

Multi-stage sampling will not be used in the study.

3. Precision Requirements:

2 (c) (i) Precision Targets

To satisfy EPA’s decision-making needs, the study has been designed to provide a minimum detectable effect (MDE) not exceeding 21 percentage points for binary measures of shop performance for the long-term evaluation and a MDE not exceeding 15 percentage points for the short-term evaluation. The MDEs are calculated using a 10 percent significance level, a one-sided hypothesis test, a power level of 80 percent, assumed population sizes of 1700 in each group, and an assumption of no improvements in precision through the use of regressors. In addition, MDE depends on shop environmental performance, telephone survey response rate, and the degree of agreement between the telephone and on-site measurements.⁸ The Agency believes that the telephone survey response rate will likely be between 20% and 40%, while the degree of agreement between telephone and on-site measurements will likely be between 80% and 95%. As the performance rate will vary across the various measures of performance, we use 50%, 70%, and 90% in our calculations.⁹ Under these assumptions, the MDE for the short-term experiment ranges from 9% to 15%, while the MDE for the long-term comparison ranges from 12% to 21% (Tables B-2 and B-3).

Table B-2: Minimum Detectable Effect for

Short-Term Comparison

Agreement between Telephone and On-Site Measurement	Performance = 50%			Performance = 70%			Performance = 90%
	RR = 20%	RR = 30%	RR = 40%	RR = 20%	RR = 30%	RR = 40%	RR = 20%	RR = 30%	RR = 40%
80%	15.2%	15.1%	14.9%	14.1%	14.0%	13.9%	n/a	n/a	n/a
85%	15.1	14.9	14.7	13.9	13.8	13.6	n/a	n/a	n/a
90%	15.0	14.7	14.4	13.8	13.5	13.3	9.3%	9.3%	9.4%
95%	14.8	14.4	14.0	13.6	13.3	12.9	9.1	9.0	8.8
Notes: RR = telephone survey response rate Performance = environmental performance rate

Table B-3: Minimum Detectable Effect for

Long-Term Comparison

Agreement between Telephone and On-Site Measurement	Performance = 50%			Performance = 70%			Performance = 90%
	RR = 20%	RR = 30%	RR = 40%	RR = 20%	RR = 30%	RR = 40%	RR = 20%	RR = 30%	RR = 40%
80%	20.5%	20.5%	20.4%	18.8%	18.8%	18.8%	n/a	n/a	n/a
85%	20.5	20.4	20.3	18.8	18.7	18.7	n/a	n/a	n/a
90%	20.4	20.3	20.3	18.7	18.7	18.6	12.4%	12.4%	12.4%
95%	20.4	20.3	20.2	18.7	18.6	18.5	12.3	12.2	12.2
Notes: RR = telephone survey response rate Performance = environmental performance rate

2 (c) (ii) Non-Sampling Error

The Agency will use several quality assurance techniques to maximize response rates, response accuracy, and processing accuracy to minimize nonsampling error:

• The study relies heavily on data collected through on-site surveys, with many of the key measurements being obtained directly through interviewer observations rather than through survey questions. This will minimize measurement error, eliminate self-reporting bias, and reduce non-response bias.

• To maximize response in on-site surveys, selected shops will be visited during normal business hours, and shops will be able to reschedule the interview if the interviewer arrives at an inconvenient time. In the past, EPA has obtained response rates near 100% during these type of facility visits.

• To maximize response in telephone surveys, initial contacts will be attempted shortly after shops open for business, as personnel are typically busiest from mid-morning to late afternoon. If the shop indicates that the time is not convenient, the interview will be rescheduled. If the shop does not answer the telephone, at least three additional callbacks will be attempted on different days and at different times.

• When telephone surveys are used, the Agency will verify telephone measurements through follow-up site visits with samples of telephone respondents and non-respondents, allowing the Agency to address any potential self-reporting bias and non-response bias.

• To minimize measurement error and interviewer bias, on-site interviewers will follow objective, written protocols in collecting data on shop environmental performance. In addition, the questions have been organized to mimic a typical shop walk-through, thus minimizing the time required to collect the necessary data elements.

• The data collection instruments have been thoroughly reviewed by numerous survey experts. A pre-test has been conducted, as discussed in section B.3.

• Standardized software will be used for sample selection.

• Data will be 100 percent independently keyed and verified.

4. Questionnaire Design:

The questions contained in the surveys are designed to obtain information on (1) environmental performance related to current hazardous waste management and training requirements under RCRA, (2) environmental performance related to air emissions control requirements associated with the recently promulgated Surface Coating Rule, (3) environmental compliance assistance received by government agencies or other entities, and (4) perceptions regarding the factors that influence shop behaviors related to environmental performance. Most of the questions are designed to produce binary (i.e., yes/no) indicators of environmental performance for use as dependent variables in the statistical analysis.

There are two different questionnaires, one for the on-site survey and one for the telephone survey. The on-site survey is designed to be consistent with the general flow of a shop walk-through, with the interviewer obtaining data on environmental performance through his or her own observations and through targeted questions of shop personnel. The telephone survey focuses on a subset of the data items that are included in the on-site survey, as the survey needs to be relatively brief to discourage hang-ups. The telephone survey focuses mainly on environmental performance measures that can be later verified independently through interviewer observations on site.

Both questionnaires were reviewed by survey experts at Industrial Economics and Abt Associates, and by EPA experts in program evaluation, program review, statistics, and survey design. Copies of the survey instruments are attached in Appendices B and C.

3. Pretests and Pilot Tests

The survey instrument was pretested on auto body shops in Boston, Massachusetts, which is not included in the sampling frame for the proposed survey. Several of the questions from the two survey modes are identical, so the pretest was limited to a total of nine shops across the two modes: the on-site survey was pretested on five shops, while the telephone instrument was pretested on four shops. The five shops used to pretest the on-site survey were selected from a list provided by the Boston Public Health Commission. The four shops used in the phone survey pretest were selected from a list derived from Dunn & Bradstreet and Reference USA (SIC 7532). The selected shops provide a range of operation sizes (from “mom-and-pop” shops to national chains) and locations within the city.¹⁰ After the pretest, the survey instruments and instructions were revised to address pretest observations regarding question wording, clarity of interviewer instructions, question flow, and survey length.

No pilot tests are planned for the survey.

4. Collection Methods and Follow-Up

1. Collection Methods:

A combination of site visits and telephone surveys will be used to collect data from the sampled auto body shops. Site visits that combine survey questions with interviewer observations will be the primary data collection approach. This approach was selected by the Agency because it avoids potential self-reporting bias and the response rate for site visits is expected to be extremely high. Interviewers for the site visits will be Agency employees for Group C ¹¹, and EPA-trained contractors for Group X. These interviewers will follow a written set of procedures during all site visits, and they will be trained in the collection of environmental performance data from auto body shops.

Due to high costs associated with site visits, the Agency prefers as a general matter, to rely on telephone surveys in collecting environmental performance data for its national compliance assistance activities. Thus, a subset of the on-site surveys will be preceded by telephone surveys as part of an evaluation of the reliability of environmental performance data collected through telephone surveys.

2. Survey Response and Follow-up:

The target response rate for the on-site survey is 95 percent, calculated as completed interviews divided by eligible shops sampled. Past experience by the Agency with site visits at auto body shops indicates that surveyors are rarely denied entry, and the non-response rate is expected to be nearly non-existent. Shops will be allowed to reschedule the site visit if the interviewer arrives at an inconvenient time.

The target response rate for the telephone survey is between 20 and 40 percent, calculated as completed interviews divided by eligible shops sampled. (Although the target response rate is somewhat low, the study will evaluate non-response bias by conducting follow-up site visits at a sample of non-respondent shops.) Up to three attempts will be made to reach each sampled shop. If an answering machine picks up during the first attempt, the interviewer will call again within the next day or two. If the second attempt is unsuccessful, the interviewer will leave a message. If the respondent does not return the call within two days, the interviewer will attempt a third and final call. As with the site visits, shops will be able to reschedule the telephone interview for a more convenient time. In addition, if no one at the shop speaks English, an attempt will be made to have an interviewer who speaks the preferred language contact the shop.

5. Analyzing and Reporting Survey Results

1. Data Preparation:

The Agency will use 100% double-keyed data entry for all survey data. Data will be entered into a Microsoft Access database. Editing will consist of automated logic and range checks, and checks for missing data. Missing environmental performance data will not be imputed and will result in sample size reductions for the relevant analyses. Missing data on covariates will be imputed using standard methods such as means and regression.

2. Analysis:

This section describes the three general types of quantitative analyses that will be conducted to evaluate (1) the short-term impact of compliance assistance, (2) the long-term impact of compliance assistance, and (3) the validity of telephone surveys. In addition to these three quantitative analyses, the Agency will analyze responses to qualitative survey questions focused on respondents’ perceptions of the factors that influence behaviors related to environmental performance. The results of the analyses will be relevant for the population of auto body shops located in risk-based clusters in Massachusetts; the Agency does not intend to use the results of the pilot study to draw conclusions about the compliance assistance program as a whole.

Short-Term Impact of Compliance Assistance Outreach and Workshops

The short-term impact of compliance assistance will be estimated as the difference in performance between Groups A and B. Depending upon the type of variable, group performance will be described either as a proportion (for binary variables, such as whether or not the shop is using appropriate spray booths for painting) or as a mean (for continuous variables, such as the number of gallons of paint a shop uses per month).

We will use two different approaches to estimate this difference. The first approach will involve a simple comparison of the two-phase performance estimate for Group A to the two-phase performance estimate for Group B. This approach allows us to take advantage of any precision gains provided by the application of the two-phase approach. For any given group, the two-phase estimate of performance is given by:

where:

r = effective sample size for telephone survey

n = initial sample size for the telephone survey

= average performance for respondent shops (telephone measurement)

= average performance for sample of respondent shops (site visit measurement)

= average performance for sample of respondent shops (telephone measurement)

= average performance for sample of nonrespondent shops (site visit measurement)

Group A will be stratified by receipt of face-to-face compliance assistance. As a result, for Group A, we will need to develop separate two-phase estimates of performance for each stratum, then estimate the overall performance for Group A as the weighted sum of these two performance estimates, where the weights are equal to the proportion of the population represented by each stratum.

The second approach will involve estimating the difference within the context of a multivariate regression, with variables included to control for factors that EPA anticipates may impact performance. Examples of such factors include quantity of coatings used, number of years in operation, and geographic location. For example, for performance measures that are continuous, the site visit results will be used to estimate the following type of model:

Where

= performance measure for shop i

= a constant

= a binary (0/1) variable equal to one if the shop is in the treatment group (Group A) and equal to zero otherwise

= a binary (0/1) variable equal to one if the shop attended a workshop and equal to zero otherwise.

= a set of J shop characteristics expected to impact performance

= a random error term representing the effect of unknown shop characteristics that may impact performance

Within this context, represents the average difference in performance between Group A shops that did not attend a workshop and Group B (after controlling for various shop characteristics) and represents the average difference in performance between Group A shops that attended a workshop and Group A shops that did not attend a workshop. The regression-adjusted estimate of the short-term impact of compliance assistance on shop performance is given by + f , where f is the fraction of shops in Group A that attended a workshop.

Long-Term Impact of Compliance Assistance Package

The impact of Region 1’s two-year compliance assistance effort (outreach, workshops, and shop visits) will be estimated as the difference between 1) the change in performance for Group C and 2) the change in performance for Group X. We will use two different approaches to estimate this difference. The first approach will involve a simple comparison of the change in performance for Group C to the change in performance for Group X.

The second approach will involve estimating the difference within the context of a multivariate regression, with variables included to control for other observable factors that EPA anticipates may impact performance. For example, for performance measures that are continuous, the analysis of compliance assistance impacts will involve estimating models of the following form:

where

= performance measure for shop i

= a binary (0/1) variable equal to one if the measurement is from the treatment group in 2009 and equal to zero otherwise

= a binary variable equal to one if the measurement is from the treatment group in 2011 and equal to zero otherwise

= a binary variable equal to one if the measurement is from the comparison group in 2009 and equal to zero otherwise

= a binary variable equal to one if the measurement is from the comparison group in 2011 and equal to zero otherwise¹²

= a set of J shop characteristics expected to impact performance (e.g., size of shop, number of years in operation, or geographic location)

= a random error term representing the effect of unknown shop characteristics that may impact performance

Within this context, represents the average performance for Group C in 2009 after controlling for shop characteristics, and the coefficients , , and have analogous interpretations. The regression-adjusted difference-in-differences estimator of the long-term impact of compliance assistance is given by:

Assessing Telephone Survey Bias

Separate estimates of telephone survey bias will be developed for three distinct sets of shops : (1) Group A shops that attended a workshop, (2) Group A shops that did not attend a workshop, and (3) Group B shops. The telephone survey bias can be expressed as the difference between the estimated performance from the telephone survey respondents ( ) and the estimated performance obtained from the two-phase sampling approach ( ):

The estimated performance from the telephone survey respondents is simply the performance rate for all shops that responded to the telephone survey. The estimated performance from the two-phase sampling approach is a weighted average of the estimated performance rates for respondents ( ) and non-respondents ( ), with the weights determined by the telephone survey response rate ( ),

The performance for non-respondents is estimated as the performance rate for the on-site visits with non-respondents. The performance for respondents is estimated by adjusting the overall average performance rate from the telephone survey. The adjustment is calculated from the subsample of telephone survey respondents that also received site visits, and it is equal to the difference between the average performance rate as measured by the telephone survey and the average performance rate as measured on-site.

3. Reporting Results:

A report containing the questionnaire, sampling plan, calculations, and results (including variances and response rates) will be prepared. The report will be made available to all parts of the Agency and the public through the following means:

• A printed report, which will be distributed to all interested offices at EPA. Compliance assistance providers from EPA and other organizations will be alerted to the report via available electronic means (e.g. listservs). Additional copies will be made available to the general public through the National Technical Information System (NTIS)

• An electronic copy of the report will be posted to EPA’s Office of Enforcement and Compliance Assurance Website (www.epa.gov/compliance/assistance), and other Agency websites will link to the report (e.g., Region 1, OAQPS).

• EPA will provide OMB with a copy of the final report, as part of its continuing commitment to “conduct a survey every three years of a statistically-valid sample of compliance assistance recipients to measure behavior changes resulting from compliance assistance.”¹³

REEFERENCES

Bloom, Howard S. Learning More from Social Experiments: Evolving Analytic Approaches. Russell Sage Foundation. 2005.

Cochran, William G. Sampling Techniques (3^rd Edition), John Wiley & Sons. New York:. 1977.

Orr, Larry L. Social Experiments: Evaluating Public Programs with Experimental Methods. Thousand Oaks, CA: Sage Publications. 1999.

1 These workshops will be organized by EPA together with local partners (e.g., fire departments), and they will vary in content, duration, and location. However, at least one hour of all workshops will be dedicated to presenting the new requirements associated with the Surface Coating Rule. A standard PowerPoint presentation will be used to cover material related to the Surface Coating Rule.

2 Workshops and mailings will be provided to the control group shops shortly after the survey described above has been completed to ensure that all shops ultimately have the opportunity to receive compliance assistance. If any shops in the control group learn about the earlier series of workshops and indicate to EPA that they would like to participate, they will be encouraged to attend workshops at a later date. If they insist on attending the earlier series of workshops, they will be allowed to do so and tracked as “crossovers” for the analysis (see, e.g., Orr 1999, pg 211-213). We anticipate that only a small number of shops, if any, will insist on attending the earlier series of workshops.

3 Group C is a combination of Groups A and B from the short-term evaluation. After the conclusion of the short-term experiment, the shops in Group B will receive outreach materials and an offer to attend a workshop identical to the workshops that were initially offered only to Group A. This will make Groups A and B equivalent with respect to compliance assistance received, making the distinction between the two groups irrelevant for the long-term experiment.

4 Compliance assistance will be delayed by approximately three months for a subset of the Group C shops (i.e., until the end of the short-term experiment).

5 Compliance assistance site visits will consist of a post-survey “debrief” with the shop representative to review areas where environmental performance could potentially be improved, including areas related to the Surface Coating Rule requirements. In addition, the interviewer will provide the shop representative with (1) a guidance document focused on multimedia compliance and environmental best practices and (2) a brochure related to the Surface Coating Rule requirements. The entire site visit (including the survey and debrief) is expected to last approximately 45 to 85 minutes.

6 We are not planning a panel approach, where we would measure the same shops at multiple points in time. A panel approach is not feasible in this case due to potential measurement effects.

7 Elevated Risk - NATA data on levels of both cancer risk and non-cancer risk was broken into five classes using the Natural Breaks method - http://webhelp.esri.com/arcgisdesktop/9.2/index.cfm?TopicName=Natural_breaks_(Jenks), Towns that intersected any of the top four categories for both cancer risk and non-cancer risk were designated as elevated risk areas to be included in the population for this study. (Source: 1999 National Air Toxics Assessment data).

8 Our calculations assume that any disagreement between the telephone and on-site measurements is due to exaggeration of environmental performance in the telephone interviews. That is, we assume that telephone respondents will not report unsatisfactory environmental performance when the shop’s performance is in fact satisfactory.

9 The impact of performance rate on MDE is symmetric around 50%, so that the MDEs associated with a 70% performance rate are identical to the MDEs associated with a 30% performance rate (and similarly with 90% and 10%).

10 Additional information regarding the selected shops is available from Kenneth Harmon, (202) 564-7049.

11 If the Agency doubles the sample size, EPA-trained contractors will need to supplement the work of Agency employees for Group C.

12 Note that although all four group dummies have been included in the model, the constant term has been excluded.

13 Expanding the Use of Outcome Measurement for EPA’s Office of Enforcement and Compliance Assurance, Report to OMB, July 31, 2006, p. 30.

Part B, Page 20