Supporting_Statement_SDE_Part_B_7-27-11

Supporting Statement
DOE Solar Decathlon Impact Evaluation Surveys

OMB Control No. 1910-New

This supporting statement provides additional information regarding the Department of Energy’s (DOE’s) Office of Energy Efficiency and Renewable Energy (EERE) request for processing of the proposed information collection, Solar Decathlon Impact Evaluation Surveys.

2. Collections of Information Employing Statistical Methods.

The surveys included in this ICR will be used for an impact evaluation of DOE’s Solar Decathlon program. The research design for the impact evaluation calls for the comparisons of post-intervention measurements on homeowners who experienced the Solar Decathlon in person with homeowners who experienced it through the media and with a comparison group of homeowners who did not experience it or know about it. These comparisons will be statistical. A fourth collection will be targeted to former college students who did not participate in a Solar Decathlon and will be statistical. Part B covers these four surveys. A fifth survey of former students who participated in a Solar Decathlon (decathletes) will not use statistical methods and is not covered in Part B except as needed to discuss certain features of the evaluation.

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used.

Potential Homeowner Respondent Universe

The respondent universe for the homeowner survey will consist of individuals over the age of 18 living in owner-occupied single-family detached or single-family attached homes on their own property lots in the Washington, D.C. and the Baltimore, MD Metropolitan Statistical Areas (MSAs) who have landline telephone service and speak English. For brevity, these individuals are referred to as “homeowners.” The qualifying population is limited to owner-occupied single-family detached or single family attached homes because one of the evaluation objectives is to estimate the effect of the Solar Decathlons on homeowner installations of solar energy systems, and these types of housing units are most likely to have the required attributes for such installations.

We used the term “Washington DC Metropolitan Statistical Area” in Part B as an abbreviation for OMB’s “Washington-Arlington-Alexandria, DC-VA-MD-WV Metropolitan Statistical Area.” This metropolitan statistical area (MSA) includes the following metropolitan divisions and principal cities:

Washington-Arlington-Alexandria, DC-VA-MD-WV Metropolitan Statistical Area Principal Cities: Washington, DC; Arlington, VA; Alexandria, VA; Reston, VA; Bethesda, MD; Rockville, MD; Frederick, MD; Gaithersburg, MD

Bethesda-Rockville-Frederick, MD Metropolitan Division: Frederick County, Montgomery County

Washington-Arlington-Alexandria, DC-VA-MD-WV Metropolitan Division: District of Columbia, DC; Calvert County, MD; Charles County, MD; Prince George's County, MD; Arlington County, VA; Clarke County, VA; Fairfax County, VA; Fauquier County, VA; Loudoun County, VA; Prince William County, VA; Spotsylvania County, VA; Stafford County, VA; Warren County, VA; Alexandria city, VA; Fairfax city, VA; Falls Church city, VA; Fredericksburg city, VA; Manassas city, VA; Manassas Park city, VA; Jefferson County, WV

(Source: Office of Management and Budget, OMB Bulletin No. 10-02, December 1, 2009. p. 52.)

This area reaches south to Fredericksburg, VA and west to Front Royal, VA. It includes Manassas, VA. We judged that this region includes all of the major urban areas that were connected through media and employment to Washington, D.C., and therefore, would be regions with higher probability for homeowners who visited to the Solar Decathlons.

The population is limited to the Washington, D.C. and Baltimore MSAs (target MSAs) because these are the geographic regions from which visitors to the Solar Decathlons are most likely to come. Restricting the study to this region increases the probability of interviewing qualified respondents considering the resources available.

The objectives of the Solar Decathlon evaluation include:

• Estimate the impact on homeowners’ knowledge of solar-home construction, installations, and energy-efficiency products resulting from attendance at a Solar Decathlon.

• Estimate the impact on homeowners’ decisions to install residential solar systems and buy energy-efficient appliances resulting from attendance at a Solar Decathlon.

• Estimate the impact on the knowledge of solar-powered home construction and installation gained by students participating in the Solar Decathlon.

• Estimate the impact on career choices of students participating in the Solar Decathlon with particular interest on:

• Solar-energy and energy-efficiency employment since participation in a Solar Decathlon

• Solar-energy and energy-efficiency installations they have influenced since participation in a Solar Decathlon.

These objectives apply to the effect of the Solar Decathlons on the behavior and knowledge of households in owner-occupied homes and by college students participating in a Solar Decathlon (decathletes).

Including mobile phone numbers would be likely to result in reaching many younger, more mobile, families living in apartments. However, as noted above the team judged that the consequences of limiting the universe of interest to homes with landline phones will not negatively affect the usefulness of the results. This segment of the population typically cannot install systems to provide solar power to their households, and their major energy-using appliances usually are provided by a landlord. These assumptions are supported by the 2010 Center for Disease Control’s National Health Interview Survey data:

• Thirty percent of U.S. households are cell-phone-only households, but,

• About half of the adults living in cell-phone-only households are renters and another 35% claim some other tenure arrangement other than ownership and renting

• Adults living in poverty and adults living near poverty were more likely than higher income adults to be living in households with cell phones only.

• Adults under the age of 29 were more likely to be living in cell-phone-only homes than older adults.

(Source: Blumberg SJ, Luke JV. Wireless substitution: Early release of estimates from the National Health Interview Survey, July=December 2010. National Center for Health Statistics. June 2011. Available from: http://www.cdc.gov/nchs/nhis.htm. )

Although these CDC data apply mostly to numbers of adults rather than numbers of homes, they tend to support the assumptions we used in making our decision that including cell phones in our sample frame would not be cost-effective.

Regarding the English-only language assumption, we decided to limit the universe of interest to English-speaking homes. Five-year data from the American Community Survey (ACS) tends to support this decision. Less than four percent of the population in the Baltimore-Towson, MD MSA and about ten percent of the population in the Washington, D.C., etc., MSA speak English less than “very well.” (Source: U.S. Census Bureau, American Fact Finder, based on ACS five-year data for 2005-2009. http://factfinder.census.gov) This language barrier suggests that many of the households in which this population lives are not likely to install solar systems or be concerned about the efficiency of appliances and lighting. Considering that this fourteen percent of the population in our sample domain probably represents much less than fourteen percent of the homes, we felt justified in avoiding the cost of translating and programming the non-English surveys.

Our screening process for the homeowner survey is designed to identify an individual who can speak in English for the energy-related decisions made on behalf of the household. If the person who answers the phone speaks only a foreign language, our screening process assumes the person will (1) hang up, or (2) remark in their native language that they do not speak English and refuse the call, or (3) call an English-speaking member of the household to the phone. In the first case we will lose the interview. For the first case (“hang up”), we do not believe this is a severe threat to the validity of the final data. In the second case, the interviewer will try to ask to speak with someone who speaks English before the call is disconnected. If the interviewer succeeds in this, he or she will apply the other screeners to the English-speaking person who comes to the phone to determine if the interview should proceed. If the interviewer does not have the opportunity to ask for an English speaking person, we will lose a potentially qualifying interview. In the third case, we speak with an English-speaking person and ask the questions regarding the other screening criteria. If they pass the screening criterion we will accept this person as someone who can speak for the energy-related decisions made by the household.

A similar screening process is applied to selecting potential respondents for the non-decathlete, former-college-student phone survey.

The homeowner universe is limited to owner-occupied single-family detached or single-family attached homes on their own property lots because this is the population that is most likely to invest in solar-energy installations.

The estimated time for a visitor to a Solar Decathlon to complete the homeowner survey is 12 minutes. It is based on the pretests of the questionnaire. No respondent will ever answer all of the questions in the questionnaire. For example, if the respondent did not install solar panels, they will skip the questions about the solar panels. If the respondent did install solar panels, they will skip questions about the installation decision process. The pretests also demonstrated that other batteries of questions will be skipped. For example, if they did not install energy-efficient lighting or appliances, they will skip the batteries of questions about these installations.

Homeowner Respondent Sample Selection Methods

The sampling method for the homeowner surveys will be random digit dialing (RDD). Landline RDD samples will be purchased from Marketing Systems Group using their Genesys Sampling System (see www.genesys-sampling.com/home.aspx). The samples provide an equal probability of selection for listed and unlisted residential telephone numbers.

The homeowner sampling approach is discussed further under Item #2 in Part B of this Supporting Statement. Genesys provides county-level geographic identification information using the Federal Information Processing Standard (FIPS) County Code with its phone numbers.

Homeowner Respondent Sample Stratification

Using 2008 American Community Survey household data, plus reasonable assumptions, we estimate the probability of finding such a household meeting the survey qualification criteria in the Washington, D.C. and Baltimore MSAs to be about 0.01.

The reasonable assumptions used in developing this probability included the following, obtained for both MSAs and combined or averaged as the need required:

• The proportion of households that are owner-occupied single-family-detached and single-family attached households with at least one person having some college or higher (ACS: 61%)

• The number of persons attending the four Solar Decathlons between 2002 and 2009 (provided by DOE: about 500,000)

• The average number of persons per household (ACS: 2.46)

• The percent of all Solar Decathlon visitors who were from the two MSAs (a subjective estimate: 60%)

• The percent of all visitors who were students on day trips (a subjective estimate: 33%)

• The proportion of all visitors who have moved from the two MSAs since visiting a Solar Decathlon (a subjective estimate: 40%).

The homeowner questionnaire contains tracks for visitors to a Solar Decathlon and non-visitors. The use of a single survey with two tracks for the homeowner surveys contributes to the project’s efficiency.

It is anticipated that visitors with some college education will be more likely to adopt renewable and energy-efficiency measures than visitors without college education.

To obtain support for this statement, we searched state and utility renewable and efficiency program evaluations and can offer the following support from these sources.

Regarding efficient lighting, a 2007 evaluation of Maine’s efficient lighting program found that, “Twenty-six percent of Maine residents have graduated from a 4-year college compared to 54% for all [lighting product rebate] coupon participants and 56% for markdown purchasers.” We infer that a similar difference carries over to non-program participants. (Source: NMR, “Process and impact Evaluation of the Efficiency Maine Lighting Program, April 10, 2007, p. 75,)

A 2004 evaluation of the efficient lighting programs of other New England states reported a similar finding for efficient lighting products. (Source: NMR, “Impact Evaluation of the Massachusetts, Rhode Island, and Vermont 2003 Residential Lighting Programs,” October 1, 2004, p. 18.)

Regarding energy-efficient appliances, we based our perception on a Wisconsin evaluation that reported that a higher percent of households with college-educated members are aware of ENERGY STAR and understand what it means than households with high school educations or less. (Source: “Glacier Consulting, “Awareness and Understanding of ENERGY STAR in Wisconsin, 2004,” Final Report, May 27, 2005, pp. G-4 & G-5. This report included Iowa and found the same results relevant to education.) We infer that higher levels of awareness and understanding correlate with higher levels of efficient appliance purchases.

Regarding residential solar systems, we based our statement on an evaluation of New Jersey’s Customer On-site Renewable Energy (CORE) Program which found that a strong plurality of renewable energy electricity systems—mostly solar—are installed by households with a young-to-middle-age (35 to 54) head of household with some college education. (Source: Aspen Systems Corporation, “Process Evaluations of the Renewable Energy Programs Administered and Managed by the New Jersey Board of Public utilities Office of Clean Energy,” November 2004, p. 4-28.) Another source from Oregon reports a similar finding for solar water heaters (Dethman & Associates, “Oregon Solar Thermal Market Characterization,” May 21, 2004, p.31.) Renewable energy installations are expensive; therefore, it is reasonable to expect that most are installed with program support and that this statement about program participants applies to the population at large.

To take advantage of this anticipated difference between visitors with and without college education and to account for other differences in the characteristics of the two subpopulations, the homeowner population will be stratified for sampling purposes into homeowners with at least some college educations or an associates’ degree and homeowners with no college education. The homeowner stratum population counts appear in Table B1.

Table B1: Estimated U.S. Owner-occupied Single-family Detached and Attached Housing Unit Population by Educational Stratum

Stratum	Estimated Household Population in Target MSA Region*
Some college education or associate’s degree, or higher	1.5 million
No college education	0.3 million
Total	1.8 million

*Source: 2008 American Community Survey

Potential Non-decathlete Former-Student Respondent Universe

The respondent universe for the non-decathlete former-student survey will be individuals who were enrolled as undergraduates between 2002 and 2009 in a four-year school of study in an academic discipline similar to those of students who participate in the Solar Decathlons. Mostly these disciplines consist of engineering and architectural fields of study. Former students who received a bachelor’s degree or higher or who left school before receiving a degree will be considered eligible for the survey if they majored in one of these disciplines. Graduate students will be eligible for the survey.

The geographic domain for the non-decathlete former-student population will be national because (1) DOE desires national estimates for this population, and (2) the results of this survey will be compared qualitatively with a comparable survey of decathletes who come from all parts of the country. The estimated population counts of non-decathlete former-students with academic majors comparable to those of decathletes appear in Table B2.

Table B2: Estimated Population Counts of Non-decathlete Former-students with Majors Comparable to those of Decathletes and Their Households

Student Group	Estimated Population
Non-decathlete former-students with comparable academic majors	723,900*
Households of non-decathlete former students	276,300**

*2008 National Center for Education Statistics Digest of Education Statistics; National Science Foundation Division of Science Resources Statistics data on Science and Engineering degrees.
**Estimated by dividing the population of non-decathlete former students by 2.62, the average number of persons per household in the U.S. (American Community Survey, 2008)

The non-decathlete former-student survey is national while the homeowner survey is limited to the Washington-Baltimore MSAs. DOE desires national estimates for the non-decathlete former-student survey results. The decathletes come from all parts of the U.S.; therefore, if a comparison is to be made between decathletes and non-decathletes, even if it will be a non-statistical comparison, the non-decathletes should also be drawn from all parts of the country.

DOE considered developing a Decathlete Survey as opposed to terminating the website survey if the respondent indicates that they participated in a Solar Decathlon while in college. The possibility of switching to a phone version of the decathlete survey was considered during the design of the survey and was rejected for the following reasons:

1. We judged the probability of getting a decathlete during this national survey would be extremely low. Therefore the extra effort required to design a separate decathlete track for the survey would produce a very small benefit for the cost of the effort, if any.

2. A separate telephone track for decathletes would either (a) increase the survey vendors unit price for a completed non-decathlete interview because the vendor would have to plan for an unpredictable number of additional completed interviews beyond the quota established for non-decathlete interviews; or (2) require a separate price for an open ended number of decathlete interviews, adding an open-ended cost to the non-decathlete survey. The vendor would also add a separate programming cost for the separate decathlete track. Both activities would increase the price of the survey for little expected benefit. For these considerations, we decided to rely on the online decathlete survey for the decathlete information.

Non-decathlete Former Student Respondent Sampling Methods

The sampling method for the non-decathlete former student survey will be random digit dialing (RDD). Landline RDD samples will be purchased from Marketing Systems Group using their Genesys Sampling System (see www.genesys-sampling.com/home.aspx). The samples provide an equal probability of selection for listed and unlisted residential telephone numbers.

The non-decathlete former-student sampling approach is discussed further under Item #2 in Part B of this Supporting Statement.

2. Describe the procedures for the collection of information including statistical methodology for stratification and sample selection, estimation procedure, degree of accuracy needed for the purpose described in the justification (Part A):

Homeowner Surveys

Statistical Methodology for Stratification and Sample Selection for the Homeowner Surveys

Only the homeowner surveys will be stratified. As described under Item #1, they will be stratified into qualifying homeowners with some college education or higher or an associate’s degree and qualifying homeowners without college education.

RDD methodology will be used to select samples of telephone numbers for the homeowner survey. Homeowners will be qualified by stratum during the interview.

The Genesys RDD methodology selects samples from 100 banks¹ that contain at least one listed telephone number. Each phone number will have the same probability of selection. The methodology sets the number of sampling intervals equal to the size of the RDD sample. A new random number is selected within each interval and the corresponding telephone number is selected for the sample. The sample of telephone numbers will be screened by Genesys to identify business, fax/modem, and non-working telephone numbers. This process eliminates about 70% of the non-productive telephone numbers in the sample.

The choice of the sample sizes for homeowners in each stratum is a balance between the target confidence interval for a variable of interest to the survey, statistical power considerations, and total survey costs.

The target confidence interval for homeowner survey estimates will be estimates that are within at least +/-10% of the true value (margin of error) with a 90% level of confidence. This confidence interval is an unofficial standard used throughout the energy-program-evaluation industry.

The variable of interest, or design variable, for the target confidence interval is the estimated percent of households in owner-occupied single-family detached and single-family attached houses that have an energy-efficient technology installed in their house. We will use an actual estimate of the value of the design variable to size the homeowner sample. The Energy Information Administration’s (EIA’s) most recent (2005) Residential Energy Consumption Survey (RECS) for which data are available estimates the national value of the design variable at 6l%. This 2005 RECS estimate is the “baseline percent” for design purposes.

All of the survey variables for which confidence intervals will be estimated are percentages.

Assuming a random sample, a minimum sample size of 64 will satisfy the target confidence interval for estimated percentages for a baseline percent of 61%. This information, plus estimates of the sample sizes needed to detect a statistically significant difference in the two populations on the design variable provides a basis for the study’s homeowner sample sizes.

Degree of Accuracy Needed and Numbers of Completed Interviews for Homeowner Samples

The evaluation design requires that certain measurements on the sample of visitor respondents be compared to comparable measurements on the sample of non-visitors and that these comparisons provide statistical estimates of the regional differences in the two populations. These differences will be interpreted as the program impacts in the region studied. Therefore, an important consideration in designing the sample is the power of the statistical tests that will be used to measure program impacts.

The word “differences” refers to measurements on variables describing visitors compared to measurements on comparable variables describing non-visitors. The words, “program impacts” mean the differential behavior and knowledge of visitors to a Solar Decathlon and non-visitors as measured for the period between the visitors’ visitation and the time of the survey. Examples of behavior include the installation of a solar system on their residence or the purchase of an energy-efficient appliance. Examples of knowledge include perceptions of the characteristics of residential solar-energy installations. The study will interpret the differential behavior and knowledge as the effect, or “impact” of the Solar Decathlon. Hopefully, the visitors will show more of the behavior and knowledge described in the project’s objectives.

A classical evaluation design would measure the variables on a random population sample before the population had an opportunity to visit a Solar Decathlon and then re-measure the same variables on the same sample at a period of time after the visit, hoping that the sample was large enough to have included some visitors. This would allow the evaluator to account for pre-Solar Decathlon differences between the visitors and non-visitors that might have influenced the visit and post-visit behavior and knowledge. This type of classical evaluation design is not possible for the Solar Decathlon because a pre-visit measurement does not exist, and it might require a sample that was too large and expensive to be cost effective. The types of questions asked might also influence the respondents to visit the Solar Decathlon and take actions that they would not otherwise have taken.

We endeavor to account for pre-Solar Decathlon differences in the visitor and non-visitor samples by asking questions that reflect a prior interest in renewable energy and energy efficiency. We will then compare the two samples on these questions and report whether pre-visit differences might have influenced the results.

“Statistical power” is the probability of detecting a real difference between the attributes or actions of the two homeowner groups with samples of given sizes. The power depends on the hypothesis being tested, the desired confidence level for comparisons, the sample sizes for each homeowner group, and the size of the difference between homeowner groups that we are trying to detect. For this study we are comparing proportions of homeowners who visited a Solar Decathlon versus a comparison group of homeowners who did not visit a Solar Decathlon. In general, a power of 80% is deemed a reasonable value. This allows for up to a 20% chance of not detecting a real difference between two groups.

The power calculations for this study design assume a 90% confidence level, one-sided tests for equality of proportions (i.e., p_{1 >} p₂, where p₁ is the probability of finding an energy-efficient installation in a Solar Decathlon visitor’s house and p₂ is the same probability for a non-visitor’s house), simple random sampling, and an estimated baseline proportion equal to 0.61 (61%) for homeowners who did not visit the Solar Decathlon.

Table B3 presents the statistical power of our sample sizes to detect homeowner group differences shown for p₁ and p₂, consistent with a 90% confidence level. The sample sizes for n₁ consider the higher survey cost of identifying visitors to a Solar Decathlon and homeowners who are aware of the Solar Decathlon but have not visited it.

Table B3: Power Analysis for Homeowner Differences on the Design Variable*

Group Compar-ison**	p1 (%) (prob of energy-eff installation in visitor’s house)	p2 (%) (prob of energy-eff installation in non-visitor’s house)	Sample Size n1	Sample Size n2	Difference to Detect	Power (%)
A vs. B	71	61	200	280	10	84%
A vs. C	70	61	200	400	9	82%
B vs. C	69	61	280	400	8	81%
A vs. B&C (visitors vs. non-visitors)	70	61	200	680	9	86%

* Subscript 1 represents visitors; subscript 2 represents non-visitors.

** Homeowner groups are defined in Part A, Item #2.

The sample sizes shown in table B3 have acceptable power for detecting homeowner group differences for absolute differences in group proportions of 0.08 (8%) to 0.10 (10%) or more (column p₁ – column p₂). Table B4 summarizes the sampling plan selected for the homeowner surveys.

Table B4; Sampling Plan for Homeowner Surveys

Education Strata	Completed Interviews Group A (Visitors)	Completed Interviews Group B (Aware)	Completed Interviews Group C (Neither visitor nor aware)	Total Completed Interviews
Some college	100	140	200	440
No college	100	140	200	440
TOTALS	200	280	400	880

A sample size of 200 provides a margin of error of +/-4% at a confidence level of 90% for the design variable, the percent of households investing in an energy-efficient technology since the first Solar Decathlon.

Estimation Procedures for the Homeowner Samples

Sampling weights will be calculated for the homeowner surveys in order to produce regional estimates. Within each stratum, the base sampling weight for each completed interview will be calculated as the inverse of the probability of selection of that household. The base sampling weight will be multiplied by a non-response adjustment factor to adjust for unit non-response. The non-response adjusted weight will then be multiplied by a post-stratification adjustment factor so that the final weights will sum to known Census Bureau totals for population characteristics (age and census region).

Non-decathlete Former-student Surveys

Statistical Methodology for Stratification and Sample Selection for the Non-decathlete Former-student Surveys

Interviews will be completed with a national random sample of former students who were enrolled in degree program similar to one of those pursued by decathletes at a 4-year U.S. college or university between 2002 and 2009 and did not participate in a Solar Decathlon. Students who are undergraduates at the time of the survey will not be eligible for participation; however, graduate students will be eligible.

Degree of Accuracy Needed and Numbers of Completed Interviews for the Non-decathlete Former-student Sample

Statistical sampling procedures are being used for the non-decathlete former-student sample because there will be findings from the non-decathlete former-student survey, e.g., findings regarding the knowledge of the non-decathlete former students about residential solar-energy systems, that will be of value to EERE and the Solar Decathlon Program.

The design variable for the non-decathlete former-student survey sample will be the proportion of non-decathlete former-students that have installed an energy-efficient technology in their houses. Assuming simple random sampling and a 61% baseline value for this variable, 110 completed interviews will allow estimation of the proportion for the design variable with a margin of error of +/-8% with 90% confidence.

A statistical comparison of differences between the non-decathlete former-student and decathlete results will not be made on any variable; therefore, a power analysis is not required.

Estimation Procedure for the Non-Decathlete Former-Student Sample

The estimation procedures for the non-decathlete former-student survey results will be the same as those described for the homeowner sample.

Unusual problems requiring specialized sampling procedures

No unusual problems requiring specialized sampling procedures are anticipated.

Use of periodic (less frequent than annual) data collection cycles to reduce burden

There are currently no plans to repeat these surveys.

3. Describe methods to maximize response rates and to deal with issues of non-response.

Methods for maximizing response rates

Inasmuch as this is a new survey, there are no historical data on response rates. A similar survey of the general public was conducted by DOE EERE in 2004 for its hydrogen program.² The 2004 survey produced a response rate of 25%³. Allowing for a continued deterioration in response rates in recent years, we are assuming a 15% response rate for the homeowner and non-decathlete former-student surveys. The final response rate will be calculated as the number of completed interviews (including screener terminations) divided by the number of households in the sample. American Association for Public Opinion Research (AAPOR) standards will be used to calculate response rates based on the final survey disposition codes⁴.

During the actual data-collection process itself, the procedures described in section B.3 for obtaining completed telephone interviews will be implemented.

After the data for the statistical surveys have been collected, we will post-stratify the respondents and calculate post-stratification weighting adjustments. These we will apply to the original sample weights for each household so that the final weights allow the sample households to sum to the target population in the respective post-stratification strata. At present, we expect post-stratification will use age and education; however, the final post-stratification scheme will be established after we receive the data.

All statistical surveys will employ CATI interviewing technology. Professionals trained in CATI interviewing techniques will conduct the homeowner and non-decathlete former-student interviews. All new interviewers receive two days of training including classroom training and live dialing interviewing exercises. Interviewers are monitored by quality control staff during the calling period. Problem areas identified during the QC monitoring will be addressed through additional interviewer training and follow-up monitoring.

The sample of RDD telephone numbers will be screened by the Genesys Sampling System to identify and remove non-residential numbers from the sample. Calls will be made during weekday evenings and on weekends when homeowners are more likely to be at home. Up to seven attempts will be made to reach a household. No answer or busy signal call dispositions will be re-dialed three hours later. Potential respondents who hang up in the introduction will be considered “soft refusals” and will be redialed.

Methods for dealing with non-response

Homeowner surveys

Several methods are available for dealing with non-response. Interviewers will leave messages on answering machines with a toll-free number for homeowners to call.

Call-backs are an important tool to minimize non-response. Call-backs are prioritized so that the CATI system will bring up the call-back record at a specified appointment time. Callbacks are scheduled when the respondent cannot speak at the current time but wants to reschedule the interview or when the respondent starts but cannot complete the interview at the current time. The CATI system also automatically schedules call-backs for attempts ending as no-answer, busy, or answering machine.

The response rate will be closely monitored. If necessary, additional actions will be taken to increase the response rate such as extending the length of the interviewing period or increasing the number of attempts.

After survey completion, base weights will be adjusted for non-response. Post-stratification adjustments will be made to the weights to account for differences in response by census region and respondent age so that weighted counts agree with Census Bureau population counts for owner-occupied single-family detached and single-family attached households.

In the report of results, we will describe the relationship between the household sample demographics and American Community Survey (ACS) household demographics. We will describe the post-stratification procedures made to adjust the weighted survey results to the ACS demographics, and we will comment on variables whose weighted results might not be representative of the population as a result of these adjustments.

Non-decathlete Former-student surveys

The same procedures will be used to deal with non-decathlete former-student non-response as were described for the homeowner surveys.

4. Describe any tests of procedures or methods to be undertaken.

No tests of procedures or methods are planned as part of the data collection.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s) or other person(s) who will actually collect and/or analyze the information for the agency.

Lockheed Martin designed the evaluation, will oversee the surveys, conduct the data analysis, and prepare the evaluation report. The principal investigator for the project is Dr. Harley Barnes, (301) 519-6322. Lockheed Martin’s statistical staff designed the sample and the analysis procedures. The principal contributor to the statistical design was Nancy Hassett.

1 Each 10-digit telephone number consists of an area code, a 3-digit prefix, and a 4-digit suffix. A 100-bank is a block of 100 consecutive numbers associated with the first two digits of the 4-digit suffix, e.g., 5200-5299.

2 The hydrogen program survey was not conducted in support of an evaluation.

3 See report ORNL/TM-2006/107, Results of the 2004 Knowledge and Opinions Surveys for the Baseline Knowledge Assessment of the U.S. Department of Energy Hydrogen Program, page 30. This report may be accessed at http://www1.eere.energy.gov/hydrogenandfuelcells/hydrogen_publications/pdfs/survey_main_report.pdf .

4 See AAPOR’s 2009 Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys.

Page 13 of 13