Sample size supplement

DRAFT
Sample Sizes Determination and Decisions for the 2015 American Housing Survey and Beyond
Shawn Bucholtz, U.S. Department of Housing and Urban Development
Stephen Ash and Ernest Lawley, U.S. Census Bureau

Purpose
The purpose of the paper is to discuss the process for determining the national and metropolitan area
samples sizes for the 2015 American Housing Survey (AHS) and beyond. This whitepaper assumes that
the 2015 AHS sample design includes a national sample integrated with metropolitan area samples from
the 15 largest metropolitan areas, by population, as of 2013. Other metropolitan area samples will be
independent samples. The integration of the national and metropolitan area samples is discussed in the
whitepaper “Metropolitan Area Selection Strategies for the American Housing: 2015 and Beyond.”

1.

AHS Sample Size History

1.1

National Longitudinal Sample Size Precision Statement

The current statistical precision statement for the national longitudinal sample, which been included in
HUD’s recent Paperwork Reduction Act (PRA) submissions to OMB, is: “A two-year change of 10% in the
median monthly costs for 5 percent subgroups will have a standard error of 5%.” Although not stated
explicitly, the population domain for this statement is the whole US. However, this statement does not
specific any particular 5 percent subgroup. The subgroups could be based on political geography (city),
statistical geography (Census Urban Area), demographic or socioeconomic characteristics (e.g.,
households with a disabled person).
The precise origin of the AHS’s statistical national longitudinal sample precision statement is unknown.
However, it certainly makes sense, from the perspective of the US Department of Housing and Urban
Development (HUD), that a statistical precision statement for the national longitudinal sample would
focus on detecting large changes in housing costs for groups of interest. Moreover, this statement of
precision assumes a repeated survey, like the AHS, although not necessarily the repeated sample that
exists in the AHS.

1.2

National Longitudinal Sample Sizes, 1985 – 2013

The national longitudinal sample size has fluctuated somewhat since its initial use in 1985. Between
1985 and 1991, the sample size was approximately 50,000. Between 1993 and 2009, the sample size

DRAFT
averaged about 55,000. Due to increased budget, the sample size was expanded for 2011 and 2013. To
expand the sample for 2011, three additions were made. First, roughly 5,200 cases that were reduced
from the 2007 sample were reinstated. Second, approximately 3,100 units that were selected during
the 2000 redesign, but were never interviewed, were introduced. Finally, an oversample of
approximately 5,300 HUD-subsidized housing units was introduced.
Most of the years between 1985 and 2013 included supplemental sample for areas of special interest.
The neighborhood supplemental samples conducted in 1985, 1989, and 1993, were not weighted, so
they did not contributed to the summary statistics. The same was true for the supplemental rural
samples in 1987 and 1991. In contrast, the Big 6 metropolitan areas samples were weighted in
accordance with the rest of the national longitudinal sample.

Year

National
Supplemental
Name of Supplements
Longitudinal Sample National Sample Size
Size
1985
49,000
12,000
Neighborhood sample
1987
47,400
6,100
Rural sample
1989
47,500
9,000
Neighborhood sample
1991
51,400
6,100
Rural sample
1993
53,000
6,000
Neighborhood sample
1995
53,500
6,000
Big 6 metropolitan area
1997
53,100
1999
54,130
6,570
Big 6 metropolitan areas
2001
55,700
2003
56,570
6,430
Big 6 metropolitan areas
2005
59,450
2007
55,000
2009
55,700
6,300
Big 6 metropolitan areas (except LA)
2011*
69,700
2,700
Big 6 metropolitan areas (only LA)
2013
68,850
15,550
Big 6 metropolitan areas (except LA)
*For 2011, 28 additional metropolitan samples totaling 114,000 cases were weighted as part of the
national sample. However, they are left out of this table so as to facilitate comparison with other years.

1.3

AHS Metropolitan Area Sample Sizes, 1985 – 2013

Throughout the history of the AHS, metropolitan area sample sizes have varied greatly. Prior to the
redesign of the survey in 1983, sample sizes for the largest metropolitan areas was approximately
15,000 housing units and 5,000 for the smaller metropolitan areas. After the 1983 redesign, the sample
size for the largest metropolitan areas was 8,250 housing units and 4,250 for smaller metropolitan
areas. A further reduction to 3,300 was implemented in the mid-1980’s.

DRAFT
In 1995, a standard of 4,500 was adopted for all metropolitan areas, although this standard was not
implemented consistently between 1995 and 2009. For example, facing budget restrictions, the sample
sizes for the independent metropolitan areas were reduced to approximately 2,600 for 2007 and 2009.
The sample sizes for the Big 6 metropolitan area, which were integrated into the national sample
starting in 1995, averaged around 2,500. Between 2011 and 2013, the AHS included 44 metropolitan
area surveys, each with a sample of approximately 4,500 housing units.
Unfortunately, it is unclear how the statistical precision standard of 4,500 housing units was
determined. This sample size is certainly not large enough to achieve the statistical precision standard
for the national sample.

2.

Considerations for AHS Sample Size

2.1

Considerations for the National Sample Size

In a discussion of AHS sample size for 2015 and beyond, there are at least five key points to recognize:
1. The AHS is a repeated survey because HUD’s goal is to measure changes in the housing stock over
time. A statistical precision statement should reflect HUD’s goal of measuring changes in the
housing stock.
2. The AHS is a longitudinal survey. If a key indicator requires responses in two or more time periods
(e.g., tenure switching), the precision of the estimates of that indicator will be impacted by
longitudinal non-response or attrition.
3. The AHS is a demographic survey, as opposed to a demographic experiment. As such, the goal is
pick a sample size sufficient to estimate a characteristic (indicator) with a given level of precision, as
opposed to testing an effect with a given power. For instance, the goal to may be to estimate
median housing cost with a margin of error of 5%.
4. The AHS covers several topics related to housing. For the purposes of determining sample size,
there may not be one particular indicator or subgroup that necessarily takes precedence over
another.
5. The AHS will never have the budget of the ACS. As such, the AHS will never have enough sample to
produce reliable estimates for small areas such as Census tracts. Furthermore, the AHS makes no
attempt to have sample in every county or city.

DRAFT
6. Historically, the domain for the national AHS sample has been the US housing stock. Establishing
additional domains, such as Census Divisions, requires multiplying the desired sample size by the
number of domains of interest.

2.2

Considerations for the Metropolitan Area Sample Size

Each of the six points for consideration for the national sample size is also true of metropolitan area
sample size. In a discussion of the AHS metropolitan area samples size for 2015 and beyond, there are a
few additional points to consider:
1. The mathematics of probability is such that, for a given desired precision, the necessary sample size
to achieve that precision will be the same if your population is 320,000 or 320,000,000. For the AHS,
it is not desirable to apply the same desired precision standard to the national sample and the
metropolitan area samples. Doing so would require metropolitan area sample sizes well beyond any
conceivable budget for the AHS.
2. Given the history of the AHS’s fluctuating budgets, it may not be prudent to develop a statistical
precision statement that assumes a repeated sample every four or six years.

3.

Options for Determining the AHS Sample Size

With these above key points in mind, there are a few different ways to develop the precision statement.
1. The sample size is based on detecting a change in an estimate of a single key indicator, for a desired
level of precision.
Example: A sample size sufficient to detect a change of 10% in median housing costs with a margin
of error of 5%.
2. The sample size is based on detecting a change in an estimate of a single key indicator for specific
subgroups, for a desired level of precision.
Example: A sample size sufficient to detect a change of 10% in median housing costs for low-income
renters, with a margin of error of 5%.
3. The sample size is based on detecting a change in an estimate of a single key indicator for general
subgroups, for a desired level of precision.

DRAFT
Example: A sample size sufficient to detect a change of 10% in median housing costs for 5%
subgroups, with a margin of error of 5%.
4. The sample size is based on detecting a change in an estimate of proportions of general subgroups,
for a desired level of precision.
Example: A sample size sufficient to detect a change of 1% in 5% subgroups, with a margin of error
of 0.5%.
5. The sample size is based on detecting a change in an estimate of a single key indicator where the
indicator is a longitudinal measure, for a desired level of precision.
Example: A sample size sufficient to detect a change of 1% change in the percent of housing units
switching tenure, with a margin of error of 0.5%.

4.

Decision on 2015 Sample Sizes

Any decision about survey sample size must be made within the context of the available budget.
Although budgets have fluctuated over the past 30 years, HUD has generally secured enough funding to
conduct the national sample, and has conducted metropolitan area samples based on the remaining
budget. The expectation is that this pattern will persist. As such, the subset of options for developing
national and metropolitan area precision statements is constrained by expected budget, and HUD
evaluated options that were considered feasible, given the expected survey budgets.

4.1

National Sample Size Precision Statement

After careful consideration of numerous options for the national sample size precision statement, HUD
decided to stick with the current national sample precision statement:
For the US as a whole, a two-year change of ten percent in median monthly housing costs for
five percent subgroups will have a standard error of five percent.
This precision statement, when applied at the 95% confidence level, will permit the detection of a 10
percent change in median monthly housing costs for five percent subgroups. This precision statement is
not designed for a particular five percent subgroup. Rather, the statement is intended to be applied to
generic five percent subgroups, whether they are based on geography, demographics, housing
characteristics, or other indicators of interest.

4.2

Base National Sample Size Calculation

DRAFT
The calculation of the national sample size necessary to achieve the precision goal is detailed in
appendix 1. In short, the sample size was determined by developing a simulation model based on data
from 2009 and 2011. The simulation model first identified five percent subgroups that had a median
monthly housing cost change of 10 percent or greater, then sub-sampled from the groups at various
sample sizes to determine the minimum sample size necessary to achieve a 5 percent standard error.
The results of the simulation showed that a five percent subgroup size of 2,000 housing units was large
enough to achieve a standard error of 5 percent. With a five percent subgroup size of 2,000 housing
units, the total sample size would be 40,000.
One more adjustment was made to determine the final national sample size. It is common with surveys
to assume a non-response rate and to increase the sample size accordingly. The single-survey nonresponse rates for the AHS over the past several survey cycles have been approximately 14 percent. As
such, to achieve a national sample size of 40,000 complete interviews, we would need to interview
approximately 46,500 housing units.
However, a single-survey non-response rate is not necessarily that appropriate adjustment for the AHS.
The AHS is longitudinal and some five percent subgroups of interest are based on a longitudinal
measure. A common example is housing units that experienced a tenure switch (renter-to-owner or
owner-to-renter). In order to calculate a longitudinal measure, the AHS housing unit must have
complete data on both years of the survey. A review of two-survey longitudinal non-response rates over
the past few survey cycles shows that the two-survey non-response rate to be approximately 18%.
In order to ensure a national sample size of 40,000 housing units with complete data in two adjacent
survey years, the national sample size must be at least 48,780 housing units. After further consideration
of sample design issues and consultations with the Census Bureau, the national sample size was slightly
increased to 50,000.

4.3

Metropolitan Sample Size Precision Statement

As previously mentioned, the AHS metropolitan area samples sizes have fluctuated greatly since the
early 1980’s. They have ranged from 2,500 to 8,500. Moreover, there does not appear to have been a
precision statement developed for metropolitan areas.
After careful consideration of numerous options, HUD decided to adopt a precision statement similar to
the national precision statement:
For the metropolitan area as a whole, a two-year change of ten percent in median monthly
housing costs will have a standard error of five percent.

DRAFT
The notable difference between the national and metropolitan area precisions statements is that the
metropolitan area statement does not include the requirement to detect a ten percent change in
median monthly housing costs for five percent subgroups. As mentioned in section 2.2, the
metropolitan area sample sizes necessary to achieve the same statistical precision as the national
sample are not feasible, given budget constraints.
It must be noted that HUD considered developing a metropolitan area precision statement based on a
single-year cross-sectional estimate, as opposed to two survey year change estimate, as is the case with
the national sample precision statement. HUD considered this option mainly because the decision on
the metropolitan area selection strategy includes 15 metropolitan areas surveyed every survey cycle, 20
metropolitan areas surveyed every other survey cycle, and other metropolitan areas surveyed as budget
permits, with no target schedule. Given the varying schedules, HUD thought it could be imprudent to
develop a precision statement based on a two-year change estimate.

4.4

Metropolitan Sample Size Calculation

The calculation of the national sample size necessary to achieve the precision goal is detailed in
appendix 2. In short, the metropolitan area sample size was determined by developing a simulation
model based on American Community Survey (ACS) data from 2005, 2007, and 2009. The ACS was used
because it had consistent data in each year for metropolitan areas, as well as sample sizes large enough
from which to subsample, for the 40 largest metropolitan areas.
The simulation model first identified metropolitan areas that had a median monthly housing cost change
of 10 percent or greater between 2005/2007 or 2007/2009. These metropolitan areas were then subsampled at various sample sizes to determine the minimum sample size necessary to achieve a 5
percent standard error for the median monthly housing cost.
Metropolitan areas are different in their housing unit profile and their housing cost distribution. As
such, the sample size necessary to achieve the desired precision statement does vary among the
metropolitan areas. The results of the simulation showed that a sample size of 2,750 housing units was
large enough to achieve a standard error of 5 percent in all of the 40 largest metropolitan areas. The
final metropolitan area sample size was increased from 2,750 to 3,000 to account for two-survey
longitudinal non-response rate.

4.5

Integrating Top 15 Metropolitan Areas into the National Sample

HUD has determined that the 2015 AHS national sample will be integrated with an oversample of
housing units from each of the 15 largest metropolitan areas.

DRAFT
HUD has determined that to achieve their desired precision standards, the AHS must have a national
sample size of 50,000, and metropolitan area sample sizes of 3,000 for each of the top 15 metropolitan
areas. A simple way to achieve this is to add 45,000 housing units (15 x 3,000) to the national sample.
However, the national sample already has a significant number of housing units from the top 15
metropolitan areas, by virtue of it being a nationally-represented sample in a nation where one-third of
the housing units are in the top 15 metropolitan areas.
As such, the achieve HUD desired integrated national sample, the national sample needs to be
augmented with just enough housing units such that each of the top 15 metropolitan areas has 3,000
housing units. The number of cases necessary to achieve this goal is approximately 30,750.

4.6

Integrating HUD-assisted Oversample into the National Sample

For the 2011 and 2013 AHS’s, HUD included an oversample of HUD-assisted units from public housing
and various multifamily programs. The sample size was approximately 5,250 cases. For 2015, HUD will
continue this practice by including approximately 5,250 cases.
4.7

Final Integrated National Sample Size

As described in the prior sections, the AHS will be an integrated national sample. This includes a
nationally-representative sample of 50,000 housing units, supplemented with approximately 30,750
cases from the top 15 metropolitan areas and 5,250 cases from HUD-assisted housing units. The final
integrated national sample size will be approximately 86,000 housing units.

DRAFT
Appendix 1. Technical Details for the National Sample Size Determination
Forthcoming.

DRAFT
Appendix 2. Technical Details for the Metropolitan Sample Size Determination
Forthcoming.