Milk Production Methodology and Quality Measures

Milk Production Methodology and
Quality Measures
ISSN: 2167-1885

Released November 5, 2020, by the National Agricultural Statistics Service (NASS), Agricultural Statistics Board, United States Department
of Agriculture (USDA).

Scope and Purpose: The Milk Production Survey, conducted in all states in January, April, July, and October, collects
data for the number of all milk cows in the herd, number of cows milked on the first day of the survey month, and total
milk produced on the first day of the survey month. In addition, milk cow replacement prices are collected each quarter
and heifer replacement prices are collected in January. In April and October, producers provide data for the amount of
milk used on the farm for food or drink and the amount fed to calves.
Survey Timeline: The reference date for the Milk Production Survey is the 1st day of each quarter (January, April, July,
and October), with a data collection period of approximately 15 calendar days. Regional Field Offices may begin data
collection one day prior to the reference date. Data collection continues until a scheduled ending date and Regional Field
Offices have four to five business days to complete editing and analysis, execute the summary, and interpret the survey
results. The national milk production statistician must review the state estimates and prepare the official estimates for
release in about four business days. Milk Production, usually released around the 18th or 19th of each month, includes
monthly estimates of milk production, milk per cow, and number of milk cows for the 24 major States. Quarterly
estimates of milk production and number of milk cows are published for all states in the January, April, July, and October
Milk Production publications. Annual milk production revisions are published for all states in February.
Sampling: The target population for the Milk Production Survey is all agricultural establishments with one or more head
of milk cows on the land operated. The Milk Production Survey is conducted in every state.
NASS uses the list frame for the Milk Production Survey. The list frame includes all known agricultural establish ments. A
profile, known as control data, of each establishment is maintained on the list frame to allow NASS to define list frame
sampling populations for specific surveys and to employ efficient sampling designs. Only list frame records with positive
milk cow control data are included in the Milk Production list frame population.
The sample size for the Milk Production Survey is approximately 12,000, though some operations will be found to be out
of business or not have the item of interest. The sample is selected using a stratified sampling design with strata defined
by the total number of milk cows. The sample is first used in the January survey quarter or the “base” survey quarter,
which is the initial quarter of the survey year. The “follow-on” quarters in the survey year use the same “base”
sample. New milk operations found during the survey year can be added to the sample if the new operation qualifies for a
strata with a probability of selection equal to 1.
Data Collection and Editing: For consistency across modes, the paper version is considered the master questionnaire and
the Computer Assisted Telephone Interview (CATI), Computed Assisted Self Interview (CASI), and Mobile Computer
Assisted Personal Interview (mCAPI) instruments are built to model the paper instrument. Questionnaire content and
format are evaluated annually through a specifications process where requests for changes are evaluated and approved or
disapproved. Input may vary from question wording or formatting to a program change involving the deletion or
modification of current questions or addition of new ones. If there are significant changes to either the content or format
proposed, a NASS survey methodologist will pre-test the changes for usability. Prior to the start of data collection, all
modes of instruments are reviewed and the paper, mCAPI, CASI, and CATI instruments are thoroughly tested.
All federal data collections require approval by the Office of Management and Budget (OMB). NASS must document the
public need for the data, apply sound statistical practice, prove the data does not already exist elsewhere, and ensure the
public is not excessively burdened. The milk questionnaire must display an active OMB number that gives NASS the
authority to conduct the survey, a statement of the purpose of the survey, and the use of the data being collected. Also, a

response burden statement that gives an estimate of the time required to complete the form, a confidentiality statement
that the respondent’s information will only be used for statistical purposes in combination with other producers, and a
statement saying that response to the survey is voluntary and not required by law must be on the questionnaire.
Sampled farms and ranches receive a presurvey letter explaining the survey and that they will be contacted for survey
purposes only. The letter provides the questions to be asked to allow respondents to prepare in advance and also provides
a pass code they can use to complete the survey on the internet (CASI). All modes of data collection are utilized for the
Milk Production Survey. Regional Field Offices are given the option of conducting a mail out/mail back phase. While
mail is the least costly mode of collection, the short data collection period and the uncertainty of postal de livery times
limit its effectiveness. Most of the data are collected by CATI by individual Regional Field Offices and Data Collection
Centers. Limited personal interviewing is done, generally for large operations or those with special handling
arrangements. A program is run to determine if any sampled farms are in multiple on-going surveys so data collection can
be coordinated.
Survey Edit: As survey data are collected and captured, they are edited for consistency and reasonableness using
automated systems. Relationships between data items on the current survey are verified and in certain situations those
items may be compared to data from earlier surveys to make sure certain relationships are logical. The edit will determine
the status of each record to be either “dirty” or “clean”. Dirty records must be updated and reedited or certified by an
analyst to be clean. If updates are needed, they are reedited interactively. Only clean records are eligible for analysis and
summary.
Analysis Tools: Edited milk production data are processed through an interactive analysis tool which displays data for all
reports by item. The tool provides scatter plots, tables, charts, and special tabulations that allow the analyst to compare an
individual record to other similar records within their state. Outliers and unusual data relationships become evident and
Regional Field Office staff will review them to determine if they are correct. The tool also allows comparison to
previously reported data to detect large changes in the operation. Suspect data found to be in error are corrected, while
data found to be correct are kept.
Nonsampling Errors: Nonsampling errors are present in any survey process. These errors include reporting, recording,
editing, and imputation errors. Steps are taken to minimize the impact of these errors, such as questionnaire testing,
comprehensive interviewer training, validation and verification of processing systems, detailed computer edits, and the
analysis tool.
Estimators: Each milk cow farm and ranch in the sample has an initial sampling weight. This weight is the inverse of the
sampling fraction. For example, if a milk cow stratum has 1,000 farms in the population and 200 are sampled for this
survey, each sampled farm has a weight of 5. In other words, each sampled farm represents 5 farms.
Response to the Milk Production Survey is voluntary. Some producers refuse to participate in the survey. Others cannot
be contacted during the data collection period, and some submit incomplete reports. These non-respondents must be
accounted for if accurate estimates of milk are to be made. For the Milk Production Survey, non-respondents are
accounted for by adjusting the weights of the respondents. The adjustment occurs by stratum as the bounded strata
represent homogeneous groupings of similar sized milk cow farms. The largest stratum is unbounded and is made up of
operations with a large number of milk cows. Non-respondents in the unbounded stratum must be manually estimated by
Regional Field Office statisticians, and their stratum sampling weights are not adjusted. The adjustment is performed by
individual item on the questionnaire (total milk cows, cows milked, milk production) so adjustments for item nonresponse (partial reports) and unit non-response (refusal and inaccessible reports) are done in a single calculation. Using
the previous example, if 160 of the original 200 respond, the weights of the 160 will be adjusted to 1,000 divided by 160,
or 6.25.
Two estimators are used to compute direct measures of the milk items. The “reweighted” estimator and the “adjusted”
estimator are computationally identical except in how the non-response adjustments are made. The reweighted estimator
uses a global weight adjustment across all complete and estimated complete (usable) reports. The non-response weight
adjustment for the adjusted estimator uses an additional piece of information, based on the presence/absence of milk cows.
When a sampled farm refuses to cooperate, interviewers can probe to determine the presence of milk cows even though
2

Milk Production Methodology and Quality Measures (November 2020)
USDA, National Agricultural Statistics Service

the number of milk cows is not known. Also, automated edit logic that is based on historic and administrative data ensures
the coding of the presence indicator value. This presence/absence indicator is used in the weight adjustment.
Point estimates, called direct expansions, for both estimators are calculated by multiplying the reported value by the nonresponse weight and summing to a stratum total. A variance estimate is also computed at the stratum level. Totals and
variances are additive across strata to form a state estimate and states are additive to a national estimate.
Ratio estimates are also computed for many items. For example, milk yield is estimated as a ratio of milk production to
total milk cows. Ratio estimates use the reweighted estimator described above for the numerator and denominator, except
a report is not usable unless both items are reported.
Estimation: Indicators from the quarterly Milk Production Survey, along with external information (administrative data)
from various sources, provide data for estimating number of milk cows and milk production on a monthly and quarterly
basis. In order to be considered, these administrative data must be deemed to be reliable and come from unbiased sources.
The most common administrative data are Federal Milk Marketing Order statistics.
When all Milk Production Survey samples are accounted for, all responses fully edited, and the analysis material is
reviewed, each Regional Field Office executes a summary for their state. When all Regional Field Offices have run
summaries, Headquarters executes the national summary. Since all states conduct identical surveys, the samples can be
pooled and national survey results computed. The summary results provide point estimates and precision measures for
each item being estimated. It also provides information used to assess the performance of the current survey and evaluate
the quality of the survey estimates, such as strata level expansions, response rates, and percent of the expansion from
usable reports.
Regional Field Offices are responsible for performing a detailed review of their survey results. Any irregularities revealed
by the summary must be investigated and, if necessary, resolved. Using the historical relationship of the survey estimates
to the official estimate, Regional Field Offices must interpret the survey results and submit recommended estimates to
Headquarters for all milk cows, milk production, and milk per cow. The data are viewed in tabular and graphical form.
Regional Field Offices see their survey results only and do not have access to other states’ results.
The national estimates are determined using the “bottom-up” approach. After the Regional Field Offices submit the
recommended estimates for each state, the national milk production statistician reviews the individual state estimates for
reasonableness. When analyzing the state estimates, the national statistician has an advantage in being able to examine
results across states and compare the state recommendations. When the review of the individual state estimates is
complete, the national estimates are determined by summing the estimates for each state. The Agricultural Statistics Board
reviews the national estimates for reasonableness.
Milk production, milk per cow, and number of milk cows are subject to revision the following month after initial
publication for monthly states or the following quarter for the quarterly states. Normally, administrative data from Federal
Market Orders, State Departments of Agriculture, or other sources are the main basis for revisions. However,
administrative data for all states may not be available in time for these revisions. Estimates are again subject to revision in
February each year based on additional administrative data. In the event that additional changes are necessary, a third
revision is possible in February the following year.
Every five years, NASS conducts the Census of Agriculture, which is an exhaustive data collection effort for all known
farm operations across the United States. Estimates are thoroughly reviewed for possible revision after data from the
Census of Agriculture are available. The information gathered from the Census of Agriculture is used to establish
benchmark levels by which the survey estimators can be compared and bias determined. Survey based estimators can also
be impacted by outliers – individual reports that have excessive influence on the results due to either improper
classification or extremely unusual data for a given operation (i.e. operation is not representative of other operations).
NASS thoroughly reviews the survey data to identify these situations and consider their impact on the surve y results when
establishing the official estimates.

Milk Production Methodology and Quality Measures (November 2020)
USDA, National Agricultural Statistics Service

3

Quality Metrics for Milk Production
Purpose and Definitions: Under the guidance of the Statistical Policy Office of the Office of Management and Budget
(OMB), the United States Department of Agriculture’s National Agricultural Statistics Service (NASS) provides data
users with quality metrics for its published data series. The metrics tables below describe the performance data for all
surveys contributing to the publication. The accuracy of data products may be evaluated through sampling and nonsampling error. The measurement of error due to sampling in the current period is irrelevant for a fully enumerated data
series. Non-sampling error is evaluated by response rates and the percent of the estimate from reported data.
Sample size is the number of observations selected from the population to represent a characteristic of the
population. Operations that did not have the item of interest or were out of business at the time of data collection
have been excluded.
Response rate is the proportion of the above sample that completed the survey.
Percent of estimate from usable reports is a ratio of survey data expanded by the original sampling weight
compared to survey data expanded by the nonresponse adjusted weight.
Coefficient of variation provides a measure of the size for the standard error relative to the point estimate and is
used to measure the precision of the results of a survey estimator.
Milk Production Survey Sample Size and Response Rate: To assist in evaluating the performance of the estimates in
the quarterly milk production report, the sample size and response rate are displayed.

Quality Metrics for Milk Production Survey - United States: January 1, 2019-2020
Sample size

Response rate

2019

2020

2019

2020

(number)

(number)

(percent)

(percent)

United States ...............................................................................

10,682

10,798

54.2

55.3

Quality Metrics for Milk Production Survey - United States: January 1, 2019-2020
Class

Percent of estimate
from usable reports

Coefficient
of variation

2019

2020

2019

2020

(percent)

(percent)

(percent)

(percent)

Milk production .............................................................................

74.8

76.0

0.7

0.6

Milk cows ......................................................................................

73.6

75.2

0.7

0.6

4

Milk Production Methodology and Quality Measures (November 2020)
USDA, National Agricultural Statistics Service

Milk Production Survey Sample Size and Response Rate – States and United States:
January 1, 2019-2020
State

Sample size
2019

Response rate
2020

(number)

2019

(number)

2020

(percent)

(percent)

Alabama .......................................................................................
Alaska ..........................................................................................
Arizona .........................................................................................
Arkansas ......................................................................................
California ......................................................................................
Colorado ......................................................................................
Connecticut ..................................................................................
Delaware ......................................................................................
Florida ..........................................................................................
Georgia ........................................................................................

38
12
61
63
445
100
101
24
88
132

42
21
62
64
429
101
100
23
95
136

65.8
58.3
55.7
57.1
42.7
53.0
58.4
50.0
58.0
62.9

61.9
61.9
50.0
64.1
58.3
52.5
39.0
60.9
54.7
58.8

Hawaii ..........................................................................................
Idaho ............................................................................................
Illinois ...........................................................................................
Indiana .........................................................................................
Iowa ..............................................................................................
Kansas .........................................................................................
Kentucky ......................................................................................
Louisiana ......................................................................................
Maine ...........................................................................................
Maryland ......................................................................................

5
228
346
370
440
158
285
98
159
276

6
220
350
365
462
158
283
106
158
253

40.0
32.0
67.9
56.8
46.6
56.3
60.4
57.1
59.1
48.2

50.0
60.9
64.3
61.4
45.7
52.5
62.5
71.7
65.2
57.7

Massachusetts .............................................................................
Michigan .......................................................................................
Minnesota ....................................................................................
Mississippi ....................................................................................
Missouri ........................................................................................
Montana .......................................................................................
Nebraska ......................................................................................
Nevada .........................................................................................
New Hampshire ...........................................................................
New Jersey ..................................................................................

104
527
728
62
332
70
130
26
112
61

112
531
739
76
321
75
128
30
112
59

48.1
57.3
47.3
58.1
55.1
72.9
48.5
50.0
50.0
45.9

48.2
50.7
40.3
53.9
51.7
65.3
50.0
50.0
57.1
45.8

New Mexico .................................................................................
New York .....................................................................................
North Carolina .............................................................................
North Dakota ................................................................................
Ohio ..............................................................................................
Oklahoma .....................................................................................
Oregon .........................................................................................
Pennsylvania ...............................................................................
Rhode Island ................................................................................
South Carolina .............................................................................

103
927
132
74
570
111
132
534
17
50

108
966
138
74
591
109
142
534
12
46

67.0
50.9
73.5
56.8
56.5
52.3
51.5
49.6
41.2
76.0

71.3
49.6
67.4
50.0
49.7
54.1
61.3
57.3
33.3
78.3

South Dakota ...............................................................................
Tennessee ...................................................................................
Texas ...........................................................................................
Utah ..............................................................................................
Vermont .......................................................................................
Virginia .........................................................................................
Washington ..................................................................................
West Virginia ................................................................................
Wisconsin .....................................................................................
Wyoming ......................................................................................

142
173
289
141
319
256
246
69
799
17

148
157
293
143
309
258
238
67
830
18

59.9
69.4
54.3
56.7
52.0
61.3
49.6
81.2
56.8
58.8

56.1
68.8
51.2
65.0
58.9
68.2
63.9
70.1
56.6
66.7

United States ...............................................................................

10,682

10,798

54.2

55.3

Milk Production Methodology and Quality Measures (November 2020)
USDA, National Agricultural Statistics Service

5

Quality Metrics for Milk Production Survey - States and United States: January 1, 2019-2020
Milk production
State

Percent of estimate
from usable reports
2019

2020

(percent)

Milk cows
Coefficient
of variation

(percent)

Percent of estimate
from usable reports

2019

2020

2019

(percent)

(percent)

Coefficient
of variation

2020

(percent)

(percent)

2019

2020

(percent)

(percent)

Alabama .............
Alaska .................
Arizona ...............
Arkansas ............
California ............
Colorado .............
Connecticut ........
Delaware ............
Florida .................
Georgia ...............

75.7
95.3
91.9
74.5
69.8
97.5
90.3
84.8
88.9
89.3

75.8
95.4
83.8
85.8
75.4
88.2
82.8
87.2
87.8
91.9

17.2
3.3
1.5
5.3
2.0
0.5
1.5
6.9
2.2
1.7

6.5
2.9
2.2
3.5
1.7
0.9
2.4
3.9
1.6
1.3

73.6
94.0
91.7
73.8
68.6
97.4
88.8
83.0
86.4
87.7

76.4
90.7
84.0
79.2
75.0
88.5
79.6
85.6
86.3
90.1

12.2
3.3
1.3
4.9
1.9
0.5
1.6
7.3
3.6
1.8

4.9
4.0
1.7
6.9
1.6
0.9
2.5
2.7
1.2
1.3

Hawaii .................
Idaho ...................
Illinois ..................
Indiana ................
Iowa ....................
Kansas ...............
Kentucky .............
Louisiana ............
Maine ..................
Maryland .............

100.0
86.9
79.4
85.2
75.5
92.9
75.7
69.7
85.4
66.2

100.0
92.0
77.3
86.4
68.1
93.9
74.9
86.1
86.0
74.8

0.0
0.9
1.9
1.6
3.9
1.6
3.2
4.8
2.0
2.3

0.0
0.8
2.0
1.6
1.8
1.3
2.7
3.9
1.8
2.2

99.9
86.3
78.2
82.6
73.9
92.4
73.3
68.9
82.5
63.4

99.8
91.6
75.9
84.6
66.8
93.3
73.4
84.4
83.1
72.5

0.0
0.9
2.0
1.8
3.5
1.4
2.9
4.3
2.2
1.8

0.0
0.8
2.0
1.7
1.9
1.1
3.0
3.7
1.7
2.1

Massachusetts ...
Michigan .............
Minnesota ...........
Mississippi ..........
Missouri ..............
Montana .............
Nebraska ............
Nevada ...............
New Hampshire .
New Jersey ........

75.0
77.3
65.9
71.1
65.6
94.7
90.3
94.2
80.4
61.1

76.5
73.7
54.3
72.7
65.4
92.3
87.5
93.9
83.8
58.2

3.0
2.5
2.3
5.1
3.9
0.8
1.2
2.4
2.3
5.1

3.2
2.2
2.2
6.6
3.5
1.5
1.6
0.9
2.6
4.8

71.5
76.6
63.7
67.4
65.3
94.0
87.8
93.9
77.5
57.6

72.5
72.5
54.7
68.5
65.5
91.1
85.5
93.5
81.2
55.7

3.2
1.9
2.3
4.6
3.8
0.8
1.3
1.9
2.0
4.3

3.7
1.8
2.0
6.5
3.1
1.5
1.5
1.1
2.2
4.9

New Mexico ........
New York ............
North Carolina ....
North Dakota ......
Ohio ....................
Oklahoma ...........
Oregon ...............
Pennsylvania ......
Rhode Island ......
South Carolina ...

86.8
65.8
87.9
89.2
73.4
89.0
91.8
54.7
95.1
92.9

87.0
67.7
88.6
86.2
69.9
89.6
92.0
65.2
90.0
89.8

1.2
1.5
2.2
2.0
2.6
1.2
1.6
4.0
4.7
2.7

2.7
1.4
1.8
1.7
1.9
1.7
1.4
3.5
4.1
3.7

86.3
64.3
86.9
84.6
71.7
84.4
90.4
53.4
94.4
92.1

83.3
65.8
87.7
81.9
68.0
87.4
91.6
64.3
85.8
88.4

1.0
1.4
2.1
2.2
2.3
2.3
1.6
3.7
3.7
3.5

4.5
1.3
1.6
1.6
2.2
1.8
1.3
3.3
5.7
4.1

South Dakota .....
Tennessee .........
Texas ..................
Utah ....................
Vermont ..............
Virginia ...............
Washington ........
West Virginia ......
Wisconsin ...........
Wyoming ............

89.0
79.5
90.7
88.0
61.9
76.5
72.3
91.0
65.5
99.8

89.3
80.0
86.0
92.3
65.5
79.4
82.8
86.7
66.0
99.9

1.2
3.9
0.8
1.7
2.9
2.3
1.9
2.9
3.5
0.2

1.1
4.0
1.0
1.2
2.7
2.1
1.3
3.0
3.3
0.3

88.3
78.3
90.1
86.2
61.3
74.8
73.0
90.2
63.8
98.9

88.9
77.9
85.5
90.9
65.1
78.0
82.2
84.6
65.3
99.5

1.0
3.4
0.7
1.7
2.7
2.2
1.7
2.0
3.5
1.1

1.0
4.7
0.9
1.2
2.5
2.0
1.2
2.2
3.4
1.0

United States .....

74.8

76.0

0.7

0.6

73.6

75.2

0.7

0.6

6

Milk Production Methodology and Quality Measures (November 2020)
USDA, National Agricultural Statistics Service

Information Contacts
Process
Estimation ......................................
Data Collection ..............................
Questionnaires ..............................
Sampling and Editing ....................
Summary and Estimators .............
Dissemination ................................
Media Contact and Webmaster ....

Unit
Livestock Branch
Survey Administration Branch
Data Collection Branch
Sampling Editing and Imputation Methodology Branch
Summary Estimation and Disclosure Methodology Branch
Data Dissemination Office
Public Affairs Office

Telephone
(202) 720-3570
(202) 720-3895
(202) 720-6201
(202) 720-5805
(202) 720-4008
(202) 720-3869
(202) 720-2639

Email
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]

Access to NASS Reports
For your convenience, you may access NASS reports and products the following ways:
 All reports are available electronically, at no cost, on the NASS web site: www.nass.usda.gov
 Both national and state specific reports are available via a free e-mail subscription. To set-up this free
subscription, visit www.nass.usda.gov and click on “National” or “State” in upper right corner above “search”
box to create an account and select the reports you would like to receive.
 Cornell’s Mann Library has launched a new website housing NASS’s and other agency’s archived reports. The
new website, https://usda.library.cornell.edu. All email subscriptions containing reports will be sent from the
new website, https://usda.library.cornell.edu. To continue receiving the reports via e-mail, you will have to go to
the new website, create a new account and re-subscribe to the reports. If you need instructions to set up an
account or subscribe, they are located at: https://usda.library.cornell.edu/help. You should whitelist
[email protected] in your email client to avoid the emails going into spam/junk
folders.
For more information on NASS surveys and reports, call the NASS Agricultural Statistics Hotline at (800) 727-9540,
7:30 a.m. to 4:00 p.m. ET, or e-mail: [email protected].
The U.S. Department of Agriculture (USDA) prohibits discrimination against its customers, employees, and applicants for
employment on the basis of race, color, national origin, age, disability, sex, gender identity, religion, reprisal, and where
applicable, political beliefs, marital status, familial or parental status, sexual orientation, or all or part of an individual's
income is derived from any public assistance program, or protected genetic information in employment or in any program
or activity conducted or funded by the Department. (Not all prohibited bases will apply to all programs and/or
employment activities.)
If you wish to file a Civil Rights program complaint of discrimination, complete the USDA Program Discrimination
Complaint Form (PDF), found online at www.ascr.usda.gov/filing-program-discrimination-complaint-usda-customer, or
at any USDA office, or call (866) 632-9992 to request the form. You may also write a letter containing all of the
information requested in the form. Send your completed complaint form or letter to us by mail at U.S. Department of
Agriculture, Director, Office of Adjudication, 1400 Independence Avenue, S.W., Washington, D.C. 20250-9410, by fax
(202) 690-7442 or email at [email protected].