Att 9_Analysis Plan

Attachment 9: RANDS Web Survey Data Analysis Plan

General Questions Guiding Web Probe Analysis

Based on themes and patterns of interpretation identified in cognitive testing, we will develop close-ended probes that will be presented immediately following the relevant survey item (i.e., the “target item”). These probes, for example, might ask respondents to select all relevant factors they considered when deciding on an answer to the target item. Our analyses of the close-ended probes will, in general, be guided by four overarching questions:

1. What is the population distribution of each pattern of interpretation of the target item(s)? How prevalent are various in-scope and potentially out-of-scope interpretations in the population?

2. Are there demographic differences in patterns of interpretation of the target item(s)? That is, do frequencies of endorsement for each pattern of interpretation differ significantly across demographic groups? Demographic characteristics of interest include race, ethnicity, age, gender, sexual identity, disability status, educational attainment, and income.

3. Do patterns of interpretation differ according to how the respondent answered the target item? Because many probe questions will allow respondents to select all factors or patterns of interpretation that apply, we will also examine whether responses to the target item are associated with the total number of interpretation patterns endorsed on the probe.

4. Is there an interaction between the above two points, such that the association between target item response and pattern of interpretation varies by demographic subpopulation?

Below, we provide further details of our methodological approach and analysis plan by survey section. Items presented in red font have been newly added since the last round of RANDS.

• General Health Rating

(Note: Assign PHSTATA to respondents in split Sample “A”. Assign PHSTATB to respondents in split Sample “B”.)

PHSTATA		Would you say your health in general is excellent, very good, good, fair, or poor?
	1	Excellent
	2	Very good
	3	Good
	4	Fair
	5	Poor
	-7	[Don’t Know]
	-9	[Refused]

PHSTATB		Would you say your health in general is very good, good, fair, bad, or very bad?
	1	Very good
	2	Good
	3	Fair
	4	Bad
	5	Very bad
	-7	[Don’t Know]
	-9	[Refused]

METHOD:

A split sample experiment will be conducted in which a randomly selected half of the web sample will receive the standard HIS general health rating item and 5 response options [PHSTATA], while the other half of the sample will receive a similar question, but with balanced response options [PHSTATB]. Based on themes derived from cognitive testing, we will develop a series of close-ended probe questions, which will characterize the factors respondents considered when selecting a specific rating on the general health item. For example, we know from prior cognitive testing that some respondents consider diet and nutrition when answering the general health item. Rather than directly asking survey respondents if diet and nutrition were among the factors they considered when selecting a response, however, we will take a more nuanced and indirect approach. We will ask respondents to rate their agreement with statements such as “My diet is generally healthy”. Each of these statements will correspond with a major theme or factor identified in cognitive interviews. The same series of probe statements will be presented to both halves of the split sample, immediately following the general health rating item.

GOALS:

We will examine whether the two versions of the general health item function similarly, and whether responses to each version capture similar types of people, as indicated by answers to the follow-up probe statements.

• ANALYSIS PLAN:

1. Determine whether the number of people who select each response category is similar across the two versions of the general health item.

• Across the two split halves of the sample, we will compare the prevalence of individuals within each equivalently-ranked response category on the two versions of the item (e.g., compare prevalence within the top response category for each version: Excellent on PHSTATA versus Very good on PHSTATB).

• We will also compare prevalence within response categories that are equivalently-worded, but not equivalently-ranked (e.g., compare prevalence of Fair ratings across the two versions, despite different placement in the response lists –3^rd on PHSTATA and 4^th on PHSTATB).

2. Determine whether the same types of people select equivalently-ranked or equivalently-worded response options.

• We will examine whether respondents who rated their health in equivalently-ranked and equivalently-worded response categories also provide similar ratings of themselves along the dimensions assessed by the follow-up probe statements.

• To provide further information about validity, we will fit models of association between self-rated health, follow-up probe responses, and other questions on the survey assessing health conditions and behaviors. We will examine whether one version of the item is more strongly associated with health conditions and behaviors compared to the other, and whether this might differ depending on demographic characteristics.

• We will also examine patterns of correlation between ratings on the general health item and follow-up probe statements to determine which of the factors identified in cognitive testing most strongly predict how one responds to the general health item. We will examine these associations in the overall survey sample, and by demographic subpopulations.

• Health Insurance

The next few questions are about health insurance, including health insurance obtained through employment or purchased directly, as well as government programs like Medicare and Medicaid that provide Medical care or help pay medical bills.

FHICOV		Are you covered by any kind of health insurance or some other kind of health care plan?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

Skip: (If FHICOV=2, Skip to WRKCOR; Otherwise, continue)

HIKIND		Do you have any of the following kinds of health insurance or health care coverage? Include those plans that pay for only one type of service, such as nursing home care, accidents, or dental care. Exclude private plans that only provide extra cash while hospitalized. (Select all that apply)
	1	Private Health Insurance
	2	Medicare
	3	Medi-Gap
	4	Medicaid
	5	SCHIP (CHIP/Children's Health Insurance Program)
	6	Military health care (TRICARE/VA/CHAMP-VA)
	7	Indian Health Service
	8	State-sponsored health plan
	9	Other government program
	10	Single service plan (e.g., dental, vision, prescriptions)
	-7	[Don’t Know]
	-9	[Refused]

Skip: (If HIKIND inc. 1, continue; otherwise, skip to WRKCOR)

PLNMGD		What type of private plan do you have?
	1	HMO (Health Maintenance Organization)
	2	IPA (Individual Practice Plan)
	3	PPO (Preferred Provider Organization)
	4	POS (Point of Service)
	5	Fee-for-Service
	6	Indemnity
	7	Some other kind of plan
	-7	[Don’t Know]
	-9	[Refused]

MGCHMD		Under your private plan, can you choose any doctor or must you choose one from a specific group or list of doctors?
	1	Choose any doctor
	2	Choose from a group or list
	-7	[Don’t Know]
	-9	[Refused]

PCPREQ		Does this plan require you to have a primary care doctor who approves all your care?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

• METHOD, GOALS, AND ANALYSIS PLAN:

The health insurance items will be used by the statistics group for estimation models. Based on themes that arise in cognitive testing, we will also introduce probes to assess response error in HIKIND. We will test for subgroup differences in the likelihood of response error, including differences based on demographic characteristics, response to HIKIND, and their interaction.

• Work Status

The next questions are about the work you do.

WRKCOR		Which of the following were you doing last week?
	1	Working for pay at a job or business
	2	With a job or business but not at work
	3	Looking for work
	4	Working, but not for pay, at a family-owned job or business
	5	Not working at a job or business and not looking for work
	-7	[Don’t Know]
	-9	[Refused]

• METHOD, GOALS, AND ANALYSIS PLAN:

The work status item will be used as a covariate in estimation models by the statistics group.

• Hypertension and High Cholesterol

The next series of questions will ask you about certain medical conditions.

HYPEV		Have you ever been told by a doctor or other health professional that you had hypertension, also called high blood pressure?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

Skip: (If code 1 in HYPEV, Continue; Otherwise, Skip to CHLEV)

HYPMDEV2		Has a doctor ever prescribed any medicine for your high blood pressure?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

HYPMED2		Are you now taking any medicine prescribed by a doctor for your high blood pressure?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

CHLEV		Have you ever been told by a doctor or other health professional that you had high cholesterol?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

CHLYR		During the past 12 months, have you had high cholesterol?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

CHLMDEV2		Was any medication ever prescribed by a doctor to help lower your cholesterol?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

CHLMDNW2		Are you now taking any medicine prescribed by a doctor to help lower your cholesterol?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

• METHOD, GOALS, AND ANALYSIS PLAN:

Prior cognitive testing has suggested possible response error for questions about hypertension and cholesterol. If we uncover additional evidence of such problems in the upcoming round of cognitive testing, we will introduce close-ended probes immediately following the target item(s) to assess the prevalence of identified response errors. We will also test for subgroup differences in the likelihood of response error, including differences based on demographic characteristics, target item response, and their interaction. (Note: items that were added since the last round appear in red font)

REQUIRES ACTION:

We will review prior CI reports on hypertension and cholesterol items that have already been tested. This review will inform the current round of cognitive testing (e.g., by probing for details that weren’t fully developed in prior rounds of testing) and will further inform our analysis plan.

• Emphysema, COPD and Chronic Bronchitis

EPHEV		Have you ever been told by a doctor or other health professional that you have emphysema?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

COPDEV		Have you ever been told by a doctor or other health professional that you had chronic obstructive pulmonary disease, also called COPD?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

CBRCHYR		During the past 12 months, have you been told by a doctor or other health professional that you had chronic bronchitis?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

NEWLUNG

Have you ever been told by a doctor or other medical professional that you have Chronic Obstructive Pulmonary Disease or COPD, emphysema, or chronic bronchitis?

1	Yes
2	No
9	Don't know

• METHOD, GOALS, AND ANALYSIS PLAN:

Prior cognitive testing has suggested possible response error for items assessing emphysema [EPHEV], COPD [COPDEV], and chronic bronchitis [CBRCHYR]. A new question has been developed that combines all three conditions into one survey item [NEWLUNG, see page 25]. This new item will be presented at the end of the survey, followed by a probe [PROBE2] asking respondents to specify which condition they have (emphysema, COPD, bronchitis, or chronic bronchitis). We will compare responses to the items asking about emphysema, COPD, and chronic bronchitis separately with responses to the new combined item and follow-up probe. In addition to examining whether the new combined item provides similar prevalence estimates compared to the 3 separate items, we are interested in whether the combined item might reduce previously observed error associated with the chronic bronchitis question. In particular, we have observed that the stand-alone question could erroneously pick up respondents diagnosed with a single episode of bronchitis rather than chronic bronchitis.

• Asthma

AASMEV		Have you ever been told by a doctor or other health professional that you had asthma?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

Skip: (If code 1 AASMEV, Continue; Otherwise, Skip to DIBPRE1)

AASSTILL		Do you still have asthma?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

AASMYR		During the past 12 months have you had an episode of asthma, or an asthma attack?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

AASMERYR		During the past 12 months have you had to visit an emergency room or urgent care center because of asthma?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

• METHOD, GOALS, AND ANALYSIS PLAN:

The asthma items will be used in estimation models by the statistics group.

• Diabetes

DIBPRE1		Have you ever been told by a doctor or other health professional that you have any of the following: prediabetes, impaired fasting glucose, impaired glucose tolerance, borderline diabetes, or high blood sugar?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

Skip: (If code 1 in DIBPRE1, Skip to INSLN; Otherwise, Continue)

DIBEV		(If Respondent is FEMALE, display:) Other than during pregnancy, have you ever been told by a doctor or other health professional that you have diabetes or sugar diabetes? (If Respondent is MALE, display:) Have you ever been told by a doctor or other health professional that you have diabetes or sugar diabetes?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

Skip: (If DIBEV=1, continue; otherwise, skip to CBRCHRY)

DIBAGE		How old were you when a doctor or other health professional first told you that you had diabetes or sugar diabetes?
	1	(OPEN: 1-120)
	-7	[Don’t Know]
	-9	[Refused]

INSLN		Are you now taking insulin?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

DIBPILL		Are you now taking diabetic pills to lower your blood sugar? These are sometimes called oral agents or oral hypoglycemic agents.
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

• METHOD, GOALS, AND ANALYSIS PLAN:

Prior cognitive testing has suggested possible response error for items assessing diabetes and prediabetes. If we uncover additional evidence of such problems in the upcoming round of cognitive testing, we will introduce close-ended probes immediately following the target item(s) to assess the prevalence of identified response errors. We will also test for subgroup differences in the likelihood of response error, including differences based on demographic characteristics, target item response, and their interaction.

REQUIRES ACTION:

We will review prior CI reports involving the diabetes items. This review will inform the current round of cognitive testing (e.g., by probing for details that weren’t fully developed in prior rounds of testing) and will further inform our analysis plan.

• Pain

(Note: Assign CHPAIN6M, PAINLMT6, and PAIN_4 to respondents in split Sample “C”. Assign PAIN_2, PAINLMT3, and PAIN_4 to respondents in split Sample “D”)

CHPAIN6M		In the past six months, how often did you have pain? Would you say...
	1	Never
	2	Some days
	3	Most days
	4	Every day
	-7	[Don’t Know]
	-9	[Refused]

Skip (If CHPAIN6M=2-4, continue; otherwise, skip to RX12M)

PAINLMT6		Over the past six months, how often did pain limit your life or work activities? Would you say...
	1	Never
	2	Some days
	3	Most days
	4	Every day
	-7	[Don’t Know]
	-9	[Refused]

PAIN_4		Thinking about the last time you had pain, how much pain did you have? Would you say…
	1	A little
	2	A lot
	3	Somewhere in between a little and a lot
	-7	[Don’t Know]
	-9	[Refused]

(Note: Assign PAIN_2, PAINLMT3, and PAIN_4 to respondents in split Sample “D”)

PAIN_2		In the past 3 months, how often did you have pain? Would you say…
	1	Never
	2	Some Days
	3	Most Days
	4	Every Day
	-7	[Don’t Know]
	-9	[Refused]

Skip (If PAIN_2=2-4, continue; otherwise, skip to RX12M)

PAINLMT3		Over the past three months, how often did pain limit your life or work activities? Would you say...
	1	Never
	2	Some days
	3	Most days
	4	Every day
	-7	[Don’t Know]
	-9	[Refused]

PAIN_4		Thinking about the last time you had pain, how much pain did you have? Would you say…
	1	A little
	2	A lot
	3	Somewhere in between a little and a lot
	-7	[Don’t Know]
	-9	[Refused]

• METHOD:

A split sample experiment will be conducted in which a randomly selected half of the web sample will receive the pain frequency and limitation items based on a six month reference period [CHPAIN6M, PAINLMT6], while the other half of the sample will receive the items with a three month reference period [PAIN_2, PAINLMT6]. All respondents will receive an identical item assessing pain intensity [PAIN_4], and will be presented with an identical series of follow-up probe questions. The probe questions will be developed based on patterns of interpretation and potential response errors that arise in cognitive testing. We will be particularly interested in probing to explore whether the pain questions, and their various reference periods, adequately differentiate between respondents who suffer chronic pain versus those who experienced an episode of acute pain (e.g., a recent injury).

• GOALS:

Our goal is to examine whether the pain frequency and limitation items capture the same information, despite their differing reference periods. If the questions capture essentially identical information, then only one version of them would need to be included as part of HIS. If any significant differences are uncovered, then the pattern of findings might suggest which reference period is superior for a given purpose.

• ANALYSIS PLAN:

We will conduct between-person comparisons to examine whether pain frequency and limitation ratings differ among respondents who receive the questions with a 3 month reference period (PAIN_2, PAINLMT3) versus a 6 month reference period (CHPAIN6M, PAINLMT6). Since an identical series of follow-up probes will be presented to both halves of the split sample, we will test whether respondents endorse similar patterns of interpretation across the two reference periods, and whether the association between pattern of interpretation and pain question response is similar for the two versions.

The item assessing pain intensity (PAIN_4) does not have a specific reference period and will thus be presented to all respondents across both split halves. This will allow us to test whether the intensity of the last pain episode is associated with pain frequency and pain limitation ratings, and whether those associations might differ depending on reference period and pattern of interpretation endorsed in the follow-up probes.

By crossing pain frequency and intensity ratings, the resulting matrix can be used to identify a continuum of functioning. We will examine whether the resulting matrix scores are associated with pain limitations, and whether that association differs depending on reference period and pattern of interpretation endorsed in the follow-up probes. We are particularly interested in examining matrix scores together with answers to the follow-up probes to determine how well the items differentiate between those who suffer chronic versus acute pain.

• Opioid Pain Medication Use

RX12M		During the past 12 months, were you prescribed medication by a doctor or other health professional?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

OPIOID1		These next questions are about the use of prescription pain relievers called opioids. When answering these questions, please do not include over-the-counter pain relievers such as aspirin, Tylenol, Advil, or Aleve. During the past 12 months, have you taken any opioid pain relievers prescribed by a doctor or dentist? Examples include hydrocodone, Vicodin, Norco, Lortab, oxycodone, OxyContin, Percocet and Percodan.
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

OPIOID2		During the past 3 months, have you taken any opioid pain relievers prescribed by a doctor or dentist?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

Skip (If OPIOID2=1, continue; otherwise skip to SMKEV)

OPIOID3		During the past 3 months, how often did you take a prescription opioid? Would you say some days, most days, or every day?
	1	Some days
	2	Most days
	3	Every day
	-7	[Don’t Know]
	-9	[Refused]

• METHOD, GOALS, AND ANALYSIS PLAN:

Prior cognitive testing has suggested that questions about opioid pain medication use might function differently among those who suffer chronic pain versus those who do not. Moreover, testing has revealed comprehension difficulties vis-à-vis the term “opioid”. Using themes that spontaneously arise during cognitive testing, we will develop follow-up probe questions that characterize patterns of interpretation of the opioid pain medication items. We will use answers to the probe questions to explore whether the pain medication items function in a similar way and capture similar information among respondents who suffer from chronic pain versus those who do not. Moreover, we will use the probes to explore respondent comprehension of the term “opioid”, as well as the usefulness of the term as a framing tool among those with and without chronic pain.

• Smoking

These next questions are about cigarette smoking.

SMKEV		Have you smoked at least 100 cigarettes in your entire life?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

Skip: (If code 1 in SMKEV, Continue; Otherwise, Skip to SMKANY)

SMKNOW		How often do you now smoke cigarettes? Every day, some days, or not at all?
	1	Every day
	2	Some days
	3	Not at all
	-7	[Don’t Know]
	-9	[Refused]

Skip: (If code 3 in SMKNOW, Continue; If code 1 or 2 in SMKNOW, Skip to CIGQTRY; Otherwise, Skip to MODNO)

SMKQTNO		How long has it been since you quit smoking cigarettes?
	1	(OPEN: 1-120)
	-7	[Don’t Know]
	-9	[Refused]

Skip: (All in SMKQTNO_N/SMKQTNO_F, Skip to MODNO)

CIGQTYR		During the past 12 months, have you stopped smoking for more than one day because you were trying to quit smoking?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

(Skip: All in CIGQTRY, Skip to MODNO)

SMKANY		Have you ever smoked a cigarette even one time?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

(Note: show ECIGEV_AE to respondents in split sample “E”; show ECIGEV_AF and PROBE1 to respondents in split sample “F”)

ECIGEV_AE		The next question is about electronic cigarettes or e-cigarettes. You may also know them as vape-pens, hookah-pens, e-hookahs, or e-vaporizers. Some look like cigarettes, and others look like pens or small pipes. These are battery-powered, usually contain liquid nicotine, and produce vapor instead of smoke. Have you ever used an e-cigarette even one time?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

(Note: show ECIGEV_AF and PROBE1 to respondents in split sample “F”)

ECIGEV_AF		Have you ever used an e-cigarette even one time?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

PROBE1		When answering the previous question were you thinking of: (select all that apply)
	1	Regular e-cigarette
	2	vape pen
	3	hookah-pen
	4	e-hookah
	5	e-vaporizer
	6	regular cigarette
	7	marijuana cigarette
	8	cigar
	9	Other (please specify)

• METHOD, GOALS, AND ANALYSIS PLAN:

The smoking questions will be used mostly for estimation purposes by the statistics group. Note that a split sample experiment will test comprehension of the term “e-cigarette”. Prior cognitive testing has suggested the term is widely used and understood; thus, we suspect burden can be reduced by eliminating the long definition of e-cigarette from the question stem of ECIGEV_AE. We will test this by presenting a random half of the web sample with the original question (ECIGEV_AE) and the other half with a significantly shortened version (ECIGEV_AF) that assumes comprehension of the term “e-cigarette”. We will examine comparability of prevalence estimates generated by the two versions of the question and will test comprehension of the shortened version with a follow-up probe (PROBE1). Additionally, we will explore familiarity with common e-cigarette synonyms among the entire web sample a probe question presented at the end of the survey (see PROBE3, page 26).

• Physical Activity

The next questions are about physical activities (exercise, sports, physically active hobbies…) that you may do in your leisure time.

MODNO		How often do you do light or moderate leisure time physical activities for at least 10 minutes that cause only light sweating or a slight to moderate increase in breathing or heart rate?
	1	(OPEN: 1-365)
	-7	[Don’t Know]
	-9	[Refused]

Skip: (If code 0 or BLANK in MODNO, Skip to VIGNO; Otherwise, Continue)

MODLNGNO		About how long do you do these light or moderate leisure-time physical activities each time?
	1	(OPEN: 1-995)
	-7	[Don’t Know]
	-9	[Refused]

VIGNO		How often do you do vigorous leisure-time physical activities for at least 10 minutes that cause heavy sweating or large increases in breathing or heart rate?
	1	(OPEN: 1-365)
	-7	[Don’t Know]
	-9	[Refused]

Skip: (If code 0 or BLANK in VIGNO, Skip to STRNGNO; Otherwise, Continue)

VIGLNGNO		About how long do you do these vigorous leisure-time physical activities each time?
	1	(OPEN: 1-995)
	-7	[Don’t Know]
	-9	[Refused]

STRNGNO		How often do you do leisure time physical activities specifically designed to strengthen your muscles such as lifting weights or doing calisthenics?
	1	(OPEN: 1-365)
	-7	[Don’t Know]
	-9	[Refused]

• METHOD, GOALS, AND ANALYSIS PLAN:

The statistics group will use the physical activity items for estimation purposes.

• Alcohol

These next questions are about drinking alcoholic beverages. Included are liquor such as whiskey or gin, beer, wine, wine coolers, and any other type of alcoholic beverage.

ALC1YR		In any one year, have you had at least 12 drinks of any type of alcoholic beverage?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

Skip: (If code 1 in ALC1YR, Skip to ALC12MNO; Otherwise, Continue)

ALC5UPNO		(If code 2 in DEMO_GENDER, display:) In the past year, on how many days did you have 4 or more drinks of any alcoholic beverage? (If code 1 in DEMO_GENDER, display:) In the past year, on how many days did you have 5 or more drinks of any alcoholic beverage?
	1	(OPEN: 1-365)
	-7	[Don’t Know]
	-9	[Refused]

BINGE		(If code 2 in DEMO_GENDER, display:) Considering all types of alcoholic beverages, during the past 30 days, how many times did you have 4 or more drinks on an occasion? (If code 1 in DEMO_GENDER, display:) Considering all types of alcoholic beverages, during the past 30 days, how many times did you have 5 or more drinks on an occasion?
	1	(OPEN: 1-60)
	-7	[Don’t Know]
	-9	[Refused]

• METHOD, GOALS, AND ANALYSIS PLAN:

Prior cognitive testing has suggested possible response error for questions about alcohol consumption. Response error comes from two sources. First, we have observed wide variation in what respondents consider “a drink” (e.g., Is a 24 oz beer one or 2 drinks? Is a self-poured drink .5 or 2 drinks? How many drinks is a pint of liquor?). Second, respondents have difficulty recalling drinking activity across lengthy periods of time (e.g., an entire year in ALC5UPNO) and may provide answers that, at best, reflect imprecise guesses. These response errors are particularly problematic to the extent that they vary systematically across meaningful subpopulations. If we uncover additional evidence of such problems in the upcoming round of cognitive testing, we will introduce close-ended probes immediately following the target item(s) to assess the prevalence of identified response errors. We will also test for subgroup differences in the likelihood of response error, including differences based on demographic characteristics, target item response, and their interaction.

• Sleep

ACISLEEP		On average, how many hours of sleep do you get in a 24-hour period?
	1	(OPEN: 1-24)
	-7	[Don’t Know]
	-9	[Refused]

ACISLPFL		In the past week, how many times did you have trouble falling asleep?
	1	(OPEN: 0-7)
	-7	[Don’t Know]
	-9	[Refused]

ACISLPST		In the past week, how many times did you have trouble staying asleep?
	1	(OPEN: 0-7)
	-7	[Don’t Know]
	-9	[Refused]

ACISLPMD		In the past week, how many times did you take medication to help you fall asleep or stay asleep?
	1	(OPEN: 0-7)
	-7	[Don’t Know]
	-9	[Refused]

ACIREST		In the past week, on how many days did you wake up feeling well rested?
	1	(OPEN: 0-7)
	-7	[Don’t Know]
	-9	[Refused]

• METHOD, GOALS, AND ANALYSIS PLAN:

The sleep questions will be used mostly for estimation purposes by the statistics group. If significant response error is identified during cognitive testing, we will introduce close-ended probes immediately following the target item(s) to assess the prevalence of identified response errors. We will also test for subgroup differences in the likelihood of response error, including differences based on demographic characteristics, target item response, and their interaction.

REQUIRES ACTION:

We will review prior CI reports involving sleep items. This review will inform the current round of cognitive testing (e.g., by probing for details that weren’t fully developed in prior rounds of testing) and will further inform our analysis plan.

• Psychological Distress

During the past 30 days, how often did you feel…

ACISAD		So sad that nothing could cheer you up?
	1	All of the time
	2	Most of the time
	3	Some of the time
	4	A little of the time
	5	None of the time
	-7	[Don’t Know]
	-9	[Refused]

ACINERV		Nervous?
	1	All of the time
	2	Most of the time
	3	Some of the time
	4	A little of the time
	5	None of the time
	-7	[Don’t Know]
	-9	[Refused]

ACIRSTLS		Restless or fidgety?
	1	All of the time
	2	Most of the time
	3	Some of the time
	4	A little of the time
	5	None of the time
	-7	[Don’t Know]
	-9	[Refused]

ACIHOPLS		Hopeless?
	1	All of the time
	2	Most of the time
	3	Some of the time
	4	A little of the time
	5	None of the time
	-7	[Don’t Know]
	-9	[Refused]

ACIEFFRT		That everything was an effort?
	1	All of the time
	2	Most of the time
	3	Some of the time
	4	A little of the time
	5	None of the time
	-7	[Don’t Know]
	-9	[Refused]

ACIWTHLS		Worthless?
	1	All of the time
	2	Most of the time
	3	Some of the time
	4	A little of the time
	5	None of the time
	-7	[Don’t Know]
	-9	[Refused]

ANX_1		How often do you feel worried, nervous, or anxious? Would you say…
	1	Daily
	2	Weekly
	3	Monthly
	4	A few times a year
	5	Never
	-7	[Don’t Know]
	-9	[Refused]

ANX_2		Do you take medication for these feelings?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

Skip: (If ANX_1=5 AND ANX_2=2, skip to DEP_1; otherwise, continue)

ANX_3		Thinking about the last time you felt worried, nervous, or anxious, how would you describe the level of these feelings? Would you say…
	1	A little
	2	A lot
	3	Somewhere in between a little and a lot
	-7	[Don’t Know]
	-9	[Refused]

DEP_1		How often do you feel depressed? Would you say…
	1	Daily
	2	Weekly
	3	Monthly
	4	A few times a year
	5	Never
	-7	[Don’t Know]
	-9	[Refused]

DEP_2		Do you take medication for depression?
	1	Yes
	2	No
	-7	[Don’t Know]
	-9	[Refused]

Skip: (If DEP_2=1,-7,-9, continue; if DEP_2=2, DEP_1=5, AND CHPAIN6M=2-4,-7,-9, skip to PAIN_2; otherwise, skip to TIRED_1 )

DEP_3		Thinking about the last time you felt depressed, how depressed did you feel? Would you say…
	1	A little
	2	A lot
	3	Somewhere in between a little and a lot
	-7	[Don’t Know]
	-9	[Refused]

Skip: (If CHPAIN6M=2-4,-7,-9, continue; otherwise skip to TIRED_1)

• METHOD:

The first 6 questions presented above (ACISAD – ACIWTHLS) make up the Kessler-6 (K6), a measure of nonspecific psychological distress administered as part of the adult HIS questionnaire. The 2019 HIS redesign introduces rotating core questions on psychological distress: the K6 will be rotated with a combination of two measures that assess symptoms of anxiety (GAD-7) and depression (PHQ-8). All three measures have recommended cut points for identifying mental health problems severe enough to cause impairment in functioning and to warrant treatment.

Additionally, items developed by The Washington Group (WG) to measure the frequency and intensity of depressed and anxious affect will appear as part of the HIS annual (non-rotating) adult core. These items, labeled ANX_1 through DEP_3 above, were developed to identify persons at risk of experiencing limited or restricted participation in society due to affect difficulties.

All web survey respondents will answer the questions composing the K6, as well as the WG questions assessing anxiety and depression. These items have a history of use on the HIS and most were included as part of the web survey in prior rounds of RANDS. Due to response burden considerations, the PHQ-8 and GAD-7 will not be answered by the full web sample. Instead, each measure will be presented to a randomly selected half of the sample. This will allow us to use the measures as described below while simultaneously minimizing burden.

The items composing the PHQ-8 and GAD-7 can be found under the heading “Anxiety and Depression (Split Sample)” on pages 23-25). We chose to separate these scales from the K6 and WG items to reduce the likelihood of confusion resulting from differing reference periods (i.e., past 30 days for K6, past 2 weeks for PHQ-8 and GAD-7).

• GOALS:

In addition to creating probes to assess response error specific to any item that is found to be problematic during cognitive testing, we will use a combination of probes and quantitative comparisons to characterize the reliability, validity, and cognitive response processes associated with each measure of psychological distress.

• ANALYSIS PLAN:

• To identify psychological distress that is severe enough to cause impairment in functioning, one goal is to minimize false positives associated with “normal” psychological reactions (e.g., anxiety about an upcoming job interview or grief following the death of a loved one). Using language that spontaneously arises during cognitive interviews, we will develop a probe that assesses whether respondents were thinking about these “normal” emotional responses when completing each of the psychological distress measures. We will then compare scores on each measure with responses to this probe to help characterize the validity of the measures.

• Using language and themes that arise during cognitive testing, we will also develop an “impact on functioning” follow-up probe (similar to PHQImp and GADImp). Impact on functioning should be greater with more severe symptomatology, so we will use these ratings to validate that the psychological distress measures are doing what they were intended to do.

• While all of the psychological distress measures assess the frequency of experiencing symptoms (albeit with differing reference periods), only the Washington Group measure includes an assessment of symptom intensity. By crossing symptom frequency and intensity, the resulting matrix can be used to identify a continuum of functioning. We will use descriptive statistics generated from the other psychological distress measures (K6, PHQ-8, GAD-7) and responses to probe questions to develop a detailed characterization of respondents who fall within each cell of the matrix. This descriptive information can then be used to inform decisions about the development of cut points for the Washington Group psychological distress items.

• We can also conduct Latent Class Analysis using cut-off scores from each measure as indicators of a “Serious Psychological Distress” latent variable and thereby estimate error rates for each measure.

• In prior cognitive testing of the Washington Group item on depression frequency (DEP_1: How often do you feel depressed?), respondents mentioned sleep problems as a primary symptom of depression (i.e., sleeping too much, always feeling tired, wanting to stay in bed all day). We will explore ways to combine information from the Washington Group fatigue and depression items to better identify those with serious psychological distress.

• Washington Group Fatigue Items

TIRED_1		In the past 3 months, how often did you feel very tired or exhausted? Would you say…
	1	Never
	2	Some Days
	3	Most Days
	4	Every Day
	-7	[Don’t Know]
	-9	[Refused]

Skip (If TIRED_1=1, skip to PHQ1 (split sample “G”) or GAD1 (split sample “H”); otherwise continue)

TIRED_2		Thinking about the last time you felt very tired or exhausted, how long did it last? Would you say…
	1	Some of the day
	2	Most of the day
	3	All of the day
	-7	[Don’t Know]
	-9	[Refused]

TIRED_3		Thinking about the last time you felt this way, how would you describe the level of tiredness? Would you say…
	1	A little
	2	A lot
	3	Somewhere in between a little and a lot
	-7	[Don’t Know]
	-9	[Refused]

• METHOD, GOALS, AND ANALYSIS PLAN:

The fatigue questions will be used mostly for estimation purposes by the statistics group. If significant response error is identified during cognitive testing, we will introduce close-ended probes immediately following the target item(s) to assess the prevalence of identified response errors. We will also test for subgroup differences in the likelihood of response error, including differences based on demographic characteristics, target item response, and their interaction.

• Anxiety and Depression (Split Sample)

(Note: assign PHQ1-PHQImp to split sample “G”; assign GAD1-GADImp to split sample “H”)

Over the last 2 weeks, how often have you been bothered by any of the following problems?

PHQ1		Little interest or pleasure in doing things
	1	Not at all
	2	Several days
	3	More than half the days
	4	Nearly every day

PHQ2		Feeling down, depressed, or hopeless
	1	Not at all
	2	Several days
	3	More than half the days
	4	Nearly every day

PHQ3		Trouble falling or staying asleep, or sleeping too much
	1	Not at all
	2	Several days
	3	More than half the days
	4	Nearly every day

PHQ4		Feeling tired or having little energy
	1	Not at all
	2	Several days
	3	More than half the days
	4	Nearly every day

PHQ5		Poor appetite or overeating
	1	Not at all
	2	Several days
	3	More than half the days
	4	Nearly every day

PHQ6		Feeling bad about yourself — or that you are a failure or have let yourself or your family down
	1	Not at all
	2	Several days
	3	More than half the days
	4	Nearly every day

PHQ7		Trouble concentrating on things, such as reading the newspaper or watching television
	1	Not at all
	2	Several days
	3	More than half the days
	4	Nearly every day

PHQ8		Moving or speaking so slowly that other people could have noticed? Or the opposite — being so fidgety or restless that you have been moving around a lot more than usual
	1	Not at all
	2	Several days
	3	More than half the days
	4	Nearly every day

Skip: (If any of PHQ1-PHQ8 = 2,3,4, continue; otherwise skip to NEWLUNG)

PHQImp		How difficult have these problems made it for you to do your work, take care of things at home, or get along with other people?
	1	Not at all difficult
	2	Somewhat difficult
	3	Very difficult
	4	Extremely difficult

(Note: assign GAD1-GADImp to split sample “H”)

Over the last 2 weeks, how often have you been bothered by the following problems?

GAD1		Feeling nervous, anxious or on edge
	1	Not at all
	2	Several days
	3	More than half the days
	4	Nearly every day

GAD2		Not being able to stop or control worrying
	1	Not at all
	2	Several days
	3	More than half the days
	4	Nearly every day

GAD3		Worrying too much about different things
	1	Not at all
	2	Several days
	3	More than half the days
	4	Nearly every day

GAD4		Trouble relaxing
	1	Not at all
	2	Several days
	3	More than half the days
	4	Nearly every day

GAD5		Being so restless that it is hard to sit still
	1	Not at all
	2	Several days
	3	More than half the days
	4	Nearly every day

GAD6		Becoming easily annoyed or irritable
	1	Not at all
	2	Several days
	3	More than half the days
	4	Nearly every day

GAD7		Feeling afraid as if something awful might happen
	1	Not at all
	2	Several days
	3	More than half the days
	4	Nearly every day

Skip: (If any of GAD1-GAD7 = 2,3,4, continue; otherwise skip to NEWLUNG)

GADImp		If you checked off any problems, how difficult have these problems made it for you to do your work, take care of things at home, or get along with other people?
	1	Not difficult at all
	2	Somewhat difficult
	3	Very difficult
	4	Extremely difficult