NCHS Survey Feedback 2020

Feedback on the National ALS Registry Surveys

General feedback:

• The instrument seems outdated and a bit clunky. The font is quite small, difficult to read and is not visually very appealing. I’m not sure if this was due to my internet connection, browser, etc. but loading the surveys often took a long time and on several occasions, I had to log off and log back in, often more than once, in order to load the next survey.

• The way that skips are handled in the instrument could be improved. It was not always clear whether one is being skipped through a field or not. On several occasions, I found myself trying to enter data into a field and then realized that I was supposed to skip it. In general, it would be best if only questions that the respondent is intended to answer are displayed.

• It is a bit overwhelming to see 18 separate surveys. Even for motivated respondents, who are interested in providing data to help with research, it may be off-putting. Some of the surveys, as noted, are quite brief but not all of them are. It would help to give respondents a clearer sense of the topics covered and possibly an estimated amount of time required for each. Consider also combining some topics such as military service with occupational history.

• Along the same vein, it seems like the instrument for several of the surveys could be made much more efficient by first asking screener questions and then only asking follow-up questions for those who report any use, exposure, etc.

• The quality of the data from the questions that ask about consumption (caffeine, alcohol, etc.), hobbies, exposure, etc. over one’s lifetime is probably questionable. It can be very difficult and burdensome to recall this type of information over such a long reference period and come up with accurate estimates, particularly when consumption, exposure, etc. has been highly variable over the course of one’s life or is highly variable within the year (seasonal), etc. It would be useful to do some cognitive interviews to see how people go about trying to answer these questions. This would help us understand better what level of accuracy can be expected, possible changes to the questions that would be more in-line with the recall strategies respondents tend to use and the level of effort they are willing to expend.

Errata, specific comments/questions:

Survey 1

7. Do you consider yourself Spanish, Hispanic, or Latino/Latina?

8. What do you consider to be your race or ethnic group? If you belong to more than one of these groups, please indicate all groups that apply to you.

Consider using race/ethnicity questions used by the Census/ACS?

What was your weight at 40 years? – Consider asking for approximate weight? Depending on the age of the respondent, this may be difficult to recall with precision.

“You’ve successfully submitted Survey.”  “the survey”? “survey x”?

Survey 2

Job title/ Industry/Industry sub-categories – Where does the list of job titles come from? How often is it updated?

For how many years were you employed in this occupation? Do people typically provide rounded answers? Is the data categorized for analysis anyway? Consider offering closed-ended categories.

How well are the job/industry/occupation questions working? How common is “other” and item nonresponse to this item? Is there a high degree of response to the job title (specify)? What is done with the open-ended responses? Other major government surveys (American Community Survey, Current Population Survey, etc.) use a more open-ended approach to job/occupation and then code the data on the back end. Presumably they get higher quality and more nuanced information but this approach is more labor intensive for the analysts that have to process the data – although much of this is probably automated now. If job/occupation information is important for ALS research, it may be worthwhile to examine these questions and the quality of the resulting data in more detail.

Survey 3

Questions 2, 3, 4 should only appear for those who answer Yes to 1 – or these items should offer a “Not applicable” response.

Were you ever deployed to a war arena?  The term “war arena” seems strange to me, perhaps combat zone? Or maybe the phrase “war arena” would be understood by those in the military?

Survey 4

During periods when you smoked, for how many years in total did you smoke cigarettes?

How do you pick up occasional or social smokers or are you primarily interested in those who have smoked regularly?

Don’t know – does it make sense to offer this as an option?

During periods when you were drinking alcoholic beverages, for how many years in total did you drink alcoholic beverages?

This may be very burdensome and challenging for some people to answer – particularly those whose consumption habits have varied and or changed a great deal over time. E.g., women factoring in periods of decreased alcohol consumption when trying to conceive, when pregnant, nursing, etc.

Survey 5

Have you ever engaged in a routine that includes vigorous leisure-time physical activity for at least 10 minutes a day that caused heavy sweating or large increases in breathing or heart rate? Please indicate the number of times per week, month or year that you engaged in vigorous activity for at least 10 minutes A DAY? for each age period (up to your current age period).

ON AVERAGE?

Does it make sense to ask in 10 year intervals? What if the pattern was very different for several years within a ten-year interval?

Survey 6

4. Offer not applicable?

Survey 7

2. Is it not possible to select more than one? Is it always only one part of the body where weakness is first felt?

Do people know what “advanced directives” are?

Disease Progression

1. Have you noticed any changes in your speech? Seems that the response options 2, 3 and 4 shown here are not necessarily mutually exclusive.

I have a noticeable speech difference

My speech has changed. I am asked often to repeat words or phrases.

My speech has changed. I sometimes need the use of alternative communication methods (e.g. computer, writing pad, letter board or eye chart).

I am unable to communicate verbally.

3. Have there been any changes in your ability to swallow?

I am unable to eat all consistencies of food and have modified the consistency of foods eaten

SHOULD THIS BE “AND HAVE TO…” OR “NEED TO MODIFY THE CONSISTENCY OF SOME(?) OR ALL FOODS”?

Seems like 2, 3, and 4 may not be mutually exclusive yet it is only possible to select one.

The skip pattern could be made more clear from question 5 to question 6 or 7.

Currently, depending on how you answer 5, the response options are just grayed out for the question that is not applicable.

6 and 7 are not questions and change from second person to first person.

7. Using a feeding tube (PEG).

I provide minimal assistance to a caregiver, WHO ASSISTS ME WITH A FEEDING TUBE (PEG)

I am unable to perform any of the manipulations, NEEDED TO USE A FEEDING TUBE (PEG)

9. Has your ability to turn in bed and adjust the bed clothes (e.g. cover yourself with the sheet or blanket) changed?

I am helpless in bed.  Maybe should be “I am unable to turn in bed or adjust the bed clothes”?

12. Do you experience shortness of breath or have difficulty breathing?

I have significant shortness of breath (all of the time) and [AM?] considering using mechanical ventilation.

What about those who are currently using mechanical ventilation?

13. Do you experience shortness of breath or have difficulty breathing while lying down on your back?

I occasional have shortness of breath while lying on MY back but don’t routinely use more that two (2) pillows to sleep. – SHOULD BE “I HAVE OCCASIONAL SHORTNESS OF BREATH WHILE LYING ON MY BACK BUT DON’T ROUTINELY USE MORE THAN TWO (2) PILLOWS…”

Survey 8

1.Over your lifetime (at least 100 days or more), have you ever had a JOB where you handled HERBICIDES (to kill weeds)?

Shouldn’t the “at least 100 days or more” go at the end?

Seems like this could be made much more efficient by first screening people for use of any of the substances.

Survey 9

Gardening

During the period when you did gardening, how many hours each month did you perform the activity? Hard to answer because it’s very seasonal in nature. – so is hunting.

Knitting – how many years did you NOT do this hobby?

This was also very hard to answer. Knitting for me has been very sporadic.

How many hours per month? Should maybe specify on average? Very difficult to recall if was a long time ago.

Survey 10

Do people know what insecticides in the home are? What counts and what doesn’t? Raid? Terra traps?

What about insecticides in the lawn or garden?

Herbicides or weed killers?

Fungicides?

Is the thinking that people who use these products know what they are, would know what counts and what doesn’t for each?

Survey 13

Espresso drinks – This could be really hard to estimate for people whose level of consumption has varied greatly over the course of their life.

Survey 14

1. Have you ever had an injury to your head or neck?

3. Have you ever injured your head or neck in a fall or from being hit by something (e.g., falling from a bike, horse, or rollerblades, falling on ice, being hit by a rock)? Have you ever injured your head or neck playing sports or on the playground?

4. Have you ever injured your head or neck in a fight, from being hit by someone, or from being shaken violently? Have you ever been shot in the head?

I would ask the more specific questions first followed by a more general one to pick up head injuries due to any other causes. E.g., “Have you ever had an injury to your head or neck caused by anything else?” Otherwise, respondents may respond yes to the first question thinking of an instance that is asked about specifically in the subsequent questions.

Both question 3 and 4 are asking more than one question. They should be broken down into separate questions.