Supporting Statement B

Supporting Statement B

National Survey of Children's Health: Research to Support the Development of Mail and Internet Questionnaires with Measurement Equivalence (NSCH Redesign Project)

OMB Control No. 0915-0379

B. Collection of Information Employing Statistical Methods

1. Respondent universe and sampling methods

The respondent universe for both the Qualitative Assessment and the Mode Effects experiment is adults age 18 or older who live in the U.S. and who are parents or guardians of at least one child aged 0-17 years living in the same household. Health care providers who will be identified by a proportion of the respondents will also be contacted as part of this study.

The target sample for the Qualitative Assessment is 120 adults. Of the 120 interviews, approximately half will be conducted specifically with parents or guardians of children with special health care needs (CSHCN). Participants will be randomly assigned to evaluate either the mail-mode (hard-copy) self-administered questionnaire (SAQ) or the web-mode (online) version of the survey. Participants will be recruited through ads posted to online message boards (e.g., Craig’s List), non-profit websites (e.g., Autism Society of Illinois; Epilepsy Foundation), and local newspapers, as well as through flyers posted in public areas (e.g., shopping malls, telephone poles, community boards, bus stops). Respondents for this component will not be selected through a random process, but rather are selected for specific characteristics such as their child’s health status or age. For the purposes of this study a randomly drawn sample is not required to assess the reliability, validity, and clarity of the survey instruments. We proposed using 20 cognitive interviews per group because research shows 20 interviews are sufficient for validation (Blair et al 2006)¹.

Qualitative Sample

Target Population	Age Group	# of Respondents
Non-CSHCN	Aged 0-5	20 respondents
Non-CSHCN	Aged 6-11	20 respondents
Non-CSHCN	Aged 12-17	20 respondents
CSHCN	Aged 0-5	20 respondents
CSHCN	Aged 6-11	20 respondents
CSHCN	Aged 12-17	20 respondents
TOTAL		120 respondents

Individuals who contact the study will be administered a brief screener over the phone to confirm eligibility and collect demographic and special-needs status information. Individuals who are screened in as eligible will be scheduled for a follow-up in-person or phone interview.

The target sample for the Mode Effects experiment is approximately 37,000 unique households, which is expected to yield 2,250 screener completes per mode (mail, web, phone). The sample will be drawn from a list obtained from a sample vendor (Marketing Systems Groups). The list will come from a national sample of households that (1) have a valid mailing address, (2) are flagged as likely having a child, and (3) have a valid telephone number. Participants from this sample will be randomly assigned to one of three mode treatments: mail, web, or telephone. No special sampling procedures other than the procedure to oversample subpopulations (when indicated) are required.

We expect to obtain approximately 6,750 child-level screener completes (~2,250 per mode) and approximately 2,526 child-level topical interview completes (~842 per mode).  The size of the overall sample was determined based on the amount of child-level screener completes that would be needed in each mode to detect differences of plus or minus three (3) percentage points between a given mode and the other two modes for four key survey variables (special needs prevalence; health insurance status; asthma prevalence rate; autism prevalence rate), and to detect differences in completion rates between the long and short versions of the questionnaires. 

Mode Effects Sample

Mode	Sample Size (# of Households Contacted)	Expected number of Completes
		Screener	Topical
Mail	6,012	2,250	842
Web	18,606	2,250	842
Phone	9,066	2,250	842
TOTAL	33,684	6,750	2,526

2. Information Collection Procedures

Data for the Qualitative Assessment will be collected via one-on-one, in-depth interviews with participants. The Qualitative Assessment interviews for the mail- and web-mode versions of the survey will consist of both a cognitive interview and usability testing component. Each interview will last approximately one hour, and will be conducted either in-person at NORC’s Chicago office, or by phone from NORC’s call center in Chicago, IL (approximately 80 percent of the total interviews will be conducted in-person (n=96); the remaining interviews will be conducted by phone (n=24)). Interviews are being conducted by phone to allow for a more diverse sample (i.e., to allow recruitment of respondents who live outside of the Chicago area and/or who are unable to travel to the interviewing location). Spanish-speaking interviewers will be available to conduct the screener and Qualitative Assessment interview with Spanish-speaking respondents.

The cognitive interview portion will consist of respondents completing the screener and topical questionnaire items. Additionally, interviewers will ask questions in order to ascertain the respondent’s understanding of terms; how confident the respondent is in his/her response; how he/she remembered the information provided in factual questions; whether the respondent found a response choice that fit his/her answer; how easy or difficult it was to answer a question, and whether there were any issues with sensitive questions. The usability testing portion will look at the time taken to answer a question, whether respondents change their answer, and whether respondents make mistakes in navigation or usage of the survey. With the consent of respondents, interviews will be recorded to allow for review of answers post-interview.

The Qualitative Assessment will be combined with predictive and criterion-based validation tests to validate household screener items as well as items related to medical diagnosis and health insurance status from the topical questionnaire.

To assess predictive validity, a test-retest approach will be used, whereby respondents will be re-administered items from the household screener and topical questionnaire two weeks subsequent to their initial Qualitative Assessment interview. Consent to contact the respondent for follow-up will be obtained at the initial interview. The follow-up interview will be conducted by phone and is expected to last approximately 20 minutes. The original and subsequent responses will be compared to assess the reliability of the instrument items and provide measures of predictive validation.

A separate, criterion-based validity test will also be conducted, which will validate questionnaire items using documentation provided by the respondent or by a health care provider. Respondents will be asked to bring documentation of their child’s medical condition(s) as well as health insurance status to their scheduled Qualitative Assessment interview. In cases where documentation is not provided, respondents will be asked to provide permission for study staff to contact the child’s health care provider to obtain the appropriate documentation. Upon receiving consent, providers will be contacted and asked to provide information on the child’s health conditions to allow for comparisons with respondent-reported data.

Approximately half of the Qualitative Assessment sample will be assigned to the predictive validation testing condition, while the other half of the sample will be assigned to the criterion-based condition. It is expected that roughly half of respondents from the criterion-based condition will require follow-up with a health care provider. The table below outlines the approximate cell size by validation testing condition/CSHCN status.

Approximate Cell Size by Validation Testing Condition/CSHCN Status

Validity Testing Condition	Non-CSHCN	CSHCN	Total
Predictive (Test-Retest)	~30	~30
Criterion-Based (Documentation/Provider Check)	~30	~30
Total	~60	~60	~120

All interviewers will be trained on the cognitive interview/usability testing protocol and will have undergone human subjects protections training and have signed a confidentiality agreement prior to conducting interviews. With the respondent’s permission, interviews will be recorded to allow for review and validation of responses.

For the Mode Effects Experiment, data will be collected from respondents via a mail, web, or phone survey. Approximately 33,700 addresses will be selected and contact will be made by one of three modes (~9,066 by phone; ~6,012 by mail; ~18,606 by mail with web URL included in letter). The screener asks respondents to verify that they are a member of the household age 18 or older; respondents who meet these eligibility criteria go on to roster the children living in their household and answer questions to determine the special needs status of each child. One child will be randomly selected within each screened and eligible household. If a child with special health care needs is present in the household, that child will automatically be selected (random selection among special needs children will occur in households with multiple special needs children). A topical survey will be administered for each selected child and will cover the following content areas: demographic information; child’s health and functional status; health insurance coverage; health care access and utilization; medical home; early childhood; middle childhood and adolescence; family functioning; parental health; neighborhood and community characteristics; and health insurance experience of uninsured children in low income families. To evaluate the optimal length of the questionnaire, two versions of the topical questionnaires will be administered: a long and a short version. Households will be randomly assigned to a questionnaire length condition (long vs. short), with the sample evenly split across the two conditions.

We expect to obtain approximately 6,750 child-level screener completes (~2,250 per mode) and approximately 2,526 child-level topical interview completes (~842 per mode).

The table below shows the original sample size targets (Column Cs reflects choosing only 1 child per Household) and the corresponding original detectable differences between one mode and the combination of the other two modes (Column D).  The table then shows the new sample sizes for any items that appear in both the short and long versions (Column E) and the new detectable differences for these items (Column F).  Finally, the table shows the sample sizes for items that appear only in the long version (Column G) and the detectable differences for these items (Column H). The detectable differences follow the inverse of the square root of the sample size. As a result, if we have 1.5 times the sample size, we achieve 1/sqrt(1.5)=0.82 times the detectable difference, and if we have 0.75 times the sample size, we will have 1/sqrt(0.75)=1.15 times the detectable difference.

A	B	C	D	E	F	G	H
Key Estimate	Anticipated Estimate	Original Target Sample Size in Each Mode	Original Detectable Difference between Modes at the α=0.05 Level with 80% Power (percentage points)	New Target Sample Size in Each Mode, Short and Long Version	New Detectable Difference between Modes at the α=0.05 Level with 80% Power (percentage points), if in Short and Long Version	New Target Sample Size in Each Mode, Only In Long Version	New Detectable Difference between Modes at the α=0.05 Level with 80% Power (percentage points), if Only in Long Version
Special-needs prevalence rate	15.10%	1,500	3.2	2,250	2.6	-	-
Proportion of children with consistent insurance in the past 12 months	88.70%	561	4.6	842	3.7	421	5.3
Proportion of children who currently have autism	1.80%	561	2.2	842	1.6	421	2.6
Proportion of children who currently have asthma	8.80%	561	4.1	842	3.4	421	4.7

Telephone interviewers will be trained on how to administer the questionnaire and will have undergone human subjects protections training and have signed a confidentiality agreement prior to conducting interviews. Approximately 10% of a given phone interviewer’s cases will be monitored by a supervisor as a validation method. Data from mailed SAQs will be data entered by trained staff who have completed human subjects protections training and have signed a confidentiality agreement prior to conducting interviews. As a quality control measure, approximately five percent of a data enterer’s caseload will be double-entered to validate entries.

3. Methods to Maximize Participation Rates and Deal with Nonresponse

In designing the various versions of the NSCH qualitative assessment, substantive questionnaire, and the screening questionnaire, attention will be placed on the following design elements to help increase cooperation by prospective respondents.

• In developing and refining specific questions, the goal will be to create a logical, clear questionnaire with concrete question wording and simple grammar

• The mail and web versions of the questionnaire will be attractive with clear and simple instructions on how to complete specific questions

• Questions will be grouped according to subject areas

• Questionnaire formatting will maximize readability, including appropriate question spacing, font type and size, and easy to follow skip instructions

• Questionnaire formatting considerations will also include the use of color, diagrams and pictures to enhance respondent comprehension

• Qualitative Assessment participants will receive up to $50 cash in remuneration.

• Mode Effects experiment respondents will receive up to $10.

Data collection for the Mode Effects experiments will involve a series of mailings and non-response follow-up activities, emphasizing questionnaire completion. Our proposed approach to data collection and non-response follow-up is based on previous project experience and recommendations made by Dillman and colleagues (2009)².

• Initial mailing. For all three mode conditions, potential respondents will first receive an advance/invitation letter providing details on the study and information on how the individual can participate. The advance letter for the phone mode will notify the potential respondent that an interviewer will be following up with them soon to complete the interview. For the mail mode, the invitation letter will be accompanied by a screener packet and return envelope; for the web mode, the invitation letter will include the URL and pin for accessing the web version of the questionnaire. Both the mail and web mode mailing will include a $1 prepaid incentive. All three versions of the letter (mail, web, and phone) will include a toll-free number for the individual can call if there are questions or comments. The letters will be printed on official letterhead, and will be signed by an official to give an unambiguous signal that the survey is sponsored by a legitimate, recognizable agency.

• Reminder postcard. Subsequent to the invitation letter mailing, NORC will send a reminder postcard to all non-responding individuals assigned to the mail and web modes. The postcard is sent at the point where response to the initial invitation begins to fall off and encourages initial non-responders to complete the questionnaire via the web. It will contain the NSCH toll-free number and e-mail address so that respondents can contact NORC with questions or if they need assistance. For the web mode, the reminder postcard will be mailed one week after the invitation mailing; for the phone mode, the postcard mailing will occur three weeks after the invitation mailing.

• Hardcopy questionnaire mailing. For mail mode cases, the topical questionnaire and accompanying cover letter will be personalized to fill in the sample child’s name and other identifying information to ensure that the survey is completed for the correct child. This level of personalization in the questionnaire improves data quality by reducing the opportunity for skip logic errors. It also results in a booklet that is as short as possible for the selected child. The shorter the questionnaire, the more likely the respondent is to complete it.

• Replacement questionnaire/invitation letter mailing (FedEx, Priority Mail, or other express mail delivery). Subsequent to the previous reminder postcard mailing, NORC will send all remaining mail non-responders a second hardcopy questionnaire and all web non-responders a second invitation letter via FedEx, Priority Mail, or other express mail delivery. This package will contain a personalized letter that urges the household to respond, and for the mail mode, will also contain a replacement questionnaire, and a business return envelope to use when returning the questionnaire via mail. The packaging, mode of delivery and speed by which it is delivered will further stress the importance and urgency for response.

• Refusal Conversion for Phone Cases. For the phone mode, respondents who refuse to participate will be offered a $10 promised incentive during the telephone interview. Up to two refusal conversion attempts will be allowed per case prior to finalizing a case as a “Final Refusal”. To maximize response rates, all interviewers receive refusal aversion/conversion training and job aides with frequently asked questions that anticipate potential questions from respondents; such as how their phone number was selected.

These operational strategies will both facilitate response rates and reduce the likelihood of differential non-response by mode, which may introduce substantial bias in the resulting estimates.

4. Tests of Procedures

We plan pilot test the Qualitative Assessment protocol for both the web- and mail-mode versions of the questionnaires. All pilot tests will involve no more than nine individuals.

5. Statistical Consultants

The contractor, NORC, will collect the information on behalf of MCHB. Contact information for NORC’s principal staff on the project is listed below:

Michael Davern

Principal Investigator

55 East Monroe Street, 30th Floor

Chicago, IL 60603

Phone: (312) 357-3770

Email: [email protected]

Michael Stern

Project Director

55 East Monroe Street, 20th Floor

Chicago, IL 60603

Phone: (312) 357-3891

Email: [email protected]

1 Blair, Johnny, Frederick Conrad, Allison Castellano Ackermann, and Greg Claxton. 2006. The effect of sample size on cognitive interview findings. Paper presented at the American Association for Public Opinion Research annual conference. Montreal, Quebec, Canada. This paper examined the number of measurement issues discovered through cognitive interview with sample sizes of 5 to 50 participants. The researchers used a balance of behavior and opinion questions. What they uncovered is that increasing the number of interviews also substantially increased the number of problems exposed. However, after 20 interviews the effect decreases, meaning diminished returns for increasing beyond a sample of 20 respondents.

2 Dillman, Don A., Jolene D. Smyth, and Leah Melani Christian. 2009. Internet, mail and mixed-mode surveys: The tailored design method, 3rd edition. Hoboken, NJ: John Wiley & Sons.

7