NASS Comments

NASS COMMENTS:

You have a very complex sampling scheme.  We expect you have the resources to plan and implement it properly.  Naturally, your explanation is very complex, too, but it is also hard to follow.  There are a lot of numbers, a lot of subsamples, and many different situations (CEO samples, for example).  There may not be much you can do to make it easier, but if you can, I’m sure OMB would appreciate it.  One specific suggestion is to make sure the numbers you mention (sample sizes, completes, etc) are clear and consistent.  Because of the complexity, it feels like you are citing different numbers in different places.  Some of that may be true, but some of it may just be the different

groups in your survey (full samples, CEO samples, and non-CEO samples ;  certified and denied students; students who apply at the beginning of the year and who apply later; complete samples, respondents; etc).  I mention a few specific places in my comments below, but there are others that I had to read several times to understand.  I’m sure OMB would appreciate anything you can do to make it more straightforward.  Perhaps you can clarify Table B2.2 in the attachments?

Specific Comments:

Part A, Item 11 – In the last paragraph, it says “receipt of public assistance receipt”.  That appears to be a typo, because the phrase as-worded is redundant.

Part B, Question 1

• Define SY before you start using the abbreviation.  I know it is defined in the other parts of the application, but this is the first time we see it in this document

• You say the universe includes those “participating in NSLP”.  What about “SBP”? 

• In the first paragraph, you say that there are 7 states participating in CEO, but in subsequent paragraphs you say you are selecting 5 states for the CEO sample.  While it is certainly possible not to sample from every state if you are only producing a national estimate, it stands out.  If that is what you are planning to do, I think it is worth addressing in more detail.

• Three paragraphs before Table B1.1, you explain how participation in SBP will affect the sample design.  In this paragraph you cite number of completes, where elsewhere you site sample sizes (for example in the Attachments in Table B2.2).  This is confusing.  In this place, it might be helpful to cite sample size as well as number of completes.

• Two paragraphs before Table B1.1, when you discuss the CEO schools, you mention collecting records data for 2,160 students.  Is this collection represented in the Attachments in Table B2.2?  If so, where?

• Just before Table B1.1, when you are explaining the columns in the text, you mention the final sample column twice.

• In Table B1.1, what is the difference between initial sample and final sample?  If it is the number of completes (i.e. the initial sample times the expected response rate), why does it need to be in the table?

• In sampling SFAs from states which are grouped together in strata, you say the states are randomly allocated 0, 1, or 2 SFAs, but then the next sentence says you will sample enough to “ensure recruiting the target number of SFAs.”  Is 0, 1, or 2 enough?  Are you actually sampling more?  Those sentences are confusing.

• Under “Sampling SFAs Participating in the CEO”, you say you want to “recruit 45 SFAs” but in the Attachments in Table B2.2, you list a CEO sample of 15.

• Under “Sampling Schools for the CEO Study” you say that you will select 135 schools but limit on-site data collection to 45 schools.  Why this limit, what are you using the larger group of schools for, and do you use a similar subset in the main sample too? 

• Under “Sampling Main Students”, you give the sample size as 6,945 (including replacements) but the sample size is listed as 5,525 (4794+731)  in Table B2.2 in the Attachments.  Is the difference the replacements?  You may want to be consistent in your numbers, or at least include both sample sizes in the text in Part B.

• Under “Sampling CEO Students” you say a quarter were directly certified, half were certified by application, and a quarter paid.  But in the response to Question 2, in Table B2.1, you list the sample size for “CEO Paid or Directly Certified” as 540.  540 was a quarter of the CEO student sample.  So do you mean you will be calculating errors for Paid and Directly Certified separately, and each has a sample size of 540?  If so, I think it would help to list them on separate lines. 

Part B, Question 2

• Table B2.1 has many footnote letters (a, b, e, f, g, h, j) but I only see footnotes for a and b.

• “Selecting Samples of Students” says you will select approximately 10 certified students and then a 60:40 ratio between the strata of denied applications.  But below in “Conducting the Household Survey” it says “approximately 8 certified households and 1 to 2 denied applicants in each participating school.”  Those numbers don’t seem to match up.

Attachments, Table B2.1 – The last section includes the term “LEAS”, but it is not defined in the acronym list below.  Perhaps it is common knowledge for your target audience, but I do not know what it means and I do not recall seeing it in Part A.

Attachments, Attachment C – The brochure refers to both “Access, Participation, Eligibility, and Certification Study” and “National School Meals Survey”.  Is that intentional?  I also see that the parental questionnaire (Attachment F) refers to the study as “National School Lunch and School Breakfast Program”.

Attachments, Attachment C – Pages 17 and 18 look like the same document, with minor formatting differences.  That seems redundant.

Attachments, Attachment F, Pages 61 and 62 – On the breakfast page, the first column is labeled “Cash Register/Line” but on the lunch page, it is just labeled “Cash Register”.  Is there a reason they are inconsistent?

Attachments, Household questionnaire survey

• Do you need to or want to differentiate between formal (court documented) alimony and child support agreements and informal situations where money is exchanged?  This may be an interviewer training issue more than a questionnaire issue.

• You ask about interest and dividends, and you ask if you get that weekly, every two weeks, or monthly.   In my (limited) experience, interest is always monthly and dividends are typically quarterly or annual.   Should your questionnaire reflect additional options?  And/or do you have interviewer training to help them handle that situation?

• In the demographics section, question M4, you list many types of advanced degrees including “Ph.D” and “Law Degree”, but you do not include an option for “M.D.”.    It may not be common among your population, but it seems to me if you have an option for the other advanced degrees, you should have a place for that too.

• In the demographics section, question M6, your question lists “Native Hawaiian or Pacific Islander” but that is not a response choice.  You do list “Hispanic” as a response choice, which was the topic of the previous question.

Attachments, Attachment I, SFA Director Survey, Question A1 – The arrows only go from choices 1 and 2 to the “How Many” boxes.  If I select choice 3, do you want numbers?  If so, you might want an arrow. 

Attachments, Attachment I, SFA Director Survey, Question B4 – The closing parenthesis is misplaced in the last response option.

Attachments, Attachment I, SFA Director Survey, Question, Question E2 – In the skip instruction here, do you really want the respondents who do not participate in the SBP to skip question E3?

Attachments, Attachment J, Page 288 -  In the second line, there is a period (.) after the word “subcontractors”, and I think it should be a comma. 

Attachments, Attachment J, Page 288 – At the end of the second paragraph, you use the acronym “OMB” but you did not identify that at the end of the first paragraph when you wrote out “Office of Management and Budget”.