Attachment III - Recommendations

Attachment III

Recommendations Received through FRN Process &

Resulting Changes to PAIMI Evaluation Surveys

The feedback received on the proposed surveys almost always requested additional questions or clarification. Below are described the changes made within each survey instrument as per feedback received from the public comment period, the PAIMI Evaluation Advisory Workgroup, and the pilot site visit to the New Jersey P&A where these instruments were field tested.

Where a new question has been added to the survey it is noted as ‘New Question #__’. For changes to existing questions, the old number is noted alongside the new question number. Where there is neither an indication of a new or old number such as the first item below, the number has not changed.

Executive Director Interview

Question #3: Is there an orientation to new Governing Board members regarding PAIMI responsibilities? A Peer Reviewer suggested including a bullet on the federal ‘restrictions’ in statue and regulation. Adopted recommendation under response choice ‘History of the PAIMI Program’ but also added ‘mandates’ to balance ‘restrictions’.

Other topics suggested for inclusion by peer reviewers that were not added to question 3 were: the state mental health policy and financial environment, the role of other mental health stakeholders, the special legislation such as the New Freedom Commission and Olmstead. HSRI did not include these suggestions as the policy and financial environment appears too broad for interpretation precisely enough for a national interview tool. The other topic areas would either be presumed under a larger topic area or not absolutely necessary for competently managing the PAIMI program.

Question #6: In the last 2 federal fiscal years, from how many sources (other than CMHS) has the P&A sought additional funding for the PAIMI program? Please list the entities solicited & whether you were successful in securing the funds or not. Include any funds received from the state by funding source. Peer reviewer suggestion was to include any state funds and source of funds. Adopted recommendation.

Question 11: How are differences resolved between what the Advisory Council recommends and what the Board adopts? Peer reviewer recommended including words ‘recommends’, ‘what’ and ‘adopts’. Adopted recommendations.

Question 13: Are policies or systems in place that address Council members having staggered terms? Peer reviewer suggested adding word ‘systems’. Adopted recommendation.

New Question 17: Is there a designated staff member to provide on-going support to the PAIMI Advisory Council? Peer reviewer recommended new question. Adopted recommendation.

Old Question 17/New Question 18: How does the level of effort vary from year to year, if any, in establishing your PAIMI Program priorities? Peer reviewer recommended clarifying language of question that was originally worded, ‘Does your PAIMI Program make roughly equivalent investment in level of effort to set priorities on an annual basis or is there year to year variation?’ Adopted recommendation.

Old Question 18/New Question 19: Which of the following sources of information is utilized by the either the Board and/or Advisory Council to establish PAIMI Program priorities? Peer reviewer suggested adding ‘Asking PAIMI Clients’ to list of information sources, and to expand I&R category to specifically note ‘including persons refused service’. Adopted both recommendations.

Old Question 19/New Question 20: Describe ways that the P&A is currently collaborating with other mental health advocacy groups. Peer reviewer recommended including ‘mental health’ to describe the type of advocacy groups with which this evaluation is concerned. Adopted recommendation.

New Question 21: What activities does your PAIMI Program not engage in as other organizations fill that role in your state/territory? New Jersey P&A site visit experience showed evaluation team how this question would inform evaluators examining types of cases, depth and diversity.

New Question 22: Are there disagreements between your PAIMI Program and other mental health advocacy groups? Peer reviewer recommended adding question. Adopted recommendation.

New Question 23: Does the PAIMI Program have a representative on the state mental health planning council? Peer reviewer recommended adding question. Adopted recommendation.

Old Question 23/New Question 27: What joint activities if any do the PAIMI Advisory Council and Governing Board engage in? Peer reviewer recommended adding to 1^st bullet on no joint activities, ‘except for cross board membership’. Adopted recommendation.

New Question 30: What do you understand to be the role and responsibilities of your GPO (Government Project Officer)? Peer reviewer recommended adding question. Adopted recommendation.

Old Question 28/New Question 33: In the past 2 years have you experienced any difficulty with the PAIMI Application? Peer reviewers recommended adding a response category of ‘Not Applicable’. Adopted recommendation.

Old Question 33/New Question 38: Are there different expectations (e.g., definition of what constitutes a case, etc.) for PAIMI Program performance or operations that arise from the involvement of federal agencies responsible for different aspects of administration of the P&A programs (CMHS, SAMHSA, ADD, RSA)? Peer reviewer recommended insertion of example to describe what is meant by different expectations. Adopted recommendation.

Old Question 36/New Question 41: What changes, if any, would improve the usefulness of the PPR for PAIMI operations? Are there other PPRs that you would recommend as better reporting tools? Peer reviewer recommended inclusion of ‘if any’. HSRI team member, Steve Haimowitz, recommended the addendum question as to whether there are other PPRs that could enhance the usefulness of PAIMI Program reporting and data. Adopted both recommendations.

Old Question 40/New Question 45: What would make the peer review reports more useful (e.g., more detailed information on strengths and areas in need of improvement)? Peer reviewer recommended changing word ‘process’ to ‘reports’ and providing example. Adopted recommendations.

Old Question 41/New Question 47: Did the site visit team recommend technical assistance or make any other recommendation(s)? Peer reviewer recommended adding ‘or make any other recommendation(s)’. Adopted recommendation.

Old Question 43/New Question 48: Was there specific information that the site visit team relayed in the exit interview that was helpful? Peer reviewer suggested rephrasing question from information to ‘specific information’ and requesting for response category ‘Yes’, that respondents provide an example. Adopted both recommendations.

Old Question 58/New Question 63: During the past 2 years, when the PAIMI has addressed systemic problems of client abuse/neglect/rights violations that could lead to litigation, what has been the typical path(s) to resolution? New Jersey P&A Executive Director and Legal Director suggested clearer response choices regarding how PAIMIs resolve systems issues. Adopted recommendations.

Old Question 60/New Question 65: How many P&A staff members are dedicated solely to PAIMI program issues? Peer reviewer recommended changing FTEs to ‘members’. Adopted recommendation.

New Question 67: Have we covered the ways in which your PAIMI utilizes consumers? New Jersey P&A Executive Director recommended including question. Adopted recommendation.

New Question 72: What do you think the PAIMI Program has accomplished in this state/territory in the past 2 years? New Jersey P&A Executive Director recommended adding this question. Adopted recommendation.

New Question 74: Which of the following does your staff visit and along what timelines? Peer reviewer suggested including ‘homeless’ before shelter and adding a response category of board and care homes and licensed assisted living facilities. Adopted recommendations.

New Question 83: How does your P&A provide supervision over the quality of work rendered to PAIMI clients? Peer reviewer recommended adding question. Adopted recommendation.

New Question 85: Does your P&A track the number of PAIMI clients dually diagnosed with mental illness and developmental disabilities? Evaluation Advisory Workgroup recommended including question and further probing as to what percent of PAIMI clients are also clients of other P&A programs. Adopted recommendations.

Old Question 81/New Question 86: How does your P&A ensure that PAIMI advocacy activities (outreach through representation) reach people who may have difficulty accessing the P&A? EAW members recommended adding to outreach categories, ‘People with different cultural background/communities of color’. Adopted recommendation

PAIMI Advisory Council Chair Survey

Instructions section: Collect calls will be accepted. Peer reviewers recommended changing ‘can be accepted’ to ‘will be’. Adopted recommendation.

Old Question 10: Does the Council have a conflict of interest policy in place that prohibits members from financially benefiting from P&A business while on the Council? PAIMI Advisory Council Chair of New Jersey P&A recommended deleting question 10 as it is covered in question 15. Adopted recommendation.

New Question 16: Is there a designated P&A staff member to provide on-going support to the PAIMI Advisory Council? Peer reviewers recommended parallel question from Executive Director’s Interview be included in PAIMI Advisory Chair survey. Adopted recommendation.

New Question 21: Do you as the PAIMI Advisory Chair have a vote on the Governing Board? New question recommended by Evaluation Advisory Workgroup members. Adopted recommendation.

Old Question 27/New Question 24: Who is primarily responsible for coming up with the Advisory Council’s PAIMI Program priorities? Peer reviewers recommended moving question to come before current question 25. Adopted recommendation.

Old Question 23/New Question 25: Describe the method the PAIMI Advisory Council uses to set priorities. Check all that apply. Evaluation Advisory Workgroup recommended adding response category, ‘Review priorities recommended by P&A staff’. Adopted recommendation.

Old Question 24/New Question 26: In what ways does the Council make use of quantitative information available for determining annual priorities? Peer reviewers recommended adding response category of ‘No data reviewed’. Adopted recommendation.

New Question 27: How does the PAIMI Advisory Council (PAC) provide opportunities for public input? Check all that apply. The PAC Chair interviewed during the New Jersey site visit recommended adding a response category of ‘Advisory Council does not know what PAIMI staff activity is related to gathering public input’. Adopted recommendation.

Old Question 34/New Question 35: Does the P&A regularly provide the Advisory Council with any of the following types of organizational and operations data? Check all that apply. Peer reviewers recommended changing response categories that addressed ‘quality of…’ to status & outcomes of…’ as well as qualifying the ‘financial information’ response to include, ‘about the PAIMI Program revenues & expenditures that you can understand’. Adopted recommendations.

Question 42: Who makes the decisions in individual PAIMI cases? Peer reviewer recommended qualifying the response ‘decisions are generally made together’ with ‘between clients and staff’. Adopted recommendation.

New Question 51: What do you think the PAIMI Program in your state/territory has accomplished in the past 2 years? Evaluation Advisory Workgroup recommended adding this question. Adopted recommendation.

Old Question 55/New Question 56: What is your experience with people with mental illness? Peer reviewer recommended adding response choice, ‘I am, or have been, a peer advocate for people with psychiatric disabilities’. Adopted recommendation.

Old Question 56/New Question 57: Are you self-identified with a disability? Peer reviewer recommended replacing ‘labeled’ with ‘self-identified’. Adopted recommendation.

Old Question 58/New Question 59: Please indicate your employment status.

Peer reviewer recommended changing ‘work’ in response categories to ‘employed’ and to add response categories ‘retired’ and ‘unemployed’. Adopted recommendations.

PAIMI Client Survey

Instructions section: Collect calls will be accepted – Peer reviewers recommended changing ‘can be accepted’ to ‘will be’. Adopted recommendation. Confidentiality – Peer reviewers recommended expanding explanation. Adopted recommendation.

Question 1: How did you learn about the Protection & Advocacy (P&A) agency? Peer reviewers recommended separating out the friends and family response into two separate categories and suggested 2 additions: I learned about it when attending a community meeting, and P&A staff visited the place I am residing. Adopted recommendation.

Question 2: Was your first contact with the P&A by telephone or did you go visit the P&A in person? Peer reviewers recommended deleting question as question 1 contains more relevant information. Adopted recommendation.

Old Question 5/New Question 4: Which of the following categories best describe the problem(s) for which you sought help from the P&A? Peer reviewers suggested removing response category of civil rights violations. Adopted recommendation.

Old Question 12/New Question 11: Are you given information about your rights? Peer reviewer recommended adding additional response categories under response ‘Yes’ to describe whether the information provided was adequate or not. Adopted recommendation.

Old Question 13/New Question 12: Have you been provided with self advocacy skills or resources? Peer reviewer recommended adding additional response categories under response ‘Yes’ to describe whether the information provided was adequate or not. Adopted recommendation.

Question 23: Have you made a complaint about the service you received from the P&A? Peer reviewer recommended adding response category if ‘No’ selected to ask why. Adopted recommendation.

Question 24: Have you made a complaint about the service you have received from the P&A? Peer reviewer suggested changing ‘objectively’ to ‘fairly’. Adopted recommendation.

Old Question 30/New Question 31: Please indicate your employment status. Peer reviewer recommended changing response categories using term ‘work’ to ‘employment’ and adding ‘retired’. Adopted recommendations.

New Question 33: What type of housing do you live in now? Peer reviewers suggested adding question and offered the response categories. Adopted recommendation.

State Mental Health Program Directors’ Survey

Public comment received from the National Association of State Mental Health Program Directors (NASMHPD) and the Assistant Attorney General from the State of Maryland’s Department of Mental Hygiene both commented that this survey should be extended to counsel for state mental health program directors. HSRI will expand the scope of the evaluation to include surveying counsel to selected mental health authorities selected in the sample. Adopted recommendation.

Old Question 6/New Question 5: When the P&A has advocated for changes in mental health regulations or policies over the past two years, how often has the State Mental Health Authority been receptive to proposed changes? New Jersey P&A Executive Director recommended collapsing question that asked about changes in regulation or policy separately. Adopted recommendation.

New Question 9: What has the PAIMI Program in your state accomplished in the past 2 years? New Jersey P&A Executive Director recommended adding this question into the Executive Director’s interview; for parallel structure, evaluation team is also placing it within surveys of other key stakeholders. Adopted recommendation.

Question 17: When—during the past two years---the P&A has dealt with systemic problems of client abuse, neglect or other rights violation that could lead to litigation, what have been the paths to resolution of the problem? Check all that apply. The National Association of State Mental Health Program Directors requested that an additional response choice be included noting that the problem has been dealt with through formal negotiations and a settlement agreement short of litigation. Adopted recommendation.

State Mental Health Advocacy Organization Directors’ Survey

Instructions section: Collect calls will be accepted. Peer reviewers recommended changing ‘can be accepted’ to ‘will be’. Adopted recommendation.

New Question 1: What is your understanding of the role of the state PAIMI Program? Experience with the New Jersey P&A site visit awareness that respondents should first be asked about their understanding of the PAIMI program before continuing with further queries.

Old Question 4/New Question 5: In advocating for changes in mental health law and policy over the past two years, the directions of the P&A and your organization have been aligned….. Respondents on the site visit were not sufficiently versed in differentiating or recalling advocacy efforts related separately to regulation, policies and practices, so the evaluation team collapsed these questions into one more general question.

New Question 12: What has the PAIMI Program in your state accomplished in the past 2 years? As per the New Jersey P&A site visit the Executive Director recommended adding this question to the Executive Director’s survey. In order to gather other stakeholder’s impression we have inserted this same question in this survey.

8