Supporting Statement for PRA Submission
B |
Collection of Information Employing Statistical Methods |
The universe for the Association of Tribal Archives, Libraries, and Museums (ATALM) Needs Assessment consists of the 420 tribal cultural organizations within 352 United States-based indigenous tribes1 that have been awarded grant funding from the Institute of Museum and Library Services (IMLS) from FY2009 thru FY2018 (inclusive). The relatively small number of awardees suggests the applicability of a census of these establishments rather than a sample. As such, estimation will not necessitate inferential statistics associated with standard error estimation.
After receiving OMB approval, ATALM will finalize preparation for and administration of the data collection. Preparation and administration includes:
Final programming of the survey instrument with SurveyMonkey;
Finalizing the database of tribal cultural institutions eligible for the survey (i.e., those that have received an IMLS grant award in FY2009-FY2018 (inclusive));
Development of likely post-collection processing steps, including protocols for coding and analyzing open-ended survey questions;
Preparation for collection of survey paradata;
Administering the survey and monitoring returns;
Data review and editing;
Data analysis and report writing;
Facilitation of report review processes that involve ATALM and IMLS; and
Posting the report online to the ATALM website.
Survey Design/Data Collection Tools
The survey data will be collected via SurveyMonkey, which minimizes user technology requirements for both hardware and software since respondents can participate via personal computers, laptops, tablets, or smart phones. Additionally, SurveyMonkey will help minimize response burden, ensure timely submissions, and will provide necessary paradata for tracking and subsequent analyses associated with response propensity. For those respondents for whom accessing SurveyMonkey is burdensome, the option to complete the paper-and-pencil PDF included as Appendix A will be provided.
As appropriate, questions include “Don’t know” and “Prefer not to answer” response categories. In addition to reducing burden, analysis of these answers will be useful in future survey planning by indicating questions that were potentially problematic or sensitive for these respondents.
ATALM will administer data collection. Subcontractor Native Nations Institute at the University of Arizona will be responsible for data cleaning and editing, data analysis, and report writing. Native Nations Institute performed these same functions for a 2012 report (Jorgenson 2012) of a similar ATALM survey.
Contacting Respondents
Sample communications are provided in Appendix B. ATALM will distribute communications about the study as described herein. Additionally, ATALM is the primary point of contact for questions about the questionnaire, with an IMLS Senior Grants Management Specialist included as a secondary point of contact for the past grantees, who constitute the pool of respondents.
The official request for data collection will be e-mailed to the director, chair, president, or other leader for each of the 352 eligible tribes. The request will introduce the survey project, ATALM, and IMLS and provide these leaders with an opportunity to ask questions. Each e-mail will include information about the individual within the tribe who will be contacted to complete the questionnaire.
A week after the official request of the tribal leader, a second email will be sent to the project director of record for the most recently-awarded IMLS grants. For 68 tribes, there was a grant awarded by both IMLS’ Office of Library Services and Office of Museum Services, therefore, two tribal cultural organizations within these tribes will be included in the study. In total, 420 individuals who were project directors of funded IMLS awards since 2009 will receive invitations to participate in the survey. These emails will include a unique identifier that will enable ATALM to track responses and a unique weblink that will permit respondents to complete a portion of the survey and then return at a later time to finish the questionnaire. Additionally, the initial email will include a PDF of the questionnaire and glossary (Appendix A). The email will stress the importance of participation in the survey as partners but assure potential respondents that future funding is not contingent on their participation.
Three reminder e-mails will be sent to non-responders (one per week after the initial survey e-mail). Additionally, phone calls will be made to non-respondents after the second e-mail reminder. Finally, all respondents will receive a thank you note from ATALM leadership and will be sent a link to the final report when it is available on the ATALM website.
This is a universe survey that does not require special considerations for inferential statistical methods. The evaluation consultant will summarize survey data by calculating descriptive statistics and coding qualitative responses, which will be organized and presented in the report. The quantitative data will likely be displayed in tables and graphs with qualitative data analyzed, as appropriate, to protect respondent confidentiality and provide information of utility to the survey stakeholders (i.e., tribal cultural institutions, IMLS, ATALM, researchers, program planners, policy makers, grant seekers, and the media.). Relatively simple cross-tabulations will be performed, with small cell sizes (<5) suppressed in reporting to protect confidentiality.
Most questions are fixed-choice items, but there are two open ended items and ten items allow respondents to supply details about an “other” category associated with the question stem. Such items will be thematically analyzed. Any quotes from these open-ended responses will be redacted to remove any identifying information if these are to be used to illustrate points in the report. The analyst at the Native Nations Institute subcontractor to ATALM completed the analyses and report for the previous survey, therefore, this analyst is already familiar with the norms of tribal cultural institutions regarding data and with principles of informant protection.
B.3.1. Maximizing Response Rates
Every effort will be made to maximize the response rate while retaining the voluntary nature of the study. Using the tailored survey design framework (Dillman et al. 2007), careful, well-timed communications with respondents from respected individuals will be key, as described in section B.2.1, above (and in the project schedule described in section A.16), these include:
Initial contact: Leaders of eligible tribes will receive an advance email from ATALM’s president. The email will explain what the survey is about, who is doing it, and why.
Survey distribution: Within 7-10 days, an email with a customized survey link will be sent via email from ATALM. The email will include a unique identifier the respondent will use to access the survey, which will include pre-populated organization and respondent information. Additionally, respondents will be sent a PDF version of the survey to facilitate information collection as well as a glossary of terms used in the survey instrument (included in Appendix A).
E-mail reminders: three email reminders will be sent during the field period.2
Additionally, throughout the data collection period, ATALM staff members will be available via email and phone to help respondents. A glossary will be provided in the survey distribution e-mail to assist respondents in answering the survey questions and reduce the burden associated with contacting a staff member.
Survey design is crucial to facilitating response, both to the questionnaire as well as item response rates (see Fan and Yan 2010 for a review of this literature). Matrix and drop-down items were used to reduce cognitive burden (Couper et al. 2001). “Don’t know” response choices were included to reduce item non-response by acknowledging there may be items about which respondents lack knowledge to answer (for a review, see Krosnick and Presser 2009). Several questionnaire items were directly taken from other IMLS surveys, therefore, there are known response profiles, including for different types and sizes of cultural institutions. Furthermore, while the Appendix A survey version is shown in PDF format, the online version will limit screen presentation to only a few questions to avoid respondents’ need for excessive scrolling to move through the survey (Rand 2002).
Finally, we designed the administration of the survey to maximize a response rate. First, the survey will be administered by ATALM, an organization trusted by many IMLS grantee tribal cultural organizations. Dual sponsorship has been shown to increase response rates and also increase the breadth of respondents, which leads to less biased results (Groves et al. 2012). Second, as a survey of previous IMLS grantees’ needs, the survey response rate is likely to be enhanced due to the potential respondents’ high interest in its results. Importantly, though, since the survey is to be administered by ATALM, IMLS will not have access to the specific responses of individual past grantees. The survey has no relationship to future grantee funding. Third, by fielding the survey over a month’s time in the early part of 2019, the survey will be received during a “slow” time of year for many cultural organizations, which tend to become busy during common vacation times such as the summer. All three of these factors are expected to facilitate higher levels of response in the face of the general trend over the past 20 years of declining rates (National Research Council 2013).
The unique identifier will be used to track respondents so that only non-respondents will receive subsequent reminders, which will be as follows:
First reminder: e-mail one week after the survey was sent from an ATALM address;
Second reminder: e-mail one week after the 1st reminder;
Third reminder: Phone calls to non-respondents by ATALM staff or by ATALM board members (based on prior knowledge of respondent);
Fourth and final reminder: e-mail within two days of the survey close date.
Survey nonresponse and its association with biasing results has been the subject of much robust research (for a comprehensive review, see National Research Council 2013). Since the universe of tribes to be included is based on IMLS grant records, information from these same records can be used to make comparisons of the institutions that respond to those that did not respond to the survey. Response propensity will be determined based on factors such as geography (region and state) and previous interactions with the sponsors. A cross-reference of previous institutional respondents to the 2012 ATALM survey and the survey universe from the IMLS tribal grantees list will also be performed. Together, these administrative records will permit testing about response propensity hypotheses such as:
Respondent grantees are more likely to be both prior (2012) respondents and current ATALM members than they are to be non-members and non-respondents;
Response propensity will follow a U-shaped distribution as a function of tribal cultural organizations’ total budgets;
Response propensity will be higher for those institutions that perform multiple functions (see survey in Appendix A, questions 1 and 2);
Respondent grantees who received more recent funding from IMLS (e.g., since FY 2016) will be more likely to respond than will those who received funding in FY 2009 – FY 2015;
The number of IMLS awards over the period FY 2009-FY 2018 will be positively related to response propensity.
The number of IMLS enhancement awards over the FY 2009-FY2018 period will be positively related to response propensity.
The 2012 ATALM survey instrument was reviewed by the ATALM Survey Advisory Board to determine the extent to which questions could be used again in the 2019 survey. Additionally, other surveys about similar topics with which IMLS is familiar were used as a means of determining how best to pose establishment-level questions in a way that would minimize respondent cognitive burden and increase response reliability and validity. IMLS administers the Public Libraries Survey (annually) and implemented the 2014 Heritage Health Information Survey in collaboration with Heritage Preservation and RMC Research Corporation. These other instruments contributed several items to the 2019 ATALM survey.
Gateway questions, judicious use of drop-down menus, and the use of matrix-style items with radio-button responses will minimize respondent burden and decrease item nonresponse. Additionally, the questionnaire used relatively large response categories (rather than asking specific numeric information), such as the one associated with annual operating revenues. Such instrumentation provides an appropriate level of analytical detail while minimizing the burden on respondents and (hopefully) mitigating the known sensitivity of this item.
Cognitive burden has been minimized by careful attention to details associated with survey design. Items that are similar have been grouped in thematic categories. Additionally, as shown in the survey copy provided in Appendix A, we have instructed the programmer to set up questions and responses to minimize respondent effort in answering the question. The survey has been pre-tested both by ATALM and IMLS in order to identify potentially confusing question wording or other issues that might impact reliability and validity of the data to be collected and to identify additional terminology that needs to be defined in the glossary (included in Appendix A).
Finally, by including “Don’t know,” “Not applicable,” and “Prefer not to answer” response options (as appropriate), the likelihood of complete item nonresponse is minimized, which enhances the utility of the information collected. Additionally, responses of “Don’t know,” “Not applicable,” and “Prefer not to answer” are vital within a needs assessment context such as this survey as well as for future survey planning (as referenced earlier). Knowing what questions are too sensitive (i.e., high rates of “prefer not to answer”) or for which respondents do not know answers is, itself, important in understanding needs.
Person |
Address |
Email / Phone |
Data analysis and report preparation: |
|
|
Miriam Jorgensen, Ph.D. Research Director Native Nations Institute University of Arizona
Study design, data collection and report: |
803 East 1st Street Tucson, AZ 85719
|
520-349-7118
|
Susan Feller President & CEO ATALM
|
6308 Harden Drive Oklahoma City, OK 73118 |
405-401-8293
|
Melissa Brodt Project Director ATALM |
6308 Harden Drive Oklahoma City, OK 73118 |
405-401-9657 |
Institute of Museum and Library Services (IMLS) |
|
|
Sandra Toro, Ph.D. Senior Grants Management Specialist
|
955 L’Enfant Plaza North, SW Suite 4000 Washington, DC 20024 |
202-653-4662 |
Bibliography
American Indian Library Association (AILA). (2018). “What is a tribal library?” online at https://ailanet.org/what-is-a-tribal-library/ access date: 2 January 2019.
Chen, H. M. and T. Ducheneaux. (2017). “How are we doing in tribal libraries?: A case study of Oglala Lakota College Library using information visualization.” Library Management 38(1): 20-44.
Couper, M. P., Traugott, M. W., and Lamias, M. J. (2001). Web survey design and
administration. Public Opinion Quarterly 65: 230–253.
Dillman, D. A., J. D. Smythe, and L. M. Christian. (2007). Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method. (3rd Edition). New York, NY: Wiley.
Fan,W., and Z. Yan. (2010). Factors affecting response rates of the web survey: A systematic review. Computers in Human Behavior 26: 132-139.
Fricker, S. and R. Tourangeau. (2010). “Examining the relationship between nonresponse propensity and data quality in two national household surveys. Public Opinion Quarterly 74(5): 935-955.
Groves, R.M., S. Presser, R. Tourangeau, B.T. West, M.P. Couper, E. Singer, and C. Toppe. (2012). “Support for the Survey Sponsor and Nonresponse Bias.” Public Opinion Quarterly 76(3): 512-524.
Jorgensen, M. (2012). “Sustaining Indigenous Culture: The Structure, Activities, and Needs of Tribal Archives, Libraries, and Museums.” Oklahoma City, OK: ATALM.
Krosnick, J. A. and S. Presser. (2009). “Question and Questionnaire Design” draft (15 February 2009) chapter for J. D. Wright and P. V. Marsden. Handbook of Survey Research (2nd edition). San Diego, CA: Elsevier.
National Research Council. (2013). Nonresponse in Social Science Surveys: A Research Agenda. Washington, DC: National Academies Press.
Roy, L., A. Bhasin, and S. Arriaga. (2011). Tribal Archives, Libraries, and Museums: Preserving Our Language, Memory, and Lifeways. (Lanham, MD: Scarecrow Press). [Book review by Terry D. Baxter in Archival Issues 34(2): 141-144].
Scheyler, T. K. L. and J. L. Forrest. (2000). “Methods for the design and administration of web-based surveys” Journal of the American Medical Information Association 7(4): 416-425.
Schonlau, M. R. D. Fricker, Jr., and M. N. Elliot. (2002). Conducting Research Surveys via E-Mail and the Web. (Santa Monica, CA: RAND).
1“Tribe" means any tribe, band, nation, or other organized group or community, including any Alaska native village, regional corporation, or village corporation (as defined in, or established pursuant to, the Alaska Native Claims Settlement Act (43 U.S.C. 1601 et seq.)), which is recognized by the Secretary of the Interior as eligible for the special programs and services provided by the United States to Indians because of their status as Indians. The 352 tribes that have received IMLS awards represent 61% of the 573 currently registered U.S. tribes.
2 Survey paradata presented in Schleyer and Forrest (2000) show the cumulative positive impact of subsequent e-mail reminders in their web-based survey of dental professionals. Each additional reminder produced an immediate positive response impact, which then stabilized, with the greatest impact after the first reminder increasing response from just under a third to just over half of the sample. Two additional reminders each netted 6-7 percentage point increases in the survey response rate.
IMLS:
ATALM Supporting Statement B |
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-15 |