BALDRIGE NATIONAL QUALITY PROGRAM RESEARCH: PRE-PHASE II TELEPHONE INTERVIEWS
FOUR STANDARD SURVEY QUESTIONS
1. Explain who will be surveyed and why the group is appropriate to survey.
In order to revitalize the Baldrige brand, it is necessary to collect the currently held perceptions and attitudes of people for whom the brand is most likely to be relevant. Because the Baldrige Performance Excellence Program and Award are focused on the quality and performance excellence in a variety of types of businesses and organizations, it is necessary to gather information from individuals who are responsible for these dimensions in their companies and organizations. Participants will be decision-makers at different levels in their organization who have some quality or performance improvement responsibilities: mid-level or senior managers, senior executives, school superintendents and principals, hospital administrators, etc. This data collection consists of a Phase I survey conducted online, followed by a series of in-depth telephone interviews, and a Phase II survey conducted online, with people participating in no more than a single research session. The in-depth telephone interviews (24) will be conducted with the same audience after the Phase I online survey, but before the follow-up Phase II online survey. The purpose of these interviews is to get reaction to the positioning concepts created that are based on the findings from the Phase I survey. This document covers the Pre-Phase II telephone interviews.
The sample for these n=24 telephone interviews will be equally split to represent n=12 current or lapsed customers and n=12 potential customers or prospects, which will allow us to better understand the feelings of those who have had past contact with Baldrige (and will have more familiarity with the organization) versus those who have never participated in the Baldrige programs or taken advantage of the products or services it offers. Current customers are defined as those who have used any Baldrige services in a specific time period (i.e., applied for or won award, used Criteria, downloaded website materials, served as a Baldrige Examiner, attended Baldrige events or Quest for Excellence Conference). Potential customers are those who may or may not be aware of Baldrige, but have not been involved in the Award process or used the materials offered by Baldrige.
A natural fall-out of customers, that is, current users (used Baldrige in past 2 years) versus lapsed users (used more than 2 years ago), will be obtained, meaning that there will be no forced quotas for
each group. Respondents will be surveyed from companies in varied industries, including manufacturing, small business, health care, services, education, and nonprofit. A mix of for-profit businesses and non-profit or government organizations will also be obtained during the research process. Aeffect, the marketing research firm contracted for this study, will also obtain a mix of respondents by business size: large (500+ employees), mid-size (100-499 employees), and small (under 100 employees). By obtaining the views of current customers, lapsed customers and potential customers across industries and company size, our analysis and recommendations will be based on feedback from a representative mix of key audience members.
The same sample criteria apply for these interviews as for the Phase I and II online surveys. This audience is appropriate to survey because all have had at least some experience with making quality and performance decisions at their company. Therefore, they are qualified to answer questions about quality and performance excellence. It is also important that we obtain a mix of respondents from different industry and business types. This effort will reduce bias so that our sample will represent different groups of current and potential customers who are providing valuable input.
2. Explain how the survey was developed including consultation with interested parties,
pre-testing, and responses to suggestions for improvement.
The telephone interview pre-recruitment screener and interview guide were developed after a thorough review of the Baldrige website, Baldrige Criteria for Performance Excellence, and previous Baldrige research studies. We also included lines of questions used in previous Aeffect research of messaging and positioning concepts, some of which have already gone through the OMB clearance process for other clients. In addition, a kick-off meeting (conference call) with Aeffect and Baldrige Program officials at NIST was held to garner a better understanding of the research objectives, to review past research materials (in order to leverage previously acquired learning), and to discuss and review the project scope, key phases of the study, and the purpose of each phase. Aeffect then submitted a draft of the recruiting screener and interview guide (along with materials for the other phases) to the Baldrige Program. All NIST feedback was incorporated into the revised version of the test material.
3. Explain how the survey will be conducted, how customers will be sampled if fewer than
all customers will be surveyed, expected response rate, and actions your agency plans to
take to improve the response rate.
Information from the Phase I survey will be used to develop branding materials, namely a series of positioning statements for the brand, as well as to help the program develop more effective products and services. A series of 24 in-depth interviews will take place over the telephone in order to fine-tune the positioning statements developed from Phase I results. Participants will be exposed to the statements in order to fine-tune the statements before they are evaluated by a larger audience during the Phase II online survey, which will gauge reaction to the new branding options and the identified needs and requirements.
Care is taken to ensure that the interview can be completed easily and quickly by participants. The interview is timed repeatedly so we understand how long it will take the average person to complete the task. Our goal is to collect quality information without fatiguing the respondent. We also look for ways to present the questions and response options in a manner that is easy to understand. This effort includes using language that the participants understand and that they consider their own.
In addition, the telephone methodology has been selected for this portion of the research because interviewers can gather more in-depth feedback from participants by exploring the test materials in a more comprehensive manner. We will still arrange for these interviews to take place at the respondent’s convenience.
Sample
A series of 24 in-depth telephone interviews will be conducted with the same audience after the Phase I online survey, but before the follow-up Phase II online survey. The purpose of these interviews is to get reaction to the positioning concepts created that are based on the findings from the Phase I survey. Recruiters will contact potential participants by phone, attempting up to 5 times to contact each person, and will use a screening questionnaire to determine participant qualification. Specifically, respondents must be responsible for making decisions regarding the improvement of their organization’s performance and must work in business, education, healthcare, government, or non-profit sectors. One-half (n=12) must be Baldrige customers while the other half (n=12) must be potential customers or prospects. Even if this person is not reached for the limited number of slots available for the in-depth interviews, he or she will still be eligible to participate in the Phase II online survey. The contractor (Aeffect) will secure the sample for potential customers.
To reach current Baldrige customers, that is, people who have participated in the Award process or have attended a Baldrige conference, Aeffect will utilize the sample list provided by NIST which will include respondent names, e-mails, (some) phone numbers, and addresses of current customers. To obtain customer prospects, that is, people without a direct association with Baldrige, additional samples will be purchased.
Respondents will be surveyed from companies in varied industries among a mix of business respondents by number of employees: large (500+ employees), mid-size (100-499 employees), and small (under 100 employees). Participants will be decision-makers at different levels in their organization with some quality and performance improvement responsibilities: mid-level or senior managers, C-level executives, the highest level of management within a company (e.g., Chief Executive Officer (CEO); Chief Operations Officer (COO), or Chief Financial Officer (CFO)); school superintendants and principals; hospital administrators; etc. It should be noted that all participants will come from companies and organizations located in the United States.
When conducting telephone interviews with executive participants, the standard procedure in the private sector is to compensate them for their time. This payment is a long-standing, widely accepted practice that assists in getting participants’ initial cooperation, their commitment to following through and actually participating on the agreed upon date and time, and developing a positive attitude toward the research experience for the future. The incentive for participating in the in-depth telephone interviews is $40 for each participant in this phase of the research (unless their organization prohibits employees from accepting such an incentive). This is the going rate for incentives paid for this type of research and length of interview among members of this audience. The incentive is not intended to affect the recipient’s opinions; these participants are not going to be swayed by such a payment. Instead, it is a measure of respect and appreciation for taking time out of their busy work schedules to contribute to this important research.
The payment makes recruiting efforts more efficient, as we are not limited to looking for people who are completely altruistic with their time. To restrict the recruiting process to only include people willing to participate without receiving an incentive would significantly increase recruiting time and expense (additional sample, calls made, time spent on the telephone, etc. will increase costs far beyond the cost of the incentives and in turn ignoring the OMB principle of reduced survey costs). More importantly, this approach would skew the audience of participants in a manner that would not make it representative of the audience that the Baldrige Program wishes to reach. This audience are specialized in that they are responsible at a relatively high level in their organizations for quality and performance excellence, thus the principle of improved coverage of specialized respondents is at stake without incentives. It should be noted that nonmonetary incentives were considered, but the fact that segments of the audience are so different (business, education, healthcare, government, not-for-profit) suggested that a uniform award of $40 would be the easiest incentive for participants to redeem and hold the most universal appeal. This approach fulfills the principle of equity outlined by OMB.
In addition, the 30 minute length of the interview could be considered burdensome to people who participate during their workday. They need to set aside the agreed upon time and they need to secure a location where they can talk on the phone in private without interruption. As such, the incentives will enhance the principle of data quality outlined by OMB.
The audience of Baldrige customers is specialized in that there is a limited universe of people in the Baldrige database, which consists of people who have participated in the process. Once these people have been approached and “used up,” there is no other sample available to acquire.
4. Describe how the results of the survey will be analyzed and used to generalize the results
to the entire customer population.
The purpose of the telephone interviews is to fine-tune the positioning statements that have been created based in part on the Phase I results. The feedback to these test materials gathered in the in-depth interviews will be used to fine-tune the positioning statements and needs and requirements before they are exposed to the larger audience of customers and non-customers via the Phase II online survey. These telephone interviews are qualitative in nature (n=24), so the results will not be projected or generalized to the total population.
| File Type | application/msword |
| File Modified | 2011-08-03 |
| File Created | 2011-08-03 |