White Papers
Principals of Good Survey Research Design
Sampling Q&A
Principals of Good Survey Research Design
While a subject as broad as good survey research design is the making of one (or more) semesters of business school classroom time, there are a number of indicators of good design that can be highlighted in this brief treatise.Survey design is not a pure science, it is part art and part science. At Sterling we have project directors with years of experience and college degrees in market research who handle the "science" side of the challenge. At the same time we have a Design Center staff who bring the art-form side to the final solution. In this brief treatise, both artistic and scientific aspects will be addressed.
Good Survey Content
Some considerations to good survey design include:Progressive Disclosure
It is important to ask questions in such an order as to gather responses that are candid at each question. Asking questions in the wrong order may tell the respondent how s/he should answer the next or following questions. A typical example of this problem is asking brand awareness. First we ask in an unaided manner what brands come to mind. Secondly we ask if the respondent has heard of a list of other brands not mentioned, and finally we bear down on the brand under consideration. The order of the questions makes this work properly. If it is done wrongly, the information cannot be interpreted.
Neutral Questions
Very closely allied to the progressive disclosure consideration is that of wording of questions in general. It is critical that questions are asked in a neutral manner if candid responses are to be obtained. For example ask "How would you rate company X on service Y?" rather than "Is company X doing a great job on service Y?" Neutral wording will not lead the respondent to a particular response.
Item Rotation
Respondents get tired of answering questions. And we're not referring here to too many different surveys, but with the questions in any survey. Almost regardless of length, the first questions in a survey obtain more responses than the last questions in the survey, and very frequently the responses to the first questions are richer (more varied) than those later in the survey. As a result, better overall responses are obtained to all questions, when each question has an equal opportunity of being the first question on the survey, the same opportunity to being the second question, and so forth. Paper surveys do not make item rotation easy, but telephone and Web-based surveys do.
Mutually Exclusive Responses That Are All-Inclusive
When providing response categories to a question, make sure that responses that are provided account for all possible situations, and at the same time do not overlap. For example, If age categories are provided use a pattern like this: "Under 18", "18-30", "31-50" "51 and Over". Everyone from "1" to "101" can answer that question accurately and for each person there is only one category that contains the appropriate response. Forgetting the "Under 18" will mean that some potential respondents will not be able to answer if they are 17, or 16, or younger. Or having categories like "18 and Under" and "18-28" could cause category errors since someone exactly 18 years of age qualifies for both "18 and Under" and "18-28".
Only Relevant Questions
When designing a survey, questions should be included in the survey if and only if the responses to that question provide needed information. Too often the "like to know" questions reflect researcher curiosity but not strategic importance. These questions may add to the cost, reduce response rates, possibly irritate respondents, and provide little in the way of strategic information to the problem at hand. One foolproof approach is to first define the objectives of the research, then write the questions that answer each objective. That keeps the survey on track and lean.
Branching Logic
One way to keep the net survey to the point and minimize respondent fatigue is to use branching logic in the survey. If the respondent indicates he has never used a particular service, then allow him/her to skip to the next section of the questionnaire rather than lead him/her through all of the questions relevant to that non-used service only. Branching logic is easy on Web-based and telephone surveys, and while still possible on printed surveys, is typically less effective.
List Deletion
On Web-based or telephone surveys, when asking a respondent to rank order items in terms of importance or satisfaction, it is useful to delete from the list the items as they are chosen. This allows the respondent to focus only on the items that remain. In a sense, the items that remain each time constitute a new list from which only the most important is to be chosen. List deletion cannot be done if paper surveys are used.
Judicious Use of Open-Ended Responses and Open-End Processing
Open-ended responses, those questions where the respondent writes his/her response rather than checking a box, are one area where qualitative and quantitative research merge. Many clients like to read open-ended responses because they provide a richness to responses that far exceed what is obtained from the tally of closed-end (listed item) responses. And while open-ended responses can make good reading, they suffer from potential abuse. The reader may focus on a well-written reply that confirms a suspicion. In this case, a single response among hundreds or thousands may weigh far disproportionately to its value, and might lead the company in seeking the wrong solution. If the study is quantitative-that is, it answers "what is the most important" or "in what order should we do things", then qualitative responses can almost only confuse the results. Stick to lists that meet the qualifications identified above. Use focus groups or in-depth interviews to obtain the richness of response but don't rely on a single or on a few responses to set corporate direction.
Good Survey Design
In addition to the scientific principles of good survey design, there are the more artistic considerations as well.Reading Comprehension Level
Not everyone has a PhD; for that matter not everyone has even graduated from elementary school. So, make sure the questionnaire is written to a comprehension level of the audience. One good way is to write most consumer surveys to a 4th grade reading level. Microsoft Word has a comprehension level measurement built-in to its "tools" section that is a useful measure of comprehension level. The Fogg Index is also a useful tool.
Tested Copy
There is an old saying that "reality has three views: he said, I thought, but can you imagine... " The same can be said about questionnaires and the survey process. Test copy to make sure that it is measuring what it is intended to measure. A technique often used is "parking lot testing ". In this manner a convenient sample of respondents (i.e. some folks in the parking lot) are asked to complete the questionnaire, and are then debriefed as to what they were answering. If major changes are made to the questionnaire, then the process should be repeated.
Type Size
Not everyone has 20/20 vision. If older people are to be included in the survey, then the smallest type that should be used is 12-point.
Use of Rulings and Color
Rulings and color can be used in a survey to help control eye tracking. By alternating bands of color in the questionnaire, particularly in grids of questions, the eye is better able to follow one line of copy across the page. Rulings can be used in the same way, as can leader dots.
Item Placement
Moving an item in a questionnaire can cause it to be answered in a different manner. Once a questionnaire is formatted, leave it alone and try not to change it once it's been in the field.
Use of Images, other marketing messages, etc.
The use of images and marketing messages should not be casual or just to jazz it up. A questionnaire is a communications document where the quality of the communication is particularly critical and anything that distracts from the communication quality of the document should be strenuously avoided.
Sampling Q & A
Many people consider the science of sampling as a necessary evil of market research. At the very least, it is a true science, it takes an advanced degree to properly understand, and is often underrated by clients in survey work. To illustrate this process as clearly as possible this section of the Sterling Website will take the Q&A format.Q. We mail 100 e-mail invitations to customers, but get back only 15-18 responses. How can I be sure the 15 that come back represent the 85 that don't come back?
A. The proper way to determine adequacy of the sample is to periodically test the sample by pursuing the 85 who do not reply by several means and testing the expanded results against the typical return. The expanded sample will still not represent all 100 customers but it will get closer. A statistical test can be performed on the data to determine whether the typical sample represents the larger sample.
Q. How large does the sample need to be?
A. The confidence we can have in the data depends upon the sample size we have to work with. If we have 100 responses, and want to know how dependable the results are, there are many factors involved. For example, lets suppose that the score on a question with a five-point scale of "very good" to "very bad" is 4.1 (slightly better than "good" which is a "4" and not nearly as high as "very good" which is a "5"). We can say statistically that if we did this test over and over again, we would expect to find the scores of the sample to be 4.1 +/-10%, 90% of the time. That means that if we repeated the test 50 times, in 45 of the tests (90%), the scores would be between 3.69 (that's 4.1 - 0.41) and 4.51 (that's 4.1 + 0.41). It is clear that the 0.41 used in both cases is 10% of 4.1.
So how big should a sample be? How much uncertainty can you handle, and how big is your budget? For the situation identified above look how the confidence interval of the data (the +/-10%) changes with sample size:
- 100 +/-10%
- 250 +/-6.3%
- 500 +/-4.5%
- 1000 +/-3.2%
For most clients, we work with larger confidence intervals, samples of 100 - 250, since the amount in improvement in confidence interval decreases more slowly as sample sizes increase. Translation: it takes a lot of money to get only a little improvement.
Q. Can we do anything to improve response rate?
A. Yes. We find that offering an incentive (cash or chance in a lottery) can improve response rates substantially. In one study we ran for several years, we found that putting a one-dollar bill in the mailed questionnaire doubled response rate from 23% to 45%. Was this worthwhile? Not from a purely economic perspective, for the same absolute number of responses it proved less expensive to simply mail more surveys out, than it did to include the incentive. But, the client was much more comfortable knowing that they were talking to almost one in every two customers sampled rather than only one in four. So, we continued the incentive during the life of the program.
Q. If we use a cash incentive, how big should it be?
A. Over the years we've enclosed everything from one dollar to twenty dollars, and we've used cash, check, or coupon for merchandise. The type of incentive varies depending upon the survey and industry being served.
Q. Does a longer survey reduce the response rate?
A. Length of questionnaire makes very little difference. We conduct thousands of customer satisfaction surveys each week, and on questionnaires that range from 6-8 questions, to some with closer to 100 questions; and if distributed using similar methodologies, the response rate remains about the same.
