How to create a good survey

BEFORE YOU CREATE A SURVEY
The first thing you should have in mind is that Internet surveying is not always the best method of collecting information you need. Some problems can be solved better with other methods like surveying at point of sale (PoS), live/telephone surveying etc. Computer interviewing has two major disadvantages, which makes it not suitable for some kind of surveys. Those are:
- Non representational sample
- Weaknesses of artificial intelligence (AI)

The users of Internet are not necessary representative for the whole society. If the result of your Internet survey tells you that 60% people likes product A, it means in the best case only that 60% of Internet users accessing your site like product A, but you cannot conclude that 60% of the population in your country has the same feelings about product A! That means that you should either use Internet surveying in situation if you need opinion of Internet users and not the whole population, or use some tricks. The first trick is to use existing base of users for a closed survey. You invite only users with certain characteristics e.g. 20 young male, 20 young female, 15 from upper middle class etc. so that the group you have chosen is representative for whole population. The second trick would be to choose randomly such a quotes from all people who responded to your survey, but it means you will need a lot more responses than normally to select the adequate sample.

The weakness of AI is clear when you consider open questions. Open questions are the ones in which you do not propose people any answer, but they may write down freely whatever they want (questions with textboxes or textareas). Let us suppose you want users to write down systems they can administrate. If it would be a live interview, you would know that Win2k, Windows2000 Prof., Ms Windows 2000 Professional or Windosw 2000 (with type-O) are the same systems. Nevertheless, a computer will treat them as 4 different possibilities. That is why Internet surveying is not the best choice to discover new coherences basing on open questions. What you can do is confirm existing coherences. If you do decide to introduce open questions in your survey, you will have to analyze it by yourself! There is one trick to spare your time/work: to write down all possible alternatives as a multiple options question plus an alternative other (and require the additional explanations in form of open questions only in the other case). That should work in most cases, but you should keep in mind that these two approaches are not quite the same! The situations in which the use of this trick can influence your results are e.g. the ones in which you want to check if user memorized something (e.g. the phrase you use in your advertising campaign). It is not the same to ask someone to choose all phrases he remember from the ones you display him/her (recall test with support), and to ask him to write down all phrases he remember without any support (recall test without support).

CREATING A SURVEY (based on Hamman P., Erichson B. (2000), Markforschung, 4. Edition, Stuttgart and on my own experience)
By creating a survey you should follow a specific schema:
1) Contact questions
2) The main questions
3) Control questions
4) Supplement questions
Contact questions should be the easy ones. You should make the person you are surveying calm. You ask few standard questions like age, *removed* etc. nothing which could scare them. Nothing what they might do not want to answer. You might want to ask if they have taken a part in any survey (and what was the subject of this survey), just to make sure you have fresh people and not the ones who already responded to 3 similar surveys, because the fact they already know a lot about surveying may affect your results. The main questions are the ones you want actually ask your surveying subject. After that, you may want to check if the responses were serious. You ask pretty much the same things but with different questions. Sometimes you need to add before these control questions some buffer questions to distract them, so that they do not really remember what they answered the last time for the similar question. At the very end come the questions, which may be problematic for some reasons for them (It's the kind of question you need to classify someone to certain group). Some people are not very likely to talk about their incomes, education, sexual habits, religion etc. so if you really need that kind of questions put them on the end, because it is more likely that you will get an answer from someone who almost completed the survey and feels accommodated that from someone who just started responding.
COMMON ERRORS:
1) Incomprehensible questions with strange vocabulary.
Do not try to impress someone with your vocabulary. KISS (Keep it simple stupid).
2) Vague questions. Do not use words, which are too general, like e.g. 'future'. Use 'In the next 3 months', 'In the next 6 months', etc. instead.
3) Ambiguous questions. Do not try to ask more things in one question. For example the question "Do you like to dance and sing? Yes/No" is wrong, because someone may like to dance but not to sing.
4) Tendencies in questions.
Avoid the questions in which you or a famous specialist from this subject express his opinion. Question like "Scientists believe that Product A is most suitable for most people. Do you think the same? " is plain wrong, because not everyone is so self-confident to be against specialists, so he/she may answer "yes " even if he/she actually means "no ".
5) Confusing/incomplete/too complicated instructions
Once again: KISS. Do not instruct them to calculate something and write down only the result. Ask for all data necessary and do it by yourself. Do not instruct them to answer question A if they answered option C in previous question. That is what the branching logic in my tool is for.
6) Questions which need deep understanding of subject.
Do not ask people things they simply don't know.
7) The questions with assumption.
The reality is more complicated that you can imagine. Be aware of that. The question "Education of your father: .. " assumes that someone actually know his/her father. What about people, who grown in families with a single mother? My pet peeves are questions related to education. My high school was in Poland, University in Germany and MBA in Spain. Let's suppose someone ask my about the type of high school I attended and give me e.g. only the ones you can attend in Germany as alternatives. Or suppose someone ask about my GPA. Hello?! You just assumed I studied in an Anglo-Saxon country. I don't even know my GPA's to begin with, because no one calculate GPA's in Germany or Poland. We use mainly a simple average or a different type of average.

PARTICULAR THINGS TO CONSIDER BY ONLINE SURVEYING:
1) Time and Location Independency
2) Multimedia
3) Problem of self-selection
4) Technical problems

It is obvious to say that Internet gives the responders certain time and location independency, which make the online surveying a very flexible research method. However many people forget about the disadvantages of this situation. Consider for example a survey in which you want to research your customers from certain country e.g. in Germany. By traditional surveying you could just go on the streets asking people and you could be sure that people asked in Germany belongs to German customers. In Internet, it is not so obvious. The responders can live in a different country. Even if they speak German, you cannot be sure because they may live in Austria, Switzerland or somewhere else. The results you will get may be inaccurate exactly because the people who were not supposed to answer it answered your survey. You cannot limit the access to the survey basing on the IP-address, because not everyone in Germany uses .de domain. The only thing you can do is to ask your responders where do they live and evaluate only the responses from Germany. The independence of timing can be also a reason of problems. Consider you want to test following hypothesis "The people are more happy at 3p.m. then at 9a.m. ". You need to mess the timing of responses to questions messing the overall happiness. The problem is you can only measure the timing on the server and not the timing in the places where the responders live. You need to know the time zone! The simplest way to do it is to ask country and post code (not everyone knows the difference between the time zone in which he live and the GMT)