“Ask and you shall know”. We’ve probably all have heard this line a time or two, going through education and career. The point it delivers is well-intentioned, but only true if you really know what you are asking. Asking the right question is the most important step on the stairway to knowledge. We have compiled a list of the most common pitfalls to avoid in panel sampling, and we start at the top:

Confirmation Bias aka “Cherry Picking”

The most common bias amongst researchers. It is the tendency to line up their questions in accordance to a certain world view and logic. Let’s say you are testing the effect of a new expensive marketing campaign. The reputation of the marketing department is at stake, and it is time for the usual pre-release testing. Even the most open minded and rational person will have trouble avoiding some level of bias in the questioning. A typical effect of confirmation bias in this setting, would be to only ask the small questions, like which variant of the campaign they like best, instead of questioning if the campaign works at all.

Ways to avoid it: Be really honest with yourself and ask; what is the most important information I can get from my survey? There isn’t good or bad news – there’s only data. Data you need to make your decisions.

Question induced answers

All questions are not created a like. Two versions of the same question with subtle differences may result in very different answers. Question induced answers is often a consequence of surveys created with a strong confirmation bias, but is also a phenomenon in its own right.

Ways to avoid it: Try to keep emotional cues out of your questioning, and save them for your answer options. You can also ask the same question several times in different versions and use a median score for the versions you used.

Next time we will be looking at Empathy Bias, stay tuned!