Survey instrument design

I was talking to a "survey expert" the other day and this person told me
something about survey design that I found disconcerting. The person is a
statistician and my spider sense database and programming background
contradicts a number of statements made during the meeting.

Is true that when you design repeat questions in a survey, for example
members of a household, that your survey should first go through

  1. get the list of names of all individual members
  2. get the list of birthdays of all members.
    n. get other attributes of all members.

Or, as I suggested:

  1. get name, birthday, other attributes for each member of the household?

I think what I found bristling was when the person stated: "all surveys are
designed this way" when asked why he insists on doing it that way when it
was quite inefficient from an algorithm design point of view.

Since I don't know where survey practitioners are, I thought I'd ask this
here.

Nikolai ゴー・ニコライ ?

That is interesting. I can see the reasons for the surveyor's approach -
scope first (get complete list), then ask drill down questions on each
item. If you do it the other way there is the risk (however small) that
they won't give you the full list - the respondent may want to abbreviate
and leave off a few family members.

Generally speaking, there is a lot of psychology that goes into survey
design and PhDs are written about how bias gets introduced or is avoided by
asking questions in different orders or using different wording, or placing
different visual clues in contradiction or support of specific answer
types. Since I worked at the Center for Social Science Computation and
Research (CSSCR) at the UW for a short time (is it still open?), I am only
mildly qualified to give you the full explanation but many of my colleagues
were expert at this sort of thing. What I recall being impressed about is
that survey bias has to be teased out through an iterative design process -
i.e. pay attention to what previous research tells you about question order
and then be sure to design around those limitations. (or not, if you're
trying to skew the data as many "polling" organizations do).

So, it could be that your surveyor acquaintance learned a "rule of thumb"
and is unwilling to revisit the original reasons for that question order,
or that this is one of those clear cut survey design things. Here's a
link http://knowledge-base.supersurvey.com/response-bias.htm to a survey
design webpage I found - I'm sure there are more (and better) ones.

James

··· On Fri, Jun 15, 2012 at 4:21 PM, ゴー・ニコライ wrote:

I was talking to a "survey expert" the other day and this person told me
something about survey design that I found disconcerting. The person is a
statistician and my spider sense database and programming background
contradicts a number of statements made during the meeting.

Is true that when you design repeat questions in a survey, for example
members of a household, that your survey should first go through

  1. get the list of names of all individual members
  2. get the list of birthdays of all members.
    n. get other attributes of all members.

Or, as I suggested:

  1. get name, birthday, other attributes for each member of the household?

I think what I found bristling was when the person stated: "all surveys
are designed this way" when asked why he insists on doing it that way when
it was quite inefficient from an algorithm design point of view.

Since I don't know where survey practitioners are, I thought I'd ask this
here.

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
James Dailey
skype: jdailey

James,

Thank you for the quick and enlightening reply and the informative link.
And as I said in that meeting, forms can be designed any way they like it,
but it wouldn't stop me from asking questions like "why are you doing it
this way, and not that way?" from time to time. :wink:

Thank you very much.

Best,
Nik

··· On Saturday, June 16, 2012, James Dailey wrote:

Nikolai ゴー・ニコライ ?

That is interesting. I can see the reasons for the surveyor's approach -
scope first (get complete list), then ask drill down questions on each
item. If you do it the other way there is the risk (however small) that
they won't give you the full list - the respondent may want to abbreviate
and leave off a few family members.

Generally speaking, there is a lot of psychology that goes into survey
design and PhDs are written about how bias gets introduced or is avoided by
asking questions in different orders or using different wording, or placing
different visual clues in contradiction or support of specific answer
types. Since I worked at the Center for Social Science Computation and
Research (CSSCR) at the UW for a short time (is it still open?), I am only
mildly qualified to give you the full explanation but many of my colleagues
were expert at this sort of thing. What I recall being impressed about is
that survey bias has to be teased out through an iterative design process -
i.e. pay attention to what previous research tells you about question order
and then be sure to design around those limitations. (or not, if you're
trying to skew the data as many "polling" organizations do).

So, it could be that your surveyor acquaintance learned a "rule of thumb"
and is unwilling to revisit the original reasons for that question order,
or that this is one of those clear cut survey design things. Here's a
link http://knowledge-base.supersurvey.com/response-bias.htm to a
survey design webpage I found - I'm sure there are more (and better) ones.

James

On Fri, Jun 15, 2012 at 4:21 PM, ゴー・ニコライ <nikolai.go@gmail.com<javascript:_e({}, 'cvml', 'nikolai.go@gmail.com');> wrote:

I was talking to a "survey expert" the other day and this person told me
something about survey design that I found disconcerting. The person is a
statistician and my spider sense database and programming background
contradicts a number of statements made during the meeting.

Is true that when you design repeat questions in a survey, for example
members of a household, that your survey should first go through

  1. get the list of names of all individual members
  2. get the list of birthdays of all members.
    n. get other attributes of all members.

Or, as I suggested:

  1. get name, birthday, other attributes for each member of the household?

I think what I found bristling was when the person stated: "all surveys
are designed this way" when asked why he insists on doing it that way when
it was quite inefficient from an algorithm design point of view.

Since I don't know where survey practitioners are, I thought I'd ask this
here.

--
Post: opendatakit@googlegroups.com <javascript:_e({}, 'cvml',
'opendatakit@googlegroups.com');>
Unsubscribe: opendatakit+unsubscribe@googlegroups.com <javascript:_e({},
'cvml', 'opendatakit%2Bunsubscribe@googlegroups.com');>
Options: http://groups.google.com/group/opendatakit?hl=en

--
James Dailey
skype: jdailey

--
Post: opendatakit@googlegroups.com <javascript:_e({}, 'cvml',
'opendatakit@googlegroups.com');>
Unsubscribe: opendatakit+unsubscribe@googlegroups.com <javascript:_e({},
'cvml', 'opendatakit%2Bunsubscribe@googlegroups.com');>
Options: http://groups.google.com/group/opendatakit?hl=en