Duplicated values in multiple select questions

We found some strange behavior on select multiple questions. Sometimes,
instances will hit the ODK Aggregate server with all choices duplicated.
Things like "choice1 choice1" and "choice3 choice 3 choice7 choice7" will
appear in the data set.

This happens very rarely, well under 1% of all form submissions.

Has anyone else experienced this issue? Is there a way to prevent it?

Thanks,

Luke

Luke,

Where do the duplicates start? Is it in the form on the device? In the
database on the server? In the export to CSV?

Yaw

··· -- Need ODK services? http://nafundi.com provides form design, server setup, professional support, and software development for ODK.

On Fri, Sep 18, 2015 at 12:40 PM, Luke MacDonald luke.h.macdonald@gmail.com wrote:

We found some strange behavior on select multiple questions. Sometimes,
instances will hit the ODK Aggregate server with all choices duplicated.
Things like "choice1 choice1" and "choice3 choice 3 choice7 choice7" will
appear in the data set.

This happens very rarely, well under 1% of all form submissions.

Has anyone else experienced this issue? Is there a way to prevent it?

Thanks,

Luke

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

It's on the server. Not sure if its on the device - we have not been able to verify this yet.

Luke

··· --- Sent from my iPhone

On Sep 19, 2015, at 8:09 AM, Yaw Anokwa yanokwa@nafundi.com wrote:

Luke,

Where do the duplicates start? Is it in the form on the device? In the
database on the server? In the export to CSV?

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Fri, Sep 18, 2015 at 12:40 PM, Luke MacDonald luke.h.macdonald@gmail.com wrote:

We found some strange behavior on select multiple questions. Sometimes,
instances will hit the ODK Aggregate server with all choices duplicated.
Things like "choice1 choice1" and "choice3 choice 3 choice7 choice7" will
appear in the data set.

This happens very rarely, well under 1% of all form submissions.

Has anyone else experienced this issue? Is there a way to prevent it?

Thanks,

Luke

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to a topic in the Google Groups "ODK Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/opendatakit/QGba6WqHGA4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hi Luke,

Until we know where the duplicates are happening, it's hard to track
down. Check every part of the pipeline and let us know when it starts.

Yaw

··· -- Need ODK services? http://nafundi.com provides form design, server setup, professional support, and software development for ODK.

On Sat, Sep 19, 2015 at 11:35 AM, Luke MacDonald luke.h.macdonald@gmail.com wrote:

It's on the server. Not sure if its on the device - we have not been able to verify this yet.

Luke

Sent from my iPhone

On Sep 19, 2015, at 8:09 AM, Yaw Anokwa yanokwa@nafundi.com wrote:

Luke,

Where do the duplicates start? Is it in the form on the device? In the
database on the server? In the export to CSV?

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Fri, Sep 18, 2015 at 12:40 PM, Luke MacDonald luke.h.macdonald@gmail.com wrote:

We found some strange behavior on select multiple questions. Sometimes,
instances will hit the ODK Aggregate server with all choices duplicated.
Things like "choice1 choice1" and "choice3 choice 3 choice7 choice7" will
appear in the data set.

This happens very rarely, well under 1% of all form submissions.

Has anyone else experienced this issue? Is there a way to prevent it?

Thanks,

Luke

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to a topic in the Google Groups "ODK Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/opendatakit/QGba6WqHGA4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

When the problem occurs, the XML file on the phone does not contain
duplicates but the database on the Aggregate server contains duplicates
values for all multi-selection. We never see more replicates (e.g.
triplicates), and we never see this problem on another question type (e.g.
single select or integers)

Interestingly, the pattern is like so:

a a b b c c d d

Not

a b c d a b c d

The problem occurs very rarely. Perhaps 1 in 5,000 forms.

Luke

··· On Sat, Sep 19, 2015 at 11:38 AM, Yaw Anokwa wrote:

Hi Luke,

Until we know where the duplicates are happening, it's hard to track
down. Check every part of the pipeline and let us know when it starts.

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Sat, Sep 19, 2015 at 11:35 AM, Luke MacDonald luke.h.macdonald@gmail.com wrote:

It's on the server. Not sure if its on the device - we have not been
able to verify this yet.

Luke

Sent from my iPhone

On Sep 19, 2015, at 8:09 AM, Yaw Anokwa yanokwa@nafundi.com wrote:

Luke,

Where do the duplicates start? Is it in the form on the device? In the
database on the server? In the export to CSV?

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Fri, Sep 18, 2015 at 12:40 PM, Luke MacDonald luke.h.macdonald@gmail.com wrote:

We found some strange behavior on select multiple questions.
Sometimes,

instances will hit the ODK Aggregate server with all choices
duplicated.

Things like "choice1 choice1" and "choice3 choice 3 choice7 choice7"
will

appear in the data set.

This happens very rarely, well under 1% of all form submissions.

Has anyone else experienced this issue? Is there a way to prevent it?

Thanks,

Luke

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups

"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an

email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.

To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/QGba6WqHGA4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/QGba6WqHGA4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

What version of Aggregate are you running? And on GAE, MySQL or PostgreSQL?

Yaw

··· -- Need ODK services? http://nafundi.com provides form design, server setup, professional support, and software development for ODK.

On Mon, Sep 21, 2015 at 3:32 PM, Luke MacDonald luke.h.macdonald@gmail.com wrote:

When the problem occurs, the XML file on the phone does not contain
duplicates but the database on the Aggregate server contains duplicates
values for all multi-selection. We never see more replicates (e.g.
triplicates), and we never see this problem on another question type (e.g.
single select or integers)

Interestingly, the pattern is like so:

a a b b c c d d

Not

a b c d a b c d

The problem occurs very rarely. Perhaps 1 in 5,000 forms.

Luke

On Sat, Sep 19, 2015 at 11:38 AM, Yaw Anokwa yanokwa@nafundi.com wrote:

Hi Luke,

Until we know where the duplicates are happening, it's hard to track
down. Check every part of the pipeline and let us know when it starts.

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Sat, Sep 19, 2015 at 11:35 AM, Luke MacDonald luke.h.macdonald@gmail.com wrote:

It's on the server. Not sure if its on the device - we have not been
able to verify this yet.

Luke

Sent from my iPhone

On Sep 19, 2015, at 8:09 AM, Yaw Anokwa yanokwa@nafundi.com wrote:

Luke,

Where do the duplicates start? Is it in the form on the device? In the
database on the server? In the export to CSV?

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Fri, Sep 18, 2015 at 12:40 PM, Luke MacDonald luke.h.macdonald@gmail.com wrote:

We found some strange behavior on select multiple questions.
Sometimes,
instances will hit the ODK Aggregate server with all choices
duplicated.
Things like "choice1 choice1" and "choice3 choice 3 choice7 choice7"
will
appear in the data set.

This happens very rarely, well under 1% of all form submissions.

Has anyone else experienced this issue? Is there a way to prevent it?

Thanks,

Luke

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/QGba6WqHGA4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/QGba6WqHGA4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

We have a bunch of servers all launched on google app engine in 2013-2014,
but let's assume most are "Aggregate v1.4.2 Production."

··· On Mon, Sep 21, 2015 at 3:34 PM, Yaw Anokwa wrote:

What version of Aggregate are you running? And on GAE, MySQL or PostgreSQL?

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Mon, Sep 21, 2015 at 3:32 PM, Luke MacDonald luke.h.macdonald@gmail.com wrote:

When the problem occurs, the XML file on the phone does not contain
duplicates but the database on the Aggregate server contains duplicates
values for all multi-selection. We never see more replicates (e.g.
triplicates), and we never see this problem on another question type
(e.g.
single select or integers)

Interestingly, the pattern is like so:

a a b b c c d d

Not

a b c d a b c d

The problem occurs very rarely. Perhaps 1 in 5,000 forms.

Luke

On Sat, Sep 19, 2015 at 11:38 AM, Yaw Anokwa yanokwa@nafundi.com wrote:

Hi Luke,

Until we know where the duplicates are happening, it's hard to track
down. Check every part of the pipeline and let us know when it starts.

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Sat, Sep 19, 2015 at 11:35 AM, Luke MacDonald luke.h.macdonald@gmail.com wrote:

It's on the server. Not sure if its on the device - we have not been
able to verify this yet.

Luke

Sent from my iPhone

On Sep 19, 2015, at 8:09 AM, Yaw Anokwa yanokwa@nafundi.com wrote:

Luke,

Where do the duplicates start? Is it in the form on the device? In
the

database on the server? In the export to CSV?

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Fri, Sep 18, 2015 at 12:40 PM, Luke MacDonald luke.h.macdonald@gmail.com wrote:

We found some strange behavior on select multiple questions.
Sometimes,
instances will hit the ODK Aggregate server with all choices
duplicated.
Things like "choice1 choice1" and "choice3 choice 3 choice7 choice7"
will
appear in the data set.

This happens very rarely, well under 1% of all form submissions.

Has anyone else experienced this issue? Is there a way to prevent
it?

Thanks,

Luke

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send

an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to a topic in
the

Google Groups "ODK Community" group.
To unsubscribe from this topic, visit

https://groups.google.com/d/topic/opendatakit/QGba6WqHGA4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/QGba6WqHGA4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/QGba6WqHGA4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I checked the appengine logs. There are no unusual submissions coming from
the device that submitted the form that was corrupted in the data set.

Here is an log entry:

"2015-06-24 01:07:43.059 /submission?deviceID=imei%3A869845002622XXX 204
66ms 0kb module=default version=1
105.167.222.XXX - - [23/Jun/2015:22:07:43 -0700] "HEAD
/submission?deviceID=imei%3A869845002622XXX HTTP/1.1" 204 158 - - "
pma2020ke.appspot.com" ms=67 cpu_ms=22 cpm_usd=0.000018
app_engine_release=1.9.22 instance=00c61b117XXXXXXXXXXXX
I 2015-06-24 01:07:43.058
org.opendatakit.aggregate.servlet.SubmissionServlet doHead: Inside doHead

··· On Mon, Sep 21, 2015 at 3:41 PM, Luke MacDonald wrote:

We have a bunch of servers all launched on google app engine in 2013-2014,
but let's assume most are "Aggregate v1.4.2 Production."

On Mon, Sep 21, 2015 at 3:34 PM, Yaw Anokwa yanokwa@nafundi.com wrote:

What version of Aggregate are you running? And on GAE, MySQL or
PostgreSQL?

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Mon, Sep 21, 2015 at 3:32 PM, Luke MacDonald luke.h.macdonald@gmail.com wrote:

When the problem occurs, the XML file on the phone does not contain
duplicates but the database on the Aggregate server contains duplicates
values for all multi-selection. We never see more replicates (e.g.
triplicates), and we never see this problem on another question type
(e.g.
single select or integers)

Interestingly, the pattern is like so:

a a b b c c d d

Not

a b c d a b c d

The problem occurs very rarely. Perhaps 1 in 5,000 forms.

Luke

On Sat, Sep 19, 2015 at 11:38 AM, Yaw Anokwa yanokwa@nafundi.com wrote:

Hi Luke,

Until we know where the duplicates are happening, it's hard to track
down. Check every part of the pipeline and let us know when it starts.

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Sat, Sep 19, 2015 at 11:35 AM, Luke MacDonald luke.h.macdonald@gmail.com wrote:

It's on the server. Not sure if its on the device - we have not been
able to verify this yet.

Luke

Sent from my iPhone

On Sep 19, 2015, at 8:09 AM, Yaw Anokwa yanokwa@nafundi.com wrote:

Luke,

Where do the duplicates start? Is it in the form on the device? In
the

database on the server? In the export to CSV?

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Fri, Sep 18, 2015 at 12:40 PM, Luke MacDonald luke.h.macdonald@gmail.com wrote:

We found some strange behavior on select multiple questions.
Sometimes,
instances will hit the ODK Aggregate server with all choices
duplicated.
Things like "choice1 choice1" and "choice3 choice 3 choice7
choice7"

will
appear in the data set.

This happens very rarely, well under 1% of all form submissions.

Has anyone else experienced this issue? Is there a way to prevent
it?

Thanks,

Luke

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send

an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to a topic in
the

Google Groups "ODK Community" group.
To unsubscribe from this topic, visit

https://groups.google.com/d/topic/opendatakit/QGba6WqHGA4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send

an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/QGba6WqHGA4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/QGba6WqHGA4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I added additional sanity checks in later versions of ODK Aggregate that
could detect double-inserts like this.

Double-inserts (corrupted submissions) can occur if the quota is reached in
the middle of a form submission.

If you do upgrade to ODK Aggregate 1.45 or later, you should see errors in
the logs for these problematic submissions (you would only see them on a
new publisher or a Briefcase pull to a new, empty, Briefcase storage
location).

The steps to resolve these issues are outlined here:

··· On Mon, Sep 21, 2015 at 1:14 PM, Luke MacDonald wrote:

I checked the appengine logs. There are no unusual submissions coming from
the device that submitted the form that was corrupted in the data set.

Here is an log entry:

"2015-06-24 01:07:43.059 /submission?deviceID=imei%3A869845002622XXX 204
66ms 0kb module=default version=1
105.167.222.XXX - - [23/Jun/2015:22:07:43 -0700] "HEAD
/submission?deviceID=imei%3A869845002622XXX HTTP/1.1" 204 158 - - "
pma2020ke.appspot.com" ms=67 cpu_ms=22 cpm_usd=0.000018
app_engine_release=1.9.22 instance=00c61b117XXXXXXXXXXXX
I 2015-06-24 01:07:43.058
org.opendatakit.aggregate.servlet.SubmissionServlet doHead: Inside doHead

On Mon, Sep 21, 2015 at 3:41 PM, Luke MacDonald < luke.h.macdonald@gmail.com> wrote:

We have a bunch of servers all launched on google app engine in
2013-2014, but let's assume most are "Aggregate v1.4.2 Production."

On Mon, Sep 21, 2015 at 3:34 PM, Yaw Anokwa yanokwa@nafundi.com wrote:

What version of Aggregate are you running? And on GAE, MySQL or
PostgreSQL?

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Mon, Sep 21, 2015 at 3:32 PM, Luke MacDonald luke.h.macdonald@gmail.com wrote:

When the problem occurs, the XML file on the phone does not contain
duplicates but the database on the Aggregate server contains duplicates
values for all multi-selection. We never see more replicates (e.g.
triplicates), and we never see this problem on another question type
(e.g.
single select or integers)

Interestingly, the pattern is like so:

a a b b c c d d

Not

a b c d a b c d

The problem occurs very rarely. Perhaps 1 in 5,000 forms.

Luke

On Sat, Sep 19, 2015 at 11:38 AM, Yaw Anokwa yanokwa@nafundi.com wrote:

Hi Luke,

Until we know where the duplicates are happening, it's hard to track
down. Check every part of the pipeline and let us know when it starts.

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Sat, Sep 19, 2015 at 11:35 AM, Luke MacDonald luke.h.macdonald@gmail.com wrote:

It's on the server. Not sure if its on the device - we have not been
able to verify this yet.

Luke

Sent from my iPhone

On Sep 19, 2015, at 8:09 AM, Yaw Anokwa yanokwa@nafundi.com wrote:

Luke,

Where do the duplicates start? Is it in the form on the device? In
the

database on the server? In the export to CSV?

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Fri, Sep 18, 2015 at 12:40 PM, Luke MacDonald luke.h.macdonald@gmail.com wrote:

We found some strange behavior on select multiple questions.
Sometimes,
instances will hit the ODK Aggregate server with all choices
duplicated.
Things like "choice1 choice1" and "choice3 choice 3 choice7
choice7"

will
appear in the data set.

This happens very rarely, well under 1% of all form submissions.

Has anyone else experienced this issue? Is there a way to prevent
it?

Thanks,

Luke

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send

an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to a topic in
the

Google Groups "ODK Community" group.
To unsubscribe from this topic, visit

https://groups.google.com/d/topic/opendatakit/QGba6WqHGA4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send

an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/QGba6WqHGA4/unsubscribe
.

To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/QGba6WqHGA4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com